[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:16:28.008011 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:43537
I20260709 12:16:28.009636 26408 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:16:28.010731 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:16:28.028951 26414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:28.028949 26415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:28.030700 26408 server_base.cc:1061] running on GCE node
W20260709 12:16:28.028944 26417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:29.248394 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:16:29.248692 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:16:29.248950 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599389248922 us; error 0 us; skew 500 ppm
I20260709 12:16:29.249810 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:16:29.255365 26408 webserver.cc:533] Webserver started at http://127.25.202.62:44439/ using document root <none> and password file <none>
I20260709 12:16:29.256309 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:16:29.256565 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:16:29.257074 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:16:29.263216 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "4b593c0701e14b1fafd7a203155de76b"
format_stamp: "Formatted at 2026-07-09 12:16:29 on dist-test-slave-m57n"
I20260709 12:16:29.271505 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:16:29.277155 26423 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.278580 26408 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:16:29.279026 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "4b593c0701e14b1fafd7a203155de76b"
format_stamp: "Formatted at 2026-07-09 12:16:29 on dist-test-slave-m57n"
I20260709 12:16:29.279548 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:16:29.343740 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:16:29.345440 26408 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:16:29.345950 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:16:29.424844 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:43537
I20260709 12:16:29.425043 26474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:43537 every 8 connection(s)
I20260709 12:16:29.433290 26475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:29.457491 26475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b: Bootstrap starting.
I20260709 12:16:29.464497 26475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:29.466457 26475 log.cc:826] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b: Log is configured to *not* fsync() on all Append() calls
I20260709 12:16:29.471443 26475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b: No bootstrap required, opened a new log
I20260709 12:16:29.494638 26475 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b593c0701e14b1fafd7a203155de76b" member_type: VOTER }
I20260709 12:16:29.495234 26475 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:29.495564 26475 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b593c0701e14b1fafd7a203155de76b, State: Initialized, Role: FOLLOWER
I20260709 12:16:29.496580 26475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b593c0701e14b1fafd7a203155de76b" member_type: VOTER }
I20260709 12:16:29.497087 26475 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:16:29.497370 26475 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:16:29.497682 26475 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:29.502296 26475 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b593c0701e14b1fafd7a203155de76b" member_type: VOTER }
I20260709 12:16:29.503058 26475 leader_election.cc:304] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b593c0701e14b1fafd7a203155de76b; no voters: 
I20260709 12:16:29.504868 26475 leader_election.cc:290] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:16:29.505246 26478 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:29.507756 26478 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 1 LEADER]: Becoming Leader. State: Replica: 4b593c0701e14b1fafd7a203155de76b, State: Running, Role: LEADER
I20260709 12:16:29.508759 26478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b593c0701e14b1fafd7a203155de76b" member_type: VOTER }
I20260709 12:16:29.509912 26475 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:16:29.522495 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b593c0701e14b1fafd7a203155de76b. Latest consensus state: current_term: 1 leader_uuid: "4b593c0701e14b1fafd7a203155de76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b593c0701e14b1fafd7a203155de76b" member_type: VOTER } }
I20260709 12:16:29.522871 26479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b593c0701e14b1fafd7a203155de76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b593c0701e14b1fafd7a203155de76b" member_type: VOTER } }
I20260709 12:16:29.523448 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [sys.catalog]: This master's current role is: LEADER
I20260709 12:16:29.523523 26479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [sys.catalog]: This master's current role is: LEADER
I20260709 12:16:29.526464 26488 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:16:29.537770 26488 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:16:29.540102 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:16:29.556516 26488 catalog_manager.cc:1383] Generated new cluster ID: 464a52e999654a61a4bd314d4ea50d2d
I20260709 12:16:29.556829 26488 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:16:29.573174 26488 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:16:29.574841 26488 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:16:29.588205 26488 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b: Generated new TSK 0
I20260709 12:16:29.589439 26488 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:16:29.606684 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:16:29.616642 26497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:29.623245 26498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:29.626995 26500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:29.627426 26408 server_base.cc:1061] running on GCE node
I20260709 12:16:29.628535 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:16:29.628861 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:16:29.629076 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599389629058 us; error 0 us; skew 500 ppm
I20260709 12:16:29.629731 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:16:29.634812 26408 webserver.cc:533] Webserver started at http://127.25.202.1:34943/ using document root <none> and password file <none>
I20260709 12:16:29.635465 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:16:29.635699 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:16:29.636020 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:16:29.637229 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
format_stamp: "Formatted at 2026-07-09 12:16:29 on dist-test-slave-m57n"
I20260709 12:16:29.643836 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.000s	sys 0.007s
I20260709 12:16:29.648965 26505 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.649922 26408 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 12:16:29.650257 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
format_stamp: "Formatted at 2026-07-09 12:16:29 on dist-test-slave-m57n"
I20260709 12:16:29.650588 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:16:29.666661 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:16:29.668239 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:16:29.671473 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:16:29.676122 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:16:29.676409 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.676780 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:16:29.676955 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.791741 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:36745
I20260709 12:16:29.791843 26567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:36745 every 8 connection(s)
I20260709 12:16:29.801250 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:16:29.811455 26572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:29.814198 26573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:29.819775 26408 server_base.cc:1061] running on GCE node
W20260709 12:16:29.820515 26575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:29.821808 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:16:29.822119 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:16:29.822350 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599389822331 us; error 0 us; skew 500 ppm
I20260709 12:16:29.823366 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:16:29.827059 26408 webserver.cc:533] Webserver started at http://127.25.202.2:36919/ using document root <none> and password file <none>
I20260709 12:16:29.827836 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:16:29.828104 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:16:29.828495 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:16:29.829962 26568 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43537
I20260709 12:16:29.830109 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
format_stamp: "Formatted at 2026-07-09 12:16:29 on dist-test-slave-m57n"
I20260709 12:16:29.830412 26568 heartbeater.cc:461] Registering TS with master...
I20260709 12:16:29.831499 26568 heartbeater.cc:507] Master 127.25.202.62:43537 requested a full tablet report, sending...
I20260709 12:16:29.834914 26440 ts_manager.cc:194] Registered new tserver with Master: 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745)
I20260709 12:16:29.838008 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.010s	sys 0.000s
I20260709 12:16:29.838201 26440 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34800
I20260709 12:16:29.843854 26580 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.845466 26408 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:16:29.845846 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
format_stamp: "Formatted at 2026-07-09 12:16:29 on dist-test-slave-m57n"
I20260709 12:16:29.846225 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:16:29.864638 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:16:29.866073 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:16:29.868971 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:16:29.872813 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:16:29.873034 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.873378 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:16:29.873538 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:29.982626 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:36495
I20260709 12:16:29.982901 26642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:36495 every 8 connection(s)
I20260709 12:16:29.989789 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:16:30.000465 26647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:30.001945 26646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:30.007143 26643 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43537
I20260709 12:16:30.007573 26643 heartbeater.cc:461] Registering TS with master...
I20260709 12:16:30.008596 26643 heartbeater.cc:507] Master 127.25.202.62:43537 requested a full tablet report, sending...
W20260709 12:16:30.009091 26649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:30.010048 26408 server_base.cc:1061] running on GCE node
I20260709 12:16:30.010977 26440 ts_manager.cc:194] Registered new tserver with Master: 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:30.011107 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:16:30.011452 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:16:30.011652 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599390011634 us; error 0 us; skew 500 ppm
I20260709 12:16:30.012447 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:16:30.012530 26440 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34812
I20260709 12:16:30.015519 26408 webserver.cc:533] Webserver started at http://127.25.202.3:39819/ using document root <none> and password file <none>
I20260709 12:16:30.016201 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:16:30.016443 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:16:30.016763 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:16:30.018265 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "274d926e226c4516a89665f833a7a626"
format_stamp: "Formatted at 2026-07-09 12:16:30 on dist-test-slave-m57n"
I20260709 12:16:30.024574 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.009s	sys 0.000s
I20260709 12:16:30.029047 26654 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:30.029920 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:16:30.030232 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "274d926e226c4516a89665f833a7a626"
format_stamp: "Formatted at 2026-07-09 12:16:30 on dist-test-slave-m57n"
I20260709 12:16:30.030519 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:16:30.042575 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:16:30.043876 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:16:30.045258 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:16:30.048250 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:16:30.048527 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:30.048801 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:16:30.048983 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:30.141176 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:33155
I20260709 12:16:30.141151 26716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:33155 every 8 connection(s)
I20260709 12:16:30.162374 26717 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43537
I20260709 12:16:30.162829 26717 heartbeater.cc:461] Registering TS with master...
I20260709 12:16:30.163800 26717 heartbeater.cc:507] Master 127.25.202.62:43537 requested a full tablet report, sending...
I20260709 12:16:30.166107 26440 ts_manager.cc:194] Registered new tserver with Master: 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:30.167400 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014766389s
I20260709 12:16:30.167917 26440 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34826
I20260709 12:16:30.230504 26440 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34838:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:16:30.235780 26440 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:16:30.347360 26681 tablet_service.cc:1511] Processing CreateTablet for tablet b7414c74b0d644f9a0a4138ac92ac641 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:16:30.349150 26681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7414c74b0d644f9a0a4138ac92ac641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.347286 26682 tablet_service.cc:1511] Processing CreateTablet for tablet 0ed028afc1a44e828f7af9f781189500 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:16:30.352774 26682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed028afc1a44e828f7af9f781189500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.357396 26680 tablet_service.cc:1511] Processing CreateTablet for tablet 671e004742704504be3eac414fa3080a (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:16:30.359040 26680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 671e004742704504be3eac414fa3080a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.361251 26679 tablet_service.cc:1511] Processing CreateTablet for tablet 6a896abd94094634a10e4ff43434a321 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:16:30.361838 26678 tablet_service.cc:1511] Processing CreateTablet for tablet 1d2cef36dd3a404fbc61811621813c5d (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:16:30.362663 26679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a896abd94094634a10e4ff43434a321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.363477 26678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2cef36dd3a404fbc61811621813c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.364949 26677 tablet_service.cc:1511] Processing CreateTablet for tablet adb287b7a1484050ac3f8d8e8f2e01c4 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:16:30.366294 26677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb287b7a1484050ac3f8d8e8f2e01c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.381285 26676 tablet_service.cc:1511] Processing CreateTablet for tablet 69697e4373f54f9388bdf85cd3c393b9 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:16:30.383546 26676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69697e4373f54f9388bdf85cd3c393b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.436481 26675 tablet_service.cc:1511] Processing CreateTablet for tablet 3002320b53d74f4cb4450e78d74d37d4 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:16:30.438784 26675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3002320b53d74f4cb4450e78d74d37d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.464818 26532 tablet_service.cc:1511] Processing CreateTablet for tablet b7414c74b0d644f9a0a4138ac92ac641 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:16:30.463373 26530 tablet_service.cc:1511] Processing CreateTablet for tablet 6a896abd94094634a10e4ff43434a321 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:16:30.466363 26533 tablet_service.cc:1511] Processing CreateTablet for tablet 0ed028afc1a44e828f7af9f781189500 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:16:30.471254 26531 tablet_service.cc:1511] Processing CreateTablet for tablet 671e004742704504be3eac414fa3080a (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:16:30.471299 26532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7414c74b0d644f9a0a4138ac92ac641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.483681 26533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed028afc1a44e828f7af9f781189500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.488011 26529 tablet_service.cc:1511] Processing CreateTablet for tablet 1d2cef36dd3a404fbc61811621813c5d (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:16:30.491510 26531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 671e004742704504be3eac414fa3080a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.491052 26527 tablet_service.cc:1511] Processing CreateTablet for tablet 69697e4373f54f9388bdf85cd3c393b9 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:16:30.496587 26528 tablet_service.cc:1511] Processing CreateTablet for tablet adb287b7a1484050ac3f8d8e8f2e01c4 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:16:30.493813 26525 tablet_service.cc:1511] Processing CreateTablet for tablet 2b91295b45e0487196251617ffcdf3f7 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:16:30.490973 26526 tablet_service.cc:1511] Processing CreateTablet for tablet 3002320b53d74f4cb4450e78d74d37d4 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:16:30.505003 26527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69697e4373f54f9388bdf85cd3c393b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.505414 26679 tablet_service.cc:1511] Processing CreateTablet for tablet 2b91295b45e0487196251617ffcdf3f7 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:16:30.510238 26605 tablet_service.cc:1511] Processing CreateTablet for tablet 6a896abd94094634a10e4ff43434a321 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:16:30.513535 26529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2cef36dd3a404fbc61811621813c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.516261 26735 tablet_bootstrap.cc:492] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:30.519997 26605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a896abd94094634a10e4ff43434a321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.521333 26525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b91295b45e0487196251617ffcdf3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.510159 26608 tablet_service.cc:1511] Processing CreateTablet for tablet 0ed028afc1a44e828f7af9f781189500 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:16:30.510970 26606 tablet_service.cc:1511] Processing CreateTablet for tablet 671e004742704504be3eac414fa3080a (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:16:30.522172 26679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b91295b45e0487196251617ffcdf3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.511626 26607 tablet_service.cc:1511] Processing CreateTablet for tablet b7414c74b0d644f9a0a4138ac92ac641 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:16:30.528918 26606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 671e004742704504be3eac414fa3080a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.518151 26604 tablet_service.cc:1511] Processing CreateTablet for tablet 1d2cef36dd3a404fbc61811621813c5d (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:16:30.521345 26601 tablet_service.cc:1511] Processing CreateTablet for tablet 3002320b53d74f4cb4450e78d74d37d4 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:16:30.536664 26607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7414c74b0d644f9a0a4138ac92ac641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.531848 26602 tablet_service.cc:1511] Processing CreateTablet for tablet 69697e4373f54f9388bdf85cd3c393b9 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:16:30.533149 26603 tablet_service.cc:1511] Processing CreateTablet for tablet adb287b7a1484050ac3f8d8e8f2e01c4 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:16:30.535079 26600 tablet_service.cc:1511] Processing CreateTablet for tablet 2b91295b45e0487196251617ffcdf3f7 (DEFAULT_TABLE table=test-workload [id=e5029a2810484aa1ab25743399cada60]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:16:30.538872 26528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb287b7a1484050ac3f8d8e8f2e01c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:16:30.543358 26487 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:16:30.541639 26608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed028afc1a44e828f7af9f781189500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.545220 26602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69697e4373f54f9388bdf85cd3c393b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.546674 26601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3002320b53d74f4cb4450e78d74d37d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.550300 26530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a896abd94094634a10e4ff43434a321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.552073 26600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b91295b45e0487196251617ffcdf3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.552771 26603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb287b7a1484050ac3f8d8e8f2e01c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.555480 26604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2cef36dd3a404fbc61811621813c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.560011 26526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3002320b53d74f4cb4450e78d74d37d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:30.660686 26737 tablet_bootstrap.cc:492] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
I20260709 12:16:30.672659 26735 tablet_bootstrap.cc:654] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.673738 26736 tablet_bootstrap.cc:492] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:30.687140 26737 tablet_bootstrap.cc:654] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.701519 26735 tablet_bootstrap.cc:492] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:30.702545 26735 ts_tablet_manager.cc:1403] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.198s	user 0.007s	sys 0.015s
I20260709 12:16:30.713683 26736 tablet_bootstrap.cc:654] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.714061 26735 raft_consensus.cc:359] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.718339 26737 tablet_bootstrap.cc:492] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:30.718405 26735 raft_consensus.cc:385] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.718724 26737 ts_tablet_manager.cc:1403] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.059s	user 0.015s	sys 0.010s
I20260709 12:16:30.719058 26735 raft_consensus.cc:740] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.720218 26735 consensus_queue.cc:260] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.721905 26737 raft_consensus.cc:359] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.723018 26737 raft_consensus.cc:385] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.723417 26737 raft_consensus.cc:740] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.724598 26737 consensus_queue.cc:260] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.730293 26717 heartbeater.cc:499] Master 127.25.202.62:43537 was elected leader, sending a full tablet report...
I20260709 12:16:30.731935 26736 tablet_bootstrap.cc:492] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:30.732540 26736 ts_tablet_manager.cc:1403] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.066s	user 0.013s	sys 0.009s
I20260709 12:16:30.736651 26736 raft_consensus.cc:359] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.737774 26736 raft_consensus.cc:385] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.738171 26736 raft_consensus.cc:740] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.740923 26568 heartbeater.cc:499] Master 127.25.202.62:43537 was elected leader, sending a full tablet report...
I20260709 12:16:30.739208 26736 consensus_queue.cc:260] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.741379 26737 ts_tablet_manager.cc:1434] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.022s	user 0.000s	sys 0.021s
I20260709 12:16:30.742506 26737 tablet_bootstrap.cc:492] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
I20260709 12:16:30.742539 26735 ts_tablet_manager.cc:1434] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.039s	user 0.010s	sys 0.005s
I20260709 12:16:30.744309 26735 tablet_bootstrap.cc:492] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:30.750927 26735 tablet_bootstrap.cc:654] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.757069 26737 tablet_bootstrap.cc:654] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.769121 26742 raft_consensus.cc:493] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.769644 26742 raft_consensus.cc:515] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.776502 26742 leader_election.cc:290] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:30.777393 26643 heartbeater.cc:499] Master 127.25.202.62:43537 was elected leader, sending a full tablet report...
I20260709 12:16:30.797173 26737 tablet_bootstrap.cc:492] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:30.797712 26737 ts_tablet_manager.cc:1403] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.055s	user 0.021s	sys 0.000s
W20260709 12:16:30.800714 26569 tablet.cc:2406] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:16:30.801409 26736 ts_tablet_manager.cc:1434] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.068s	user 0.012s	sys 0.023s
I20260709 12:16:30.801797 26737 raft_consensus.cc:359] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.803465 26736 tablet_bootstrap.cc:492] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:30.803191 26737 raft_consensus.cc:385] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.804790 26737 raft_consensus.cc:740] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.804726 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:30.806453 26735 tablet_bootstrap.cc:492] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:30.806928 26735 ts_tablet_manager.cc:1403] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.063s	user 0.005s	sys 0.010s
I20260709 12:16:30.806540 26692 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 0.
I20260709 12:16:30.809010 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:30.810029 26735 raft_consensus.cc:359] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.810669 26735 raft_consensus.cc:385] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.810992 26735 raft_consensus.cc:740] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
W20260709 12:16:30.811017 26507 leader_election.cc:343] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:30.812004 26735 consensus_queue.cc:260] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.812122 26506 leader_election.cc:304] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:30.814011 26735 ts_tablet_manager.cc:1434] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:16:30.814639 26742 raft_consensus.cc:2804] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:30.814922 26735 tablet_bootstrap.cc:492] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:30.815004 26742 raft_consensus.cc:493] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:30.815452 26742 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:30.820360 26736 tablet_bootstrap.cc:654] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.805835 26737 consensus_queue.cc:260] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.822795 26742 raft_consensus.cc:515] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.828727 26742 leader_election.cc:290] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:30.828918 26735 tablet_bootstrap.cc:654] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.830648 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:30.831018 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:30.837082 26692 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:30.833197 26737 ts_tablet_manager.cc:1434] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.010s
I20260709 12:16:30.838444 26737 tablet_bootstrap.cc:492] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
W20260709 12:16:30.843478 26507 leader_election.cc:343] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:30.845046 26737 tablet_bootstrap.cc:654] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.845583 26736 tablet_bootstrap.cc:492] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:30.846073 26736 ts_tablet_manager.cc:1403] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.043s	user 0.011s	sys 0.006s
I20260709 12:16:30.848661 26692 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 1.
I20260709 12:16:30.852240 26506 leader_election.cc:304] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:30.853139 26742 raft_consensus.cc:2804] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:30.853793 26742 raft_consensus.cc:697] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:30.854871 26742 consensus_queue.cc:237] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.856071 26736 raft_consensus.cc:359] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.858325 26736 raft_consensus.cc:385] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.858768 26736 raft_consensus.cc:740] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.859867 26736 consensus_queue.cc:260] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.867643 26743 raft_consensus.cc:493] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.868471 26743 raft_consensus.cc:515] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.870954 26735 tablet_bootstrap.cc:492] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:30.871560 26735 ts_tablet_manager.cc:1403] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.057s	user 0.024s	sys 0.010s
I20260709 12:16:30.877115 26743 leader_election.cc:290] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:30.877223 26748 raft_consensus.cc:493] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.877036 26735 raft_consensus.cc:359] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.878165 26735 raft_consensus.cc:385] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.877912 26748 raft_consensus.cc:515] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.878609 26735 raft_consensus.cc:740] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.879724 26735 consensus_queue.cc:260] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.880985 26748 leader_election.cc:290] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:30.882541 26735 ts_tablet_manager.cc:1434] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:16:30.883431 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
W20260709 12:16:30.884931 26507 leader_election.cc:343] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:30.888371 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:30.888937 26736 ts_tablet_manager.cc:1434] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.043s	user 0.010s	sys 0.001s
W20260709 12:16:30.890164 26506 leader_election.cc:343] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:30.890730 26735 tablet_bootstrap.cc:492] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:30.890882 26506 leader_election.cc:304] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626
I20260709 12:16:30.892035 26748 raft_consensus.cc:2749] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:30.894341 26741 raft_consensus.cc:493] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.894876 26741 raft_consensus.cc:515] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.898026 26736 tablet_bootstrap.cc:492] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:30.900144 26741 leader_election.cc:290] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:30.921895 26737 tablet_bootstrap.cc:492] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:30.922518 26737 ts_tablet_manager.cc:1403] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.084s	user 0.026s	sys 0.024s
I20260709 12:16:30.930371 26737 raft_consensus.cc:359] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.932209 26737 raft_consensus.cc:385] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:30.932682 26737 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:30.938285 26736 tablet_bootstrap.cc:654] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.949558 26743 raft_consensus.cc:493] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.953171 26743 raft_consensus.cc:515] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.956820 26743 leader_election.cc:290] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:30.958988 26737 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.964968 26741 raft_consensus.cc:493] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.947175 26438 catalog_manager.cc:5697] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 0 to 1, leader changed from <none> to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 1 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:30.968653 26741 raft_consensus.cc:515] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:30.974179 26737 ts_tablet_manager.cc:1434] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.001s
I20260709 12:16:30.975791 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:30.976065 26741 leader_election.cc:290] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:30.977491 26735 tablet_bootstrap.cc:654] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:30.977264 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:30.978017 26691 raft_consensus.cc:2468] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 0.
I20260709 12:16:30.978919 26737 tablet_bootstrap.cc:492] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
W20260709 12:16:30.979796 26581 leader_election.cc:343] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:30.981469 26581 leader_election.cc:304] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:30.983363 26743 raft_consensus.cc:2804] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:30.984396 26743 raft_consensus.cc:493] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:30.984793 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:30.986699 26743 raft_consensus.cc:3060] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:30.988296 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
W20260709 12:16:30.992503 26584 leader_election.cc:343] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:16:30.994027 26584 leader_election.cc:343] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:30.994649 26584 leader_election.cc:304] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.004829 26736 tablet_bootstrap.cc:492] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.006394 26736 ts_tablet_manager.cc:1403] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.109s	user 0.017s	sys 0.011s
I20260709 12:16:31.005450 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.007239 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
W20260709 12:16:31.014914 26656 leader_election.cc:343] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.015873 26759 raft_consensus.cc:2749] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.018723 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.019354 26736 raft_consensus.cc:359] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.018759 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.020190 26736 raft_consensus.cc:385] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.020040 26743 raft_consensus.cc:515] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
W20260709 12:16:31.022149 26658 leader_election.cc:343] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.023439 26617 raft_consensus.cc:2393] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 274d926e226c4516a89665f833a7a626 in current term 1: Already voted for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in this term.
W20260709 12:16:31.022941 26658 leader_election.cc:343] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.024297 26658 leader_election.cc:304] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:31.020540 26736 raft_consensus.cc:740] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.022938 26743 leader_election.cc:290] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.026494 26741 raft_consensus.cc:2749] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.027551 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:31.028640 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:31.028767 26656 leader_election.cc:304] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:31.029565 26691 raft_consensus.cc:3060] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.028849 26736 consensus_queue.cc:260] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
W20260709 12:16:31.031080 26584 leader_election.cc:343] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.032608 26736 ts_tablet_manager.cc:1434] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.002s
I20260709 12:16:31.034015 26736 tablet_bootstrap.cc:492] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:31.035391 26735 tablet_bootstrap.cc:492] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:31.035876 26735 ts_tablet_manager.cc:1403] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.145s	user 0.020s	sys 0.011s
I20260709 12:16:31.042078 26736 tablet_bootstrap.cc:654] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.042567 26691 raft_consensus.cc:2468] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 1.
I20260709 12:16:31.044162 26741 raft_consensus.cc:2749] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.044838 26735 raft_consensus.cc:359] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.045833 26735 raft_consensus.cc:385] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.046263 26735 raft_consensus.cc:740] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.046919 26581 leader_election.cc:304] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.048818 26743 raft_consensus.cc:2804] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.049342 26743 raft_consensus.cc:697] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:31.049439 26737 tablet_bootstrap.cc:654] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.050473 26743 consensus_queue.cc:237] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.050473 26735 consensus_queue.cc:260] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.065847 26735 ts_tablet_manager.cc:1434] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.002s
I20260709 12:16:31.066830 26735 tablet_bootstrap.cc:492] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:31.074395 26737 tablet_bootstrap.cc:492] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:31.073447 26437 catalog_manager.cc:5697] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:31.074998 26737 ts_tablet_manager.cc:1403] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.096s	user 0.019s	sys 0.004s
I20260709 12:16:31.077718 26735 tablet_bootstrap.cc:654] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.078022 26737 raft_consensus.cc:359] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.078743 26737 raft_consensus.cc:385] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.079051 26737 raft_consensus.cc:740] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.079797 26737 consensus_queue.cc:260] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.086905 26737 ts_tablet_manager.cc:1434] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:16:31.087962 26737 tablet_bootstrap.cc:492] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
I20260709 12:16:31.097967 26736 tablet_bootstrap.cc:492] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.098616 26736 ts_tablet_manager.cc:1403] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.065s	user 0.014s	sys 0.007s
I20260709 12:16:31.115743 26737 tablet_bootstrap.cc:654] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.115734 26736 raft_consensus.cc:359] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.120307 26736 raft_consensus.cc:385] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.120688 26736 raft_consensus.cc:740] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.121459 26736 consensus_queue.cc:260] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.128484 26736 ts_tablet_manager.cc:1434] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 12:16:31.129851 26736 tablet_bootstrap.cc:492] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:31.130564 26735 tablet_bootstrap.cc:492] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:31.131055 26735 ts_tablet_manager.cc:1403] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.064s	user 0.016s	sys 0.005s
I20260709 12:16:31.134240 26735 raft_consensus.cc:359] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.135054 26735 raft_consensus.cc:385] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.135500 26735 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.136508 26735 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.138520 26736 tablet_bootstrap.cc:654] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.139173 26735 ts_tablet_manager.cc:1434] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:16:31.140618 26735 tablet_bootstrap.cc:492] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:31.142143 26737 tablet_bootstrap.cc:492] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:31.142699 26737 ts_tablet_manager.cc:1403] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.055s	user 0.014s	sys 0.029s
I20260709 12:16:31.145692 26737 raft_consensus.cc:359] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.146489 26737 raft_consensus.cc:385] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.146837 26737 raft_consensus.cc:740] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.147598 26737 consensus_queue.cc:260] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.149812 26743 raft_consensus.cc:493] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.150262 26743 raft_consensus.cc:515] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.152431 26743 leader_election.cc:290] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.153700 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.154317 26691 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 0.
I20260709 12:16:31.155622 26581 leader_election.cc:304] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:31.157106 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
W20260709 12:16:31.159973 26584 leader_election.cc:343] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.165159 26743 raft_consensus.cc:2804] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.165625 26743 raft_consensus.cc:493] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.166040 26743 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.172647 26735 tablet_bootstrap.cc:654] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.173545 26737 ts_tablet_manager.cc:1434] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.000s
I20260709 12:16:31.176172 26737 tablet_bootstrap.cc:492] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
I20260709 12:16:31.182422 26737 tablet_bootstrap.cc:654] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.174943 26743 raft_consensus.cc:515] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.184973 26743 leader_election.cc:290] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.185997 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:31.187390 26691 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.190338 26736 tablet_bootstrap.cc:492] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.190960 26736 ts_tablet_manager.cc:1403] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.061s	user 0.012s	sys 0.036s
I20260709 12:16:31.186739 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
W20260709 12:16:31.192541 26584 leader_election.cc:343] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.194862 26736 raft_consensus.cc:359] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.195533 26691 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 1.
I20260709 12:16:31.195886 26736 raft_consensus.cc:385] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.196348 26736 raft_consensus.cc:740] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.197124 26581 leader_election.cc:304] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.197292 26736 consensus_queue.cc:260] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.198499 26743 raft_consensus.cc:2804] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.199025 26743 raft_consensus.cc:697] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:31.199925 26743 consensus_queue.cc:237] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.202685 26736 ts_tablet_manager.cc:1434] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.002s
I20260709 12:16:31.204150 26736 tablet_bootstrap.cc:492] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:31.211416 26737 tablet_bootstrap.cc:492] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:31.215061 26737 ts_tablet_manager.cc:1403] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.039s	user 0.008s	sys 0.005s
I20260709 12:16:31.218253 26737 raft_consensus.cc:359] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.222081 26737 raft_consensus.cc:385] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.222649 26737 raft_consensus.cc:740] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.223573 26735 tablet_bootstrap.cc:492] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:31.224089 26735 ts_tablet_manager.cc:1403] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.084s	user 0.029s	sys 0.012s
I20260709 12:16:31.223681 26737 consensus_queue.cc:260] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.213546 26736 tablet_bootstrap.cc:654] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.227021 26742 raft_consensus.cc:493] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.229230 26742 raft_consensus.cc:515] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.235484 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
W20260709 12:16:31.237607 26507 leader_election.cc:343] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.238795 26742 leader_election.cc:290] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.240355 26737 ts_tablet_manager.cc:1434] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.002s
I20260709 12:16:31.240933 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.241654 26737 tablet_bootstrap.cc:492] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
W20260709 12:16:31.242851 26506 leader_election.cc:343] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.243541 26506 leader_election.cc:304] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.246417 26742 raft_consensus.cc:2749] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.229355 26438 catalog_manager.cc:5697] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:31.254951 26736 tablet_bootstrap.cc:492] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.255527 26736 ts_tablet_manager.cc:1403] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.052s	user 0.020s	sys 0.011s
I20260709 12:16:31.256625 26737 tablet_bootstrap.cc:654] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.258033 26741 raft_consensus.cc:493] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.258536 26741 raft_consensus.cc:515] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.258739 26736 raft_consensus.cc:359] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.259436 26736 raft_consensus.cc:385] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.259859 26736 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.260815 26736 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.262365 26741 leader_election.cc:290] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.263219 26736 ts_tablet_manager.cc:1434] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:16:31.262948 26743 raft_consensus.cc:493] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.264333 26736 tablet_bootstrap.cc:492] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:31.263974 26743 raft_consensus.cc:515] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.264705 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
W20260709 12:16:31.266398 26658 leader_election.cc:343] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.266831 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.267510 26617 raft_consensus.cc:2468] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 0.
I20260709 12:16:31.268945 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.269454 26656 leader_election.cc:304] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.273365 26741 raft_consensus.cc:2804] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.273515 26743 leader_election.cc:290] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.274703 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.275800 26691 raft_consensus.cc:2393] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in current term 1: Already voted for candidate 0102899a9b8c4d798a1ef5a9d20c30db in this term.
I20260709 12:16:31.277264 26581 leader_election.cc:304] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.273800 26741 raft_consensus.cc:493] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.278506 26741 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.278178 26743 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.292944 26735 raft_consensus.cc:359] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.293725 26735 raft_consensus.cc:385] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.294014 26735 raft_consensus.cc:740] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.294888 26737 tablet_bootstrap.cc:492] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:31.295848 26737 ts_tablet_manager.cc:1403] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.054s	user 0.006s	sys 0.010s
I20260709 12:16:31.296144 26743 raft_consensus.cc:2749] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.292526 26736 tablet_bootstrap.cc:654] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.297129 26735 consensus_queue.cc:260] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.299249 26741 raft_consensus.cc:515] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.302348 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:31.302644 26735 ts_tablet_manager.cc:1434] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.078s	user 0.007s	sys 0.000s
W20260709 12:16:31.303750 26658 leader_election.cc:343] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.303840 26777 raft_consensus.cc:493] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.304241 26735 tablet_bootstrap.cc:492] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:31.308418 26748 raft_consensus.cc:493] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.308936 26748 raft_consensus.cc:515] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.310789 26748 leader_election.cc:290] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.319011 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.319211 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:31.317757 26741 leader_election.cc:290] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.319701 26692 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 0.
I20260709 12:16:31.319901 26617 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.322176 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.322860 26618 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 0.
I20260709 12:16:31.323351 26737 raft_consensus.cc:359] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.324074 26737 raft_consensus.cc:385] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.324587 26507 leader_election.cc:304] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:31.325063 26737 raft_consensus.cc:740] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.304639 26777 raft_consensus.cc:515] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.325718 26748 raft_consensus.cc:2804] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.326130 26748 raft_consensus.cc:493] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.326472 26748 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.327634 26742 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.345793 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.346493 26543 raft_consensus.cc:2468] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 0.
I20260709 12:16:31.347944 26658 leader_election.cc:304] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:31.348878 26741 raft_consensus.cc:2804] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.349262 26741 raft_consensus.cc:493] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.349638 26741 raft_consensus.cc:3060] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.350167 26777 leader_election.cc:290] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.349067 26737 consensus_queue.cc:260] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.352133 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.352789 26737 ts_tablet_manager.cc:1434] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.000s
I20260709 12:16:31.353940 26737 tablet_bootstrap.cc:492] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
I20260709 12:16:31.355126 26735 tablet_bootstrap.cc:654] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
W20260709 12:16:31.356033 26656 leader_election.cc:343] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.356724 26743 raft_consensus.cc:493] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.372090 26743 raft_consensus.cc:515] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.374333 26743 leader_election.cc:290] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.356730 26741 raft_consensus.cc:515] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.360532 26737 tablet_bootstrap.cc:654] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.362239 26748 raft_consensus.cc:515] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.362996 26742 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.363597 26617 raft_consensus.cc:2468] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 1.
I20260709 12:16:31.376230 26741 leader_election.cc:290] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.371307 26736 tablet_bootstrap.cc:492] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.377566 26736 ts_tablet_manager.cc:1403] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.113s	user 0.016s	sys 0.002s
I20260709 12:16:31.378011 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:31.378352 26656 leader_election.cc:304] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.378575 26618 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.380139 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.381742 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:31.381865 26616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
W20260709 12:16:31.383158 26581 leader_election.cc:343] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.376214 26748 leader_election.cc:290] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.384454 26741 raft_consensus.cc:2804] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.384935 26741 raft_consensus.cc:697] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:31.385989 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:31.385746 26741 consensus_queue.cc:237] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.386615 26692 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.382329 26543 raft_consensus.cc:3060] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:16:31.390292 26656 leader_election.cc:343] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.394522 26736 raft_consensus.cc:359] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.395486 26736 raft_consensus.cc:385] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.396968 26736 raft_consensus.cc:740] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.398391 26736 consensus_queue.cc:260] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.402539 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.385756 26618 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 1.
I20260709 12:16:31.403628 26542 raft_consensus.cc:2468] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 0.
I20260709 12:16:31.409111 26507 leader_election.cc:304] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:31.410435 26742 raft_consensus.cc:2804] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.410918 26742 raft_consensus.cc:697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:31.411983 26584 leader_election.cc:304] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.412032 26742 consensus_queue.cc:237] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.414134 26736 ts_tablet_manager.cc:1434] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.002s
I20260709 12:16:31.417001 26543 raft_consensus.cc:2468] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 1.
I20260709 12:16:31.418403 26658 leader_election.cc:304] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:31.415261 26743 raft_consensus.cc:2804] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.433965 26743 raft_consensus.cc:493] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.434451 26743 raft_consensus.cc:3060] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.435102 26736 tablet_bootstrap.cc:492] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:31.423106 26737 tablet_bootstrap.cc:492] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:31.439471 26737 ts_tablet_manager.cc:1403] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.086s	user 0.011s	sys 0.015s
I20260709 12:16:31.439990 26741 raft_consensus.cc:2804] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.448908 26741 raft_consensus.cc:697] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:31.440758 26759 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.438498 26692 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 1.
I20260709 12:16:31.449892 26741 consensus_queue.cc:237] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.459851 26543 raft_consensus.cc:3060] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.450688 26759 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.468822 26748 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.473655 26759 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155), 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745)
I20260709 12:16:31.474392 26748 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.463963 26439 catalog_manager.cc:5697] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:31.475944 26743 raft_consensus.cc:515] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.476387 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.478125 26743 leader_election.cc:290] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.484294 26542 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 0.
I20260709 12:16:31.485419 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.487166 26737 raft_consensus.cc:359] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.487987 26737 raft_consensus.cc:385] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.488270 26737 raft_consensus.cc:740] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.489125 26541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
W20260709 12:16:31.489670 26581 leader_election.cc:343] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.489738 26541 raft_consensus.cc:3060] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.489900 26748 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.491129 26737 consensus_queue.cc:260] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.493271 26737 ts_tablet_manager.cc:1434] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.053s	user 0.005s	sys 0.000s
I20260709 12:16:31.493682 26736 tablet_bootstrap.cc:654] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.500509 26690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:31.502548 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
W20260709 12:16:31.513303 26581 leader_election.cc:343] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.504340 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.519068 26735 tablet_bootstrap.cc:492] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:31.515947 26618 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 0.
I20260709 12:16:31.521502 26735 ts_tablet_manager.cc:1403] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.217s	user 0.007s	sys 0.015s
I20260709 12:16:31.522316 26759 consensus_queue.cc:1048] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:16:31.520918 26506 leader_election.cc:343] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:16:31.528461 26735 raft_consensus.cc:359] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.529213 26735 raft_consensus.cc:385] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.529506 26735 raft_consensus.cc:740] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.529939 26541 raft_consensus.cc:2468] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 1.
I20260709 12:16:31.530328 26735 consensus_queue.cc:260] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.531050 26584 leader_election.cc:304] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.532502 26759 raft_consensus.cc:2804] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.532528 26743 consensus_queue.cc:1048] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.533103 26759 raft_consensus.cc:697] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:31.538280 26735 ts_tablet_manager.cc:1434] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:16:31.538205 26759 consensus_queue.cc:237] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.539228 26735 tablet_bootstrap.cc:492] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:31.541149 26736 tablet_bootstrap.cc:492] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.541714 26736 ts_tablet_manager.cc:1403] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.107s	user 0.015s	sys 0.010s
I20260709 12:16:31.560227 26736 raft_consensus.cc:359] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.560989 26736 raft_consensus.cc:385] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.561288 26736 raft_consensus.cc:740] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.562122 26736 consensus_queue.cc:260] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.566761 26736 ts_tablet_manager.cc:1434] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.015s
I20260709 12:16:31.567749 26736 tablet_bootstrap.cc:492] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Bootstrap starting.
I20260709 12:16:31.571893 26736 tablet_bootstrap.cc:654] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.572952 26735 tablet_bootstrap.cc:654] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.574208 26584 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.579924 26787 raft_consensus.cc:2804] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.580435 26787 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.581063 26787 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.581780 26759 raft_consensus.cc:493] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.583387 26759 raft_consensus.cc:515] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.587922 26736 tablet_bootstrap.cc:492] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: No bootstrap required, opened a new log
I20260709 12:16:31.592732 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.593820 26543 raft_consensus.cc:2393] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in current term 1: Already voted for candidate 274d926e226c4516a89665f833a7a626 in this term.
I20260709 12:16:31.595955 26507 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.597326 26742 raft_consensus.cc:2804] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:16:31.597272 26507 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:36495, user_credentials={real_user=slave}} blocked reactor thread for 73528.3us
I20260709 12:16:31.597834 26742 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.598192 26742 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:16:31.600857 26487 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:16:31.609247 26780 raft_consensus.cc:493] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.609891 26780 raft_consensus.cc:515] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.609338 26742 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.614719 26737 tablet_bootstrap.cc:492] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: Bootstrap starting.
I20260709 12:16:31.614951 26787 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.626658 26759 leader_election.cc:290] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.637656 26787 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Requested vote from peers 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155), 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745)
I20260709 12:16:31.650215 26736 ts_tablet_manager.cc:1403] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent bootstrapping tablet: real 0.083s	user 0.035s	sys 0.040s
I20260709 12:16:31.651178 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.651155 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:31.653999 26780 leader_election.cc:290] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:31.654464 26542 raft_consensus.cc:2393] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in current term 1: Already voted for candidate 0102899a9b8c4d798a1ef5a9d20c30db in this term.
I20260709 12:16:31.654330 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626" is_pre_election: true
I20260709 12:16:31.655489 26735 tablet_bootstrap.cc:492] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:31.652248 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.655910 26735 ts_tablet_manager.cc:1403] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.117s	user 0.016s	sys 0.004s
I20260709 12:16:31.656883 26618 raft_consensus.cc:2393] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0102899a9b8c4d798a1ef5a9d20c30db in current term 1: Already voted for candidate 274d926e226c4516a89665f833a7a626 in this term.
I20260709 12:16:31.657070 26742 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Requested vote from peers 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.658861 26507 leader_election.cc:304] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.659785 26439 catalog_manager.cc:5697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 0 to 1, leader changed from <none> to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 1 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:31.660501 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:31.662714 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:31.664867 26618 raft_consensus.cc:2393] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0102899a9b8c4d798a1ef5a9d20c30db in current term 1: Already voted for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in this term.
I20260709 12:16:31.664834 26692 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.665189 26736 raft_consensus.cc:359] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.665853 26736 raft_consensus.cc:385] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.666297 26736 raft_consensus.cc:740] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.664156 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:31.667039 26736 consensus_queue.cc:260] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.668962 26735 raft_consensus.cc:359] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.669667 26735 raft_consensus.cc:385] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.669946 26735 raft_consensus.cc:740] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.670598 26735 consensus_queue.cc:260] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.668993 26506 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db; no voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.672237 26736 ts_tablet_manager.cc:1434] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 12:16:31.665822 26440 catalog_manager.cc:5697] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:31.672819 26735 ts_tablet_manager.cc:1434] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:16:31.673748 26735 tablet_bootstrap.cc:492] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626: Bootstrap starting.
I20260709 12:16:31.674820 26780 raft_consensus.cc:2749] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.667083 26742 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.679426 26737 tablet_bootstrap.cc:654] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.679847 26581 leader_election.cc:304] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626
I20260709 12:16:31.690057 26799 raft_consensus.cc:3060] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.680120 26692 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 1.
I20260709 12:16:31.694087 26737 tablet_bootstrap.cc:492] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: No bootstrap required, opened a new log
I20260709 12:16:31.694628 26737 ts_tablet_manager.cc:1403] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent bootstrapping tablet: real 0.080s	user 0.010s	sys 0.007s
I20260709 12:16:31.693934 26438 catalog_manager.cc:5697] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:31.695209 26581 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.696285 26799 raft_consensus.cc:2749] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.697037 26799 raft_consensus.cc:2804] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.697484 26799 raft_consensus.cc:697] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:31.698342 26799 consensus_queue.cc:237] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:31.699406 26737 raft_consensus.cc:359] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.700229 26737 raft_consensus.cc:385] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.700776 26737 raft_consensus.cc:740] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.685969 26742 raft_consensus.cc:2749] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.680941 26735 tablet_bootstrap.cc:654] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:31.708081 26737 consensus_queue.cc:260] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.716619 26799 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.718967 26735 tablet_bootstrap.cc:492] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626: No bootstrap required, opened a new log
I20260709 12:16:31.719458 26735 ts_tablet_manager.cc:1403] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626: Time spent bootstrapping tablet: real 0.046s	user 0.009s	sys 0.005s
I20260709 12:16:31.724725 26735 raft_consensus.cc:359] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.725487 26735 raft_consensus.cc:385] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:31.725826 26735 raft_consensus.cc:740] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Initialized, Role: FOLLOWER
I20260709 12:16:31.726549 26735 consensus_queue.cc:260] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.729077 26735 ts_tablet_manager.cc:1434] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:16:31.735136 26542 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.726583 26737 ts_tablet_manager.cc:1434] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:16:31.743451 26438 catalog_manager.cc:5697] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:31.747704 26787 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.778828 26741 raft_consensus.cc:493] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.779505 26741 raft_consensus.cc:515] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.781872 26741 leader_election.cc:290] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.782573 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.783010 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.783180 26543 raft_consensus.cc:2468] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 0.
I20260709 12:16:31.783674 26618 raft_consensus.cc:2468] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 0.
I20260709 12:16:31.784233 26658 leader_election.cc:304] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:31.784965 26741 raft_consensus.cc:2804] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:16:31.785327 26741 raft_consensus.cc:493] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.785604 26741 raft_consensus.cc:3060] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.790015 26741 raft_consensus.cc:515] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.791791 26741 leader_election.cc:290] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.792586 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:31.792937 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:31.793202 26543 raft_consensus.cc:3060] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.793484 26618 raft_consensus.cc:3060] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.799736 26618 raft_consensus.cc:2468] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 1.
I20260709 12:16:31.799741 26543 raft_consensus.cc:2468] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 1.
I20260709 12:16:31.801148 26658 leader_election.cc:304] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:31.801923 26741 raft_consensus.cc:2804] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:31.802461 26741 raft_consensus.cc:697] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:31.803347 26741 consensus_queue.cc:237] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.813270 26437 catalog_manager.cc:5697] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:31.818919 26741 consensus_queue.cc:1048] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.837265 26742 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.839286 26741 consensus_queue.cc:1048] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.854483 26780 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:31.953076 26777 raft_consensus.cc:493] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.953588 26777 raft_consensus.cc:515] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:31.954246 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:59492
I20260709 12:16:31.954928 26618 raft_consensus.cc:606] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Received request to transfer leadership to TS 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.955752 26777 leader_election.cc:290] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:31.956887 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" is_pre_election: true
I20260709 12:16:31.957373 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" is_pre_election: true
I20260709 12:16:31.958168 26543 raft_consensus.cc:2393] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 274d926e226c4516a89665f833a7a626 in current term 1: Already voted for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in this term.
I20260709 12:16:31.959182 26658 leader_election.cc:304] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:31.960269 26777 raft_consensus.cc:3060] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:31.967562 26777 raft_consensus.cc:2749] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:16:31.974735 26692 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641"
dest_uuid: "274d926e226c4516a89665f833a7a626"
mode: GRACEFUL
new_leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:56938
I20260709 12:16:31.975531 26692 raft_consensus.cc:606] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Received request to transfer leadership to TS 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.985178 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a896abd94094634a10e4ff43434a321"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:59492
I20260709 12:16:31.985728 26618 raft_consensus.cc:606] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Received request to transfer leadership to TS 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:31.991475 26692 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d"
dest_uuid: "274d926e226c4516a89665f833a7a626"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:56938
I20260709 12:16:31.991968 26692 raft_consensus.cc:606] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:32.014487 26543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:46512
I20260709 12:16:32.015043 26543 raft_consensus.cc:606] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:32.024138 26692 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4"
dest_uuid: "274d926e226c4516a89665f833a7a626"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:56938
I20260709 12:16:32.024781 26692 raft_consensus.cc:606] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:32.030393 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:59492
I20260709 12:16:32.030921 26618 raft_consensus.cc:606] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Received request to transfer leadership to TS 274d926e226c4516a89665f833a7a626
I20260709 12:16:32.053910 26777 consensus_queue.cc:1048] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.068786 26777 consensus_queue.cc:1048] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.079135 26777 raft_consensus.cc:993] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:32.079815 26809 raft_consensus.cc:1081] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:32.082120 26618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59480
I20260709 12:16:32.082710 26618 raft_consensus.cc:493] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.083106 26618 raft_consensus.cc:3060] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.089879 26618 raft_consensus.cc:515] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.092993 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:32.093201 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d2cef36dd3a404fbc61811621813c5d" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:32.093575 26543 raft_consensus.cc:3060] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.093783 26692 raft_consensus.cc:3055] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.094169 26692 raft_consensus.cc:740] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:32.094983 26692 consensus_queue.cc:260] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.096262 26692 raft_consensus.cc:3060] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.098150 26618 leader_election.cc:290] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 2 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:32.100199 26543 raft_consensus.cc:2468] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 2.
I20260709 12:16:32.101233 26584 leader_election.cc:304] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:32.101914 26799 raft_consensus.cc:2804] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.102334 26799 raft_consensus.cc:697] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:32.102578 26692 raft_consensus.cc:2468] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 2.
I20260709 12:16:32.103219 26799 consensus_queue.cc:237] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.110937 26437 catalog_manager.cc:5697] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 1 to 2, leader changed from 274d926e226c4516a89665f833a7a626 (127.25.202.3) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:32.124351 26799 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.131565 26799 consensus_queue.cc:1048] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:16:32.139405 26759 raft_consensus.cc:993] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:32.139919 26799 raft_consensus.cc:993] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:32.140533 26795 raft_consensus.cc:1081] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Signalling peer 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:32.139837 26759 raft_consensus.cc:1081] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Signalling peer 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:32.147428 26689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7"
dest_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:56932
I20260709 12:16:32.148036 26689 raft_consensus.cc:493] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.148451 26689 raft_consensus.cc:3060] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.151394 26543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:46482
I20260709 12:16:32.151973 26543 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.152463 26543 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.158715 26795 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.159521 26543 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:32.160174 26795 consensus_queue.cc:1048] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.161473 26795 raft_consensus.cc:993] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:32.161849 26795 raft_consensus.cc:1081] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Signalling peer 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:32.161849 26543 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 2 election: Requested vote from peers 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:32.163177 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:32.163717 26618 raft_consensus.cc:3055] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.163954 26618 raft_consensus.cc:740] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:32.166694 26691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:32.164714 26618 consensus_queue.cc:260] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:32.168730 26618 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.174582 26542 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a896abd94094634a10e4ff43434a321"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:46482
I20260709 12:16:32.175182 26542 raft_consensus.cc:493] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.175594 26542 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.175402 26689 raft_consensus.cc:515] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.178189 26692 raft_consensus.cc:1240] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Rejecting Update request from peer 2555e0bc64ea4db28865ed9d4dc1e1e4 for earlier term 1. Current term is 2. Ops: []
I20260709 12:16:32.178521 26541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:32.180007 26799 consensus_queue.cc:1059] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:16:32.181751 26743 raft_consensus.cc:3055] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.182201 26743 raft_consensus.cc:740] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:32.183280 26743 consensus_queue.cc:260] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.184676 26743 raft_consensus.cc:3060] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.186744 26618 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 2.
I20260709 12:16:32.188300 26507 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 274d926e226c4516a89665f833a7a626
I20260709 12:16:32.190263 26742 raft_consensus.cc:2804] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.190791 26742 raft_consensus.cc:697] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:32.191736 26742 consensus_queue.cc:237] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:32.192414 26689 leader_election.cc:290] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 2 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:32.193079 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b91295b45e0487196251617ffcdf3f7" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:32.200158 26617 raft_consensus.cc:2468] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 2.
I20260709 12:16:32.205677 26542 raft_consensus.cc:515] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.208587 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:32.209535 26618 raft_consensus.cc:3055] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.209573 26656 leader_election.cc:304] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:32.210070 26618 raft_consensus.cc:740] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:32.210767 26809 raft_consensus.cc:2804] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.211249 26809 raft_consensus.cc:697] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:32.210951 26618 consensus_queue.cc:260] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.212034 26809 consensus_queue.cc:237] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.213240 26618 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.214310 26689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:32.214926 26689 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.223011 26809 consensus_queue.cc:1048] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.222846 26438 catalog_manager.cc:5697] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 1 to 2, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 2 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:32.225172 26689 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 2.
I20260709 12:16:32.226286 26506 leader_election.cc:304] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:32.227514 26780 raft_consensus.cc:2804] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.228013 26780 raft_consensus.cc:697] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:32.229130 26780 consensus_queue.cc:237] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.236310 26618 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 2.
I20260709 12:16:32.244867 26542 leader_election.cc:290] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 2 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:32.249722 26809 consensus_queue.cc:1048] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:16:32.240074 26437 catalog_manager.cc:5697] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 1 to 2, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:32.258755 26741 raft_consensus.cc:993] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:32.259296 26809 raft_consensus.cc:1081] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:32.262737 26438 catalog_manager.cc:5697] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 1 to 2, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 2 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:32.264676 26618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59480
I20260709 12:16:32.265326 26618 raft_consensus.cc:493] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.265889 26618 raft_consensus.cc:3060] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.274405 26618 raft_consensus.cc:515] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.277175 26618 leader_election.cc:290] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 2 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:32.277851 26689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:32.278450 26689 raft_consensus.cc:3055] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.278764 26689 raft_consensus.cc:740] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:32.280135 26689 consensus_queue.cc:260] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.281389 26689 raft_consensus.cc:3060] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.289229 26689 raft_consensus.cc:2468] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 2.
I20260709 12:16:32.290285 26581 leader_election.cc:304] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:32.291199 26743 raft_consensus.cc:2804] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.291715 26743 raft_consensus.cc:697] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:32.286237 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3002320b53d74f4cb4450e78d74d37d4" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:32.292502 26743 consensus_queue.cc:237] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.293393 26543 raft_consensus.cc:3060] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.300611 26439 catalog_manager.cc:5697] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 1 to 2, leader changed from 274d926e226c4516a89665f833a7a626 (127.25.202.3) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:32.301599 26543 raft_consensus.cc:2468] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 2.
I20260709 12:16:32.314198 26741 raft_consensus.cc:993] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: : Instructing follower 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:32.314659 26809 raft_consensus.cc:1081] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Signalling peer 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:32.316255 26543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:46498
I20260709 12:16:32.316807 26543 raft_consensus.cc:493] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.317338 26543 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.324033 26543 raft_consensus.cc:515] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.326104 26543 leader_election.cc:290] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 2 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:32.326841 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:32.327359 26689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:32.327451 26618 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.327996 26689 raft_consensus.cc:3055] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.328338 26689 raft_consensus.cc:740] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:32.329207 26689 consensus_queue.cc:260] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.330397 26689 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.334299 26618 raft_consensus.cc:2468] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 2.
I20260709 12:16:32.335646 26507 leader_election.cc:304] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:32.336416 26789 raft_consensus.cc:2804] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.336875 26789 raft_consensus.cc:697] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:32.337667 26689 raft_consensus.cc:2468] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 2.
I20260709 12:16:32.337700 26789 consensus_queue.cc:237] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.344854 26439 catalog_manager.cc:5697] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 1 to 2, leader changed from 274d926e226c4516a89665f833a7a626 (127.25.202.3) to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 2 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:32.471092 26742 raft_consensus.cc:993] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:32.471634 26789 raft_consensus.cc:1081] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:32.473026 26618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59476
I20260709 12:16:32.473742 26618 raft_consensus.cc:493] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:32.474072 26618 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.478720 26618 raft_consensus.cc:515] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.480538 26618 leader_election.cc:290] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 2 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:32.481822 26689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:32.481717 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:32.482347 26689 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.482453 26543 raft_consensus.cc:3055] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:32.482824 26543 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:32.483654 26543 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.484869 26543 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.489137 26689 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 2.
I20260709 12:16:32.490345 26581 leader_election.cc:304] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:32.491164 26743 raft_consensus.cc:2804] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:32.491586 26543 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 2.
I20260709 12:16:32.491654 26743 raft_consensus.cc:697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:32.493304 26743 consensus_queue.cc:237] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:32.497196 26689 raft_consensus.cc:1275] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.498507 26743 consensus_queue.cc:1048] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.501060 26439 catalog_manager.cc:5697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 1 to 2, leader changed from 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:32.512507 26543 raft_consensus.cc:1275] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.516461 26795 consensus_queue.cc:1048] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.661650 26543 raft_consensus.cc:3060] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.670497 26543 raft_consensus.cc:1275] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.672292 26809 consensus_queue.cc:1048] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.681437 26618 raft_consensus.cc:1275] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.683485 26741 consensus_queue.cc:1048] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.707270 26618 raft_consensus.cc:1275] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.708614 26789 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.717566 26689 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:32.725044 26689 raft_consensus.cc:1275] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.726553 26742 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.755285 26618 raft_consensus.cc:1275] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.756637 26742 consensus_queue.cc:1048] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.767745 26617 raft_consensus.cc:1275] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.770133 26780 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.773638 26689 raft_consensus.cc:1275] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.775211 26742 consensus_queue.cc:1048] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.792932 26692 raft_consensus.cc:1275] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.794539 26742 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.870698 26692 raft_consensus.cc:1275] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.872004 26795 consensus_queue.cc:1048] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.881335 26543 raft_consensus.cc:1275] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.882890 26799 consensus_queue.cc:1048] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.885476 26543 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.887569 26799 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:32.896770 26692 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:32.898664 26799 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:35.062775 26408 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:16:35.063596 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:35.084987 26542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "671e004742704504be3eac414fa3080a"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
mode: GRACEFUL
new_leader_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:33812
I20260709 12:16:35.085678 26542 raft_consensus.cc:606] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Received request to transfer leadership to TS 274d926e226c4516a89665f833a7a626
I20260709 12:16:35.097656 26617 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:32792
I20260709 12:16:35.098191 26617 raft_consensus.cc:606] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Received request to transfer leadership to TS 274d926e226c4516a89665f833a7a626
I20260709 12:16:35.099536 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:16:35.100247 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:35.100302 26831 raft_consensus.cc:993] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:35.100751 26837 raft_consensus.cc:1081] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Signalling peer 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:35.102401 26689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:56932
I20260709 12:16:35.102936 26689 raft_consensus.cc:493] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:35.103236 26689 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:35.107779 26689 raft_consensus.cc:515] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:35.109355 26689 leader_election.cc:290] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 3 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:35.110422 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:35.110422 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:35.111248 26617 raft_consensus.cc:3055] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:16:35.111245 26542 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:35.111742 26617 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:35.112572 26617 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:35.113776 26617 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:35.118129 26542 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 3.
I20260709 12:16:35.119575 26658 leader_election.cc:304] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:35.120636 26617 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 3.
I20260709 12:16:35.121089 26832 raft_consensus.cc:2804] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:16:35.122486 26832 raft_consensus.cc:697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:35.123246 26832 consensus_queue.cc:237] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:35.131559 26437 catalog_manager.cc:5697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 2 to 3, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:35.377139 26826 raft_consensus.cc:993] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: : Instructing follower 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:35.377590 26844 raft_consensus.cc:1081] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Signalling peer 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:35.379240 26689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "671e004742704504be3eac414fa3080a"
dest_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:56928
I20260709 12:16:35.379783 26689 raft_consensus.cc:493] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:35.380086 26689 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:35.384635 26689 raft_consensus.cc:515] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:35.386241 26689 leader_election.cc:290] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 2 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:35.387295 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:35.387315 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:35.388178 26542 raft_consensus.cc:3055] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:35.388192 26617 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:35.388819 26542 raft_consensus.cc:740] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:35.389657 26542 consensus_queue.cc:260] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:35.390856 26542 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:35.395370 26617 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 2.
I20260709 12:16:35.396718 26656 leader_election.cc:304] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:35.397535 26832 raft_consensus.cc:2804] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:35.397529 26542 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 2.
I20260709 12:16:35.398151 26832 raft_consensus.cc:697] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:35.399117 26832 consensus_queue.cc:237] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:35.408428 26437 catalog_manager.cc:5697] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 1 to 2, leader changed from 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1) to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:35.568024 26617 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:35.569392 26832 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:35.579981 26542 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:35.581600 26838 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:35.840991 26617 raft_consensus.cc:1275] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:35.842285 26838 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:35.850639 26542 raft_consensus.cc:1275] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:35.852905 26848 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:36.101364 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:36.106374 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:36.106756 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:37.890281 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:37.895849 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:37.896291 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:16:38.604732 26713 debug-util.cc:398] Leaking SignalData structure 0x7b080016d060 after lost signal to thread 26411
W20260709 12:16:38.605790 26713 debug-util.cc:398] Leaking SignalData structure 0x7b080016d740 after lost signal to thread 26474
W20260709 12:16:38.606812 26713 debug-util.cc:398] Leaking SignalData structure 0x7b08001ba680 after lost signal to thread 26567
W20260709 12:16:38.608039 26713 debug-util.cc:398] Leaking SignalData structure 0x7b08001a0e20 after lost signal to thread 26642
W20260709 12:16:38.609171 26713 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf340 after lost signal to thread 26716
I20260709 12:16:38.897445 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:38.903645 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:38.904129 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:39.905331 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:39.910121 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:39.910522 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:40.911489 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:40.916621 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:40.917048 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:41.918005 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:41.924525 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:41.924983 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:42.926091 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:42.931293 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:42.931664 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:43.932567 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:43.938033 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:43.938450 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:44.939509 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:44.944844 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:44.945284 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:45.977655 26408 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:16:45.982962 26543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:46512
I20260709 12:16:45.983806 26543 raft_consensus.cc:606] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:45.989310 26543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:46512
I20260709 12:16:45.989723 26543 raft_consensus.cc:606] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:45.994999 26689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "671e004742704504be3eac414fa3080a"
dest_uuid: "274d926e226c4516a89665f833a7a626"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:56938
I20260709 12:16:45.995483 26689 raft_consensus.cc:606] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:46.000581 26543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a896abd94094634a10e4ff43434a321"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:46512
I20260709 12:16:46.001132 26543 raft_consensus.cc:606] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:46.009137 26689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "274d926e226c4516a89665f833a7a626"
mode: GRACEFUL
new_leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:56938
I20260709 12:16:46.009642 26689 raft_consensus.cc:606] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 LEADER]: Received request to transfer leadership to TS 2555e0bc64ea4db28865ed9d4dc1e1e4
I20260709 12:16:46.067377 26864 raft_consensus.cc:993] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.067883 26867 raft_consensus.cc:1081] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.069849 26617 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "671e004742704504be3eac414fa3080a"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59480
I20260709 12:16:46.070416 26617 raft_consensus.cc:493] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:46.070791 26617 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.076273 26617 raft_consensus.cc:515] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.078677 26617 leader_election.cc:290] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:46.079821 26689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:46.079730 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:46.080672 26689 raft_consensus.cc:3055] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:16:46.080830 26543 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.080997 26689 raft_consensus.cc:740] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:46.081862 26689 consensus_queue.cc:260] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.083084 26689 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.085479 26543 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.086622 26584 leader_election.cc:304] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:46.087549 26862 raft_consensus.cc:2804] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:16:46.088651 26862 raft_consensus.cc:697] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:46.088798 26689 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.089811 26862 consensus_queue.cc:237] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.102739 26438 catalog_manager.cc:5697] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 2 to 3, leader changed from 274d926e226c4516a89665f833a7a626 (127.25.202.3) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:46.359849 26855 raft_consensus.cc:993] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.360297 26855 raft_consensus.cc:1081] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.360862 26874 raft_consensus.cc:993] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.361389 26855 raft_consensus.cc:1081] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.362095 26617 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59476
I20260709 12:16:46.362599 26617 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:46.362923 26617 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.363198 26618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a896abd94094634a10e4ff43434a321"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59476
I20260709 12:16:46.363698 26618 raft_consensus.cc:493] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:46.364014 26618 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.369855 26617 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:46.370257 26618 raft_consensus.cc:515] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.371898 26618 leader_election.cc:290] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:46.371927 26617 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Requested vote from peers 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155), 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745)
I20260709 12:16:46.372655 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:46.373106 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:46.373385 26543 raft_consensus.cc:3055] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:16:46.373764 26542 raft_consensus.cc:3055] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:16:46.373914 26543 raft_consensus.cc:740] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:46.374181 26542 raft_consensus.cc:740] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:46.374374 26689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:46.374768 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:46.374697 26543 consensus_queue.cc:260] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.374905 26689 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.375451 26692 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.375000 26542 consensus_queue.cc:260] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:46.375946 26543 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.376317 26542 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.381541 26689 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.381541 26543 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.382326 26692 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.382987 26584 leader_election.cc:304] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:46.382567 26542 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.383080 26581 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:46.384066 26862 raft_consensus.cc:2804] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:16:46.384148 26868 raft_consensus.cc:2804] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:16:46.385541 26862 raft_consensus.cc:697] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:46.385890 26868 raft_consensus.cc:697] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:46.386627 26862 consensus_queue.cc:237] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:46.386788 26868 consensus_queue.cc:237] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.397998 26438 catalog_manager.cc:5697] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 2 to 3, leader changed from 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:46.399161 26438 catalog_manager.cc:5697] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 2 to 3, leader changed from 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:46.428236 26874 raft_consensus.cc:993] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.428790 26855 raft_consensus.cc:1081] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.430269 26618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59476
I20260709 12:16:46.430743 26618 raft_consensus.cc:493] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:46.431001 26618 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.434852 26618 raft_consensus.cc:515] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.436340 26618 leader_election.cc:290] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:46.437167 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:46.437449 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7414c74b0d644f9a0a4138ac92ac641" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:46.437752 26542 raft_consensus.cc:3055] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:16:46.437947 26692 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.438155 26542 raft_consensus.cc:740] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:46.438896 26542 consensus_queue.cc:260] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.439867 26542 raft_consensus.cc:3060] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:16:46.442754 26692 raft_consensus.cc:2468] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.443799 26581 leader_election.cc:304] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555e0bc64ea4db28865ed9d4dc1e1e4, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:46.444253 26542 raft_consensus.cc:2468] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 3.
I20260709 12:16:46.444510 26868 raft_consensus.cc:2804] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:16:46.445068 26868 raft_consensus.cc:697] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:46.446022 26868 consensus_queue.cc:237] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.452674 26438 catalog_manager.cc:5697] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 2 to 3, leader changed from 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:46.470775 26864 raft_consensus.cc:993] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: : Instructing follower 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.471010 26542 raft_consensus.cc:1275] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.471243 26864 raft_consensus.cc:1081] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 LEADER]: Signalling peer 2555e0bc64ea4db28865ed9d4dc1e1e4 to start an election
I20260709 12:16:46.472373 26868 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:46.473114 26618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
 from {username='slave'} at 127.0.0.1:59480
I20260709 12:16:46.473623 26618 raft_consensus.cc:493] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:46.474004 26618 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:46.479652 26618 raft_consensus.cc:515] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.482439 26543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:46.483218 26543 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:46.492465 26543 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 4.
I20260709 12:16:46.493896 26618 leader_election.cc:290] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 4 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:46.495568 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:46.497882 26692 raft_consensus.cc:3055] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:16:46.498462 26692 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:46.499271 26692 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.495998 26689 raft_consensus.cc:1275] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.500207 26692 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:46.501991 26584 leader_election.cc:304] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:46.502630 26862 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:46.502825 26868 raft_consensus.cc:2804] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:16:46.503449 26868 raft_consensus.cc:697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 LEADER]: Becoming Leader. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:46.504456 26868 consensus_queue.cc:237] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:46.508816 26692 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2555e0bc64ea4db28865ed9d4dc1e1e4 in term 4.
I20260709 12:16:46.515532 26437 catalog_manager.cc:5697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 reported cstate change: term changed from 3 to 4, leader changed from 274d926e226c4516a89665f833a7a626 (127.25.202.3) to 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:46.843272 26543 raft_consensus.cc:1275] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.844538 26862 consensus_queue.cc:1048] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:46.853693 26543 raft_consensus.cc:1275] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.856164 26689 raft_consensus.cc:1275] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.856158 26862 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:16:46.857882 26877 consensus_queue.cc:1048] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:46.872491 26689 raft_consensus.cc:1275] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.874063 26868 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:46.895818 26542 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:46.897089 26862 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:16:46.907681 26692 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:46.909056 26868 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:16:46.989794 26542 raft_consensus.cc:1275] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:46.991001 26868 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:46.999617 26692 raft_consensus.cc:1275] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Refusing update from remote peer 2555e0bc64ea4db28865ed9d4dc1e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:16:47.001748 26862 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:16:49.051707 26408 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:16:49.052599 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:49.059737 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:32792
I20260709 12:16:49.060535 26618 raft_consensus.cc:606] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Received request to transfer leadership to TS 274d926e226c4516a89665f833a7a626
I20260709 12:16:49.063601 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "671e004742704504be3eac414fa3080a"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:32792
I20260709 12:16:49.064122 26618 raft_consensus.cc:606] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Received request to transfer leadership to TS 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:49.066617 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:32792
I20260709 12:16:49.067059 26618 raft_consensus.cc:606] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Received request to transfer leadership to TS 274d926e226c4516a89665f833a7a626
I20260709 12:16:49.069427 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a896abd94094634a10e4ff43434a321"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:32792
I20260709 12:16:49.069871 26618 raft_consensus.cc:606] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Received request to transfer leadership to TS 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:49.072448 26618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
mode: GRACEFUL
new_leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:32792
I20260709 12:16:49.072942 26618 raft_consensus.cc:606] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 LEADER]: Received request to transfer leadership to TS 0102899a9b8c4d798a1ef5a9d20c30db
I20260709 12:16:49.073915 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:16:49.074508 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:49.115829 26876 raft_consensus.cc:993] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:49.116379 26868 raft_consensus.cc:1081] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Signalling peer 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:49.117844 26543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "671e004742704504be3eac414fa3080a"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:46482
I20260709 12:16:49.118260 26543 raft_consensus.cc:493] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:49.118577 26543 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.124121 26543 raft_consensus.cc:515] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.126253 26543 leader_election.cc:290] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 4 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:49.127307 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:49.127552 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e004742704504be3eac414fa3080a" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:49.128116 26618 raft_consensus.cc:3055] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:16:49.128149 26692 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.128566 26618 raft_consensus.cc:740] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:49.129462 26618 consensus_queue.cc:260] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.130713 26618 raft_consensus.cc:3060] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.134975 26692 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 4.
I20260709 12:16:49.136412 26506 leader_election.cc:304] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:49.136782 26618 raft_consensus.cc:2468] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 4.
I20260709 12:16:49.138090 26896 raft_consensus.cc:2804] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:16:49.138895 26896 raft_consensus.cc:697] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:49.139868 26896 consensus_queue.cc:237] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.146736 26438 catalog_manager.cc:5697] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 3 to 4, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 4 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:49.172286 26876 raft_consensus.cc:993] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:49.172715 26868 raft_consensus.cc:1081] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 LEADER]: Signalling peer 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:49.174223 26543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:46482
I20260709 12:16:49.174690 26543 raft_consensus.cc:493] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:49.174970 26543 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:16:49.180994 26543 raft_consensus.cc:515] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.182507 26543 leader_election.cc:290] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 5 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:49.183517 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:49.183480 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adb287b7a1484050ac3f8d8e8f2e01c4" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:49.184212 26692 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:16:49.184249 26618 raft_consensus.cc:3055] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:16:49.184711 26618 raft_consensus.cc:740] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:49.185523 26618 consensus_queue.cc:260] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.186739 26618 raft_consensus.cc:3060] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:16:49.190827 26692 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 5.
I20260709 12:16:49.192003 26506 leader_election.cc:304] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:49.192755 26896 raft_consensus.cc:2804] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:16:49.192965 26618 raft_consensus.cc:2468] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 5.
I20260709 12:16:49.193449 26896 raft_consensus.cc:697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 5 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:49.194351 26896 consensus_queue.cc:237] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.202178 26438 catalog_manager.cc:5697] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 4 to 5, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 5 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:49.303903 26876 raft_consensus.cc:993] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:49.304423 26868 raft_consensus.cc:1081] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Signalling peer 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:49.306399 26692 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9"
dest_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:56932
I20260709 12:16:49.307078 26692 raft_consensus.cc:493] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:49.307533 26692 raft_consensus.cc:3060] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:49.310308 26876 raft_consensus.cc:993] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:49.310772 26868 raft_consensus.cc:1081] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Signalling peer 0102899a9b8c4d798a1ef5a9d20c30db to start an election
I20260709 12:16:49.312427 26543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a896abd94094634a10e4ff43434a321"
dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
 from {username='slave'} at 127.0.0.1:46482
I20260709 12:16:49.312866 26543 raft_consensus.cc:493] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:49.313141 26543 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.313277 26692 raft_consensus.cc:515] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.315548 26692 leader_election.cc:290] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 2 election: Requested vote from peers 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745), 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495)
I20260709 12:16:49.316529 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:49.316581 26618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69697e4373f54f9388bdf85cd3c393b9" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:49.317271 26618 raft_consensus.cc:3055] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:16:49.317257 26542 raft_consensus.cc:3060] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:49.317780 26618 raft_consensus.cc:740] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:49.318630 26618 consensus_queue.cc:260] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.318989 26543 raft_consensus.cc:515] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.320006 26618 raft_consensus.cc:3060] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:16:49.320950 26543 leader_election.cc:290] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 4 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 274d926e226c4516a89665f833a7a626 (127.25.202.3:33155)
I20260709 12:16:49.321664 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:49.322206 26617 raft_consensus.cc:3055] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:16:49.322461 26617 raft_consensus.cc:740] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:49.323151 26617 consensus_queue.cc:260] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.324090 26617 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.324733 26542 raft_consensus.cc:2468] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 2.
I20260709 12:16:49.326128 26658 leader_election.cc:304] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:49.326862 26618 raft_consensus.cc:2468] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 2.
I20260709 12:16:49.327086 26864 raft_consensus.cc:2804] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:16:49.328035 26692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a896abd94094634a10e4ff43434a321" candidate_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "274d926e226c4516a89665f833a7a626"
I20260709 12:16:49.328779 26692 raft_consensus.cc:3060] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.329015 26864 raft_consensus.cc:697] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:49.329947 26864 consensus_queue.cc:237] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.331372 26617 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 4.
I20260709 12:16:49.332513 26507 leader_election.cc:304] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 2555e0bc64ea4db28865ed9d4dc1e1e4; no voters: 
I20260709 12:16:49.333302 26692 raft_consensus.cc:2468] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102899a9b8c4d798a1ef5a9d20c30db in term 4.
I20260709 12:16:49.333377 26896 raft_consensus.cc:2804] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:16:49.333905 26896 raft_consensus.cc:697] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 LEADER]: Becoming Leader. State: Replica: 0102899a9b8c4d798a1ef5a9d20c30db, State: Running, Role: LEADER
I20260709 12:16:49.334749 26896 consensus_queue.cc:237] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } }
I20260709 12:16:49.339473 26438 catalog_manager.cc:5697] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 1 to 2, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:49.343003 26439 catalog_manager.cc:5697] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db reported cstate change: term changed from 3 to 4, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1). New cstate: current_term: 4 leader_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:49.491552 26876 raft_consensus.cc:993] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: : Instructing follower 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:49.492224 26868 raft_consensus.cc:1081] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Signalling peer 274d926e226c4516a89665f833a7a626 to start an election
I20260709 12:16:49.493785 26692 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500"
dest_uuid: "274d926e226c4516a89665f833a7a626"
 from {username='slave'} at 127.0.0.1:56932
I20260709 12:16:49.494254 26692 raft_consensus.cc:493] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:49.494535 26692 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.498829 26692 raft_consensus.cc:515] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:49.500336 26692 leader_election.cc:290] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 4 election: Requested vote from peers 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495), 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745)
I20260709 12:16:49.501302 26617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4"
I20260709 12:16:49.501380 26542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ed028afc1a44e828f7af9f781189500" candidate_uuid: "274d926e226c4516a89665f833a7a626" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0102899a9b8c4d798a1ef5a9d20c30db"
I20260709 12:16:49.501808 26617 raft_consensus.cc:3055] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:16:49.501956 26542 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.502213 26617 raft_consensus.cc:740] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2555e0bc64ea4db28865ed9d4dc1e1e4, State: Running, Role: LEADER
I20260709 12:16:49.502995 26617 consensus_queue.cc:260] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:49.503975 26617 raft_consensus.cc:3060] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:16:49.506672 26542 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 4.
I20260709 12:16:49.507712 26658 leader_election.cc:304] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102899a9b8c4d798a1ef5a9d20c30db, 274d926e226c4516a89665f833a7a626; no voters: 
I20260709 12:16:49.508677 26864 raft_consensus.cc:2804] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:16:49.508980 26617 raft_consensus.cc:2468] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274d926e226c4516a89665f833a7a626 in term 4.
I20260709 12:16:49.509135 26864 raft_consensus.cc:697] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 4 LEADER]: Becoming Leader. State: Replica: 274d926e226c4516a89665f833a7a626, State: Running, Role: LEADER
I20260709 12:16:49.510089 26864 consensus_queue.cc:237] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } }
I20260709 12:16:49.516649 26439 catalog_manager.cc:5697] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 reported cstate change: term changed from 3 to 4, leader changed from 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2) to 274d926e226c4516a89665f833a7a626 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "274d926e226c4516a89665f833a7a626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:49.589164 26617 raft_consensus.cc:1275] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:49.590559 26896 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:16:49.596479 26692 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 5 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:16:49.598208 26896 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:16:49.611897 26618 raft_consensus.cc:1275] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 5 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:16:49.614288 26896 consensus_queue.cc:1048] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:16:49.615393 26692 raft_consensus.cc:1275] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:49.620450 26897 consensus_queue.cc:1048] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:16:49.786604 26617 raft_consensus.cc:1275] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:49.788081 26864 consensus_queue.cc:1048] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:49.798872 26542 raft_consensus.cc:1275] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:16:49.800583 26864 consensus_queue.cc:1048] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:16:49.894474 26617 raft_consensus.cc:1275] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:49.896050 26912 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:16:49.906273 26689 raft_consensus.cc:1275] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Refusing update from remote peer 0102899a9b8c4d798a1ef5a9d20c30db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:49.910178 26912 consensus_queue.cc:1048] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d926e226c4516a89665f833a7a626" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:16:49.915591 26617 raft_consensus.cc:1275] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:49.919482 26864 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555e0bc64ea4db28865ed9d4dc1e1e4" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:16:49.929723 26542 raft_consensus.cc:1275] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Refusing update from remote peer 274d926e226c4516a89665f833a7a626: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:16:49.931234 26919 consensus_queue.cc:1048] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102899a9b8c4d798a1ef5a9d20c30db" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:16:50.075655 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:50.080715 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:50.081203 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:51.082357 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:51.087790 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:51.088241 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:52.089324 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:52.094600 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:52.095042 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:53.096104 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:53.101186 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:53.101683 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:54.102842 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:54.108654 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:54.109143 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:55.110225 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:55.115043 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:55.115502 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:56.116540 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:56.121953 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:56.122390 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:57.123574 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:57.128683 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:57.129137 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:58.131083 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:58.137414 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:16:58.137893 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:59.166143 26408 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:16:59.166749 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
W20260709 12:16:59.179610 26584 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:16:59.188313 26584 consensus_peers.cc:597] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 -> Peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Couldn't send request to peer 0102899a9b8c4d798a1ef5a9d20c30db. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:16:59.196921 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:16:59.197772 26408 tablet_replica.cc:333] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.198470 26408 raft_consensus.cc:2243] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.199103 26408 raft_consensus.cc:2272] T b7414c74b0d644f9a0a4138ac92ac641 P 0102899a9b8c4d798a1ef5a9d20c30db [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.201534 26408 tablet_replica.cc:333] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.202035 26408 raft_consensus.cc:2243] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.202801 26408 raft_consensus.cc:2272] T 6a896abd94094634a10e4ff43434a321 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.204459 26408 tablet_replica.cc:333] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.204902 26408 raft_consensus.cc:2243] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.205277 26408 raft_consensus.cc:2272] T 69697e4373f54f9388bdf85cd3c393b9 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.207564 26408 tablet_replica.cc:333] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.208206 26408 raft_consensus.cc:2243] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.208779 26408 raft_consensus.cc:2272] T 1d2cef36dd3a404fbc61811621813c5d P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.210567 26408 tablet_replica.cc:333] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.211097 26408 raft_consensus.cc:2243] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.211567 26408 raft_consensus.cc:2272] T 2b91295b45e0487196251617ffcdf3f7 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.213570 26408 tablet_replica.cc:333] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.214107 26408 raft_consensus.cc:2243] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.214596 26408 raft_consensus.cc:2272] T 0ed028afc1a44e828f7af9f781189500 P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.216404 26408 tablet_replica.cc:333] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.217190 26408 raft_consensus.cc:2243] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.218258 26408 raft_consensus.cc:2272] T adb287b7a1484050ac3f8d8e8f2e01c4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.220395 26408 tablet_replica.cc:333] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.220816 26408 raft_consensus.cc:2243] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.221171 26408 raft_consensus.cc:2272] T 3002320b53d74f4cb4450e78d74d37d4 P 0102899a9b8c4d798a1ef5a9d20c30db [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.222761 26408 tablet_replica.cc:333] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db: stopping tablet replica
I20260709 12:16:59.223160 26408 raft_consensus.cc:2243] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.223935 26408 raft_consensus.cc:2272] T 671e004742704504be3eac414fa3080a P 0102899a9b8c4d798a1ef5a9d20c30db [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.319847 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
I20260709 12:16:59.339520 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
W20260709 12:16:59.345978 26584 consensus_peers.cc:597] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 -> Peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Couldn't send request to peer 0102899a9b8c4d798a1ef5a9d20c30db. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:36745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:16:59.351132 26656 consensus_peers.cc:597] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 -> Peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Couldn't send request to peer 2555e0bc64ea4db28865ed9d4dc1e1e4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:16:59.372216 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:16:59.373008 26408 tablet_replica.cc:333] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.373679 26408 raft_consensus.cc:2243] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.374518 26408 raft_consensus.cc:2272] T 1d2cef36dd3a404fbc61811621813c5d P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.376302 26408 tablet_replica.cc:333] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.376835 26408 raft_consensus.cc:2243] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.377321 26408 raft_consensus.cc:2272] T adb287b7a1484050ac3f8d8e8f2e01c4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.378973 26408 tablet_replica.cc:333] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.379524 26408 raft_consensus.cc:2243] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.380244 26408 raft_consensus.cc:2272] T 3002320b53d74f4cb4450e78d74d37d4 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.381824 26408 tablet_replica.cc:333] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.382303 26408 raft_consensus.cc:2243] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.382697 26408 raft_consensus.cc:2272] T 671e004742704504be3eac414fa3080a P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.384393 26408 tablet_replica.cc:333] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.384919 26408 raft_consensus.cc:2243] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.385314 26408 raft_consensus.cc:2272] T 6a896abd94094634a10e4ff43434a321 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.386960 26408 tablet_replica.cc:333] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.387491 26408 raft_consensus.cc:2243] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.387936 26408 raft_consensus.cc:2272] T 69697e4373f54f9388bdf85cd3c393b9 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.389503 26408 tablet_replica.cc:333] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.390000 26408 raft_consensus.cc:2243] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.390432 26408 raft_consensus.cc:2272] T 0ed028afc1a44e828f7af9f781189500 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.392073 26408 tablet_replica.cc:333] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.392613 26408 raft_consensus.cc:2243] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.393045 26408 raft_consensus.cc:2272] T 2b91295b45e0487196251617ffcdf3f7 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.394654 26408 tablet_replica.cc:333] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4: stopping tablet replica
I20260709 12:16:59.395095 26408 raft_consensus.cc:2243] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.395869 26408 raft_consensus.cc:2272] T b7414c74b0d644f9a0a4138ac92ac641 P 2555e0bc64ea4db28865ed9d4dc1e1e4 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:16:59.404546 26658 consensus_peers.cc:597] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 -> Peer 0102899a9b8c4d798a1ef5a9d20c30db (127.25.202.1:36745): Couldn't send request to peer 0102899a9b8c4d798a1ef5a9d20c30db. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:36745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:16:59.425783 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
W20260709 12:16:59.440092 26656 consensus_peers.cc:597] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 -> Peer 2555e0bc64ea4db28865ed9d4dc1e1e4 (127.25.202.2:36495): Couldn't send request to peer 2555e0bc64ea4db28865ed9d4dc1e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:36495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:16:59.446588 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
I20260709 12:16:59.477536 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:16:59.478318 26408 tablet_replica.cc:333] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.478966 26408 raft_consensus.cc:2243] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.479893 26408 raft_consensus.cc:2272] T 2b91295b45e0487196251617ffcdf3f7 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.481773 26408 tablet_replica.cc:333] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.482312 26408 raft_consensus.cc:2243] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.483136 26408 raft_consensus.cc:2272] T 0ed028afc1a44e828f7af9f781189500 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.485052 26408 tablet_replica.cc:333] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.485579 26408 raft_consensus.cc:2243] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.486091 26408 raft_consensus.cc:2272] T 1d2cef36dd3a404fbc61811621813c5d P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.487985 26408 tablet_replica.cc:333] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.488508 26408 raft_consensus.cc:2243] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.488958 26408 raft_consensus.cc:2272] T adb287b7a1484050ac3f8d8e8f2e01c4 P 274d926e226c4516a89665f833a7a626 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.490782 26408 tablet_replica.cc:333] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.491291 26408 raft_consensus.cc:2243] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.491914 26408 raft_consensus.cc:2272] T b7414c74b0d644f9a0a4138ac92ac641 P 274d926e226c4516a89665f833a7a626 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.493502 26408 tablet_replica.cc:333] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.493963 26408 raft_consensus.cc:2243] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.494851 26408 raft_consensus.cc:2272] T 69697e4373f54f9388bdf85cd3c393b9 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.496517 26408 tablet_replica.cc:333] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.496924 26408 raft_consensus.cc:2243] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.497334 26408 raft_consensus.cc:2272] T 6a896abd94094634a10e4ff43434a321 P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.498950 26408 tablet_replica.cc:333] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.499442 26408 raft_consensus.cc:2243] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.499845 26408 raft_consensus.cc:2272] T 3002320b53d74f4cb4450e78d74d37d4 P 274d926e226c4516a89665f833a7a626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.501307 26408 tablet_replica.cc:333] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626: stopping tablet replica
I20260709 12:16:59.501713 26408 raft_consensus.cc:2243] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:16:59.502092 26408 raft_consensus.cc:2272] T 671e004742704504be3eac414fa3080a P 274d926e226c4516a89665f833a7a626 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.546247 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:16:59.564282 26408 master.cc:562] Master@127.25.202.62:43537 shutting down...
I20260709 12:16:59.635178 26495 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260709 12:16:59.772126 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:16:59.772722 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:16:59.773099 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4b593c0701e14b1fafd7a203155de76b: stopping tablet replica
I20260709 12:16:59.794143 26408 master.cc:584] Master@127.25.202.62:43537 shutdown complete.
0102899a9b8c4d798a1ef5a9d20c30db  4
2555e0bc64ea4db28865ed9d4dc1e1e4  4
274d926e226c4516a89665f833a7a626  1
0102899a9b8c4d798a1ef5a9d20c30db  3
2555e0bc64ea4db28865ed9d4dc1e1e4  3
274d926e226c4516a89665f833a7a626  3
0102899a9b8c4d798a1ef5a9d20c30db  0
2555e0bc64ea4db28865ed9d4dc1e1e4  8
274d926e226c4516a89665f833a7a626  1
0102899a9b8c4d798a1ef5a9d20c30db  3
2555e0bc64ea4db28865ed9d4dc1e1e4  3
274d926e226c4516a89665f833a7a626  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (31830 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:16:59.835659 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:45067
I20260709 12:16:59.836836 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:16:59.842599 26934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:59.843544 26935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:16:59.843807 26937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:16:59.845628 26408 server_base.cc:1061] running on GCE node
I20260709 12:16:59.846685 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:16:59.846910 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:16:59.847074 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599419847056 us; error 0 us; skew 500 ppm
I20260709 12:16:59.847997 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:16:59.850739 26408 webserver.cc:533] Webserver started at http://127.25.202.62:39191/ using document root <none> and password file <none>
I20260709 12:16:59.851269 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:16:59.851528 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:16:59.851807 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:16:59.853084 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "625de6d6b13746aabaca5197f0ec7bbb"
format_stamp: "Formatted at 2026-07-09 12:16:59 on dist-test-slave-m57n"
I20260709 12:16:59.857749 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:16:59.861541 26942 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:16:59.862403 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:16:59.862645 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "625de6d6b13746aabaca5197f0ec7bbb"
format_stamp: "Formatted at 2026-07-09 12:16:59 on dist-test-slave-m57n"
I20260709 12:16:59.862891 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:16:59.880569 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:16:59.881798 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:16:59.920075 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:45067
I20260709 12:16:59.920173 26993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:45067 every 8 connection(s)
I20260709 12:16:59.924196 26994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:16:59.933737 26994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb: Bootstrap starting.
I20260709 12:16:59.938339 26994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:16:59.942744 26994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb: No bootstrap required, opened a new log
I20260709 12:16:59.944876 26994 raft_consensus.cc:359] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625de6d6b13746aabaca5197f0ec7bbb" member_type: VOTER }
I20260709 12:16:59.945349 26994 raft_consensus.cc:385] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:16:59.945598 26994 raft_consensus.cc:740] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625de6d6b13746aabaca5197f0ec7bbb, State: Initialized, Role: FOLLOWER
I20260709 12:16:59.946157 26994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625de6d6b13746aabaca5197f0ec7bbb" member_type: VOTER }
I20260709 12:16:59.946621 26994 raft_consensus.cc:399] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:16:59.946835 26994 raft_consensus.cc:493] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:16:59.947129 26994 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:16:59.951067 26994 raft_consensus.cc:515] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625de6d6b13746aabaca5197f0ec7bbb" member_type: VOTER }
I20260709 12:16:59.951612 26994 leader_election.cc:304] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625de6d6b13746aabaca5197f0ec7bbb; no voters: 
I20260709 12:16:59.952729 26994 leader_election.cc:290] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:16:59.953050 26997 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:16:59.954490 26997 raft_consensus.cc:697] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 1 LEADER]: Becoming Leader. State: Replica: 625de6d6b13746aabaca5197f0ec7bbb, State: Running, Role: LEADER
I20260709 12:16:59.955217 26997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625de6d6b13746aabaca5197f0ec7bbb" member_type: VOTER }
I20260709 12:16:59.955771 26994 sys_catalog.cc:565] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:16:59.960762 26998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625de6d6b13746aabaca5197f0ec7bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625de6d6b13746aabaca5197f0ec7bbb" member_type: VOTER } }
I20260709 12:16:59.960695 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625de6d6b13746aabaca5197f0ec7bbb. Latest consensus state: current_term: 1 leader_uuid: "625de6d6b13746aabaca5197f0ec7bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625de6d6b13746aabaca5197f0ec7bbb" member_type: VOTER } }
I20260709 12:16:59.961758 26998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [sys.catalog]: This master's current role is: LEADER
I20260709 12:16:59.961854 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [sys.catalog]: This master's current role is: LEADER
I20260709 12:16:59.964811 27004 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:16:59.973318 27004 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:16:59.974318 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:16:59.982652 27004 catalog_manager.cc:1383] Generated new cluster ID: 2a4185c6200e4d1fbeb49d54d42c0d19
I20260709 12:16:59.982892 27004 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:17:00.014416 27004 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:17:00.015750 27004 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:17:00.028432 27004 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb: Generated new TSK 0
I20260709 12:17:00.029064 27004 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:17:00.040884 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:00.047788 27015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:00.048669 27016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:00.050868 26408 server_base.cc:1061] running on GCE node
W20260709 12:17:00.051182 27018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:00.052172 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:00.052361 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:00.052484 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599420052472 us; error 0 us; skew 500 ppm
I20260709 12:17:00.052981 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:00.055202 26408 webserver.cc:533] Webserver started at http://127.25.202.1:33851/ using document root <none> and password file <none>
I20260709 12:17:00.055714 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:00.055874 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:00.056077 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:00.057207 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "f6318bc9ad074866ba228c136b4c8426"
format_stamp: "Formatted at 2026-07-09 12:17:00 on dist-test-slave-m57n"
I20260709 12:17:00.061657 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20260709 12:17:00.065474 27023 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.066449 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:17:00.066772 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "f6318bc9ad074866ba228c136b4c8426"
format_stamp: "Formatted at 2026-07-09 12:17:00 on dist-test-slave-m57n"
I20260709 12:17:00.067094 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:00.079134 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:00.080410 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:00.081877 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:00.084281 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:00.084506 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.084750 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:00.084945 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.126963 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:45039
I20260709 12:17:00.127075 27085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:45039 every 8 connection(s)
I20260709 12:17:00.131819 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:00.140153 27090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:00.140903 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:00.144613 27093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:00.146021 26408 server_base.cc:1061] running on GCE node
I20260709 12:17:00.146973 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:00.147158 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:00.147441 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599420147421 us; error 0 us; skew 500 ppm
I20260709 12:17:00.147964 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:00.150776 26408 webserver.cc:533] Webserver started at http://127.25.202.2:44795/ using document root <none> and password file <none>
I20260709 12:17:00.151413 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:00.151646 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:00.151939 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:00.152326 27086 heartbeater.cc:344] Connected to a master server at 127.25.202.62:45067
I20260709 12:17:00.152714 27086 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:00.153437 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "aeb5150d7d474113b386036e444b80d7"
format_stamp: "Formatted at 2026-07-09 12:17:00 on dist-test-slave-m57n"
I20260709 12:17:00.153589 27086 heartbeater.cc:507] Master 127.25.202.62:45067 requested a full tablet report, sending...
I20260709 12:17:00.155836 26959 ts_manager.cc:194] Registered new tserver with Master: f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039)
I20260709 12:17:00.157811 26959 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36442
I20260709 12:17:00.158648 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:17:00.162631 27098 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.163590 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:17:00.163929 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "aeb5150d7d474113b386036e444b80d7"
format_stamp: "Formatted at 2026-07-09 12:17:00 on dist-test-slave-m57n"
I20260709 12:17:00.164229 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:00.180773 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:00.181993 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:00.183634 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:00.185942 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:00.186167 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.186422 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:00.186594 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.230154 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:39609
I20260709 12:17:00.230443 27160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:39609 every 8 connection(s)
I20260709 12:17:00.234778 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:00.242374 27164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:00.242913 27165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:00.249137 27161 heartbeater.cc:344] Connected to a master server at 127.25.202.62:45067
I20260709 12:17:00.249526 27161 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:00.250226 27161 heartbeater.cc:507] Master 127.25.202.62:45067 requested a full tablet report, sending...
I20260709 12:17:00.252530 26959 ts_manager.cc:194] Registered new tserver with Master: aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:00.254849 26959 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36454
W20260709 12:17:00.255875 27167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:00.257593 26408 server_base.cc:1061] running on GCE node
I20260709 12:17:00.258639 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:00.258824 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:00.258951 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599420258941 us; error 0 us; skew 500 ppm
I20260709 12:17:00.259490 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:00.261894 26408 webserver.cc:533] Webserver started at http://127.25.202.3:37167/ using document root <none> and password file <none>
I20260709 12:17:00.262337 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:00.262491 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:00.262693 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:00.263823 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
format_stamp: "Formatted at 2026-07-09 12:17:00 on dist-test-slave-m57n"
I20260709 12:17:00.268585 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:17:00.272493 27172 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.273409 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:17:00.273785 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
format_stamp: "Formatted at 2026-07-09 12:17:00 on dist-test-slave-m57n"
I20260709 12:17:00.274098 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:00.300453 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:00.301730 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:00.303218 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:00.305420 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:00.305655 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.305904 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:00.306094 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:00.348462 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:43207
I20260709 12:17:00.348590 27234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:43207 every 8 connection(s)
I20260709 12:17:00.362990 27235 heartbeater.cc:344] Connected to a master server at 127.25.202.62:45067
I20260709 12:17:00.363348 27235 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:00.364058 27235 heartbeater.cc:507] Master 127.25.202.62:45067 requested a full tablet report, sending...
I20260709 12:17:00.365933 26959 ts_manager.cc:194] Registered new tserver with Master: 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.366274 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014500122s
I20260709 12:17:00.367729 26959 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36466
I20260709 12:17:00.392309 26959 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36482:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:17:00.394990 26959 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:17:00.472716 27051 tablet_service.cc:1511] Processing CreateTablet for tablet 9013e4bd5c25415a977c1a085842d0ac (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:17:00.474213 27051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9013e4bd5c25415a977c1a085842d0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.478694 27050 tablet_service.cc:1511] Processing CreateTablet for tablet cae1b0bb8aba4ea69fed0507b2cdd16f (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:17:00.481323 27124 tablet_service.cc:1511] Processing CreateTablet for tablet 88553a5581b04f03be2745006b011395 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:17:00.481818 27049 tablet_service.cc:1511] Processing CreateTablet for tablet 88553a5581b04f03be2745006b011395 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:17:00.481956 27121 tablet_service.cc:1511] Processing CreateTablet for tablet 943c0718dc5d44889b02cdbbe48b1318 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:17:00.482895 27124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88553a5581b04f03be2745006b011395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.483546 27121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943c0718dc5d44889b02cdbbe48b1318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.484110 27048 tablet_service.cc:1511] Processing CreateTablet for tablet d7f747d704794e3fb8dde0d7ba27da7a (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:17:00.484663 27123 tablet_service.cc:1511] Processing CreateTablet for tablet d7f747d704794e3fb8dde0d7ba27da7a (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:17:00.486135 27123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f747d704794e3fb8dde0d7ba27da7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.486444 27047 tablet_service.cc:1511] Processing CreateTablet for tablet b7b4d71766084c9caafdaaaa2243a249 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:17:00.488955 27046 tablet_service.cc:1511] Processing CreateTablet for tablet 943c0718dc5d44889b02cdbbe48b1318 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:17:00.489725 27126 tablet_service.cc:1511] Processing CreateTablet for tablet 9013e4bd5c25415a977c1a085842d0ac (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:17:00.491130 27126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9013e4bd5c25415a977c1a085842d0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.491633 27050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae1b0bb8aba4ea69fed0507b2cdd16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.495716 27048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f747d704794e3fb8dde0d7ba27da7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.495755 27125 tablet_service.cc:1511] Processing CreateTablet for tablet cae1b0bb8aba4ea69fed0507b2cdd16f (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:17:00.497318 27125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae1b0bb8aba4ea69fed0507b2cdd16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.499636 27047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b4d71766084c9caafdaaaa2243a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.488701 27122 tablet_service.cc:1511] Processing CreateTablet for tablet b7b4d71766084c9caafdaaaa2243a249 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:17:00.500689 27122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b4d71766084c9caafdaaaa2243a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.503768 27046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943c0718dc5d44889b02cdbbe48b1318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.507746 27049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88553a5581b04f03be2745006b011395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.516078 27250 tablet_bootstrap.cc:492] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.526341 27250 tablet_bootstrap.cc:654] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.527524 27200 tablet_service.cc:1511] Processing CreateTablet for tablet 9013e4bd5c25415a977c1a085842d0ac (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:17:00.534112 27200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9013e4bd5c25415a977c1a085842d0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.528983 27196 tablet_service.cc:1511] Processing CreateTablet for tablet b7b4d71766084c9caafdaaaa2243a249 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:17:00.539815 27251 tablet_bootstrap.cc:492] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:00.540367 27196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b4d71766084c9caafdaaaa2243a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.529417 27199 tablet_service.cc:1511] Processing CreateTablet for tablet cae1b0bb8aba4ea69fed0507b2cdd16f (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:17:00.542387 27199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae1b0bb8aba4ea69fed0507b2cdd16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.546286 27251 tablet_bootstrap.cc:654] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.548259 27197 tablet_service.cc:1511] Processing CreateTablet for tablet d7f747d704794e3fb8dde0d7ba27da7a (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:17:00.549693 27197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f747d704794e3fb8dde0d7ba27da7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.531551 27198 tablet_service.cc:1511] Processing CreateTablet for tablet 88553a5581b04f03be2745006b011395 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:17:00.550920 27198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88553a5581b04f03be2745006b011395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.566402 27195 tablet_service.cc:1511] Processing CreateTablet for tablet 943c0718dc5d44889b02cdbbe48b1318 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:17:00.568008 27195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943c0718dc5d44889b02cdbbe48b1318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.570382 27194 tablet_service.cc:1511] Processing CreateTablet for tablet ceeac06a7cf54e40b81ce70b21a222e3 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:17:00.572288 27194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceeac06a7cf54e40b81ce70b21a222e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.580374 27125 tablet_service.cc:1511] Processing CreateTablet for tablet ceeac06a7cf54e40b81ce70b21a222e3 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:17:00.585561 27048 tablet_service.cc:1511] Processing CreateTablet for tablet d7d884995e6e40c6bf8be25283343717 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:17:00.586659 27250 tablet_bootstrap.cc:492] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:00.586824 27048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d884995e6e40c6bf8be25283343717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.587267 27250 ts_tablet_manager.cc:1403] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.072s	user 0.017s	sys 0.000s
I20260709 12:17:00.592885 27250 raft_consensus.cc:359] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.589354 27122 tablet_service.cc:1511] Processing CreateTablet for tablet d7d884995e6e40c6bf8be25283343717 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:17:00.597682 27250 raft_consensus.cc:385] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.598023 27250 raft_consensus.cc:740] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.598340 27122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d884995e6e40c6bf8be25283343717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.598920 27250 consensus_queue.cc:260] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.598999 27049 tablet_service.cc:1511] Processing CreateTablet for tablet 88847fc535ed499cb53f82a470f81ded (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:17:00.583199 27050 tablet_service.cc:1511] Processing CreateTablet for tablet ceeac06a7cf54e40b81ce70b21a222e3 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:17:00.602849 27049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88847fc535ed499cb53f82a470f81ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.603515 27050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceeac06a7cf54e40b81ce70b21a222e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.603935 27126 tablet_service.cc:1511] Processing CreateTablet for tablet 88847fc535ed499cb53f82a470f81ded (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:17:00.605212 27126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88847fc535ed499cb53f82a470f81ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.610344 27200 tablet_service.cc:1511] Processing CreateTablet for tablet 88847fc535ed499cb53f82a470f81ded (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:17:00.611251 27125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceeac06a7cf54e40b81ce70b21a222e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.611658 27200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88847fc535ed499cb53f82a470f81ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.619112 27251 tablet_bootstrap.cc:492] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:00.634105 27251 ts_tablet_manager.cc:1403] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.106s	user 0.009s	sys 0.005s
I20260709 12:17:00.625479 27197 tablet_service.cc:1511] Processing CreateTablet for tablet 9af6b5759a1a40a2854bd1ed9df3ec8a (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:17:00.635432 27197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af6b5759a1a40a2854bd1ed9df3ec8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.620128 27196 tablet_service.cc:1511] Processing CreateTablet for tablet d7d884995e6e40c6bf8be25283343717 (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:17:00.620535 27254 tablet_bootstrap.cc:492] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:00.647580 27161 heartbeater.cc:499] Master 127.25.202.62:45067 was elected leader, sending a full tablet report...
I20260709 12:17:00.632416 27046 tablet_service.cc:1511] Processing CreateTablet for tablet 9af6b5759a1a40a2854bd1ed9df3ec8a (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:17:00.621012 27124 tablet_service.cc:1511] Processing CreateTablet for tablet 9af6b5759a1a40a2854bd1ed9df3ec8a (DEFAULT_TABLE table=test-workload [id=f7750c5bf9f94d6daf9f8a8897758942]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:17:00.648917 27124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af6b5759a1a40a2854bd1ed9df3ec8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.650213 27254 tablet_bootstrap.cc:654] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.650295 27250 ts_tablet_manager.cc:1434] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.000s
I20260709 12:17:00.652101 27250 tablet_bootstrap.cc:492] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.652138 27046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af6b5759a1a40a2854bd1ed9df3ec8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.656013 27196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d884995e6e40c6bf8be25283343717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:00.665391 27251 raft_consensus.cc:359] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.666206 27251 raft_consensus.cc:385] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.668946 27251 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.669760 27251 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.665944 27250 tablet_bootstrap.cc:654] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.685833 27251 ts_tablet_manager.cc:1434] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.003s
I20260709 12:17:00.686707 27086 heartbeater.cc:499] Master 127.25.202.62:45067 was elected leader, sending a full tablet report...
I20260709 12:17:00.688071 27254 tablet_bootstrap.cc:492] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:00.688383 27251 tablet_bootstrap.cc:492] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:00.688576 27254 ts_tablet_manager.cc:1403] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.068s	user 0.014s	sys 0.020s
I20260709 12:17:00.691555 27254 raft_consensus.cc:359] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.692305 27254 raft_consensus.cc:385] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.692608 27254 raft_consensus.cc:740] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.693312 27254 consensus_queue.cc:260] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.694808 27251 tablet_bootstrap.cc:654] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.697218 27250 tablet_bootstrap.cc:492] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:00.698307 27250 ts_tablet_manager.cc:1403] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.046s	user 0.015s	sys 0.000s
I20260709 12:17:00.702517 27250 raft_consensus.cc:359] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.703725 27250 raft_consensus.cc:385] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.704268 27250 raft_consensus.cc:740] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.705672 27250 consensus_queue.cc:260] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.709345 27250 ts_tablet_manager.cc:1434] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:17:00.710894 27250 tablet_bootstrap.cc:492] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.714037 27235 heartbeater.cc:499] Master 127.25.202.62:45067 was elected leader, sending a full tablet report...
I20260709 12:17:00.715956 27254 ts_tablet_manager.cc:1434] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.027s	user 0.013s	sys 0.008s
I20260709 12:17:00.717247 27254 tablet_bootstrap.cc:492] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:00.718946 27250 tablet_bootstrap.cc:654] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.722594 27251 tablet_bootstrap.cc:492] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:00.723129 27251 ts_tablet_manager.cc:1403] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.035s	user 0.016s	sys 0.008s
I20260709 12:17:00.725350 27254 tablet_bootstrap.cc:654] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.726213 27251 raft_consensus.cc:359] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.727013 27251 raft_consensus.cc:385] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.727494 27251 raft_consensus.cc:740] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.728291 27251 consensus_queue.cc:260] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.730641 27251 ts_tablet_manager.cc:1434] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:17:00.731648 27251 tablet_bootstrap.cc:492] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:00.739059 27251 tablet_bootstrap.cc:654] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.763870 27250 tablet_bootstrap.cc:492] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:00.764374 27250 ts_tablet_manager.cc:1403] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.054s	user 0.029s	sys 0.019s
I20260709 12:17:00.764824 27254 tablet_bootstrap.cc:492] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:00.765192 27251 tablet_bootstrap.cc:492] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:00.765288 27254 ts_tablet_manager.cc:1403] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.048s	user 0.035s	sys 0.011s
I20260709 12:17:00.765700 27251 ts_tablet_manager.cc:1403] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.012s
I20260709 12:17:00.767114 27250 raft_consensus.cc:359] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.767833 27250 raft_consensus.cc:385] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.768203 27250 raft_consensus.cc:740] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.768395 27254 raft_consensus.cc:359] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.768558 27251 raft_consensus.cc:359] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.769146 27254 raft_consensus.cc:385] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.769273 27251 raft_consensus.cc:385] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.769563 27254 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.769718 27251 raft_consensus.cc:740] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.769160 27250 consensus_queue.cc:260] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.770462 27254 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.770439 27251 consensus_queue.cc:260] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.773061 27254 ts_tablet_manager.cc:1434] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:17:00.773842 27250 ts_tablet_manager.cc:1434] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.003s
I20260709 12:17:00.773991 27254 tablet_bootstrap.cc:492] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:00.774883 27250 tablet_bootstrap.cc:492] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.775039 27251 ts_tablet_manager.cc:1434] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.005s
I20260709 12:17:00.776247 27251 tablet_bootstrap.cc:492] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:00.779091 27255 raft_consensus.cc:493] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.779659 27255 raft_consensus.cc:515] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.781104 27254 tablet_bootstrap.cc:654] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.782258 27250 tablet_bootstrap.cc:654] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.783034 27251 tablet_bootstrap.cc:654] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.802716 27250 tablet_bootstrap.cc:492] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:00.803260 27250 ts_tablet_manager.cc:1403] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.003s
I20260709 12:17:00.806202 27250 raft_consensus.cc:359] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.806943 27250 raft_consensus.cc:385] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.807255 27250 raft_consensus.cc:740] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.808261 27250 consensus_queue.cc:260] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.811678 27255 leader_election.cc:290] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.815980 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:00.816820 27061 raft_consensus.cc:2468] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 0.
I20260709 12:17:00.818478 27102 leader_election.cc:304] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:00.822935 27255 raft_consensus.cc:2804] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:00.823410 27255 raft_consensus.cc:493] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.823788 27255 raft_consensus.cc:3060] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:00.828671 27250 ts_tablet_manager.cc:1434] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.005s
I20260709 12:17:00.830116 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:00.830868 27257 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.831398 27257 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
W20260709 12:17:00.831799 27102 leader_election.cc:343] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:00.834564 27250 tablet_bootstrap.cc:492] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.836936 27257 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.842182 27250 tablet_bootstrap.cc:654] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.849402 27255 raft_consensus.cc:515] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
W20260709 12:17:00.853137 27236 tablet.cc:2406] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:17:00.855621 27250 tablet_bootstrap.cc:492] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:00.856185 27250 ts_tablet_manager.cc:1403] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.027s	user 0.009s	sys 0.011s
I20260709 12:17:00.859593 27250 raft_consensus.cc:359] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.860466 27250 raft_consensus.cc:385] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.860786 27250 raft_consensus.cc:740] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.861549 27250 consensus_queue.cc:260] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.862346 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:00.862977 27061 raft_consensus.cc:3060] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:00.870611 27250 ts_tablet_manager.cc:1434] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:17:00.871819 27250 tablet_bootstrap.cc:492] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.872926 27061 raft_consensus.cc:2468] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:00.874383 27268 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.874872 27268 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.877319 27268 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.878784 27102 leader_election.cc:304] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:00.880436 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:00.882705 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:00.884583 27268 raft_consensus.cc:2804] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:17:00.885643 27102 leader_election.cc:343] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:00.886361 27102 leader_election.cc:343] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:00.886734 27102 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aeb5150d7d474113b386036e444b80d7; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:00.888137 27250 tablet_bootstrap.cc:654] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.898313 27259 raft_consensus.cc:493] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.898960 27259 raft_consensus.cc:515] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.905670 27259 leader_election.cc:290] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:00.910005 27272 raft_consensus.cc:2749] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:00.912469 27273 raft_consensus.cc:493] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.913041 27250 tablet_bootstrap.cc:492] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:00.912985 27273 raft_consensus.cc:515] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.913592 27250 ts_tablet_manager.cc:1403] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.042s	user 0.009s	sys 0.008s
I20260709 12:17:00.916908 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:00.916908 27250 raft_consensus.cc:359] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
W20260709 12:17:00.919976 27102 leader_election.cc:343] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:00.919152 27250 raft_consensus.cc:385] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.920684 27250 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.922438 27250 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.921356 27268 raft_consensus.cc:697] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:00.924311 27268 consensus_queue.cc:237] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.924850 27250 ts_tablet_manager.cc:1434] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:17:00.925853 27250 tablet_bootstrap.cc:492] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:00.921481 27273 leader_election.cc:290] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.919178 27255 leader_election.cc:290] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.935355 27251 tablet_bootstrap.cc:492] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:00.935913 27251 ts_tablet_manager.cc:1403] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.160s	user 0.044s	sys 0.069s
I20260709 12:17:00.941501 26959 catalog_manager.cc:5697] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:00.949331 27254 tablet_bootstrap.cc:492] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:00.949990 27254 ts_tablet_manager.cc:1403] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.176s	user 0.039s	sys 0.120s
I20260709 12:17:00.950229 27268 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.950724 27268 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.952870 27268 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.953989 27254 raft_consensus.cc:359] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.954814 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:00.955240 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:00.955533 27136 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 0.
I20260709 12:17:00.955847 27061 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 0.
I20260709 12:17:00.957031 27027 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:00.955046 27254 raft_consensus.cc:385] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.958043 27257 raft_consensus.cc:2804] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:00.958247 27254 raft_consensus.cc:740] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.958590 27257 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.959069 27257 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:00.959362 27254 consensus_queue.cc:260] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.960621 27102 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:00.961820 27268 raft_consensus.cc:2804] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:00.962314 27268 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.962738 27268 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:00.970548 27254 ts_tablet_manager.cc:1434] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.020s	user 0.011s	sys 0.000s
I20260709 12:17:00.971680 27254 tablet_bootstrap.cc:492] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
W20260709 12:17:00.971917 27007 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:00.978488 27250 tablet_bootstrap.cc:654] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.981554 27257 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.982929 27254 tablet_bootstrap.cc:654] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:00.983716 27257 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:00.984856 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:00.994241 27251 raft_consensus.cc:359] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.995285 27251 raft_consensus.cc:385] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:00.995694 27251 raft_consensus.cc:740] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:00.995811 27268 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:00.997025 27136 raft_consensus.cc:2393] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6318bc9ad074866ba228c136b4c8426 in current term 1: Already voted for candidate aeb5150d7d474113b386036e444b80d7 in this term.
I20260709 12:17:00.997745 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:00.998732 27210 raft_consensus.cc:2468] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 0.
I20260709 12:17:01.002468 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.002403 27251 consensus_queue.cc:260] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.003223 27254 tablet_bootstrap.cc:492] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.003509 27061 raft_consensus.cc:2393] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aeb5150d7d474113b386036e444b80d7 in current term 1: Already voted for candidate f6318bc9ad074866ba228c136b4c8426 in this term.
I20260709 12:17:01.003724 27254 ts_tablet_manager.cc:1403] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.032s	user 0.016s	sys 0.010s
I20260709 12:17:01.004962 27251 ts_tablet_manager.cc:1434] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.069s	user 0.008s	sys 0.000s
I20260709 12:17:01.004967 27250 tablet_bootstrap.cc:492] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:01.006649 27250 ts_tablet_manager.cc:1403] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.081s	user 0.015s	sys 0.001s
I20260709 12:17:01.006569 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.007814 27210 raft_consensus.cc:3060] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.007982 27209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.009203 27102 leader_election.cc:304] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.009397 27208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.010764 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
W20260709 12:17:01.010950 27102 leader_election.cc:343] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.011513 27255 raft_consensus.cc:2804] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.011901 27255 raft_consensus.cc:493] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.006894 27254 raft_consensus.cc:359] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.012262 27255 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.012187 27206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
W20260709 12:17:01.012226 27102 leader_election.cc:343] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.013751 27102 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aeb5150d7d474113b386036e444b80d7; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.012501 27254 raft_consensus.cc:385] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.014528 27254 raft_consensus.cc:740] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.014573 27272 raft_consensus.cc:2749] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:17:01.016556 27027 leader_election.cc:343] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.016505 27254 consensus_queue.cc:260] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
W20260709 12:17:01.017967 27027 leader_election.cc:343] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.018474 27027 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6318bc9ad074866ba228c136b4c8426; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7
I20260709 12:17:01.019229 27210 raft_consensus.cc:2468] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:01.020093 27254 ts_tablet_manager.cc:1434] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.004s
I20260709 12:17:01.005936 27251 tablet_bootstrap.cc:492] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:01.021485 27254 tablet_bootstrap.cc:492] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:01.018844 27250 raft_consensus.cc:359] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.021991 27250 raft_consensus.cc:385] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.022378 27250 raft_consensus.cc:740] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.019706 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.023027 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.023167 27250 consensus_queue.cc:260] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.025352 27257 raft_consensus.cc:2749] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.025626 27250 ts_tablet_manager.cc:1434] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.003s
I20260709 12:17:01.028442 27250 tablet_bootstrap.cc:492] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:01.032015 27251 tablet_bootstrap.cc:654] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.026520 27255 raft_consensus.cc:515] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.036202 27254 tablet_bootstrap.cc:654] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.036757 27136 raft_consensus.cc:2393] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in current term 1: Already voted for candidate aeb5150d7d474113b386036e444b80d7 in this term.
I20260709 12:17:01.026620 27268 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.038678 27251 tablet_bootstrap.cc:492] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:01.039168 27251 ts_tablet_manager.cc:1403] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.004s
I20260709 12:17:01.040053 27255 leader_election.cc:290] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
W20260709 12:17:01.027029 27176 leader_election.cc:343] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.041304 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.041757 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.035116 27250 tablet_bootstrap.cc:654] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.042456 27210 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:01.042608 27102 leader_election.cc:343] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.041754 27176 leader_election.cc:304] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3; no voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.052742 27210 raft_consensus.cc:2468] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:01.055687 27259 raft_consensus.cc:2749] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.060657 27102 leader_election.cc:304] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.062314 27255 raft_consensus.cc:2804] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.063091 27255 raft_consensus.cc:697] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:01.064308 27255 consensus_queue.cc:237] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.054457 27251 raft_consensus.cc:359] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.069617 27251 raft_consensus.cc:385] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.070088 27251 raft_consensus.cc:740] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.071530 27251 consensus_queue.cc:260] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.074486 27251 ts_tablet_manager.cc:1434] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.000s
I20260709 12:17:01.075944 27251 tablet_bootstrap.cc:492] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:01.080575 27255 raft_consensus.cc:493] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.081156 27255 raft_consensus.cc:515] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.083515 27255 leader_election.cc:290] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.083671 27250 tablet_bootstrap.cc:492] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:01.084249 27250 ts_tablet_manager.cc:1403] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.056s	user 0.026s	sys 0.019s
I20260709 12:17:01.084196 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.085007 27061 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 0.
I20260709 12:17:01.086454 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.087055 27210 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 0.
I20260709 12:17:01.087229 27250 raft_consensus.cc:359] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.087990 27250 raft_consensus.cc:385] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.088681 27250 raft_consensus.cc:740] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.088765 27102 leader_election.cc:304] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:01.089596 27255 raft_consensus.cc:2804] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.089975 27255 raft_consensus.cc:493] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.089504 27250 consensus_queue.cc:260] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.090312 27255 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.091676 27268 raft_consensus.cc:493] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.092119 27268 raft_consensus.cc:515] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.094278 27268 leader_election.cc:290] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.096403 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.097808 27250 ts_tablet_manager.cc:1434] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:17:01.097937 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
W20260709 12:17:01.098573 27102 leader_election.cc:343] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.101343 27250 tablet_bootstrap.cc:492] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
W20260709 12:17:01.101840 27102 leader_election.cc:343] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.102254 27102 leader_election.cc:304] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aeb5150d7d474113b386036e444b80d7; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.102968 27268 raft_consensus.cc:2749] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.103391 27255 raft_consensus.cc:515] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.102896 26956 catalog_manager.cc:5697] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:01.106801 27268 raft_consensus.cc:493] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.107280 27268 raft_consensus.cc:515] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.107939 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.108469 27061 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.109274 27268 leader_election.cc:290] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.111694 27250 tablet_bootstrap.cc:654] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.117040 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.120179 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.100298 27251 tablet_bootstrap.cc:654] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:01.121935 27102 leader_election.cc:343] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.122321 27255 leader_election.cc:290] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
W20260709 12:17:01.122923 27102 leader_election.cc:343] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.123435 27102 leader_election.cc:304] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aeb5150d7d474113b386036e444b80d7; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.125298 27061 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:01.125496 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.128575 27210 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.132035 27102 leader_election.cc:304] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.132915 27268 raft_consensus.cc:2804] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.133399 27268 raft_consensus.cc:697] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:01.134202 27268 consensus_queue.cc:237] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.137238 27210 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:01.126012 27255 raft_consensus.cc:2749] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.127940 27250 tablet_bootstrap.cc:492] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:01.138451 27250 ts_tablet_manager.cc:1403] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.004s
I20260709 12:17:01.141328 27250 raft_consensus.cc:359] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.142254 27250 raft_consensus.cc:385] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.142642 27250 raft_consensus.cc:740] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.143572 27250 consensus_queue.cc:260] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.146373 27250 ts_tablet_manager.cc:1434] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:17:01.147662 27250 tablet_bootstrap.cc:492] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: Bootstrap starting.
I20260709 12:17:01.151682 27254 tablet_bootstrap.cc:492] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.152204 27254 ts_tablet_manager.cc:1403] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.131s	user 0.052s	sys 0.061s
I20260709 12:17:01.154466 26956 catalog_manager.cc:5697] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:01.155440 27250 tablet_bootstrap.cc:654] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.155427 27254 raft_consensus.cc:359] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.156438 27254 raft_consensus.cc:385] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.156790 27254 raft_consensus.cc:740] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.157732 27254 consensus_queue.cc:260] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.158499 27251 tablet_bootstrap.cc:492] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:01.159181 27251 ts_tablet_manager.cc:1403] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.084s	user 0.015s	sys 0.011s
I20260709 12:17:01.163182 27251 raft_consensus.cc:359] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.164268 27251 raft_consensus.cc:385] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.164729 27251 raft_consensus.cc:740] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.166049 27251 consensus_queue.cc:260] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.167889 27254 ts_tablet_manager.cc:1434] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.002s
I20260709 12:17:01.169000 27254 tablet_bootstrap.cc:492] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:01.174062 27251 ts_tablet_manager.cc:1434] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:17:01.176132 27251 tablet_bootstrap.cc:492] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:01.177314 27250 tablet_bootstrap.cc:492] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: No bootstrap required, opened a new log
I20260709 12:17:01.177840 27250 ts_tablet_manager.cc:1403] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.012s
I20260709 12:17:01.180569 27250 raft_consensus.cc:359] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.181329 27250 raft_consensus.cc:385] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.181641 27250 raft_consensus.cc:740] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.182372 27251 tablet_bootstrap.cc:654] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.182770 27254 tablet_bootstrap.cc:654] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.182415 27250 consensus_queue.cc:260] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.185119 27250 ts_tablet_manager.cc:1434] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:17:01.189049 27254 tablet_bootstrap.cc:492] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.189546 27254 ts_tablet_manager.cc:1403] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.004s
I20260709 12:17:01.190433 27251 tablet_bootstrap.cc:492] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:01.190881 27251 ts_tablet_manager.cc:1403] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20260709 12:17:01.192473 27254 raft_consensus.cc:359] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.193199 27254 raft_consensus.cc:385] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.193499 27254 raft_consensus.cc:740] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.193503 27251 raft_consensus.cc:359] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.194286 27251 raft_consensus.cc:385] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.194656 27251 raft_consensus.cc:740] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.194339 27254 consensus_queue.cc:260] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.195411 27251 consensus_queue.cc:260] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.196655 27254 ts_tablet_manager.cc:1434] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:17:01.197619 27251 ts_tablet_manager.cc:1434] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:17:01.197911 27254 tablet_bootstrap.cc:492] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:01.200378 27251 tablet_bootstrap.cc:492] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:01.205024 27254 tablet_bootstrap.cc:654] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.207223 27251 tablet_bootstrap.cc:654] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.210556 27254 tablet_bootstrap.cc:492] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.211051 27254 ts_tablet_manager.cc:1403] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:17:01.213798 27254 raft_consensus.cc:359] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.214360 27254 raft_consensus.cc:385] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.214541 27251 tablet_bootstrap.cc:492] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:01.214710 27254 raft_consensus.cc:740] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.215045 27251 ts_tablet_manager.cc:1403] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.001s
I20260709 12:17:01.215590 27254 consensus_queue.cc:260] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.217170 27251 raft_consensus.cc:359] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.217608 27251 raft_consensus.cc:385] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.217809 27254 ts_tablet_manager.cc:1434] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:17:01.217800 27251 raft_consensus.cc:740] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.218940 27254 tablet_bootstrap.cc:492] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:01.218660 27251 consensus_queue.cc:260] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.220839 27251 ts_tablet_manager.cc:1434] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:17:01.221890 27251 tablet_bootstrap.cc:492] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:01.225778 27254 tablet_bootstrap.cc:654] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.227756 27251 tablet_bootstrap.cc:654] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.238009 27251 tablet_bootstrap.cc:492] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:01.238463 27251 ts_tablet_manager.cc:1403] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.003s
I20260709 12:17:01.239714 27254 tablet_bootstrap.cc:492] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.240166 27254 ts_tablet_manager.cc:1403] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.003s
I20260709 12:17:01.240634 27251 raft_consensus.cc:359] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.241191 27251 raft_consensus.cc:385] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.241449 27251 raft_consensus.cc:740] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.242005 27251 consensus_queue.cc:260] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.242313 27254 raft_consensus.cc:359] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.242838 27254 raft_consensus.cc:385] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.243067 27254 raft_consensus.cc:740] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.244024 27251 ts_tablet_manager.cc:1434] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:17:01.243813 27254 consensus_queue.cc:260] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.244947 27251 tablet_bootstrap.cc:492] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: Bootstrap starting.
I20260709 12:17:01.245945 27254 ts_tablet_manager.cc:1434] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:17:01.246517 27268 raft_consensus.cc:493] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.246980 27268 raft_consensus.cc:515] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.246989 27254 tablet_bootstrap.cc:492] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:01.249210 27268 leader_election.cc:290] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.250748 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.251101 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
W20260709 12:17:01.252156 27102 leader_election.cc:343] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:01.252838 27102 leader_election.cc:343] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:01.253211 27102 leader_election.cc:304] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aeb5150d7d474113b386036e444b80d7; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.253631 27254 tablet_bootstrap.cc:654] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.254335 27268 raft_consensus.cc:2749] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.256072 27259 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.256634 27259 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.257684 27251 tablet_bootstrap.cc:654] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.259367 27259 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039)
I20260709 12:17:01.260279 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.261271 27136 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 0.
I20260709 12:17:01.261610 27284 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.262785 27259 raft_consensus.cc:493] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.262356 27284 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.263422 27259 raft_consensus.cc:515] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.265596 27284 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.267300 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.267681 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.268172 27136 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 0.
I20260709 12:17:01.268621 27061 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 0.
I20260709 12:17:01.273880 27027 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.276003 27259 leader_election.cc:290] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:01.279198 27254 tablet_bootstrap.cc:492] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.279757 27254 ts_tablet_manager.cc:1403] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.033s	user 0.006s	sys 0.024s
I20260709 12:17:01.279695 27284 raft_consensus.cc:2804] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.280138 27284 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.280512 27284 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.281358 27251 tablet_bootstrap.cc:492] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: No bootstrap required, opened a new log
I20260709 12:17:01.281839 27251 ts_tablet_manager.cc:1403] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.013s
I20260709 12:17:01.282634 27254 raft_consensus.cc:359] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.283371 27254 raft_consensus.cc:385] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.283635 27254 raft_consensus.cc:740] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.284092 27251 raft_consensus.cc:359] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.284708 27251 raft_consensus.cc:385] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.285043 27251 raft_consensus.cc:740] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.284541 27254 consensus_queue.cc:260] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.285712 27251 consensus_queue.cc:260] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.287796 27254 ts_tablet_manager.cc:1434] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:17:01.288744 27254 tablet_bootstrap.cc:492] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Bootstrap starting.
I20260709 12:17:01.290300 27251 ts_tablet_manager.cc:1434] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:17:01.295519 27254 tablet_bootstrap.cc:654] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:01.296016 27284 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.296777 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.297443 27061 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 0.
I20260709 12:17:01.298470 27284 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.299991 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:01.300807 27136 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.309634 27136 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 1.
I20260709 12:17:01.310607 27254 tablet_bootstrap.cc:492] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: No bootstrap required, opened a new log
I20260709 12:17:01.310714 27027 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.311249 27254 ts_tablet_manager.cc:1403] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.011s
I20260709 12:17:01.311785 27284 raft_consensus.cc:2804] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.312693 27176 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.313612 27176 leader_election.cc:304] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.313668 27284 raft_consensus.cc:697] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:01.314095 27254 raft_consensus.cc:359] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.314560 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.314827 27254 raft_consensus.cc:385] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:01.315096 27254 raft_consensus.cc:740] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Initialized, Role: FOLLOWER
I20260709 12:17:01.315135 27136 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 0.
I20260709 12:17:01.315374 27284 consensus_queue.cc:237] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.316198 27259 raft_consensus.cc:2804] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.316571 27259 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.316512 27254 consensus_queue.cc:260] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.317000 27259 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.318657 27304 raft_consensus.cc:2804] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.319424 27254 ts_tablet_manager.cc:1434] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:17:01.319231 27304 raft_consensus.cc:493] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.320012 27304 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.324247 27268 raft_consensus.cc:493] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.324743 27268 raft_consensus.cc:515] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.326886 27268 leader_election.cc:290] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.328334 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.328933 27061 raft_consensus.cc:2468] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 0.
I20260709 12:17:01.329519 27259 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.330063 27102 leader_election.cc:304] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.330808 27268 raft_consensus.cc:2804] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.331146 27268 raft_consensus.cc:493] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.331458 27268 raft_consensus.cc:3060] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.332851 27304 raft_consensus.cc:515] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.335732 27304 leader_election.cc:290] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:01.335924 27268 raft_consensus.cc:515] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.338739 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.338876 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.339524 27061 raft_consensus.cc:3060] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.339843 27210 raft_consensus.cc:2393] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6318bc9ad074866ba228c136b4c8426 in current term 1: Already voted for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in this term.
I20260709 12:17:01.340461 27259 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039)
I20260709 12:17:01.343811 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:01.343925 27059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.344594 27135 raft_consensus.cc:2393] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in current term 1: Already voted for candidate f6318bc9ad074866ba228c136b4c8426 in this term.
I20260709 12:17:01.341295 27206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.342552 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.345431 27060 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.345619 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:01.346309 27136 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.346457 26956 catalog_manager.cc:5697] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:01.349079 27268 leader_election.cc:290] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.350673 27061 raft_consensus.cc:2468] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:01.351692 27206 raft_consensus.cc:2393] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6318bc9ad074866ba228c136b4c8426 in current term 1: Already voted for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in this term.
I20260709 12:17:01.351871 27102 leader_election.cc:304] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.353006 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.353102 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.353646 27207 raft_consensus.cc:3060] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.353842 27268 raft_consensus.cc:2804] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.354436 27268 raft_consensus.cc:697] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:01.355418 27268 consensus_queue.cc:237] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.355847 27060 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 1.
I20260709 12:17:01.361049 27284 raft_consensus.cc:493] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.361141 27176 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3; no voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.361677 27284 raft_consensus.cc:515] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.353646 27136 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 1.
I20260709 12:17:01.362452 27176 leader_election.cc:304] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.363219 27304 raft_consensus.cc:2749] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.363376 27259 raft_consensus.cc:2804] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.362481 27207 raft_consensus.cc:2468] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:01.364002 27259 raft_consensus.cc:697] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:01.365983 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.366550 27210 raft_consensus.cc:2468] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 0.
I20260709 12:17:01.366681 27259 consensus_queue.cc:237] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.367169 27255 consensus_queue.cc:1048] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:01.368377 27027 leader_election.cc:304] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.369135 27257 raft_consensus.cc:2804] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.369530 27257 raft_consensus.cc:493] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.364598 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.363998 27284 leader_election.cc:290] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.369885 27257 raft_consensus.cc:3060] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.370426 27135 raft_consensus.cc:2468] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 0.
I20260709 12:17:01.378187 27257 raft_consensus.cc:515] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.382151 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:01.382763 27135 raft_consensus.cc:3060] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.381527 26957 catalog_manager.cc:5697] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:01.382452 26956 catalog_manager.cc:5697] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:01.390695 27255 consensus_queue.cc:1048] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.392252 27135 raft_consensus.cc:2468] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 1.
I20260709 12:17:01.393441 27257 leader_election.cc:290] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.395213 27027 leader_election.cc:304] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.395835 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.396240 27257 raft_consensus.cc:2804] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.396409 27207 raft_consensus.cc:3060] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.396719 27257 raft_consensus.cc:697] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:01.397610 27257 consensus_queue.cc:237] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.406783 27207 raft_consensus.cc:2468] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 1.
I20260709 12:17:01.413935 27259 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.414484 27259 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.414376 26959 catalog_manager.cc:5697] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:01.416823 27259 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:01.418409 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.419140 27135 raft_consensus.cc:2393] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in current term 1: Already voted for candidate aeb5150d7d474113b386036e444b80d7 in this term.
I20260709 12:17:01.420645 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.421356 27061 raft_consensus.cc:2393] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in current term 1: Already voted for candidate f6318bc9ad074866ba228c136b4c8426 in this term.
I20260709 12:17:01.422430 27176 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3; no voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:01.423486 27259 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.428771 27259 raft_consensus.cc:2749] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.469959 27257 raft_consensus.cc:493] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.470541 27257 raft_consensus.cc:515] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.472936 27257 leader_election.cc:290] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.473732 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.474361 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.474426 27135 raft_consensus.cc:2468] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 0.
I20260709 12:17:01.474932 27210 raft_consensus.cc:2468] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 0.
I20260709 12:17:01.475965 27027 leader_election.cc:304] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.476631 27257 raft_consensus.cc:2804] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.476970 27257 raft_consensus.cc:493] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.477329 27257 raft_consensus.cc:3060] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.483207 27257 raft_consensus.cc:515] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.484956 27257 leader_election.cc:290] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.485750 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:01.486166 27259 raft_consensus.cc:493] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.486304 27135 raft_consensus.cc:3060] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.486073 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:01.486907 27210 raft_consensus.cc:3060] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.486780 27259 raft_consensus.cc:515] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.488931 27259 leader_election.cc:290] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:01.490556 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.491305 27136 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 0.
I20260709 12:17:01.492695 27176 leader_election.cc:304] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:01.493816 27210 raft_consensus.cc:2468] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 1.
I20260709 12:17:01.494014 27259 raft_consensus.cc:2804] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:01.494354 27259 raft_consensus.cc:493] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.494686 27259 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.494926 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:01.495553 27061 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 0.
I20260709 12:17:01.500067 27259 raft_consensus.cc:515] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.500583 27027 leader_election.cc:304] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.500883 27135 raft_consensus.cc:2468] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 1.
I20260709 12:17:01.501585 27257 raft_consensus.cc:2804] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.502133 27257 raft_consensus.cc:697] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:01.502878 27257 consensus_queue.cc:237] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.503577 27259 leader_election.cc:290] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:01.503891 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:01.504439 27061 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.504578 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:01.505244 27135 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.511190 27061 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 1.
I20260709 12:17:01.511071 26956 catalog_manager.cc:5697] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:01.512398 27176 leader_election.cc:304] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:01.512817 27135 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 1.
I20260709 12:17:01.513294 27259 raft_consensus.cc:2804] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:01.513924 27259 raft_consensus.cc:697] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:01.514609 27259 consensus_queue.cc:237] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.522148 26959 catalog_manager.cc:5697] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:01.526238 27257 raft_consensus.cc:493] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.526749 27257 raft_consensus.cc:515] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:01.528621 27257 leader_election.cc:290] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:01.529806 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb5150d7d474113b386036e444b80d7" is_pre_election: true
I20260709 12:17:01.529920 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:01.530730 27210 raft_consensus.cc:2393] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6318bc9ad074866ba228c136b4c8426 in current term 1: Already voted for candidate aeb5150d7d474113b386036e444b80d7 in this term.
I20260709 12:17:01.531672 27027 leader_election.cc:304] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6318bc9ad074866ba228c136b4c8426; no voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7
I20260709 12:17:01.532456 27257 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:01.537541 27257 raft_consensus.cc:2749] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:01.553954 27255 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.564370 27268 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.582438 27272 consensus_queue.cc:1048] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.592732 27255 consensus_queue.cc:1048] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.762235 27257 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:01.777747 27257 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.788270 27259 consensus_queue.cc:1048] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.801389 27304 consensus_queue.cc:1048] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.890679 27255 consensus_queue.cc:1048] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.902065 27272 consensus_queue.cc:1048] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.928804 27257 consensus_queue.cc:1048] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.940136 27257 consensus_queue.cc:1048] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.946838 27257 consensus_queue.cc:1048] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:01.957813 27257 consensus_queue.cc:1048] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:17:01.975665 27007 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:01.999447 27304 consensus_queue.cc:1048] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:02.010598 27302 consensus_queue.cc:1048] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:02.518147 27268 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.518533 27268 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.520366 27268 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:02.521167 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426" is_pre_election: true
I20260709 12:17:02.521449 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" is_pre_election: true
I20260709 12:17:02.521747 27061 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:02.522152 27210 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 1.
I20260709 12:17:02.522857 27102 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:02.523731 27268 raft_consensus.cc:2804] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:02.524189 27268 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.524649 27268 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.529834 27268 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.531661 27268 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:02.532548 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:02.532907 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:02.533133 27061 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.533582 27207 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.538455 27061 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.538458 27207 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.540036 27102 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:02.540720 27268 raft_consensus.cc:2804] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:02.541204 27268 raft_consensus.cc:697] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:02.541874 27268 consensus_queue.cc:237] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.549305 26959 catalog_manager.cc:5697] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 0 to 2, leader changed from <none> to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:02.655993 27136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:02.656759 27136 raft_consensus.cc:606] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:02.661835 27136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "88553a5581b04f03be2745006b011395"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:02.662258 27136 raft_consensus.cc:606] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:02.667086 27136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:02.667649 27136 raft_consensus.cc:606] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:02.672812 27136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:02.673422 27136 raft_consensus.cc:606] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:02.687366 27207 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
mode: GRACEFUL
new_leader_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:43620
I20260709 12:17:02.687968 27207 raft_consensus.cc:606] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Received request to transfer leadership to TS aeb5150d7d474113b386036e444b80d7
I20260709 12:17:02.695858 27207 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
mode: GRACEFUL
new_leader_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:43620
I20260709 12:17:02.696398 27207 raft_consensus.cc:606] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Received request to transfer leadership to TS aeb5150d7d474113b386036e444b80d7
I20260709 12:17:02.703826 27255 raft_consensus.cc:993] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:02.704288 27268 raft_consensus.cc:1081] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:02.705870 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:02.706622 27061 raft_consensus.cc:493] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:02.706969 27061 raft_consensus.cc:3060] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.710711 27060 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:43228
I20260709 12:17:02.711429 27060 raft_consensus.cc:606] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Received request to transfer leadership to TS aeb5150d7d474113b386036e444b80d7
I20260709 12:17:02.714294 27061 raft_consensus.cc:515] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.716455 27061 leader_election.cc:290] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:02.716890 27060 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:43228
I20260709 12:17:02.717461 27060 raft_consensus.cc:606] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Received request to transfer leadership to TS 91c8ef68f2cb4b6dbf3db4084441a6f3
I20260709 12:17:02.717581 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:02.718259 27136 raft_consensus.cc:3055] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:02.718573 27136 raft_consensus.cc:740] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:02.718473 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f747d704794e3fb8dde0d7ba27da7a" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:02.719242 27207 raft_consensus.cc:3060] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.719663 27136 consensus_queue.cc:260] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.721058 27136 raft_consensus.cc:3060] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.727308 27207 raft_consensus.cc:2468] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:02.728638 27136 raft_consensus.cc:2468] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:02.728890 27027 leader_election.cc:304] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:02.729970 27284 raft_consensus.cc:2804] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:02.730507 27284 raft_consensus.cc:697] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:02.731421 27284 consensus_queue.cc:237] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.738554 26959 catalog_manager.cc:5697] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 1 to 2, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:02.830812 27304 raft_consensus.cc:993] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: : Instructing follower aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:02.831290 27259 raft_consensus.cc:1081] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Signalling peer aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:02.832676 27136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:39030
I20260709 12:17:02.833216 27136 raft_consensus.cc:493] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:02.833535 27136 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.837982 27136 raft_consensus.cc:515] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.839495 27136 leader_election.cc:290] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:02.840322 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:02.840930 27060 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.840844 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:02.841511 27207 raft_consensus.cc:3055] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:02.841862 27207 raft_consensus.cc:740] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:02.842665 27207 consensus_queue.cc:260] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.843947 27207 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.845234 27257 raft_consensus.cc:993] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: : Instructing follower aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:02.845649 27284 raft_consensus.cc:1081] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Signalling peer aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:02.847070 27136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:39018
I20260709 12:17:02.847560 27136 raft_consensus.cc:493] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:02.847887 27136 raft_consensus.cc:3060] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.847885 27060 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.849380 27102 leader_election.cc:304] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:02.850163 27268 raft_consensus.cc:2804] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:02.850615 27268 raft_consensus.cc:697] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:02.850819 27207 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.851576 27268 consensus_queue.cc:237] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.854493 27136 raft_consensus.cc:515] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.856415 27136 leader_election.cc:290] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:02.857630 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:02.857605 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88847fc535ed499cb53f82a470f81ded" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:02.858307 27207 raft_consensus.cc:3060] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.858367 27060 raft_consensus.cc:3055] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:02.858804 27060 raft_consensus.cc:740] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:02.859534 26959 catalog_manager.cc:5697] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 1 to 2, leader changed from 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3) to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:02.859629 27060 consensus_queue.cc:260] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.860951 27060 raft_consensus.cc:3060] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.865264 27207 raft_consensus.cc:2468] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.866593 27102 leader_election.cc:304] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:02.867475 27268 raft_consensus.cc:2804] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:02.867995 27060 raft_consensus.cc:2468] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.868139 27268 raft_consensus.cc:697] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:02.869231 27268 consensus_queue.cc:237] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.875411 26959 catalog_manager.cc:5697] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 1 to 2, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:02.911520 27257 raft_consensus.cc:993] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: : Instructing follower 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:02.911988 27284 raft_consensus.cc:1081] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Signalling peer 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:02.913451 27207 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:43604
I20260709 12:17:02.913933 27207 raft_consensus.cc:493] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:02.914443 27207 raft_consensus.cc:3060] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.919284 27207 raft_consensus.cc:515] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.921197 27207 leader_election.cc:290] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 2 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:02.921907 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:02.922315 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9af6b5759a1a40a2854bd1ed9df3ec8a" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:02.922505 27060 raft_consensus.cc:3055] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:02.922874 27136 raft_consensus.cc:3060] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.922894 27060 raft_consensus.cc:740] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:02.923832 27060 consensus_queue.cc:260] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.925045 27060 raft_consensus.cc:3060] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.929813 27136 raft_consensus.cc:2468] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 2.
I20260709 12:17:02.931157 27176 leader_election.cc:304] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:02.931578 27060 raft_consensus.cc:2468] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 2.
I20260709 12:17:02.931947 27259 raft_consensus.cc:2804] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:02.932436 27259 raft_consensus.cc:697] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:02.933250 27259 consensus_queue.cc:237] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.939502 26959 catalog_manager.cc:5697] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 1 to 2, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:02.964367 27304 raft_consensus.cc:993] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: : Instructing follower aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:02.964820 27259 raft_consensus.cc:1081] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Signalling peer aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:02.966390 27136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:39030
I20260709 12:17:02.966828 27136 raft_consensus.cc:493] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:02.967084 27136 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.972059 27136 raft_consensus.cc:515] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.973619 27136 leader_election.cc:290] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:02.974582 27060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:02.975076 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:02.975261 27060 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.975600 27207 raft_consensus.cc:3055] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:02.975940 27207 raft_consensus.cc:740] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:02.976828 27207 consensus_queue.cc:260] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.977787 27207 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:02.982343 27207 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.982422 27060 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 2.
I20260709 12:17:02.983626 27102 leader_election.cc:304] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:02.984441 27268 raft_consensus.cc:2804] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:02.984899 27268 raft_consensus.cc:697] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:02.985581 27268 consensus_queue.cc:237] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:02.993202 26959 catalog_manager.cc:5697] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 1 to 2, leader changed from 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3) to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:03.034525 27060 raft_consensus.cc:1275] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:03.035903 27268 consensus_queue.cc:1048] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:03.043459 27255 raft_consensus.cc:993] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:03.044498 27255 raft_consensus.cc:1081] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:03.047117 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:03.047928 27061 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:03.048616 27061 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:03.050910 27207 raft_consensus.cc:1275] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:03.052199 27345 consensus_queue.cc:1048] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:03.055910 27061 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.059705 27345 raft_consensus.cc:993] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:03.060238 27268 raft_consensus.cc:1081] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:03.061331 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:03.061904 27136 raft_consensus.cc:3055] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:03.062238 27136 raft_consensus.cc:740] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:03.063033 27136 consensus_queue.cc:260] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.063951 27060 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "88553a5581b04f03be2745006b011395"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:03.064529 27060 raft_consensus.cc:493] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:03.061331 27061 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:03.064981 27060 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:03.063969 27136 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:03.072309 27207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:03.072888 27207 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:03.073421 27136 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:03.074397 27060 raft_consensus.cc:515] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.074733 27027 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:03.075649 27284 raft_consensus.cc:2804] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:03.076097 27284 raft_consensus.cc:697] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:03.076601 27060 leader_election.cc:290] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:03.076928 27284 consensus_queue.cc:237] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.079614 27207 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:03.080149 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:03.080727 27210 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:03.085250 26956 catalog_manager.cc:5697] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 2 to 3, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:03.084244 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:03.087255 27136 raft_consensus.cc:3055] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:03.087507 27210 raft_consensus.cc:2468] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:03.087628 27136 raft_consensus.cc:740] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:03.088696 27027 leader_election.cc:304] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:03.088559 27136 consensus_queue.cc:260] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.089460 27284 raft_consensus.cc:2804] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:03.089913 27284 raft_consensus.cc:697] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:03.090610 27284 consensus_queue.cc:237] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.091027 27136 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:03.097877 27136 raft_consensus.cc:2468] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:03.099690 26956 catalog_manager.cc:5697] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 1 to 2, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:03.147178 27345 raft_consensus.cc:993] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:03.147710 27268 raft_consensus.cc:1081] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:03.149313 27060 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:03.149868 27060 raft_consensus.cc:493] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:03.150182 27060 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:03.155166 27060 raft_consensus.cc:515] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.156868 27060 leader_election.cc:290] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:03.157853 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:03.158433 27136 raft_consensus.cc:3055] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:03.158452 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:03.158763 27136 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:03.159199 27210 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:03.159672 27136 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.160604 27136 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:03.164800 27210 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:03.165277 27136 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:03.165953 27027 leader_election.cc:304] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:03.166623 27284 raft_consensus.cc:2804] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:03.167054 27284 raft_consensus.cc:697] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:03.167809 27284 consensus_queue.cc:237] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:03.174377 26956 catalog_manager.cc:5697] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 1 to 2, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:03.207516 27136 raft_consensus.cc:1275] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.208804 27284 consensus_queue.cc:1048] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.219832 27210 raft_consensus.cc:1275] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.221678 27284 consensus_queue.cc:1048] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.257545 27210 raft_consensus.cc:1275] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.258993 27268 consensus_queue.cc:1048] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.269063 27060 raft_consensus.cc:1275] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.270642 27268 consensus_queue.cc:1048] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.350638 27060 raft_consensus.cc:1275] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.352324 27259 consensus_queue.cc:1048] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.361366 27136 raft_consensus.cc:1275] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.363098 27358 consensus_queue.cc:1048] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:03.363790 27061 raft_consensus.cc:1275] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.365507 27268 consensus_queue.cc:1048] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.376616 27210 raft_consensus.cc:1275] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.380208 27255 consensus_queue.cc:1048] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.438437 27210 raft_consensus.cc:1275] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.439829 27255 consensus_queue.cc:1048] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.449640 27061 raft_consensus.cc:1275] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.451030 27255 consensus_queue.cc:1048] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.509500 27136 raft_consensus.cc:1275] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:17:03.510903 27284 consensus_queue.cc:1048] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.519034 27210 raft_consensus.cc:1275] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:17:03.520915 27353 consensus_queue.cc:1048] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.595084 27136 raft_consensus.cc:1275] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.596324 27353 consensus_queue.cc:1048] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.605192 27210 raft_consensus.cc:1275] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.606879 27284 consensus_queue.cc:1048] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.647168 27210 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.648450 27284 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:03.657728 27136 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:03.659499 27257 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:05.751959 26408 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:17:05.752673 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:05.769570 27061 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:33456
I20260709 12:17:05.770192 27061 raft_consensus.cc:606] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Received request to transfer leadership to TS 91c8ef68f2cb4b6dbf3db4084441a6f3
I20260709 12:17:05.772877 27061 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:33456
I20260709 12:17:05.773308 27061 raft_consensus.cc:606] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Received request to transfer leadership to TS 91c8ef68f2cb4b6dbf3db4084441a6f3
I20260709 12:17:05.774384 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:17:05.774757 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:06.149457 27353 raft_consensus.cc:993] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: : Instructing follower 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:06.149887 27353 raft_consensus.cc:1081] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Signalling peer 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:06.151162 27210 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:43604
I20260709 12:17:06.151616 27210 raft_consensus.cc:493] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:06.151840 27210 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:06.156199 27210 raft_consensus.cc:515] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:06.157738 27210 leader_election.cc:290] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 3 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:06.158954 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:06.159099 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:06.159540 27136 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:06.159540 27061 raft_consensus.cc:3055] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:06.160053 27061 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:06.160934 27061 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:06.161834 27061 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:06.164332 27136 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 3.
I20260709 12:17:06.165421 27176 leader_election.cc:304] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:06.166081 27358 raft_consensus.cc:2804] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:06.167181 27358 raft_consensus.cc:697] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:06.168046 27061 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 3.
I20260709 12:17:06.168071 27358 consensus_queue.cc:237] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:06.175510 26959 catalog_manager.cc:5697] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 2 to 3, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:06.272907 27370 raft_consensus.cc:993] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: : Instructing follower 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:06.273631 27353 raft_consensus.cc:1081] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Signalling peer 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:06.275233 27210 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:43604
I20260709 12:17:06.275712 27210 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:06.276093 27210 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:06.280022 27210 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:06.281396 27210 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 2 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039)
I20260709 12:17:06.282055 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:06.282358 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:06.282517 27136 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:06.282900 27061 raft_consensus.cc:3055] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:06.283154 27061 raft_consensus.cc:740] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:06.283857 27061 consensus_queue.cc:260] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:06.284885 27061 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:06.287607 27136 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 2.
I20260709 12:17:06.288537 27176 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:06.289410 27358 raft_consensus.cc:2804] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:06.289557 27061 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 2.
I20260709 12:17:06.289844 27358 raft_consensus.cc:697] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:06.290772 27358 consensus_queue.cc:237] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:06.296627 26959 catalog_manager.cc:5697] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 1 to 2, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:06.668529 27136 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:06.669893 27358 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:06.679643 27061 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:06.681335 27371 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:06.775832 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:06.781365 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:06.781838 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:06.839398 27061 raft_consensus.cc:1275] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:06.840842 27380 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:06.850469 27136 raft_consensus.cc:1275] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:06.851716 27385 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:07.782881 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:07.788197 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:07.788643 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:08.790175 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:08.797252 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:08.797760 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:09.798784 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:09.804116 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:09.804566 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:10.805670 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:10.811300 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:10.811765 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:11.812759 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:11.818774 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:11.819296 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:12.820441 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:12.826035 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:12.826493 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:13.827441 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:13.832666 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:13.833105 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:14.834167 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:14.839394 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:14.839830 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:15.885545 26408 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:17:15.891100 27210 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43620
I20260709 12:17:15.891952 27210 raft_consensus.cc:606] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:15.905890 27210 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43620
I20260709 12:17:15.906319 27210 raft_consensus.cc:606] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:15.911298 27135 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:15.911988 27135 raft_consensus.cc:606] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:15.917593 27135 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:15.918255 27135 raft_consensus.cc:606] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:15.924307 27135 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
mode: GRACEFUL
new_leader_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:39038
I20260709 12:17:15.924916 27135 raft_consensus.cc:606] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Received request to transfer leadership to TS f6318bc9ad074866ba228c136b4c8426
I20260709 12:17:15.975147 27398 raft_consensus.cc:993] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:15.975739 27398 raft_consensus.cc:1081] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:15.977653 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:15.978158 27061 raft_consensus.cc:493] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:15.978487 27061 raft_consensus.cc:3060] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:15.983798 27061 raft_consensus.cc:515] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:15.986016 27061 leader_election.cc:290] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:15.986980 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:15.987236 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ceeac06a7cf54e40b81ce70b21a222e3" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:15.987999 27135 raft_consensus.cc:3055] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:15.988058 27210 raft_consensus.cc:3060] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:15.988348 27135 raft_consensus.cc:740] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:15.989316 27135 consensus_queue.cc:260] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:15.990419 27135 raft_consensus.cc:3060] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:15.991360 27393 raft_consensus.cc:993] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:15.991819 27398 raft_consensus.cc:1081] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:15.993439 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:15.993995 27061 raft_consensus.cc:493] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:15.994364 27061 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:15.995200 27210 raft_consensus.cc:2468] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:15.996086 27135 raft_consensus.cc:2468] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 2.
I20260709 12:17:15.996541 27027 leader_election.cc:304] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:15.997375 27397 raft_consensus.cc:2804] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:15.998513 27397 raft_consensus.cc:697] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:15.999661 27397 consensus_queue.cc:237] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:15.999974 27061 raft_consensus.cc:515] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.001751 27061 leader_election.cc:290] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:16.003407 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:16.003680 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943c0718dc5d44889b02cdbbe48b1318" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:16.004199 27135 raft_consensus.cc:3055] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:16.004522 27210 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.004578 27135 raft_consensus.cc:740] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:16.005550 27135 consensus_queue.cc:260] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.006870 27135 raft_consensus.cc:3060] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.009534 27210 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:16.008967 26959 catalog_manager.cc:5697] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 1 to 2, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:16.010501 27027 leader_election.cc:304] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:16.011201 27397 raft_consensus.cc:2804] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:16.011761 27397 raft_consensus.cc:697] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:16.013063 27397 consensus_queue.cc:237] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.015115 27135 raft_consensus.cc:2468] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:16.028136 27402 raft_consensus.cc:993] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:16.028970 27396 raft_consensus.cc:1081] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:16.030921 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43222
I20260709 12:17:16.031467 27061 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:16.031783 27061 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.036665 27061 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.038147 27061 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:16.037612 26959 catalog_manager.cc:5697] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 2 to 3, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:16.038874 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:16.039314 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:16.039494 27135 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.039803 27210 raft_consensus.cc:3055] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:16.040148 27210 raft_consensus.cc:740] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:16.040802 27210 consensus_queue.cc:260] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.041967 27210 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.045572 27135 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:16.047302 27027 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:16.047740 27210 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:16.048112 27397 raft_consensus.cc:2804] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:16.048581 27397 raft_consensus.cc:697] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:16.049523 27397 consensus_queue.cc:237] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.056361 26959 catalog_manager.cc:5697] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 2 to 3, leader changed from 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:16.357864 27402 raft_consensus.cc:993] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:16.358327 27396 raft_consensus.cc:1081] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:16.359710 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43222
I20260709 12:17:16.360129 27061 raft_consensus.cc:493] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:16.360548 27061 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:16.364564 27061 raft_consensus.cc:515] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.366144 27061 leader_election.cc:290] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 4 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:16.367018 27135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:16.367224 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:16.367574 27135 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:16.367820 27210 raft_consensus.cc:3055] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:17:16.368144 27210 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:16.368813 27210 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.369776 27210 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:16.372826 27135 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 4.
I20260709 12:17:16.373862 27027 leader_election.cc:304] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb5150d7d474113b386036e444b80d7, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:16.373962 27210 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 4.
I20260709 12:17:16.374594 27397 raft_consensus.cc:2804] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:17:16.375097 27397 raft_consensus.cc:697] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:16.375976 27397 consensus_queue.cc:237] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.382817 26959 catalog_manager.cc:5697] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 3 to 4, leader changed from 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 4 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:16.440754 27135 raft_consensus.cc:1275] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:16.442627 27393 raft_consensus.cc:993] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: : Instructing follower f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:16.442281 27397 consensus_queue.cc:1048] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:16.443039 27398 raft_consensus.cc:1081] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Signalling peer f6318bc9ad074866ba228c136b4c8426 to start an election
I20260709 12:17:16.444749 27061 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
 from {username='slave'} at 127.0.0.1:43206
I20260709 12:17:16.445191 27061 raft_consensus.cc:493] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:16.445514 27061 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.446480 27136 raft_consensus.cc:1275] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:16.448000 27397 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:16.452787 27061 raft_consensus.cc:515] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.455170 27061 leader_election.cc:290] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Requested vote from peers aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:16.456060 27134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:16.456851 27134 raft_consensus.cc:3055] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:16.457257 27134 raft_consensus.cc:740] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:16.458187 27134 consensus_queue.cc:260] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.459769 27210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7d884995e6e40c6bf8be25283343717" candidate_uuid: "f6318bc9ad074866ba228c136b4c8426" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:16.460362 27210 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.467357 27134 raft_consensus.cc:3060] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:16.468235 27207 raft_consensus.cc:1275] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:16.469887 27206 raft_consensus.cc:1275] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:16.471441 27399 consensus_queue.cc:1048] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:16.472751 27399 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:16.474249 27134 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:16.466722 27210 raft_consensus.cc:2468] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6318bc9ad074866ba228c136b4c8426 in term 3.
I20260709 12:17:16.485620 27027 leader_election.cc:304] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, f6318bc9ad074866ba228c136b4c8426; no voters: 
I20260709 12:17:16.487869 27413 raft_consensus.cc:2804] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:16.488503 27413 raft_consensus.cc:697] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Becoming Leader. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:16.489692 27413 consensus_queue.cc:237] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:16.501078 26959 catalog_manager.cc:5697] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 reported cstate change: term changed from 2 to 3, leader changed from aeb5150d7d474113b386036e444b80d7 (127.25.202.2) to f6318bc9ad074866ba228c136b4c8426 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "f6318bc9ad074866ba228c136b4c8426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:16.577646 27210 raft_consensus.cc:1275] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:16.578935 27397 consensus_queue.cc:1048] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:16.588941 27136 raft_consensus.cc:1275] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:16.590561 27413 consensus_queue.cc:1048] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:16.823089 27206 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:17:16.824436 27413 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:16.833312 27136 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:17:16.835551 27399 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:17.031754 27136 raft_consensus.cc:1275] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:17.033001 27413 consensus_queue.cc:1048] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:17.041775 27206 raft_consensus.cc:1275] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Refusing update from remote peer f6318bc9ad074866ba228c136b4c8426: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:17.043840 27425 consensus_queue.cc:1048] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:18.965188 26408 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:17:18.966156 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:18.974382 27061 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "88553a5581b04f03be2745006b011395"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:33456
I20260709 12:17:18.975044 27061 raft_consensus.cc:606] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Received request to transfer leadership to TS 91c8ef68f2cb4b6dbf3db4084441a6f3
I20260709 12:17:18.978045 27061 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:33456
I20260709 12:17:18.978667 27061 raft_consensus.cc:606] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Received request to transfer leadership to TS aeb5150d7d474113b386036e444b80d7
I20260709 12:17:18.981336 27061 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:33456
I20260709 12:17:18.981849 27061 raft_consensus.cc:606] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 LEADER]: Received request to transfer leadership to TS 91c8ef68f2cb4b6dbf3db4084441a6f3
I20260709 12:17:18.984427 27061 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f"
dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
mode: GRACEFUL
new_leader_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:33456
I20260709 12:17:18.984894 27061 raft_consensus.cc:606] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Received request to transfer leadership to TS aeb5150d7d474113b386036e444b80d7
I20260709 12:17:18.986080 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:17:18.986693 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:19.159679 27428 raft_consensus.cc:993] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: : Instructing follower aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:19.160338 27397 raft_consensus.cc:1081] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Signalling peer aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:19.162081 27136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:39018
I20260709 12:17:19.162695 27136 raft_consensus.cc:493] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:19.163103 27136 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:19.170784 27136 raft_consensus.cc:515] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.172830 27136 leader_election.cc:290] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 4 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:19.173856 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:19.174379 27206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9013e4bd5c25415a977c1a085842d0ac" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:19.174623 27061 raft_consensus.cc:3055] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:17:19.174966 27206 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:19.174916 27061 raft_consensus.cc:740] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:19.176187 27061 consensus_queue.cc:260] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.177433 27061 raft_consensus.cc:3060] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:19.182839 27206 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 4.
I20260709 12:17:19.184206 27102 leader_election.cc:304] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:19.184540 27061 raft_consensus.cc:2468] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 4.
I20260709 12:17:19.185266 27398 raft_consensus.cc:2804] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:17:19.185913 27398 raft_consensus.cc:697] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 4 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:19.186990 27398 consensus_queue.cc:237] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.195776 26959 catalog_manager.cc:5697] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 3 to 4, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:19.344453 27428 raft_consensus.cc:993] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: : Instructing follower 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:19.344877 27397 raft_consensus.cc:1081] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 LEADER]: Signalling peer 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:19.346526 27206 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:43604
I20260709 12:17:19.347043 27206 raft_consensus.cc:493] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:19.347411 27206 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:17:19.352159 27206 raft_consensus.cc:515] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.354372 27206 leader_election.cc:290] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 5 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:19.355340 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:19.355588 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b4d71766084c9caafdaaaa2243a249" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:19.355919 27061 raft_consensus.cc:3055] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:17:19.356194 27136 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:17:19.356240 27061 raft_consensus.cc:740] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:19.357021 27061 consensus_queue.cc:260] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.357975 27061 raft_consensus.cc:3060] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:17:19.360838 27136 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 5.
I20260709 12:17:19.362080 27176 leader_election.cc:304] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:19.362377 27061 raft_consensus.cc:2468] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 5.
I20260709 12:17:19.362946 27396 raft_consensus.cc:2804] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:17:19.364494 27396 raft_consensus.cc:697] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 5 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:19.365769 27428 raft_consensus.cc:993] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: : Instructing follower aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:19.365518 27396 consensus_queue.cc:237] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.366242 27397 raft_consensus.cc:1081] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Signalling peer aeb5150d7d474113b386036e444b80d7 to start an election
I20260709 12:17:19.367844 27136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f"
dest_uuid: "aeb5150d7d474113b386036e444b80d7"
 from {username='slave'} at 127.0.0.1:39018
I20260709 12:17:19.368337 27136 raft_consensus.cc:493] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:19.368716 27136 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:19.373801 27136 raft_consensus.cc:515] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.374205 26959 catalog_manager.cc:5697] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 4 to 5, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 5 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:19.375134 27136 leader_election.cc:290] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 4 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3:43207)
I20260709 12:17:19.376024 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:19.376565 27061 raft_consensus.cc:3055] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:17:19.376519 27206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cae1b0bb8aba4ea69fed0507b2cdd16f" candidate_uuid: "aeb5150d7d474113b386036e444b80d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
I20260709 12:17:19.376946 27061 raft_consensus.cc:740] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:19.377133 27206 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:19.377704 27061 consensus_queue.cc:260] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.378798 27061 raft_consensus.cc:3060] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:17:19.383059 27206 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 4.
I20260709 12:17:19.383721 27061 raft_consensus.cc:2468] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb5150d7d474113b386036e444b80d7 in term 4.
I20260709 12:17:19.384203 27102 leader_election.cc:304] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:19.384933 27398 raft_consensus.cc:2804] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:17:19.385371 27398 raft_consensus.cc:697] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 4 LEADER]: Becoming Leader. State: Replica: aeb5150d7d474113b386036e444b80d7, State: Running, Role: LEADER
I20260709 12:17:19.386190 27398 consensus_queue.cc:237] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.392848 26959 catalog_manager.cc:5697] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 reported cstate change: term changed from 3 to 4, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to aeb5150d7d474113b386036e444b80d7 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "aeb5150d7d474113b386036e444b80d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:19.469041 27428 raft_consensus.cc:993] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: : Instructing follower 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:19.469602 27397 raft_consensus.cc:1081] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Signalling peer 91c8ef68f2cb4b6dbf3db4084441a6f3 to start an election
I20260709 12:17:19.471055 27206 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "88553a5581b04f03be2745006b011395"
dest_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3"
 from {username='slave'} at 127.0.0.1:43604
I20260709 12:17:19.471562 27206 raft_consensus.cc:493] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:19.471906 27206 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:19.477232 27206 raft_consensus.cc:515] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.478785 27206 leader_election.cc:290] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 3 election: Requested vote from peers f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039), aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609)
I20260709 12:17:19.479604 27061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6318bc9ad074866ba228c136b4c8426"
I20260709 12:17:19.479998 27136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88553a5581b04f03be2745006b011395" candidate_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aeb5150d7d474113b386036e444b80d7"
I20260709 12:17:19.480172 27061 raft_consensus.cc:3055] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:19.480482 27061 raft_consensus.cc:740] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6318bc9ad074866ba228c136b4c8426, State: Running, Role: LEADER
I20260709 12:17:19.480545 27136 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:19.481174 27061 consensus_queue.cc:260] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.482036 27061 raft_consensus.cc:3060] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:19.485324 27136 raft_consensus.cc:2468] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 3.
I20260709 12:17:19.486289 27061 raft_consensus.cc:2468] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c8ef68f2cb4b6dbf3db4084441a6f3 in term 3.
I20260709 12:17:19.486629 27176 leader_election.cc:304] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c8ef68f2cb4b6dbf3db4084441a6f3, aeb5150d7d474113b386036e444b80d7; no voters: 
I20260709 12:17:19.487437 27396 raft_consensus.cc:2804] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:19.487900 27396 raft_consensus.cc:697] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Becoming Leader. State: Replica: 91c8ef68f2cb4b6dbf3db4084441a6f3, State: Running, Role: LEADER
I20260709 12:17:19.488724 27396 consensus_queue.cc:237] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } }
I20260709 12:17:19.495811 26959 catalog_manager.cc:5697] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 reported cstate change: term changed from 2 to 3, leader changed from f6318bc9ad074866ba228c136b4c8426 (127.25.202.1) to 91c8ef68f2cb4b6dbf3db4084441a6f3 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:19.603807 27206 raft_consensus.cc:1275] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:17:19.605904 27398 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:19.618866 27061 raft_consensus.cc:1275] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:17:19.621913 27442 consensus_queue.cc:1048] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:17:19.781942 27061 raft_consensus.cc:1275] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:17:19.783265 27442 consensus_queue.cc:1048] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:19.793277 27210 raft_consensus.cc:1275] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Refusing update from remote peer aeb5150d7d474113b386036e444b80d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:17:19.794802 27442 consensus_queue.cc:1048] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8ef68f2cb4b6dbf3db4084441a6f3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:19.832728 27061 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 5 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:17:19.834379 27435 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:17:19.844326 27136 raft_consensus.cc:1275] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 5 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:17:19.845757 27396 consensus_queue.cc:1048] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:17:19.903235 27061 raft_consensus.cc:1275] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:19.904479 27396 consensus_queue.cc:1048] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6318bc9ad074866ba228c136b4c8426" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:19.913890 27136 raft_consensus.cc:1275] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Refusing update from remote peer 91c8ef68f2cb4b6dbf3db4084441a6f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:19.915607 27396 consensus_queue.cc:1048] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb5150d7d474113b386036e444b80d7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:19.987926 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:19.994093 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:19.994593 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:20.995710 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:21.001364 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:21.001827 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:22.002924 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:22.008574 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:22.009025 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:23.009981 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:23.016499 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:23.016937 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:24.017882 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:24.023599 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:24.024030 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:25.025007 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:25.030491 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:25.030992 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:26.032006 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:26.038111 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:26.038591 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:27.039697 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:27.045033 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:27.045473 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:28.046681 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:28.053251 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:17:28.053728 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:29.086920 26408 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:17:29.087604 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
I20260709 12:17:29.122257 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:17:29.123196 26408 tablet_replica.cc:333] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.124037 26408 raft_consensus.cc:2243] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.124981 26408 raft_consensus.cc:2272] T ceeac06a7cf54e40b81ce70b21a222e3 P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.127315 26408 tablet_replica.cc:333] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.127935 26408 raft_consensus.cc:2243] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.128377 26408 raft_consensus.cc:2272] T 88847fc535ed499cb53f82a470f81ded P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.130250 26408 tablet_replica.cc:333] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.130848 26408 raft_consensus.cc:2243] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.131727 26408 raft_consensus.cc:2272] T d7d884995e6e40c6bf8be25283343717 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.133694 26408 tablet_replica.cc:333] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.134207 26408 raft_consensus.cc:2243] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.134975 26408 raft_consensus.cc:2272] T 943c0718dc5d44889b02cdbbe48b1318 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.137006 26408 tablet_replica.cc:333] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.137518 26408 raft_consensus.cc:2243] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.137925 26408 raft_consensus.cc:2272] T b7b4d71766084c9caafdaaaa2243a249 P f6318bc9ad074866ba228c136b4c8426 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.140021 26408 tablet_replica.cc:333] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.140645 26408 raft_consensus.cc:2243] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.141120 26408 raft_consensus.cc:2272] T 88553a5581b04f03be2745006b011395 P f6318bc9ad074866ba228c136b4c8426 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.143018 26408 tablet_replica.cc:333] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.143625 26408 raft_consensus.cc:2243] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.144080 26408 raft_consensus.cc:2272] T 9013e4bd5c25415a977c1a085842d0ac P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.146168 26408 tablet_replica.cc:333] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.146694 26408 raft_consensus.cc:2243] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.147426 26408 raft_consensus.cc:2272] T cae1b0bb8aba4ea69fed0507b2cdd16f P f6318bc9ad074866ba228c136b4c8426 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.149474 26408 tablet_replica.cc:333] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.150067 26408 raft_consensus.cc:2243] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.150571 26408 raft_consensus.cc:2272] T 9af6b5759a1a40a2854bd1ed9df3ec8a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.152932 26408 tablet_replica.cc:333] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426: stopping tablet replica
I20260709 12:17:29.153427 26408 raft_consensus.cc:2243] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.154399 26408 raft_consensus.cc:2272] T d7f747d704794e3fb8dde0d7ba27da7a P f6318bc9ad074866ba228c136b4c8426 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.189781 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
W20260709 12:17:29.189941 27176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.1:45039: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260709 12:17:29.194201 27176 consensus_peers.cc:597] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 -> Peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Couldn't send request to peer f6318bc9ad074866ba228c136b4c8426. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:45039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:29.211846 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
I20260709 12:17:29.240800 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:17:29.241719 26408 tablet_replica.cc:333] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.242478 26408 raft_consensus.cc:2243] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.243021 26408 raft_consensus.cc:2272] T ceeac06a7cf54e40b81ce70b21a222e3 P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.245594 26408 tablet_replica.cc:333] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.246368 26408 raft_consensus.cc:2243] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.247426 26408 raft_consensus.cc:2272] T 88847fc535ed499cb53f82a470f81ded P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:17:29.249177 27176 consensus_peers.cc:597] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 -> Peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Couldn't send request to peer f6318bc9ad074866ba228c136b4c8426. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:45039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:29.249933 26408 tablet_replica.cc:333] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.250592 26408 raft_consensus.cc:2243] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.251086 26408 raft_consensus.cc:2272] T d7d884995e6e40c6bf8be25283343717 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.252910 26408 tablet_replica.cc:333] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.253453 26408 raft_consensus.cc:2243] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.253940 26408 raft_consensus.cc:2272] T b7b4d71766084c9caafdaaaa2243a249 P aeb5150d7d474113b386036e444b80d7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.255784 26408 tablet_replica.cc:333] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.256319 26408 raft_consensus.cc:2243] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.256763 26408 raft_consensus.cc:2272] T 88553a5581b04f03be2745006b011395 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.258546 26408 tablet_replica.cc:333] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.259038 26408 raft_consensus.cc:2243] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.259863 26408 raft_consensus.cc:2272] T cae1b0bb8aba4ea69fed0507b2cdd16f P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.261823 26408 tablet_replica.cc:333] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.262341 26408 raft_consensus.cc:2243] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.262780 26408 raft_consensus.cc:2272] T 9af6b5759a1a40a2854bd1ed9df3ec8a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.264729 26408 tablet_replica.cc:333] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
W20260709 12:17:29.265076 27176 consensus_peers.cc:597] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 -> Peer aeb5150d7d474113b386036e444b80d7 (127.25.202.2:39609): Couldn't send request to peer aeb5150d7d474113b386036e444b80d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:29.265381 26408 raft_consensus.cc:2243] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.265877 26408 raft_consensus.cc:2272] T d7f747d704794e3fb8dde0d7ba27da7a P aeb5150d7d474113b386036e444b80d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.267961 26408 tablet_replica.cc:333] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.268445 26408 raft_consensus.cc:2243] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.269181 26408 raft_consensus.cc:2272] T 9013e4bd5c25415a977c1a085842d0ac P aeb5150d7d474113b386036e444b80d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.270875 26408 tablet_replica.cc:333] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7: stopping tablet replica
I20260709 12:17:29.271435 26408 raft_consensus.cc:2243] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.271867 26408 raft_consensus.cc:2272] T 943c0718dc5d44889b02cdbbe48b1318 P aeb5150d7d474113b386036e444b80d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.302846 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
I20260709 12:17:29.324167 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
I20260709 12:17:29.354805 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:17:29.355597 26408 tablet_replica.cc:333] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.356175 26408 raft_consensus.cc:2243] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.356664 26408 raft_consensus.cc:2272] T d7d884995e6e40c6bf8be25283343717 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.359202 26408 tablet_replica.cc:333] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.359930 26408 raft_consensus.cc:2243] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.360448 26408 raft_consensus.cc:2272] T ceeac06a7cf54e40b81ce70b21a222e3 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:17:29.362288 27176 consensus_peers.cc:597] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 -> Peer f6318bc9ad074866ba228c136b4c8426 (127.25.202.1:45039): Couldn't send request to peer f6318bc9ad074866ba228c136b4c8426. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:45039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:29.362787 26408 tablet_replica.cc:333] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.363421 26408 raft_consensus.cc:2243] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.363842 26408 raft_consensus.cc:2272] T 943c0718dc5d44889b02cdbbe48b1318 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.365545 26408 tablet_replica.cc:333] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.366036 26408 raft_consensus.cc:2243] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.366459 26408 raft_consensus.cc:2272] T 88847fc535ed499cb53f82a470f81ded P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.368221 26408 tablet_replica.cc:333] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.368714 26408 raft_consensus.cc:2243] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.369154 26408 raft_consensus.cc:2272] T cae1b0bb8aba4ea69fed0507b2cdd16f P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.370697 26408 tablet_replica.cc:333] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.371179 26408 raft_consensus.cc:2243] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.371665 26408 raft_consensus.cc:2272] T 9013e4bd5c25415a977c1a085842d0ac P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.373308 26408 tablet_replica.cc:333] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.373816 26408 raft_consensus.cc:2243] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.374567 26408 raft_consensus.cc:2272] T 9af6b5759a1a40a2854bd1ed9df3ec8a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.376518 26408 tablet_replica.cc:333] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.376987 26408 raft_consensus.cc:2243] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:29.377404 26408 raft_consensus.cc:2272] T d7f747d704794e3fb8dde0d7ba27da7a P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.378973 26408 tablet_replica.cc:333] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.379493 26408 raft_consensus.cc:2243] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.380213 26408 raft_consensus.cc:2272] T b7b4d71766084c9caafdaaaa2243a249 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.381856 26408 tablet_replica.cc:333] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3: stopping tablet replica
I20260709 12:17:29.382280 26408 raft_consensus.cc:2243] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.382997 26408 raft_consensus.cc:2272] T 88553a5581b04f03be2745006b011395 P 91c8ef68f2cb4b6dbf3db4084441a6f3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.410678 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:17:29.430979 26408 master.cc:562] Master@127.25.202.62:45067 shutting down...
I20260709 12:17:29.455968 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:17:29.456620 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:29.456951 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P 625de6d6b13746aabaca5197f0ec7bbb: stopping tablet replica
I20260709 12:17:29.477500 26408 master.cc:584] Master@127.25.202.62:45067 shutdown complete.
91c8ef68f2cb4b6dbf3db4084441a6f3  1
aeb5150d7d474113b386036e444b80d7  4
f6318bc9ad074866ba228c136b4c8426  5
91c8ef68f2cb4b6dbf3db4084441a6f3  3
aeb5150d7d474113b386036e444b80d7  4
f6318bc9ad074866ba228c136b4c8426  3
91c8ef68f2cb4b6dbf3db4084441a6f3  1
aeb5150d7d474113b386036e444b80d7  1
f6318bc9ad074866ba228c136b4c8426  8
91c8ef68f2cb4b6dbf3db4084441a6f3  3
aeb5150d7d474113b386036e444b80d7  3
f6318bc9ad074866ba228c136b4c8426  4
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29689 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:17:29.525298 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:43715
I20260709 12:17:29.526638 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:29.533450 27462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.534142 27463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.534735 27465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:29.535455 26408 server_base.cc:1061] running on GCE node
I20260709 12:17:29.536418 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:29.536590 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:29.536701 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599449536691 us; error 0 us; skew 500 ppm
I20260709 12:17:29.537159 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:29.539777 26408 webserver.cc:533] Webserver started at http://127.25.202.62:43007/ using document root <none> and password file <none>
I20260709 12:17:29.540225 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:29.540392 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:29.540624 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:29.541752 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "04e0ca717d6b41608e4d35acaee1cac8"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:29.546329 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:17:29.550283 27470 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.551669 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:17:29.552017 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "04e0ca717d6b41608e4d35acaee1cac8"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:29.552356 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:29.578830 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:29.580201 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:29.627853 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:43715
I20260709 12:17:29.628261 27521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:43715 every 8 connection(s)
I20260709 12:17:29.632948 27522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:29.643255 27522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Bootstrap starting.
I20260709 12:17:29.647991 27522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:29.652559 27522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: No bootstrap required, opened a new log
I20260709 12:17:29.655033 27522 raft_consensus.cc:359] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e0ca717d6b41608e4d35acaee1cac8" member_type: VOTER }
I20260709 12:17:29.655541 27522 raft_consensus.cc:385] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:29.655792 27522 raft_consensus.cc:740] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e0ca717d6b41608e4d35acaee1cac8, State: Initialized, Role: FOLLOWER
I20260709 12:17:29.656560 27522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e0ca717d6b41608e4d35acaee1cac8" member_type: VOTER }
I20260709 12:17:29.657037 27522 raft_consensus.cc:399] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:17:29.657253 27522 raft_consensus.cc:493] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:17:29.657562 27522 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:29.662423 27522 raft_consensus.cc:515] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e0ca717d6b41608e4d35acaee1cac8" member_type: VOTER }
I20260709 12:17:29.662987 27522 leader_election.cc:304] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e0ca717d6b41608e4d35acaee1cac8; no voters: 
I20260709 12:17:29.664343 27522 leader_election.cc:290] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:17:29.664760 27525 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:29.666079 27525 raft_consensus.cc:697] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 1 LEADER]: Becoming Leader. State: Replica: 04e0ca717d6b41608e4d35acaee1cac8, State: Running, Role: LEADER
I20260709 12:17:29.666814 27525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e0ca717d6b41608e4d35acaee1cac8" member_type: VOTER }
I20260709 12:17:29.667375 27522 sys_catalog.cc:565] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:17:29.670001 27527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e0ca717d6b41608e4d35acaee1cac8. Latest consensus state: current_term: 1 leader_uuid: "04e0ca717d6b41608e4d35acaee1cac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e0ca717d6b41608e4d35acaee1cac8" member_type: VOTER } }
I20260709 12:17:29.669966 27526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e0ca717d6b41608e4d35acaee1cac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e0ca717d6b41608e4d35acaee1cac8" member_type: VOTER } }
I20260709 12:17:29.670646 27527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [sys.catalog]: This master's current role is: LEADER
I20260709 12:17:29.670709 27526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [sys.catalog]: This master's current role is: LEADER
I20260709 12:17:29.673501 27530 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:17:29.682037 27530 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:17:29.684778 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:17:29.692235 27530 catalog_manager.cc:1383] Generated new cluster ID: d189f7dbe07b48b5b77f5307c6d19bfd
I20260709 12:17:29.692680 27530 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:17:29.724740 27530 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:17:29.726095 27530 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:17:29.740831 27530 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Generated new TSK 0
I20260709 12:17:29.741529 27530 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:17:29.752031 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:29.758778 27543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.760243 27544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:29.763674 26408 server_base.cc:1061] running on GCE node
W20260709 12:17:29.763501 27546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:29.765161 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:29.765506 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:29.765722 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599449765704 us; error 0 us; skew 500 ppm
I20260709 12:17:29.766361 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:29.769716 26408 webserver.cc:533] Webserver started at http://127.25.202.1:46041/ using document root <none> and password file <none>
I20260709 12:17:29.770370 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:29.770720 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:29.771034 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:29.772375 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "3f90188458da46f6ba138d5f82c55b69"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:29.778007 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:17:29.781983 27551 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.783226 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 12:17:29.783749 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "3f90188458da46f6ba138d5f82c55b69"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:29.784096 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:29.808038 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:29.809320 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:29.810925 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:29.813331 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:29.813572 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.813819 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:29.814010 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.861215 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:41657
I20260709 12:17:29.861423 27613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:41657 every 8 connection(s)
I20260709 12:17:29.866506 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:29.875669 27618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.876852 27619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.878087 27621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:29.878942 26408 server_base.cc:1061] running on GCE node
I20260709 12:17:29.880264 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:29.880471 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:29.880615 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599449880603 us; error 0 us; skew 500 ppm
I20260709 12:17:29.881213 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:29.884215 26408 webserver.cc:533] Webserver started at http://127.25.202.2:40413/ using document root <none> and password file <none>
I20260709 12:17:29.884896 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:29.885183 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:29.885541 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:29.887245 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "9c09719959a54e8eb66d2041dd9c3113"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:29.887528 27614 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43715
I20260709 12:17:29.887935 27614 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:29.888654 27614 heartbeater.cc:507] Master 127.25.202.62:43715 requested a full tablet report, sending...
I20260709 12:17:29.891132 27487 ts_manager.cc:194] Registered new tserver with Master: 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:29.894040 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:17:29.894042 27487 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38754
I20260709 12:17:29.898634 27626 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.899801 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:17:29.900238 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "9c09719959a54e8eb66d2041dd9c3113"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:29.900529 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:29.919440 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:29.920755 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:29.922247 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:29.924741 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:29.924966 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.925705 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:29.925884 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:29.971726 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:33051
I20260709 12:17:29.971906 27688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:33051 every 8 connection(s)
I20260709 12:17:29.977275 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:29.985745 27692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.986894 27693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:29.990429 27695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:29.991487 26408 server_base.cc:1061] running on GCE node
I20260709 12:17:29.992100 27689 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43715
I20260709 12:17:29.992467 27689 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:29.992515 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:29.992856 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:29.993040 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599449993004 us; error 0 us; skew 500 ppm
I20260709 12:17:29.993498 27689 heartbeater.cc:507] Master 127.25.202.62:43715 requested a full tablet report, sending...
I20260709 12:17:29.993613 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:29.996084 27487 ts_manager.cc:194] Registered new tserver with Master: 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:29.996881 26408 webserver.cc:533] Webserver started at http://127.25.202.3:38751/ using document root <none> and password file <none>
I20260709 12:17:29.997442 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:29.997648 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:29.997642 27487 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38756
I20260709 12:17:29.998140 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:29.999620 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:30.004877 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:17:30.008757 27700 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:30.009709 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:17:30.010025 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
format_stamp: "Formatted at 2026-07-09 12:17:29 on dist-test-slave-m57n"
I20260709 12:17:30.010358 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:30.027664 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:30.028918 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:30.030447 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:30.032841 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:30.033071 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:30.033340 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:30.033517 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:30.077618 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:41211
I20260709 12:17:30.077822 27762 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:41211 every 8 connection(s)
I20260709 12:17:30.092885 27763 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43715
I20260709 12:17:30.093281 27763 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:30.094043 27763 heartbeater.cc:507] Master 127.25.202.62:43715 requested a full tablet report, sending...
I20260709 12:17:30.096145 27487 ts_manager.cc:194] Registered new tserver with Master: ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:30.096956 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015991006s
I20260709 12:17:30.097499 27487 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38758
I20260709 12:17:30.124886 27487 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38762:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:17:30.131307 27487 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:17:30.339560 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 85b7862879764c33b1db83d8596e3e87 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:17:30.340693 27654 tablet_service.cc:1511] Processing CreateTablet for tablet 85b7862879764c33b1db83d8596e3e87 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:17:30.341478 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b7862879764c33b1db83d8596e3e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.342206 27654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b7862879764c33b1db83d8596e3e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.341835 27653 tablet_service.cc:1511] Processing CreateTablet for tablet 37a0fdc57f6240869f0977e359f258c0 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:17:30.343304 27653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a0fdc57f6240869f0977e359f258c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.344660 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 37a0fdc57f6240869f0977e359f258c0 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:17:30.346117 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a0fdc57f6240869f0977e359f258c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.351006 27726 tablet_service.cc:1511] Processing CreateTablet for tablet 6491d84db64f4d48b3b3d5be8aec385b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:17:30.352560 27726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491d84db64f4d48b3b3d5be8aec385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.356719 27725 tablet_service.cc:1511] Processing CreateTablet for tablet 22597fcae204491f84839e680c083ecc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:17:30.363392 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 68dbab4c1e1442b699465f7b3e728acd (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:17:30.364289 27725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22597fcae204491f84839e680c083ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.364830 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68dbab4c1e1442b699465f7b3e728acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.364562 27651 tablet_service.cc:1511] Processing CreateTablet for tablet 22597fcae204491f84839e680c083ecc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:17:30.365890 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22597fcae204491f84839e680c083ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.363385 27652 tablet_service.cc:1511] Processing CreateTablet for tablet 6491d84db64f4d48b3b3d5be8aec385b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:17:30.367107 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491d84db64f4d48b3b3d5be8aec385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.370215 27650 tablet_service.cc:1511] Processing CreateTablet for tablet 0d80be98cdd043ffa8d122ba0ac29447 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:17:30.371634 27650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d80be98cdd043ffa8d122ba0ac29447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.360281 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 0d80be98cdd043ffa8d122ba0ac29447 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:17:30.373401 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d80be98cdd043ffa8d122ba0ac29447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.380867 27578 tablet_service.cc:1511] Processing CreateTablet for tablet 37a0fdc57f6240869f0977e359f258c0 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:17:30.382177 27576 tablet_service.cc:1511] Processing CreateTablet for tablet 22597fcae204491f84839e680c083ecc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:17:30.382632 27578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a0fdc57f6240869f0977e359f258c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.383543 27577 tablet_service.cc:1511] Processing CreateTablet for tablet 6491d84db64f4d48b3b3d5be8aec385b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:17:30.385105 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491d84db64f4d48b3b3d5be8aec385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.385524 27575 tablet_service.cc:1511] Processing CreateTablet for tablet 0d80be98cdd043ffa8d122ba0ac29447 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:17:30.386431 27579 tablet_service.cc:1511] Processing CreateTablet for tablet 85b7862879764c33b1db83d8596e3e87 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:17:30.389987 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d80be98cdd043ffa8d122ba0ac29447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.392925 27574 tablet_service.cc:1511] Processing CreateTablet for tablet 68dbab4c1e1442b699465f7b3e728acd (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:17:30.396027 27579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b7862879764c33b1db83d8596e3e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.399923 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22597fcae204491f84839e680c083ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.403540 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 68dbab4c1e1442b699465f7b3e728acd (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:17:30.405027 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68dbab4c1e1442b699465f7b3e728acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.411718 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68dbab4c1e1442b699465f7b3e728acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.414991 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 70b48af2ec82404986e5e18d544db5b4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:17:30.416599 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b48af2ec82404986e5e18d544db5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.417042 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 70b48af2ec82404986e5e18d544db5b4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:17:30.418556 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b48af2ec82404986e5e18d544db5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.430308 27723 tablet_service.cc:1511] Processing CreateTablet for tablet 70b48af2ec82404986e5e18d544db5b4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:17:30.431874 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b48af2ec82404986e5e18d544db5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.434930 27572 tablet_service.cc:1511] Processing CreateTablet for tablet 27d7343bc8174a5f89f4652b4401c3ec (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:17:30.436554 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d7343bc8174a5f89f4652b4401c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.443637 27778 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:30.462651 27647 tablet_service.cc:1511] Processing CreateTablet for tablet 27d7343bc8174a5f89f4652b4401c3ec (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:17:30.464365 27647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d7343bc8174a5f89f4652b4401c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.473625 27778 tablet_bootstrap.cc:654] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.477933 27780 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:30.481455 27779 tablet_bootstrap.cc:492] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:30.483798 27576 tablet_service.cc:1511] Processing CreateTablet for tablet 5accfdbc68cd4cf399c98e2a533f500e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:17:30.485121 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5accfdbc68cd4cf399c98e2a533f500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.496994 27780 tablet_bootstrap.cc:654] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.482472 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 27d7343bc8174a5f89f4652b4401c3ec (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:17:30.499272 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d7343bc8174a5f89f4652b4401c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.505174 27653 tablet_service.cc:1511] Processing CreateTablet for tablet 5accfdbc68cd4cf399c98e2a533f500e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:17:30.507030 27653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5accfdbc68cd4cf399c98e2a533f500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.511467 27725 tablet_service.cc:1511] Processing CreateTablet for tablet 5accfdbc68cd4cf399c98e2a533f500e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:17:30.512845 27725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5accfdbc68cd4cf399c98e2a533f500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.524220 27577 tablet_service.cc:1511] Processing CreateTablet for tablet 74b289dab6a94832ad11d8ac1e30d802 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:17:30.525692 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b289dab6a94832ad11d8ac1e30d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.537599 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 74b289dab6a94832ad11d8ac1e30d802 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:17:30.538973 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b289dab6a94832ad11d8ac1e30d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.539480 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 310b1b37298244c0bd5bbe2b0c7328d7 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:17:30.540812 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 310b1b37298244c0bd5bbe2b0c7328d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.544512 27778 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:30.544984 27778 ts_tablet_manager.cc:1403] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.102s	user 0.012s	sys 0.000s
I20260709 12:17:30.545002 27652 tablet_service.cc:1511] Processing CreateTablet for tablet b4e3fe6577354786a64bed5843d9a6a4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:17:30.546361 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e3fe6577354786a64bed5843d9a6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.548813 27780 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:30.549288 27780 ts_tablet_manager.cc:1403] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.072s	user 0.013s	sys 0.001s
I20260709 12:17:30.552359 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 74b289dab6a94832ad11d8ac1e30d802 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:17:30.553725 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b289dab6a94832ad11d8ac1e30d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.554136 27726 tablet_service.cc:1511] Processing CreateTablet for tablet 310b1b37298244c0bd5bbe2b0c7328d7 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:17:30.564210 27726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 310b1b37298244c0bd5bbe2b0c7328d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.567953 27728 tablet_service.cc:1511] Processing CreateTablet for tablet b4e3fe6577354786a64bed5843d9a6a4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:17:30.569299 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e3fe6577354786a64bed5843d9a6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.570921 27779 tablet_bootstrap.cc:654] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.572048 27576 tablet_service.cc:1511] Processing CreateTablet for tablet 4f392cf87198499d9c064d1ec232d3d7 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:17:30.573457 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f392cf87198499d9c064d1ec232d3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.574592 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 4f392cf87198499d9c064d1ec232d3d7 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:17:30.576097 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f392cf87198499d9c064d1ec232d3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.562234 27780 raft_consensus.cc:359] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.585983 27780 raft_consensus.cc:385] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:30.586376 27780 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:30.563762 27778 raft_consensus.cc:359] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:30.560496 27575 tablet_service.cc:1511] Processing CreateTablet for tablet 310b1b37298244c0bd5bbe2b0c7328d7 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:17:30.588336 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 310b1b37298244c0bd5bbe2b0c7328d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.593457 27725 tablet_service.cc:1511] Processing CreateTablet for tablet 4f392cf87198499d9c064d1ec232d3d7 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:17:30.594759 27725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f392cf87198499d9c064d1ec232d3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.587522 27778 raft_consensus.cc:385] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:30.597383 27778 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:30.597522 27578 tablet_service.cc:1511] Processing CreateTablet for tablet 7c204963842941859d4313114fd92c93 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:17:30.598248 27778 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:30.587306 27780 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.588661 27572 tablet_service.cc:1511] Processing CreateTablet for tablet b4e3fe6577354786a64bed5843d9a6a4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:17:30.606132 27653 tablet_service.cc:1511] Processing CreateTablet for tablet 7c204963842941859d4313114fd92c93 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:17:30.607547 27653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c204963842941859d4313114fd92c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.607820 27577 tablet_service.cc:1511] Processing CreateTablet for tablet 5f21db64000c434ca36b2b9021c13a44 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:17:30.612054 27578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c204963842941859d4313114fd92c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.613093 27654 tablet_service.cc:1511] Processing CreateTablet for tablet 5f21db64000c434ca36b2b9021c13a44 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:17:30.614750 27654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f21db64000c434ca36b2b9021c13a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.621610 27723 tablet_service.cc:1511] Processing CreateTablet for tablet 5f21db64000c434ca36b2b9021c13a44 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:17:30.621954 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 7c204963842941859d4313114fd92c93 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:17:30.625098 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f21db64000c434ca36b2b9021c13a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.624615 27574 tablet_service.cc:1511] Processing CreateTablet for tablet b8ac8041c87e49308784e1cad6aeec03 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:17:30.626444 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ac8041c87e49308784e1cad6aeec03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.629201 27647 tablet_service.cc:1511] Processing CreateTablet for tablet b8ac8041c87e49308784e1cad6aeec03 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:17:30.630987 27647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ac8041c87e49308784e1cad6aeec03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.636142 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c204963842941859d4313114fd92c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.641296 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e3fe6577354786a64bed5843d9a6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.641661 27778 ts_tablet_manager.cc:1434] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.096s	user 0.005s	sys 0.008s
I20260709 12:17:30.642597 27763 heartbeater.cc:499] Master 127.25.202.62:43715 was elected leader, sending a full tablet report...
I20260709 12:17:30.642275 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:17:30.643054 27778 tablet_bootstrap.cc:492] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:30.643738 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f21db64000c434ca36b2b9021c13a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.647802 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.656239 27726 tablet_service.cc:1511] Processing CreateTablet for tablet b8ac8041c87e49308784e1cad6aeec03 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:17:30.671036 27779 tablet_bootstrap.cc:492] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:30.671919 27779 ts_tablet_manager.cc:1403] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.191s	user 0.012s	sys 0.025s
I20260709 12:17:30.673593 27780 ts_tablet_manager.cc:1434] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.124s	user 0.019s	sys 0.021s
I20260709 12:17:30.674399 27614 heartbeater.cc:499] Master 127.25.202.62:43715 was elected leader, sending a full tablet report...
I20260709 12:17:30.674749 27780 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:30.678182 27778 tablet_bootstrap.cc:654] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.679858 27726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ac8041c87e49308784e1cad6aeec03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.680420 27652 tablet_service.cc:1511] Processing CreateTablet for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:17:30.681839 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.686528 27780 tablet_bootstrap.cc:654] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:30.697142 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:30.699313 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:17:30.700801 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.704257 27576 tablet_service.cc:1511] Processing CreateTablet for tablet ecd16aab46634913b20f3501e9a6389f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:17:30.705653 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd16aab46634913b20f3501e9a6389f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.707948 27785 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.708424 27785 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.675103 27779 raft_consensus.cc:359] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.713683 27779 raft_consensus.cc:385] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:30.723838 27779 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:30.730104 27779 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.711177 27785 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:30.734364 27780 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:30.734896 27780 ts_tablet_manager.cc:1403] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.060s	user 0.027s	sys 0.003s
I20260709 12:17:30.735448 27648 tablet_service.cc:1511] Processing CreateTablet for tablet ecd16aab46634913b20f3501e9a6389f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:17:30.736838 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd16aab46634913b20f3501e9a6389f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.740338 27728 tablet_service.cc:1511] Processing CreateTablet for tablet ecd16aab46634913b20f3501e9a6389f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:17:30.741870 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd16aab46634913b20f3501e9a6389f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.741477 27575 tablet_service.cc:1511] Processing CreateTablet for tablet b0b3d61cfc0d48f98b3cc7cc7cb63370 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:17:30.743194 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b3d61cfc0d48f98b3cc7cc7cb63370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.750484 27650 tablet_service.cc:1511] Processing CreateTablet for tablet b0b3d61cfc0d48f98b3cc7cc7cb63370 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:17:30.751843 27650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b3d61cfc0d48f98b3cc7cc7cb63370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.767462 27577 tablet_service.cc:1511] Processing CreateTablet for tablet 42e7d195f3f545b98a8dedb26c479c9c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:17:30.768906 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e7d195f3f545b98a8dedb26c479c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.769925 27724 tablet_service.cc:1511] Processing CreateTablet for tablet b0b3d61cfc0d48f98b3cc7cc7cb63370 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:17:30.774797 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b3d61cfc0d48f98b3cc7cc7cb63370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.777837 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:30.780520 27552 leader_election.cc:343] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:30.781684 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 42e7d195f3f545b98a8dedb26c479c9c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:17:30.782205 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:30.784615 27552 leader_election.cc:343] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:30.785070 27552 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:30.785981 27787 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:30.792753 27725 tablet_service.cc:1511] Processing CreateTablet for tablet 42e7d195f3f545b98a8dedb26c479c9c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:17:30.794044 27725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e7d195f3f545b98a8dedb26c479c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.778896 27780 raft_consensus.cc:359] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:30.795145 27780 raft_consensus.cc:385] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:30.795553 27780 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:30.795918 27653 tablet_service.cc:1511] Processing CreateTablet for tablet a6d40162b2bb4b61adc69c9fa3f03326 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:17:30.796334 27780 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:30.798511 27780 ts_tablet_manager.cc:1434] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.063s	user 0.003s	sys 0.004s
I20260709 12:17:30.799842 27723 tablet_service.cc:1511] Processing CreateTablet for tablet a6d40162b2bb4b61adc69c9fa3f03326 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:17:30.801234 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d40162b2bb4b61adc69c9fa3f03326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.801173 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 1c3943cf144748fca8d2892751757c1a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:17:30.802740 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3943cf144748fca8d2892751757c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.804432 27574 tablet_service.cc:1511] Processing CreateTablet for tablet a6d40162b2bb4b61adc69c9fa3f03326 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:17:30.807240 27780 tablet_bootstrap.cc:492] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:30.813045 27653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d40162b2bb4b61adc69c9fa3f03326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.818564 27651 tablet_service.cc:1511] Processing CreateTablet for tablet 1c3943cf144748fca8d2892751757c1a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:17:30.814234 27778 tablet_bootstrap.cc:492] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:30.820030 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3943cf144748fca8d2892751757c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.823020 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e7d195f3f545b98a8dedb26c479c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.820030 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d40162b2bb4b61adc69c9fa3f03326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.831632 27778 ts_tablet_manager.cc:1403] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.189s	user 0.017s	sys 0.000s
I20260709 12:17:30.853652 27778 raft_consensus.cc:359] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:30.866510 27615 tablet.cc:2406] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:17:30.868086 27778 raft_consensus.cc:385] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:30.868906 27778 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:30.870482 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 1c3943cf144748fca8d2892751757c1a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:17:30.875115 27778 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.877595 27652 tablet_service.cc:1511] Processing CreateTablet for tablet 22efa3bef6634e7a811baa3c42d9cd18 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:17:30.878888 27778 ts_tablet_manager.cc:1434] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.000s
I20260709 12:17:30.878954 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22efa3bef6634e7a811baa3c42d9cd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.884747 27576 tablet_service.cc:1511] Processing CreateTablet for tablet 22efa3bef6634e7a811baa3c42d9cd18 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:17:30.885748 27791 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.886555 27689 heartbeater.cc:499] Master 127.25.202.62:43715 was elected leader, sending a full tablet report...
I20260709 12:17:30.886739 27791 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:30.887976 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22efa3bef6634e7a811baa3c42d9cd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.894145 27572 tablet_service.cc:1511] Processing CreateTablet for tablet dbdcf17dc981450f989b93d262e9a5d0 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:17:30.895460 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdcf17dc981450f989b93d262e9a5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.897553 27648 tablet_service.cc:1511] Processing CreateTablet for tablet dbdcf17dc981450f989b93d262e9a5d0 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:17:30.885759 27723 tablet_service.cc:1511] Processing CreateTablet for tablet 22efa3bef6634e7a811baa3c42d9cd18 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:17:30.898947 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdcf17dc981450f989b93d262e9a5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.901436 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3943cf144748fca8d2892751757c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.902604 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22efa3bef6634e7a811baa3c42d9cd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.908320 27779 ts_tablet_manager.cc:1434] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.236s	user 0.040s	sys 0.033s
I20260709 12:17:30.910909 27778 tablet_bootstrap.cc:492] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:30.911615 27784 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.912118 27784 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:30.914544 27784 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:30.915442 27779 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:30.918128 27650 tablet_service.cc:1511] Processing CreateTablet for tablet bdb219c7e3524fc9b4a91a96610cee8b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:17:30.919376 27728 tablet_service.cc:1511] Processing CreateTablet for tablet bdb219c7e3524fc9b4a91a96610cee8b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:17:30.919397 27650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdb219c7e3524fc9b4a91a96610cee8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.920677 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdb219c7e3524fc9b4a91a96610cee8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.919673 27780 tablet_bootstrap.cc:654] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.939101 27577 tablet_service.cc:1511] Processing CreateTablet for tablet bdb219c7e3524fc9b4a91a96610cee8b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:17:30.940418 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdb219c7e3524fc9b4a91a96610cee8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.949570 27726 tablet_service.cc:1511] Processing CreateTablet for tablet 7274fd8a2115405c9b78c61214bff736 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:17:30.945576 27725 tablet_service.cc:1511] Processing CreateTablet for tablet dbdcf17dc981450f989b93d262e9a5d0 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:17:30.962126 27726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7274fd8a2115405c9b78c61214bff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.963455 27725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdcf17dc981450f989b93d262e9a5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.949637 27578 tablet_service.cc:1511] Processing CreateTablet for tablet 7274fd8a2115405c9b78c61214bff736 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:17:30.964421 27578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7274fd8a2115405c9b78c61214bff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.964974 27779 tablet_bootstrap.cc:654] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.970103 27575 tablet_service.cc:1511] Processing CreateTablet for tablet 3b881b4e55d543f9ade316642ea1b7e3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:17:30.971487 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b881b4e55d543f9ade316642ea1b7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.978842 27778 tablet_bootstrap.cc:654] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:30.958750 27791 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:30.986423 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 3b881b4e55d543f9ade316642ea1b7e3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:17:30.986423 27653 tablet_service.cc:1511] Processing CreateTablet for tablet 7274fd8a2115405c9b78c61214bff736 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:17:30.993427 27787 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.996608 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b881b4e55d543f9ade316642ea1b7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:30.997012 27787 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:30.991699 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 3b881b4e55d543f9ade316642ea1b7e3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:17:31.008846 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b881b4e55d543f9ade316642ea1b7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.010596 27787 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:31.012655 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:31.013358 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:31.014834 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:31.014961 27579 tablet_service.cc:1511] Processing CreateTablet for tablet 456e2eac35d647a3bf9eeb45a3d7596f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:17:31.015715 27787 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:31.015974 27653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7274fd8a2115405c9b78c61214bff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.016098 27787 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.016484 27787 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.016347 27579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456e2eac35d647a3bf9eeb45a3d7596f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.021879 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:31.023002 27652 tablet_service.cc:1511] Processing CreateTablet for tablet 456e2eac35d647a3bf9eeb45a3d7596f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:17:31.024392 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456e2eac35d647a3bf9eeb45a3d7596f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:17:31.023380 27552 leader_election.cc:343] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.030112 27787 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:31.033013 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:31.034492 27738 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.056286 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 456e2eac35d647a3bf9eeb45a3d7596f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:17:31.057657 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456e2eac35d647a3bf9eeb45a3d7596f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.057511 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:17:31.058910 27552 leader_election.cc:343] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.060426 27787 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:31.061892 27573 tablet_service.cc:1511] Processing CreateTablet for tablet e76581eba9454a229c556095ac8b3e23 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:17:31.063252 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76581eba9454a229c556095ac8b3e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.064368 27780 tablet_bootstrap.cc:492] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:31.064962 27780 ts_tablet_manager.cc:1403] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.266s	user 0.026s	sys 0.012s
I20260709 12:17:31.066035 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:31.067478 27701 leader_election.cc:343] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.067490 27651 tablet_service.cc:1511] Processing CreateTablet for tablet e76581eba9454a229c556095ac8b3e23 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:17:31.069026 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76581eba9454a229c556095ac8b3e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.081769 27780 raft_consensus.cc:359] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.082759 27780 raft_consensus.cc:385] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.083065 27780 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.083871 27780 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.091396 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:31.092725 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:31.093830 27722 tablet_service.cc:1511] Processing CreateTablet for tablet e76581eba9454a229c556095ac8b3e23 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:17:31.096654 27780 ts_tablet_manager.cc:1434] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.004s
I20260709 12:17:31.097599 27780 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:31.098374 27785 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:31.100934 27574 tablet_service.cc:1511] Processing CreateTablet for tablet c596ddd50b894549b567a5b66ac1a31e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:17:31.102227 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c596ddd50b894549b567a5b66ac1a31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.102957 27722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76581eba9454a229c556095ac8b3e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.103157 27649 tablet_service.cc:1511] Processing CreateTablet for tablet c596ddd50b894549b567a5b66ac1a31e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:17:31.104401 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c596ddd50b894549b567a5b66ac1a31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.123896 27779 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:31.126048 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:31.126906 27738 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:31.128654 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:31.130009 27787 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.130127 27791 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:31.130681 27791 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.130522 27787 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.131196 27791 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.132680 27787 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.135011 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:31.136385 27552 leader_election.cc:343] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.137185 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:31.138567 27552 leader_election.cc:343] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.139014 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:31.140142 27787 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:31.124550 27779 ts_tablet_manager.cc:1403] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.209s	user 0.016s	sys 0.023s
I20260709 12:17:31.143802 27779 raft_consensus.cc:359] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.144493 27779 raft_consensus.cc:385] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.144781 27779 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.145473 27779 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.147810 27780 tablet_bootstrap.cc:654] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.150336 27779 ts_tablet_manager.cc:1434] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:17:31.153385 27779 tablet_bootstrap.cc:492] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:31.160959 27779 tablet_bootstrap.cc:654] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.167723 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:31.172379 27792 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.172868 27792 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.175069 27792 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.178579 27785 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:31.182983 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:31.183174 27785 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:31.184378 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:31.185772 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:31.186503 27587 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:31.188683 27575 tablet_service.cc:1511] Processing CreateTablet for tablet 723b2024e4414796ba50ece654596156 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
W20260709 12:17:31.189401 27627 leader_election.cc:343] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.189977 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723b2024e4414796ba50ece654596156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.191236 27579 tablet_service.cc:1511] Processing CreateTablet for tablet c3f34c916e524bfa9fcae2044da089fc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:17:31.191705 27721 tablet_service.cc:1511] Processing CreateTablet for tablet 723b2024e4414796ba50ece654596156 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:17:31.193001 27579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f34c916e524bfa9fcae2044da089fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:17:31.194770 27630 leader_election.cc:343] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.196065 27630 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:31.199615 27724 tablet_service.cc:1511] Processing CreateTablet for tablet c3f34c916e524bfa9fcae2044da089fc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:17:31.200354 27572 tablet_service.cc:1511] Processing CreateTablet for tablet d3d15932c0a64b668179d6868ab4707c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:17:31.201808 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d15932c0a64b668179d6868ab4707c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.203020 27792 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:31.203545 27792 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.203997 27792 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.193526 27721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723b2024e4414796ba50ece654596156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.189939 27650 tablet_service.cc:1511] Processing CreateTablet for tablet 723b2024e4414796ba50ece654596156 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:17:31.207038 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f34c916e524bfa9fcae2044da089fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.202571 27723 tablet_service.cc:1511] Processing CreateTablet for tablet c596ddd50b894549b567a5b66ac1a31e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:17:31.208945 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c596ddd50b894549b567a5b66ac1a31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.209180 27650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723b2024e4414796ba50ece654596156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.210475 27778 tablet_bootstrap.cc:492] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:31.210978 27778 ts_tablet_manager.cc:1403] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.300s	user 0.021s	sys 0.045s
I20260709 12:17:31.211602 27649 tablet_service.cc:1511] Processing CreateTablet for tablet c3f34c916e524bfa9fcae2044da089fc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:17:31.213272 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f34c916e524bfa9fcae2044da089fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.222340 27652 tablet_service.cc:1511] Processing CreateTablet for tablet d3d15932c0a64b668179d6868ab4707c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:17:31.223793 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d15932c0a64b668179d6868ab4707c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.236544 27778 raft_consensus.cc:359] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.253324 27727 tablet_service.cc:1511] Processing CreateTablet for tablet d3d15932c0a64b668179d6868ab4707c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:17:31.254709 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d15932c0a64b668179d6868ab4707c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.204458 27791 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.245720 27778 raft_consensus.cc:385] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.286557 27575 tablet_service.cc:1511] Processing CreateTablet for tablet a07754f58c3a421ba559f56d67d87bb4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:17:31.292783 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:31.293301 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07754f58c3a421ba559f56d67d87bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.293941 27784 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:31.276764 27792 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.287078 27778 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.299530 27778 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.302870 27778 ts_tablet_manager.cc:1434] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.092s	user 0.010s	sys 0.000s
I20260709 12:17:31.316162 27485 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:31.320590 27791 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.321391 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:17:31.323187 27630 leader_election.cc:343] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.330088 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:31.330698 27738 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.305330 27648 tablet_service.cc:1511] Processing CreateTablet for tablet a07754f58c3a421ba559f56d67d87bb4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:17:31.336651 27779 tablet_bootstrap.cc:492] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:31.337131 27779 ts_tablet_manager.cc:1403] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.186s	user 0.009s	sys 0.023s
I20260709 12:17:31.338719 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07754f58c3a421ba559f56d67d87bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.340374 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:31.341190 27589 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.341344 27779 raft_consensus.cc:359] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.342067 27779 raft_consensus.cc:385] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.342366 27779 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.350473 27779 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.352944 27780 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:31.353454 27780 ts_tablet_manager.cc:1403] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.256s	user 0.017s	sys 0.047s
I20260709 12:17:31.354843 27738 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:31.356202 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:31.356726 27779 ts_tablet_manager.cc:1434] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 12:17:31.357306 27791 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:31.366022 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:17:31.367498 27627 leader_election.cc:343] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.368451 27778 tablet_bootstrap.cc:492] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:31.370975 27780 raft_consensus.cc:359] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.371419 27779 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:31.371739 27780 raft_consensus.cc:385] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.372180 27780 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.372870 27780 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.371680 27792 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.375905 27589 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:31.376540 27727 tablet_service.cc:1511] Processing CreateTablet for tablet a07754f58c3a421ba559f56d67d87bb4 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:17:31.377082 27630 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:31.377954 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07754f58c3a421ba559f56d67d87bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.378207 27792 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:31.378660 27792 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:31.379539 27792 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.381832 27780 ts_tablet_manager.cc:1434] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:17:31.382097 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 8c52fdc88d8646dd8e6584f8dbd137a3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:17:31.382854 27780 tablet_bootstrap.cc:492] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:31.383385 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c52fdc88d8646dd8e6584f8dbd137a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.391486 27791 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:31.392831 27791 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.393923 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 8c52fdc88d8646dd8e6584f8dbd137a3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:17:31.394894 27780 tablet_bootstrap.cc:654] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.395462 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c52fdc88d8646dd8e6584f8dbd137a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.406939 27575 tablet_service.cc:1511] Processing CreateTablet for tablet 6f3a9a00f0db424982bf0ea90e7ae286 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:17:31.408730 27575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3a9a00f0db424982bf0ea90e7ae286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.414299 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 6f3a9a00f0db424982bf0ea90e7ae286 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:17:31.416170 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3a9a00f0db424982bf0ea90e7ae286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.405267 27723 tablet_service.cc:1511] Processing CreateTablet for tablet 8c52fdc88d8646dd8e6584f8dbd137a3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:17:31.424434 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c52fdc88d8646dd8e6584f8dbd137a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.420884 27652 tablet_service.cc:1511] Processing CreateTablet for tablet 6f3a9a00f0db424982bf0ea90e7ae286 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:17:31.431591 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3a9a00f0db424982bf0ea90e7ae286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.432909 27572 tablet_service.cc:1511] Processing CreateTablet for tablet dad2444fb3474aee8e7dbbd7e673ab01 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:17:31.434749 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad2444fb3474aee8e7dbbd7e673ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.446972 27649 tablet_service.cc:1511] Processing CreateTablet for tablet dad2444fb3474aee8e7dbbd7e673ab01 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:17:31.448930 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad2444fb3474aee8e7dbbd7e673ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.456871 27779 tablet_bootstrap.cc:654] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.471405 27778 tablet_bootstrap.cc:654] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.477973 27726 tablet_service.cc:1511] Processing CreateTablet for tablet dad2444fb3474aee8e7dbbd7e673ab01 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:17:31.479631 27726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad2444fb3474aee8e7dbbd7e673ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.495086 27574 tablet_service.cc:1511] Processing CreateTablet for tablet e634fbff5fb24e2fa133b95599b0db1f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:17:31.496929 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e634fbff5fb24e2fa133b95599b0db1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.500590 27721 tablet_service.cc:1511] Processing CreateTablet for tablet e634fbff5fb24e2fa133b95599b0db1f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:17:31.504071 27721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e634fbff5fb24e2fa133b95599b0db1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.504580 27649 tablet_service.cc:1511] Processing CreateTablet for tablet e634fbff5fb24e2fa133b95599b0db1f (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:17:31.505820 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e634fbff5fb24e2fa133b95599b0db1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.506925 27486 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:31.510939 27579 tablet_service.cc:1511] Processing CreateTablet for tablet 73b2d9da2f874f958d24ff32c2097c0a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:17:31.512291 27579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b2d9da2f874f958d24ff32c2097c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.512357 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 73b2d9da2f874f958d24ff32c2097c0a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:17:31.514775 27578 tablet_service.cc:1511] Processing CreateTablet for tablet 073b12eba02f4990a4fbe7080d8b8aff (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:17:31.515595 27577 tablet_service.cc:1511] Processing CreateTablet for tablet 81e6b2dabf8f4e6cb549ca250430e568 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:17:31.516042 27578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b12eba02f4990a4fbe7080d8b8aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.516782 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e6b2dabf8f4e6cb549ca250430e568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.516862 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b2d9da2f874f958d24ff32c2097c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.516978 27651 tablet_service.cc:1511] Processing CreateTablet for tablet 073b12eba02f4990a4fbe7080d8b8aff (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:17:31.518303 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b12eba02f4990a4fbe7080d8b8aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.522985 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 073b12eba02f4990a4fbe7080d8b8aff (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:17:31.524266 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b12eba02f4990a4fbe7080d8b8aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.533381 27653 tablet_service.cc:1511] Processing CreateTablet for tablet 81e6b2dabf8f4e6cb549ca250430e568 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:17:31.544292 27653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e6b2dabf8f4e6cb549ca250430e568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.545171 27576 tablet_service.cc:1511] Processing CreateTablet for tablet 27b4972f682948f4981cb0dd1a74b4f5 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:17:31.546407 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b4972f682948f4981cb0dd1a74b4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.548465 27784 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.548941 27784 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.551014 27784 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:31.552963 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:31.553591 27780 tablet_bootstrap.cc:492] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:31.553952 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:31.555392 27701 leader_election.cc:343] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.553607 27589 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:31.557199 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:31.554143 27780 ts_tablet_manager.cc:1403] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.171s	user 0.039s	sys 0.047s
I20260709 12:17:31.559491 27784 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:31.559847 27784 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.560912 27724 tablet_service.cc:1511] Processing CreateTablet for tablet 81e6b2dabf8f4e6cb549ca250430e568 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:17:31.561780 27724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e6b2dabf8f4e6cb549ca250430e568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.560173 27784 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.573760 27780 raft_consensus.cc:359] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.573959 27725 tablet_service.cc:1511] Processing CreateTablet for tablet 27b4972f682948f4981cb0dd1a74b4f5 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:17:31.574503 27780 raft_consensus.cc:385] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.574824 27780 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.570777 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 27b4972f682948f4981cb0dd1a74b4f5 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:17:31.575624 27780 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.576324 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b4972f682948f4981cb0dd1a74b4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.578133 27780 ts_tablet_manager.cc:1434] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.001s
I20260709 12:17:31.579073 27780 tablet_bootstrap.cc:492] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:31.575250 27725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b4972f682948f4981cb0dd1a74b4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.585114 27780 tablet_bootstrap.cc:654] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.586983 27784 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.589020 27784 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:31.589732 27571 tablet_service.cc:1511] Processing CreateTablet for tablet e1ffc143bb914f039c5b8bb615692aa3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:17:31.593364 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:31.594615 27652 tablet_service.cc:1511] Processing CreateTablet for tablet e1ffc143bb914f039c5b8bb615692aa3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:17:31.595893 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ffc143bb914f039c5b8bb615692aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.598146 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:17:31.595067 27701 leader_election.cc:343] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.598739 27589 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.598690 27570 tablet_service.cc:1511] Processing CreateTablet for tablet 74155a9e0ffe48e5ac2f5dd6d0e8aac8 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:17:31.591156 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ffc143bb914f039c5b8bb615692aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.598471 27650 tablet_service.cc:1511] Processing CreateTablet for tablet 73b2d9da2f874f958d24ff32c2097c0a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:17:31.606871 27650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b2d9da2f874f958d24ff32c2097c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.607887 27570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74155a9e0ffe48e5ac2f5dd6d0e8aac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.647713 27723 tablet_service.cc:1511] Processing CreateTablet for tablet e1ffc143bb914f039c5b8bb615692aa3 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:17:31.648386 27779 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:31.648833 27779 ts_tablet_manager.cc:1403] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.278s	user 0.017s	sys 0.006s
I20260709 12:17:31.649063 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ffc143bb914f039c5b8bb615692aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.652654 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 74155a9e0ffe48e5ac2f5dd6d0e8aac8 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:17:31.653551 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74155a9e0ffe48e5ac2f5dd6d0e8aac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.679777 27791 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.683563 27791 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.685578 27791 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.660543 27779 raft_consensus.cc:359] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:31.694849 27552 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:31.695439 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 74155a9e0ffe48e5ac2f5dd6d0e8aac8 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:17:31.699126 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:31.692862 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:31.701764 27627 leader_election.cc:343] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.702471 27587 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:31.692736 27779 raft_consensus.cc:385] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.705917 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:31.705962 27779 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.706928 27791 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:31.707437 27791 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.707901 27791 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.708889 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 70329e70bb724d43a3185508e1186f4b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:17:31.709116 27726 tablet_service.cc:1511] Processing CreateTablet for tablet 70329e70bb724d43a3185508e1186f4b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:17:31.710107 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70329e70bb724d43a3185508e1186f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.712844 27651 tablet_service.cc:1511] Processing CreateTablet for tablet 70329e70bb724d43a3185508e1186f4b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:17:31.713769 27589 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:31.714181 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70329e70bb724d43a3185508e1186f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.715112 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:31.716271 27784 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:31.721374 27778 tablet_bootstrap.cc:492] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:31.722091 27778 ts_tablet_manager.cc:1403] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.354s	user 0.018s	sys 0.012s
I20260709 12:17:31.732779 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74155a9e0ffe48e5ac2f5dd6d0e8aac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.707187 27779 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.741228 27726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70329e70bb724d43a3185508e1186f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.744730 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 83433b8ed26448aca1d62a3b80ba4198 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:17:31.746237 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83433b8ed26448aca1d62a3b80ba4198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.751559 27791 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.762787 27791 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.763563 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:31.764439 27589 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.766265 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:31.766571 27721 tablet_service.cc:1511] Processing CreateTablet for tablet 83433b8ed26448aca1d62a3b80ba4198 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:17:31.766848 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:17:31.768391 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 83433b8ed26448aca1d62a3b80ba4198 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:17:31.768391 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:17:31.770010 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b423a3b3ee1452fb419df4c9c0a8d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.770687 27721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83433b8ed26448aca1d62a3b80ba4198. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:17:31.773996 27627 leader_election.cc:343] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.775027 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b423a3b3ee1452fb419df4c9c0a8d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.775537 27778 raft_consensus.cc:359] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.776289 27778 raft_consensus.cc:385] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.776588 27778 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.777386 27778 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.778339 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83433b8ed26448aca1d62a3b80ba4198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.781558 27778 ts_tablet_manager.cc:1434] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.059s	user 0.005s	sys 0.004s
I20260709 12:17:31.783030 27778 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:31.784396 27572 tablet_service.cc:1511] Processing CreateTablet for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:17:31.773942 27779 ts_tablet_manager.cc:1434] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.125s	user 0.006s	sys 0.003s
I20260709 12:17:31.798563 27785 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:17:31.805274 27779 tablet_bootstrap.cc:492] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:31.808171 27807 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:31.797814 27780 tablet_bootstrap.cc:492] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:31.809705 27780 ts_tablet_manager.cc:1403] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.231s	user 0.009s	sys 0.030s
W20260709 12:17:31.791282 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:31.812186 27589 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:31.815761 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:31.816654 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b423a3b3ee1452fb419df4c9c0a8d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.817054 27791 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:31.817994 27778 tablet_bootstrap.cc:654] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.818442 27791 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:31.818595 27779 tablet_bootstrap.cc:654] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:31.819164 27791 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:31.820042 27791 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.815647 27780 raft_consensus.cc:359] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.797019 27487 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:31.755393 27784 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:31.828927 27784 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.841470 27574 tablet_service.cc:1511] Processing CreateTablet for tablet 5357eb349b4b4934a5b0d36f90884fc6 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:17:31.857311 27723 tablet_service.cc:1511] Processing CreateTablet for tablet 5357eb349b4b4934a5b0d36f90884fc6 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:17:31.831847 27780 raft_consensus.cc:385] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.845839 27785 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.854197 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 5357eb349b4b4934a5b0d36f90884fc6 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:17:31.858949 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5357eb349b4b4934a5b0d36f90884fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.859997 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5357eb349b4b4934a5b0d36f90884fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.863435 27780 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.858717 27785 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.864668 27780 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.889597 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5357eb349b4b4934a5b0d36f90884fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.907943 27785 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:31.921365 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:31.922127 27630 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:31.922688 27577 tablet_service.cc:1511] Processing CreateTablet for tablet 7efb19c4fe844c2ea043bd3638faae4e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260709 12:17:31.923067 27552 leader_election.cc:343] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:31.923978 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efb19c4fe844c2ea043bd3638faae4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.924896 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 4811ee81866e469abca31f212c08ace2 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
W20260709 12:17:31.925271 27627 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:31.921409 27779 tablet_bootstrap.cc:492] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:31.926142 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4811ee81866e469abca31f212c08ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.926537 27779 ts_tablet_manager.cc:1403] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.121s	user 0.015s	sys 0.000s
I20260709 12:17:31.919709 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:31.927930 27664 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:31.929795 27779 raft_consensus.cc:359] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.930576 27779 raft_consensus.cc:385] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:31.930933 27779 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:31.938416 27778 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:31.938946 27778 ts_tablet_manager.cc:1403] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.156s	user 0.048s	sys 0.044s
I20260709 12:17:31.939659 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:31.940621 27785 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:31.941042 27785 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.941417 27785 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.942559 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 4811ee81866e469abca31f212c08ace2 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:17:31.944200 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4811ee81866e469abca31f212c08ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.944269 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 4811ee81866e469abca31f212c08ace2 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:17:31.945518 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4811ee81866e469abca31f212c08ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.946444 27487 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:31.948944 27571 tablet_service.cc:1511] Processing CreateTablet for tablet d2b187795e304ffd9c1293d922ff5b38 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:17:31.950397 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b187795e304ffd9c1293d922ff5b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.956066 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 7efb19c4fe844c2ea043bd3638faae4e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:17:31.957769 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efb19c4fe844c2ea043bd3638faae4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.972795 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 7efb19c4fe844c2ea043bd3638faae4e (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:17:31.979116 27779 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.979825 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efb19c4fe844c2ea043bd3638faae4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:31.987485 27779 ts_tablet_manager.cc:1434] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.061s	user 0.011s	sys 0.000s
I20260709 12:17:31.990198 27784 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.989921 27780 ts_tablet_manager.cc:1434] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.180s	user 0.017s	sys 0.017s
I20260709 12:17:31.995491 27784 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:31.997637 27487 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:31.996436 27785 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.000324 27784 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.002077 27780 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.002380 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.003180 27589 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:32.009859 27780 tablet_bootstrap.cc:654] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.013487 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:32.014276 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.014403 27664 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.021425 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.021497 27649 tablet_service.cc:1511] Processing CreateTablet for tablet d2b187795e304ffd9c1293d922ff5b38 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:17:32.022759 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b187795e304ffd9c1293d922ff5b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.023288 27784 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:31.980022 27778 raft_consensus.cc:359] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.034904 27778 raft_consensus.cc:385] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.035369 27778 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.042609 27779 tablet_bootstrap.cc:492] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.044234 27648 tablet_service.cc:1511] Processing CreateTablet for tablet fe68c465ca4346b9b9db89a5be4e639c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:17:32.045845 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe68c465ca4346b9b9db89a5be4e639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.050704 27728 tablet_service.cc:1511] Processing CreateTablet for tablet d2b187795e304ffd9c1293d922ff5b38 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:17:32.052621 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b187795e304ffd9c1293d922ff5b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.059760 27778 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.062943 27778 ts_tablet_manager.cc:1434] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.124s	user 0.009s	sys 0.000s
I20260709 12:17:32.069706 27778 tablet_bootstrap.cc:492] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.071604 27785 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.072837 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.073119 27664 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:32.074326 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:17:32.075516 27552 leader_election.cc:343] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.076694 27785 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.077299 27785 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:32.078488 27785 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.077349 27727 tablet_service.cc:1511] Processing CreateTablet for tablet fe68c465ca4346b9b9db89a5be4e639c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:17:32.080499 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe68c465ca4346b9b9db89a5be4e639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.085534 27573 tablet_service.cc:1511] Processing CreateTablet for tablet fd03dfdd894243c3af75d463b6913332 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:17:32.087095 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd03dfdd894243c3af75d463b6913332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.089144 27572 tablet_service.cc:1511] Processing CreateTablet for tablet fe68c465ca4346b9b9db89a5be4e639c (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:17:32.089177 27649 tablet_service.cc:1511] Processing CreateTablet for tablet fd03dfdd894243c3af75d463b6913332 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:17:32.090919 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe68c465ca4346b9b9db89a5be4e639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.092043 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd03dfdd894243c3af75d463b6913332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.102710 27830 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.103231 27830 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.116714 27784 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.117499 27830 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.119386 27723 tablet_service.cc:1511] Processing CreateTablet for tablet fd03dfdd894243c3af75d463b6913332 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:17:32.119580 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.120640 27779 tablet_bootstrap.cc:654] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:32.121075 27552 leader_election.cc:343] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.125671 27648 tablet_service.cc:1511] Processing CreateTablet for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:17:32.130458 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd03dfdd894243c3af75d463b6913332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.131073 27571 tablet_service.cc:1511] Processing CreateTablet for tablet 5865d310fddd49789e9de0a6a7bb260a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:17:32.131811 27778 tablet_bootstrap.cc:654] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.132457 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5865d310fddd49789e9de0a6a7bb260a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.126880 27648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.124526 27574 tablet_service.cc:1511] Processing CreateTablet for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:17:32.148129 27574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.149494 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:32.151273 27552 leader_election.cc:343] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.151696 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:32.154654 27834 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.164921 27649 tablet_service.cc:1511] Processing CreateTablet for tablet 5865d310fddd49789e9de0a6a7bb260a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:17:32.165423 27651 tablet_service.cc:1511] Processing CreateTablet for tablet eb215cc72c384ec8b9421c81d953589b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:17:32.166160 27649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5865d310fddd49789e9de0a6a7bb260a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.168362 27721 tablet_service.cc:1511] Processing CreateTablet for tablet eb215cc72c384ec8b9421c81d953589b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:17:32.169080 27780 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.169557 27780 ts_tablet_manager.cc:1403] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.168s	user 0.017s	sys 0.027s
I20260709 12:17:32.168362 27727 tablet_service.cc:1511] Processing CreateTablet for tablet 5865d310fddd49789e9de0a6a7bb260a (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:17:32.169972 27723 tablet_service.cc:1511] Processing CreateTablet for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:17:32.169631 27721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb215cc72c384ec8b9421c81d953589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.172751 27780 raft_consensus.cc:359] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.173447 27780 raft_consensus.cc:385] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.173635 27727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5865d310fddd49789e9de0a6a7bb260a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.173755 27780 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.174510 27780 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.176417 27650 tablet_service.cc:1511] Processing CreateTablet for tablet 4ab0ff10d53741dea50e33f971484854 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:17:32.177659 27650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab0ff10d53741dea50e33f971484854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.179256 27723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.181247 27577 tablet_service.cc:1511] Processing CreateTablet for tablet eb215cc72c384ec8b9421c81d953589b (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:17:32.181402 27572 tablet_service.cc:1511] Processing CreateTablet for tablet 4ab0ff10d53741dea50e33f971484854 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:17:32.183046 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb215cc72c384ec8b9421c81d953589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.198191 27780 ts_tablet_manager.cc:1434] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.004s
I20260709 12:17:32.204046 27779 tablet_bootstrap.cc:492] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.203267 27485 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:32.204550 27779 ts_tablet_manager.cc:1403] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.162s	user 0.023s	sys 0.017s
I20260709 12:17:32.207394 27779 raft_consensus.cc:359] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.208357 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 4ab0ff10d53741dea50e33f971484854 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:17:32.209522 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab0ff10d53741dea50e33f971484854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.207888 27780 tablet_bootstrap.cc:492] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.213653 27573 tablet_service.cc:1511] Processing CreateTablet for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:17:32.201500 27577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb215cc72c384ec8b9421c81d953589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.216152 27573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.208091 27779 raft_consensus.cc:385] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.221019 27779 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.221920 27779 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.223628 27652 tablet_service.cc:1511] Processing CreateTablet for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:17:32.225286 27652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.226931 27779 ts_tablet_manager.cc:1434] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.004s
I20260709 12:17:32.228994 27779 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.229295 27572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab0ff10d53741dea50e33f971484854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.236371 27778 tablet_bootstrap.cc:492] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.236867 27778 ts_tablet_manager.cc:1403] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.167s	user 0.014s	sys 0.006s
I20260709 12:17:32.250393 27780 tablet_bootstrap.cc:654] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.251801 27778 raft_consensus.cc:359] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.252578 27778 raft_consensus.cc:385] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.252874 27778 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.253901 27778 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.262472 27778 ts_tablet_manager.cc:1434] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.002s
I20260709 12:17:32.263537 27778 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.268163 27792 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.268728 27792 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.271143 27792 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.272964 27779 tablet_bootstrap.cc:654] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.273489 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:17:32.274778 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.275906 27587 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:32.278254 27576 tablet_service.cc:1511] Processing CreateTablet for tablet 39fef9cf211c4b858a135faa1b00e1b1 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:17:32.279659 27576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fef9cf211c4b858a135faa1b00e1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.280874 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.285413 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.292902 27651 tablet_service.cc:1511] Processing CreateTablet for tablet 39fef9cf211c4b858a135faa1b00e1b1 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:17:32.294775 27651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fef9cf211c4b858a135faa1b00e1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.295689 27780 tablet_bootstrap.cc:492] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.296173 27780 ts_tablet_manager.cc:1403] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.090s	user 0.013s	sys 0.007s
I20260709 12:17:32.296401 27778 tablet_bootstrap.cc:654] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.296515 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:32.298185 27791 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.300367 27780 raft_consensus.cc:359] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.303053 27780 raft_consensus.cc:385] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.303391 27780 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.304200 27780 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.308595 27791 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.310046 27780 ts_tablet_manager.cc:1434] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260709 12:17:32.311023 27780 tablet_bootstrap.cc:492] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.312676 27791 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:32.316465 27780 tablet_bootstrap.cc:654] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.321667 27728 tablet_service.cc:1511] Processing CreateTablet for tablet 39fef9cf211c4b858a135faa1b00e1b1 (DEFAULT_TABLE table=test-workload [id=7d627b8d55164602aecda877945cd940]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:17:32.323279 27728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fef9cf211c4b858a135faa1b00e1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:32.331348 27778 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.339651 27778 ts_tablet_manager.cc:1403] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.076s	user 0.010s	sys 0.004s
I20260709 12:17:32.341104 27792 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:32.342975 27780 tablet_bootstrap.cc:492] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.343715 27780 ts_tablet_manager.cc:1403] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.033s	user 0.022s	sys 0.007s
I20260709 12:17:32.344542 27778 raft_consensus.cc:359] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.354293 27778 raft_consensus.cc:385] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.354714 27778 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.355633 27780 raft_consensus.cc:359] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.356125 27818 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:32.359025 27778 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.358295 27779 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.365427 27780 raft_consensus.cc:385] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.365849 27780 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.366670 27780 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.367544 27778 ts_tablet_manager.cc:1434] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.001s
I20260709 12:17:32.368772 27778 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.370754 27779 ts_tablet_manager.cc:1403] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.142s	user 0.027s	sys 0.050s
I20260709 12:17:32.378213 27780 ts_tablet_manager.cc:1434] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.000s
I20260709 12:17:32.379354 27780 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.380501 27832 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.381075 27832 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.384423 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.386230 27780 tablet_bootstrap.cc:654] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:32.386392 27552 leader_election.cc:343] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.387825 27832 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.389706 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.391466 27587 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:32.391732 27552 leader_election.cc:343] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.392551 27552 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:32.394475 27832 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.391889 27784 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:17:32.401252 27778 tablet_bootstrap.cc:654] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.387840 27779 raft_consensus.cc:359] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.411698 27779 raft_consensus.cc:385] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.412096 27779 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.424350 27778 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.425021 27778 ts_tablet_manager.cc:1403] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.057s	user 0.006s	sys 0.009s
I20260709 12:17:32.425688 27818 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.426383 27818 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.429422 27779 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.431906 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.432953 27779 ts_tablet_manager.cc:1434] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260709 12:17:32.434607 27818 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.438486 27778 raft_consensus.cc:359] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.439401 27778 raft_consensus.cc:385] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.439762 27778 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.440979 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.443091 27704 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.444595 27818 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.450054 27791 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.450927 27778 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.450522 27791 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.453676 27792 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.454257 27778 ts_tablet_manager.cc:1434] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:17:32.454576 27792 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.455302 27778 tablet_bootstrap.cc:492] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.455631 27791 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.456347 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.456985 27587 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.461058 27778 tablet_bootstrap.cc:654] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.462020 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.462884 27738 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.464761 27792 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.466228 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:32.468992 27818 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.469204 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.469663 27818 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.470126 27738 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.472023 27778 tablet_bootstrap.cc:492] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.471812 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:32.472663 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.473188 27792 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.473670 27792 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.474099 27792 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.474359 27778 ts_tablet_manager.cc:1403] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.005s
I20260709 12:17:32.476177 27791 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.476588 27791 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.476994 27791 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.477066 27778 raft_consensus.cc:359] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.477766 27778 raft_consensus.cc:385] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.478044 27778 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.479030 27778 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.479966 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.480669 27664 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:32.481067 27778 ts_tablet_manager.cc:1434] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:17:32.481963 27778 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.482210 27701 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.482997 27784 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.486399 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.487560 27778 tablet_bootstrap.cc:654] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.487895 27587 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.490440 27779 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.501157 27792 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.502964 27818 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.505921 27792 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.507092 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.507694 27738 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.508105 27780 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.508657 27780 ts_tablet_manager.cc:1403] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.130s	user 0.064s	sys 0.059s
I20260709 12:17:32.509569 27737 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.511204 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.511803 27587 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.515412 27791 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.517602 27791 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.523416 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.524101 27589 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.526986 27784 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.528122 27832 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:32.536825 27589 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:32.538790 27630 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:32.540096 27791 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.540716 27791 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:32.541805 27791 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.561206 27587 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:32.563365 27780 raft_consensus.cc:359] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.564069 27780 raft_consensus.cc:385] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.564329 27780 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.567844 27779 tablet_bootstrap.cc:654] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.564965 27780 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.566552 27778 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.583803 27778 ts_tablet_manager.cc:1403] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.102s	user 0.023s	sys 0.016s
I20260709 12:17:32.583235 27780 ts_tablet_manager.cc:1434] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.074s	user 0.020s	sys 0.030s
I20260709 12:17:32.571878 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:32.585275 27792 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.585784 27792 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:32.586637 27792 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.588043 27778 raft_consensus.cc:359] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.589020 27778 raft_consensus.cc:385] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.589448 27778 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.590883 27780 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.590469 27778 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.561205 27738 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:32.593698 27778 ts_tablet_manager.cc:1434] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:17:32.595037 27778 tablet_bootstrap.cc:492] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.568187 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.600864 27736 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.606801 27778 tablet_bootstrap.cc:654] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.611362 27780 tablet_bootstrap.cc:654] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.616950 27779 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.617074 27830 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:32.612251 27736 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:32.617424 27779 ts_tablet_manager.cc:1403] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.127s	user 0.007s	sys 0.008s
I20260709 12:17:32.623018 27779 raft_consensus.cc:359] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:32.623718 27779 raft_consensus.cc:385] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.623991 27779 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.624874 27779 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:32.627691 27779 ts_tablet_manager.cc:1434] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:17:32.628707 27779 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.629792 27778 tablet_bootstrap.cc:492] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.630311 27778 ts_tablet_manager.cc:1403] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.036s	user 0.007s	sys 0.007s
I20260709 12:17:32.630330 27485 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:32.633066 27780 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.633502 27780 ts_tablet_manager.cc:1403] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.043s	user 0.015s	sys 0.000s
I20260709 12:17:32.633826 27778 raft_consensus.cc:359] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.634466 27778 raft_consensus.cc:385] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.634625 27779 tablet_bootstrap.cc:654] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.634749 27778 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.635449 27778 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.636106 27780 raft_consensus.cc:359] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.636721 27780 raft_consensus.cc:385] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.636961 27780 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.637411 27778 ts_tablet_manager.cc:1434] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:17:32.637696 27780 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.638643 27778 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.650279 27778 tablet_bootstrap.cc:654] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.652444 27780 ts_tablet_manager.cc:1434] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:17:32.653400 27780 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.659086 27780 tablet_bootstrap.cc:654] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.652143 27486 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:32.670339 27780 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.670809 27780 ts_tablet_manager.cc:1403] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.006s
I20260709 12:17:32.675194 27780 raft_consensus.cc:359] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.675880 27780 raft_consensus.cc:385] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.676143 27780 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.676640 27779 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.677102 27779 ts_tablet_manager.cc:1403] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.049s	user 0.009s	sys 0.031s
I20260709 12:17:32.676752 27780 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.678915 27780 ts_tablet_manager.cc:1434] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:17:32.680454 27778 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.680919 27778 ts_tablet_manager.cc:1403] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.043s	user 0.015s	sys 0.015s
I20260709 12:17:32.681586 27780 tablet_bootstrap.cc:492] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.683604 27779 raft_consensus.cc:359] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.684278 27779 raft_consensus.cc:385] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.684573 27779 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.685273 27779 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.686548 27778 raft_consensus.cc:359] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.687170 27778 raft_consensus.cc:385] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.687472 27778 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.687794 27780 tablet_bootstrap.cc:654] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.688624 27778 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.691239 27779 ts_tablet_manager.cc:1434] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.002s
I20260709 12:17:32.692400 27778 ts_tablet_manager.cc:1434] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:17:32.693645 27778 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.694416 27830 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.694913 27830 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.696985 27830 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.699674 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.701697 27778 tablet_bootstrap.cc:654] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.702900 27779 tablet_bootstrap.cc:492] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.703441 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.704054 27736 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:32.704641 27791 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:32.705330 27552 leader_election.cc:343] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.705101 27791 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.706455 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.707180 27791 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.708367 27828 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.708734 27828 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.709072 27828 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.708884 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.709587 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.709764 27587 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.710208 27736 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.711475 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:32.712697 27664 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.712771 27791 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.715603 27791 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.716156 27791 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.717906 27779 tablet_bootstrap.cc:654] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.715557 27778 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.724071 27778 ts_tablet_manager.cc:1403] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.031s	user 0.016s	sys 0.001s
I20260709 12:17:32.725224 27791 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.728449 27780 tablet_bootstrap.cc:492] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.727409 27778 raft_consensus.cc:359] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.728891 27780 ts_tablet_manager.cc:1403] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.048s	user 0.032s	sys 0.009s
I20260709 12:17:32.728875 27778 raft_consensus.cc:385] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.730796 27778 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.732491 27780 raft_consensus.cc:359] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.733021 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.733217 27780 raft_consensus.cc:385] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.733593 27780 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.733619 27587 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.734197 27778 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.736754 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.737308 27736 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.742025 27587 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:32.743672 27630 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:32.744679 27792 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.744824 27736 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:32.745236 27792 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:32.746236 27791 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.746142 27792 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.748780 27778 ts_tablet_manager.cc:1434] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.000s
I20260709 12:17:32.749482 27784 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.749986 27784 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.752419 27784 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.754029 27778 tablet_bootstrap.cc:492] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.754204 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.733600 27828 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:32.755918 27701 leader_election.cc:343] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.751971 27780 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.760339 27779 tablet_bootstrap.cc:492] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.760812 27779 ts_tablet_manager.cc:1403] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.058s	user 0.026s	sys 0.014s
I20260709 12:17:32.762390 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:17:32.763850 27552 leader_election.cc:343] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.764236 27780 ts_tablet_manager.cc:1434] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.000s
I20260709 12:17:32.765224 27780 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.766275 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.766863 27736 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.769184 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.769850 27587 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:32.772090 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.773284 27784 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.773663 27784 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.773988 27784 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.774847 27778 tablet_bootstrap.cc:654] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.774780 27779 raft_consensus.cc:359] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.775539 27779 raft_consensus.cc:385] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.775817 27779 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.776562 27779 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.778791 27779 ts_tablet_manager.cc:1434] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 12:17:32.779819 27866 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.780305 27866 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.783452 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.784114 27780 tablet_bootstrap.cc:654] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.784057 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.784114 27589 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:32.784832 27866 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.785629 27779 tablet_bootstrap.cc:492] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.787609 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:32.788949 27866 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.789543 27866 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.790091 27866 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.791887 27779 tablet_bootstrap.cc:654] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.795310 27784 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.796507 27738 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:32.798103 27736 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:32.800062 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.800918 27830 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.800963 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.801455 27830 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:32.801592 27589 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.802352 27830 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.807746 27828 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.808449 27589 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:32.809715 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:32.810563 27818 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.812228 27778 tablet_bootstrap.cc:492] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.812747 27778 ts_tablet_manager.cc:1403] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.059s	user 0.016s	sys 0.009s
I20260709 12:17:32.815582 27778 raft_consensus.cc:359] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.820026 27778 raft_consensus.cc:385] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.820853 27778 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.820992 27784 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.821733 27778 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:32.822360 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:32.819267 27866 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.819392 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:32.825378 27780 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.826270 27780 ts_tablet_manager.cc:1403] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.061s	user 0.008s	sys 0.015s
I20260709 12:17:32.826357 27664 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:32.828145 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.828403 27866 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.817495 27486 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:32.828893 27589 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:32.829806 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.830524 27818 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:32.831452 27818 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.833999 27780 raft_consensus.cc:359] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.834914 27780 raft_consensus.cc:385] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.835266 27780 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.836174 27780 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.839152 27780 ts_tablet_manager.cc:1434] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:17:32.841912 27778 ts_tablet_manager.cc:1434] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.015s
I20260709 12:17:32.842923 27778 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.845319 27780 tablet_bootstrap.cc:492] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.848368 27779 tablet_bootstrap.cc:492] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.848805 27779 ts_tablet_manager.cc:1403] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.063s	user 0.015s	sys 0.000s
I20260709 12:17:32.848941 27778 tablet_bootstrap.cc:654] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.851513 27779 raft_consensus.cc:359] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.852166 27779 raft_consensus.cc:385] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.852428 27779 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.852332 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:32.853097 27780 tablet_bootstrap.cc:654] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.844825 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:32.853150 27866 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:32.853103 27779 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.855419 27779 ts_tablet_manager.cc:1434] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:17:32.858753 27486 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:32.860596 27779 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.871887 27779 tablet_bootstrap.cc:654] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.875557 27780 tablet_bootstrap.cc:492] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.876183 27780 ts_tablet_manager.cc:1403] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.002s
I20260709 12:17:32.877619 27778 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.878266 27818 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.878736 27818 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.880730 27818 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.881062 27872 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.881628 27872 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.882388 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.883018 27589 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:32.885186 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.885015 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:17:32.886746 27701 leader_election.cc:343] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.887183 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:32.887710 27778 ts_tablet_manager.cc:1403] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.045s	user 0.008s	sys 0.008s
I20260709 12:17:32.888131 27589 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:32.889930 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:32.891288 27871 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.891765 27871 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.892068 27871 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.892380 27872 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.880143 27780 raft_consensus.cc:359] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.893509 27780 raft_consensus.cc:385] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.893923 27780 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.894812 27780 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.895733 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.891191 27818 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.896802 27818 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:32.897176 27701 leader_election.cc:343] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.897300 27818 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.898691 27871 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.899866 27778 raft_consensus.cc:359] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.901420 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.902218 27778 raft_consensus.cc:385] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.902304 27589 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.901865 27779 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.903249 27779 ts_tablet_manager.cc:1403] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.043s	user 0.016s	sys 0.017s
I20260709 12:17:32.905107 27818 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.902649 27778 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.906435 27778 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.908155 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:32.908766 27587 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.909073 27589 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:32.909617 27778 ts_tablet_manager.cc:1434] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.001s
I20260709 12:17:32.911446 27778 tablet_bootstrap.cc:492] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.911549 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:32.910018 27779 raft_consensus.cc:359] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.912117 27779 raft_consensus.cc:385] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.912636 27779 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.916949 27871 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.917582 27779 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.918002 27780 ts_tablet_manager.cc:1434] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.041s	user 0.009s	sys 0.000s
I20260709 12:17:32.918448 27830 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.919149 27780 tablet_bootstrap.cc:492] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:32.918955 27830 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.919970 27779 ts_tablet_manager.cc:1434] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260709 12:17:32.921185 27830 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.923496 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:32.911975 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:32.925549 27780 tablet_bootstrap.cc:654] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:32.925645 27701 leader_election.cc:343] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:32.926378 27552 leader_election.cc:343] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.927842 27779 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:32.924578 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:32.928800 27736 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:32.929785 27871 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.930130 27778 tablet_bootstrap.cc:654] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.930327 27871 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:32.930414 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.934250 27587 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:32.935515 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:32.936317 27872 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.936766 27872 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:32.937342 27830 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:32.937902 27830 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.937623 27872 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.938267 27830 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.942260 27871 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.948664 27818 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:32.948696 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:17:32.950938 27701 leader_election.cc:343] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.956843 27780 tablet_bootstrap.cc:492] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:32.957368 27780 ts_tablet_manager.cc:1403] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.001s
I20260709 12:17:32.958169 27779 tablet_bootstrap.cc:654] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.958947 27778 tablet_bootstrap.cc:492] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:32.959426 27778 ts_tablet_manager.cc:1403] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.048s	user 0.017s	sys 0.016s
I20260709 12:17:32.960177 27780 raft_consensus.cc:359] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.960896 27780 raft_consensus.cc:385] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.961161 27780 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.957368 27830 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.962062 27778 raft_consensus.cc:359] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.962666 27778 raft_consensus.cc:385] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.962947 27778 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.963627 27778 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.963991 27830 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:32.964495 27780 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.966027 27778 ts_tablet_manager.cc:1434] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:17:32.966951 27778 tablet_bootstrap.cc:492] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:32.969347 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:32.970034 27779 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:32.970503 27779 ts_tablet_manager.cc:1403] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.043s	user 0.007s	sys 0.010s
I20260709 12:17:32.971701 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:32.972347 27736 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:32.973430 27779 raft_consensus.cc:359] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:32.974012 27778 tablet_bootstrap.cc:654] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:32.974205 27779 raft_consensus.cc:385] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:32.974545 27779 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:32.975538 27779 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:32.980577 27552 leader_election.cc:343] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:32.966476 27485 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:32.986315 27779 ts_tablet_manager.cc:1434] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:17:32.986974 27485 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:32.989919 27780 ts_tablet_manager.cc:1434] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.001s
I20260709 12:17:32.989077 27736 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:32.991297 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:32.995239 27830 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:32.997123 27779 tablet_bootstrap.cc:492] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:33.001920 27830 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:33.003188 27830 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.006521 27779 tablet_bootstrap.cc:654] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.008481 27780 tablet_bootstrap.cc:492] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:33.013036 27778 tablet_bootstrap.cc:492] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:33.013962 27778 ts_tablet_manager.cc:1403] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.047s	user 0.025s	sys 0.019s
I20260709 12:17:33.018596 27778 raft_consensus.cc:359] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.020073 27778 raft_consensus.cc:385] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.020534 27778 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.018615 27780 tablet_bootstrap.cc:654] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.021435 27778 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.024454 27778 ts_tablet_manager.cc:1434] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:17:33.024906 27779 tablet_bootstrap.cc:492] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:33.025390 27779 ts_tablet_manager.cc:1403] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.008s
I20260709 12:17:33.028002 27778 tablet_bootstrap.cc:492] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:33.028350 27779 raft_consensus.cc:359] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.029096 27779 raft_consensus.cc:385] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.029372 27779 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.030076 27779 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.035727 27780 tablet_bootstrap.cc:492] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:33.036257 27780 ts_tablet_manager.cc:1403] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.006s
I20260709 12:17:33.038932 27780 raft_consensus.cc:359] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.039569 27780 raft_consensus.cc:385] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.039824 27780 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.040570 27778 tablet_bootstrap.cc:654] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.040668 27780 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.041086 27779 ts_tablet_manager.cc:1434] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.002s
I20260709 12:17:33.042831 27780 ts_tablet_manager.cc:1434] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:17:33.042309 27484 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:33.043910 27780 tablet_bootstrap.cc:492] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:33.046437 27779 tablet_bootstrap.cc:492] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:33.048357 27872 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.048882 27872 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.052353 27780 tablet_bootstrap.cc:654] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.053664 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:33.053977 27779 tablet_bootstrap.cc:654] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.054473 27664 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:33.055756 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:33.056484 27587 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:33.057667 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:33.058620 27818 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:33.058810 27872 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:33.059051 27818 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.059441 27818 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.065507 27780 tablet_bootstrap.cc:492] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:33.066114 27780 ts_tablet_manager.cc:1403] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.004s
I20260709 12:17:33.069690 27780 raft_consensus.cc:359] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.070585 27780 raft_consensus.cc:385] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.070932 27780 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.070982 27792 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:33.072149 27780 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.076006 27780 ts_tablet_manager.cc:1434] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.001s
I20260709 12:17:33.077195 27780 tablet_bootstrap.cc:492] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:33.083108 27807 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:17:33.087252 27780 tablet_bootstrap.cc:654] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.095111 27779 tablet_bootstrap.cc:492] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:33.095728 27779 ts_tablet_manager.cc:1403] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.049s	user 0.014s	sys 0.016s
I20260709 12:17:33.096143 27818 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.100270 27779 raft_consensus.cc:359] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.101279 27779 raft_consensus.cc:385] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.101603 27779 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.102964 27780 tablet_bootstrap.cc:492] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:33.102237 27779 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.103438 27780 ts_tablet_manager.cc:1403] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.027s	user 0.017s	sys 0.005s
I20260709 12:17:33.104614 27792 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.105321 27792 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.106012 27780 raft_consensus.cc:359] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.106685 27780 raft_consensus.cc:385] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.110275 27780 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.110951 27780 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.112129 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:33.113013 27780 ts_tablet_manager.cc:1434] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20260709 12:17:33.109634 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:33.113932 27780 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:33.114385 27589 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:33.109341 27779 ts_tablet_manager.cc:1434] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.002s
I20260709 12:17:33.115804 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:33.114950 27792 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.117002 27807 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.120805 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:33.121501 27664 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.126451 27780 tablet_bootstrap.cc:654] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.129021 27818 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:33.151660 27830 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.163970 27872 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.172461 27830 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.172736 27872 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.175547 27779 tablet_bootstrap.cc:492] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:33.175889 27872 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:33.176383 27778 tablet_bootstrap.cc:492] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:33.176851 27778 ts_tablet_manager.cc:1403] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.149s	user 0.040s	sys 0.048s
I20260709 12:17:33.177155 27792 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.177630 27792 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.175787 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:33.128505 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:33.180539 27778 raft_consensus.cc:359] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.181716 27778 raft_consensus.cc:385] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.182245 27778 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.181149 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:33.187676 27778 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.169193 27807 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:33.179895 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:33.192126 27830 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.194049 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:33.201900 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:33.178812 27791 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:17:33.202325 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:33.203236 27830 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:33.203649 27830 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.203992 27830 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.189942 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:33.210110 27588 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:33.215816 27663 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.196681 27792 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.202344 27736 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:33.197764 27778 ts_tablet_manager.cc:1434] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.001s
W20260709 12:17:33.199368 27701 leader_election.cc:343] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:33.220968 27778 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:33.221764 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:33.200671 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:33.223951 27738 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:33.225795 27627 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:33.227345 27792 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.189400 27589 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.158456 27664 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:33.239246 27872 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:33.243603 27872 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.243995 27872 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.256345 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:33.257299 27818 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:33.257815 27818 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:33.258708 27818 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.245694 27791 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:17:33.277454 27792 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:33.281097 27780 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:33.281605 27780 ts_tablet_manager.cc:1403] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.168s	user 0.043s	sys 0.054s
I20260709 12:17:33.290868 27779 tablet_bootstrap.cc:654] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.291283 27780 raft_consensus.cc:359] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.292135 27780 raft_consensus.cc:385] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.292390 27780 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.294912 27871 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:33.295862 27780 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.299016 27780 ts_tablet_manager.cc:1434] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.002s
I20260709 12:17:33.300261 27780 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:33.312464 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:33.315232 27828 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:33.316166 27780 tablet_bootstrap.cc:654] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.318917 27807 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:33.324795 27485 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:33.325542 27872 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.328289 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:33.327251 27872 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:33.337110 27828 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.337241 27586 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.332511 27830 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.346397 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:33.347066 27661 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.347008 27828 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.339962 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:33.357113 27779 tablet_bootstrap.cc:492] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:33.357764 27779 ts_tablet_manager.cc:1403] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.182s	user 0.012s	sys 0.016s
I20260709 12:17:33.361287 27778 tablet_bootstrap.cc:654] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.368542 27830 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.370606 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:33.371439 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:33.372045 27738 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:33.360769 27828 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.383494 27779 raft_consensus.cc:359] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.384194 27779 raft_consensus.cc:385] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.384451 27779 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.387729 27779 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:33.380450 27701 leader_election.cc:343] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:33.617455 27779 ts_tablet_manager.cc:1434] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.259s	user 0.027s	sys 0.030s
I20260709 12:17:33.618397 27779 tablet_bootstrap.cc:492] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
W20260709 12:17:33.611387 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 162155us
I20260709 12:17:33.394475 27828 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:33.622272 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 242706us
I20260709 12:17:33.396313 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:33.626452 27791 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:33.627516 27807 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.507798 27834 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.628051 27807 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.633240 27780 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:33.633858 27780 ts_tablet_manager.cc:1403] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.334s	user 0.018s	sys 0.013s
I20260709 12:17:33.630895 27807 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.522245 27738 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:33.506394 27661 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:33.622788 27828 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.455468 27871 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.177s
I20260709 12:17:33.440969 27818 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.191s
I20260709 12:17:33.438724 27872 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.194s
I20260709 12:17:33.631247 27737 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:33.662020 27780 raft_consensus.cc:359] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.662802 27780 raft_consensus.cc:385] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.663084 27780 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.664238 27780 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.638710 27834 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.667055 27780 ts_tablet_manager.cc:1434] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.002s
I20260709 12:17:33.669025 27834 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.672904 27780 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:33.673833 27828 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:17:33.674471 27865 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.204s	user 0.003s	sys 0.009s
I20260709 12:17:33.690587 27778 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:33.711580 27779 tablet_bootstrap.cc:654] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.713001 27778 ts_tablet_manager.cc:1403] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.492s	user 0.036s	sys 0.020s
I20260709 12:17:33.713688 27780 tablet_bootstrap.cc:654] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:33.716142 27778 raft_consensus.cc:359] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.717129 27778 raft_consensus.cc:385] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:33.717480 27778 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:33.718348 27778 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.772116 27778 ts_tablet_manager.cc:1434] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260709 12:17:33.773942 27903 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.774677 27903 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.776926 27779 tablet_bootstrap.cc:492] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:33.777417 27903 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:33.777580 27779 ts_tablet_manager.cc:1403] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.159s	user 0.018s	sys 0.020s
I20260709 12:17:34.474905 27912 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:34.475811 27912 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.480611 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.516965 27912 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.547896 27484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 38) took 1046 ms. Trace:
I20260709 12:17:34.548537 27484 rpcz_store.cc:276] 0709 12:17:33.501596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.501818 (+   222us) service_pool.cc:224] Handling call
0709 12:17:34.547802 (+1045984us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11199,"reported_tablets":2}
W20260709 12:17:34.553844 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:17:33.382725 27552 leader_election.cc:343] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:34.558848 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.17669e+06us
W20260709 12:17:33.617084 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132631us
I20260709 12:17:34.603055 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 22) took 1281 ms. Trace:
I20260709 12:17:34.603633 27664 rpcz_store.cc:276] 0709 12:17:33.321863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.322010 (+   147us) service_pool.cc:224] Handling call
0709 12:17:33.322331 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:33.322701 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:33.322728 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:33.323106 (+   378us) log.cc:844] Serialized 26 byte log entry
0709 12:17:33.379182 (+ 56076us) raft_consensus.cc:1645] Marking committed up to 0
0709 12:17:33.381587 (+  2405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:33.385133 (+  3546us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:34.602735 (+1217602us) raft_consensus.cc:1685] finished
0709 12:17:34.602769 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:34.602984 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":46135,"threads_started":1,"wal-append.queue_time_us":285760}
I20260709 12:17:34.604976 27913 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.605471 27913 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.606344 27911 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:34.606909 27911 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.609057 27911 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.610824 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.611238 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.612571 27914 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:17:34.612838 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.613394 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:34.613103 27914 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.615833 27913 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.616530 27914 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.620657 27779 raft_consensus.cc:359] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:33.612296 27586 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:34.621408 27779 raft_consensus.cc:385] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:34.621608 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 18) took 1293 ms. Trace:
I20260709 12:17:34.621737 27779 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:34.622007 27586 rpcz_store.cc:276] 0709 12:17:33.327910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.328050 (+   140us) service_pool.cc:224] Handling call
0709 12:17:34.621545 (+1293495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:34.622431 27779 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.623448 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.625131 27911 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:34.625909 27911 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:34.627409 27807 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.627067 27911 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.627970 27807 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.629802 27807 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.633953 27779 ts_tablet_manager.cc:1434] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.856s	user 0.008s	sys 0.005s
W20260709 12:17:34.634287 27779 ts_tablet_manager.cc:1467] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Tablet startup took 856ms
W20260709 12:17:34.634562 27779 ts_tablet_manager.cc:1469] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Trace:
0709 12:17:30.749348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.749551 (+   203us) service_pool.cc:224] Handling call
0709 12:17:30.751357 (+  1806us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:17:30.751449 (+    92us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:17:30.780612 (+ 29163us) tablet_metadata.cc:665] Metadata flushed
0709 12:17:30.846519 (+ 65907us) tablet_replica.cc:206] Creating consensus instance
0709 12:17:30.846862 (+   343us) inbound_call.cc:177] Queueing success response
0709 12:17:33.618167 (+2771305us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:17:33.778255 (+160088us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:17:34.618426 (+840171us) tablet_replica.cc:261] Starting instrumentation
0709 12:17:34.618921 (+   495us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":721,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":111,"tablet-open.queue_time_us":2771369}
I20260709 12:17:34.636795 27779 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:34.637449 27778 tablet_bootstrap.cc:492] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
W20260709 12:17:34.640108 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.112s	user 0.014s	sys 0.021s
W20260709 12:17:34.640475 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.112s	user 0.015s	sys 0.022s
I20260709 12:17:34.641708 27903 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:34.642196 27903 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.642952 27779 tablet_bootstrap.cc:654] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.643039 27911 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.643700 27911 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.644867 27903 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.646303 27911 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.650875 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.652689 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.657398 27778 tablet_bootstrap.cc:654] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.658547 27779 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:34.658967 27779 ts_tablet_manager.cc:1403] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.022s	user 0.008s	sys 0.007s
I20260709 12:17:34.660678 27918 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.661226 27918 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.661650 27779 raft_consensus.cc:359] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.662279 27779 raft_consensus.cc:385] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:34.662518 27779 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:34.663151 27779 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.664719 27779 ts_tablet_manager.cc:1434] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:17:34.664569 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.665674 27779 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
W20260709 12:17:34.665628 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.042s	user 0.013s	sys 0.041s
W20260709 12:17:34.666735 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.043s	user 0.014s	sys 0.042s
W20260709 12:17:34.667091 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.04392e+06us
I20260709 12:17:34.671766 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.672465 27779 tablet_bootstrap.cc:654] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.673240 27911 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.673190 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.673779 27911 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.674571 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.675832 27911 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.679307 27660 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.684085 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.685824 27779 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:34.683472 27918 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.686508 27779 ts_tablet_manager.cc:1403] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.000s
I20260709 12:17:34.688230 27918 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:34.690132 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.691031 27911 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:34.691452 27871 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.693222 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.694126 27911 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.695551 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.695674 27779 raft_consensus.cc:359] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.696360 27779 raft_consensus.cc:385] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:34.696501 27911 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.696683 27779 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:34.698426 27779 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.699790 27779 ts_tablet_manager.cc:1434] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:17:34.700634 27779 tablet_bootstrap.cc:492] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:34.701772 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 22) took 1004 ms. Trace:
I20260709 12:17:34.702158 27589 rpcz_store.cc:276] 0709 12:17:33.697694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.677757 (+980063us) service_pool.cc:224] Handling call
0709 12:17:34.678250 (+   493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:34.678584 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:34.678642 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:34.679282 (+   640us) log.cc:844] Serialized 26 byte log entry
0709 12:17:34.701375 (+ 22093us) raft_consensus.cc:1645] Marking committed up to 0
0709 12:17:34.701432 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:34.701514 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:34.701566 (+    52us) raft_consensus.cc:1685] finished
0709 12:17:34.701583 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:34.701714 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":538,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":1593,"thread_start_us":21633,"threads_started":1,"wal-append.queue_time_us":19954}
I20260709 12:17:34.705791 27779 tablet_bootstrap.cc:654] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.706528 27911 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:34.684223 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.708884 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.709523 27663 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:34.709837 27834 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:34.710295 27834 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.711773 27866 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.712651 27834 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.712450 27866 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.712949 27701 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.717975 27872 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.718676 27701 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:34.721804 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.721812 27866 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.727419 27918 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:34.728863 27918 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.729398 27918 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:34.729437 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:34.733593 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.107s	user 0.000s	sys 0.023s
W20260709 12:17:34.734290 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.000s	sys 0.024s
I20260709 12:17:34.739499 27784 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.740258 27784 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:34.744627 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.745018 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.746726 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.747349 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.747437 27663 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:34.748943 27784 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:34.732576 27872 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:34.748239 27589 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:34.754232 27778 tablet_bootstrap.cc:492] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:34.754689 27778 ts_tablet_manager.cc:1403] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.117s	user 0.009s	sys 0.008s
I20260709 12:17:34.762679 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:34.761551 27627 leader_election.cc:343] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:34.773021 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.773909 27663 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:34.777210 27866 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.783411 27866 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.786482 27866 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.788329 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.763717 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:17:34.792531 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9240 after lost signal to thread 26411
W20260709 12:17:34.793519 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800195d20 after lost signal to thread 27521
I20260709 12:17:34.793632 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.791394 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.797260 27778 raft_consensus.cc:359] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.797367 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:34.797978 27778 raft_consensus.cc:385] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:34.798332 27778 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:34.799191 27828 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:34.799705 27828 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:34.800628 27778 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.802155 27828 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.813972 27780 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:34.814124 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.814725 27780 ts_tablet_manager.cc:1403] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 1.142s	user 0.057s	sys 0.035s
I20260709 12:17:34.818866 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.791611 27866 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.820952 27818 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.821671 27866 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:34.822062 27818 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.822657 27779 tablet_bootstrap.cc:492] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:34.822912 27663 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
W20260709 12:17:34.823431 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800158320 after lost signal to thread 27613
I20260709 12:17:34.823310 27779 ts_tablet_manager.cc:1403] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.123s	user 0.021s	sys 0.027s
I20260709 12:17:34.824486 27893 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:17:34.825047 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080010a920 after lost signal to thread 27688
W20260709 12:17:34.825395 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800129a40 after lost signal to thread 27689
I20260709 12:17:34.825126 27893 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.827844 27778 ts_tablet_manager.cc:1434] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.073s	user 0.006s	sys 0.000s
I20260709 12:17:34.833660 27778 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:34.833513 27779 raft_consensus.cc:359] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.834110 27779 raft_consensus.cc:385] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:34.834448 27779 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
W20260709 12:17:34.832653 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080018b100 after lost signal to thread 27762
W20260709 12:17:34.835848 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080018b540 after lost signal to thread 27763
I20260709 12:17:34.835368 27779 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.832742 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.838675 27779 ts_tablet_manager.cc:1434] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:17:34.838809 27893 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.839731 27779 tablet_bootstrap.cc:492] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:34.844229 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.846483 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.849604 27586 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:34.850992 27818 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.852808 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.853536 27661 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:34.854120 27485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 36) took 1414 ms. Trace:
I20260709 12:17:34.854452 27485 rpcz_store.cc:276] 0709 12:17:33.440181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.440325 (+   144us) service_pool.cc:224] Handling call
0709 12:17:34.854063 (+1413738us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41620,"reported_tablets":2}
I20260709 12:17:34.854604 27780 raft_consensus.cc:359] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.832870 27828 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:34.850889 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:34.855742 27828 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.857749 27828 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.858780 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:34.859531 27552 leader_election.cc:343] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:34.860320 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:34.859550 27818 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:34.861466 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:34.861763 27818 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.862154 27818 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.862879 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:34.864383 27807 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.864543 27630 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:34.865996 27792 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.861307 27830 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:17:34.870498 27866 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:34.872077 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.872818 27918 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.873659 27780 raft_consensus.cc:385] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:34.874053 27780 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:34.879909 27807 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:34.880159 27818 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.873449 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.875449 27834 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:34.882143 27780 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.884120 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.885188 27780 ts_tablet_manager.cc:1434] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.001s
I20260709 12:17:34.877693 27830 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:34.887418 27780 tablet_bootstrap.cc:492] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:34.887425 27830 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.887949 27830 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.889000 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.890298 27914 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:34.890686 27914 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.890861 27701 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:34.891422 27818 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:34.892275 27818 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:34.892498 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:34.891039 27914 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.893046 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:34.894196 27913 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:34.896373 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.875524 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.902735 27736 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:17:34.903380 27893 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:34.905098 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:34.882627 27866 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.883443 27834 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.909626 27893 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.913683 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.913362 27485 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:34.894716 27818 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.918079 27834 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.921730 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.919339 27913 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:17:34.919531 27893 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:34.927174 27818 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:34.925022 27914 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.926801 27734 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:34.929915 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:34.931465 27627 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:34.918509 27866 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.933758 27918 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:17:34.934751 27552 leader_election.cc:343] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:34.938344 27779 tablet_bootstrap.cc:654] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.939978 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.929998 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:34.940706 27734 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:34.941404 27792 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.935720 27866 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.935935 27807 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:34.952396 27807 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:34.952535 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:34.952888 27807 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:34.953439 27778 tablet_bootstrap.cc:654] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.948244 27914 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:17:34.949502 27552 leader_election.cc:343] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:34.955281 27792 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.949684 27830 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.956477 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:34.959754 27893 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.961539 27934 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:34.961853 27664 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:34.962406 27627 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:34.963363 27791 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:34.963742 27791 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.964079 27791 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:34.979393 27780 tablet_bootstrap.cc:654] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:34.981654 27701 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:34.983620 27918 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:17:34.984388 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:34.988381 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:34.989189 27734 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:34.989427 27830 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.989965 27893 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.990881 27791 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.992725 27807 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.993327 27791 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.956552 27792 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:34.994731 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:34.995374 27734 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:34.995673 27792 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.996029 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:34.993542 27818 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:34.997675 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.002447 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:34.994776 27807 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:34.999236 27830 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260709 12:17:35.002961 27664 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:17:35.009622 27663 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:17:35.010164 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.053834 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:35.057320 27834 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.058391 27586 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:35.066036 27818 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:35.067755 27914 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:35.068233 27914 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.059372 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.069012 27659 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.069315 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:35.069465 27914 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.070276 27737 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:35.071050 27914 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:35.075703 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.077205 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:35.078825 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.082182 27780 tablet_bootstrap.cc:492] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:35.082157 27701 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:35.082854 27780 ts_tablet_manager.cc:1403] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.196s	user 0.030s	sys 0.008s
I20260709 12:17:35.082752 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.059260 27834 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.083464 27663 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.063508 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.083550 27818 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.096493 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.097285 27589 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:35.092473 27659 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:35.095345 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:35.098177 27733 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.096174 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.096256 27630 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.080456 27734 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:35.100835 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:35.103607 27738 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:17:35.103814 27737 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.104041 27627 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:35.101780 27792 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.105034 27791 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:35.105536 27791 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:35.105829 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:35.106380 27791 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.106956 27818 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.110266 27630 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.111804 27663 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:35.099386 27807 raft_consensus.cc:2764] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:17:35.114399 27779 tablet_bootstrap.cc:492] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:35.114809 27779 ts_tablet_manager.cc:1403] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.275s	user 0.015s	sys 0.022s
I20260709 12:17:35.117720 27818 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.119057 27818 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.122522 27733 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:35.122629 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.120513 27779 raft_consensus.cc:359] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.118331 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.124368 27779 raft_consensus.cc:385] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.126080 27818 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.116029 27893 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.128288 27779 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.131986 27780 raft_consensus.cc:359] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.130949 27914 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.133213 27780 raft_consensus.cc:385] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.134994 27779 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.135962 27780 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.129474 27737 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:35.146169 27779 ts_tablet_manager.cc:1434] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.005s
I20260709 12:17:35.148345 27779 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:35.148615 27933 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.134660 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.149153 27933 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.152550 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.154546 27933 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.156059 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:35.157333 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.159956 27914 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:17:35.159461 27627 leader_election.cc:343] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.164088 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.159060 27933 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.164641 27871 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.166105 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.156545 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:35.167521 27914 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.169509 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:35.169589 27552 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.169816 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.171512 27871 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:35.172120 27871 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.172310 27914 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.172528 27871 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:35.172691 27780 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.171725 27778 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:35.173818 27585 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:35.174052 27778 ts_tablet_manager.cc:1403] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.341s	user 0.020s	sys 0.010s
I20260709 12:17:35.176611 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.177249 27914 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.178038 27807 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:35.178514 27807 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.178619 27914 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.178063 27834 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:35.179733 27834 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.180442 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:35.181290 27934 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:35.180806 27834 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.181399 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:35.181814 27934 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.182408 27828 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:35.179790 27830 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:35.182905 27828 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.183180 27830 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.183878 27828 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.184247 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.184105 27830 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.185321 27913 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:35.185365 27830 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:35.182752 27934 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.187007 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:35.182240 27834 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:35.187927 27733 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:35.189051 27779 tablet_bootstrap.cc:654] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.190188 27733 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:17:35.178980 27807 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.192286 27780 ts_tablet_manager.cc:1434] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.109s	user 0.006s	sys 0.005s
I20260709 12:17:35.193559 27780 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:35.228868 27778 raft_consensus.cc:359] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.229692 27778 raft_consensus.cc:385] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.231611 27778 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.230189 27903 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:35.235278 27903 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:35.235690 27903 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.236658 27903 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:35.238098 27903 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:35.241591 27913 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.242251 27913 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.247762 27872 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.226980 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.251973 27663 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:35.252104 27913 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.253854 27778 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.258123 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.259368 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.260990 27778 ts_tablet_manager.cc:1434] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.087s	user 0.005s	sys 0.003s
W20260709 12:17:35.261080 27552 leader_election.cc:343] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.262300 27778 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:35.262632 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:35.266345 27552 leader_election.cc:343] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.266757 27872 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.266861 27552 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.267549 27872 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.276201 27872 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.278554 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.234068 27871 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.284160 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:35.284394 27701 leader_election.cc:343] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.284878 27589 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:35.271875 27934 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.281075 27828 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.279232 27807 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.287850 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:35.305908 27907 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:35.319308 27780 tablet_bootstrap.cc:654] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.323565 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.336958 27664 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:17:35.319988 27946 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.056s	user 0.002s	sys 0.002s
I20260709 12:17:35.331619 27871 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.342813 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.343050 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.328189 27907 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.343880 27587 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.343979 27907 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.345690 27778 tablet_bootstrap.cc:654] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.347721 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:17:35.349273 27627 leader_election.cc:343] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.349332 27484 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:35.342839 27807 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.354892 27779 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:35.355434 27779 ts_tablet_manager.cc:1403] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.207s	user 0.037s	sys 0.015s
I20260709 12:17:35.360605 27907 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.363088 27907 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.363786 27587 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:35.363508 27779 raft_consensus.cc:359] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.364408 27779 raft_consensus.cc:385] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.364758 27779 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.365020 27737 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:17:35.365626 27779 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.368044 27779 ts_tablet_manager.cc:1434] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:17:35.365049 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.369133 27779 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:35.375560 27807 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:35.376502 27807 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:35.372071 27830 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:17:35.377713 27830 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:17:35.378165 27830 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:35.377812 27807 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.379233 27830 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.381031 27830 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:35.387938 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:35.391983 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.392693 27587 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.398694 27663 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:17:35.390738 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.407639 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:35.409642 27907 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:35.410223 27907 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:35.411165 27907 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:35.415153 27701 leader_election.cc:343] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.416965 27587 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:35.424471 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:35.427145 27818 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:35.428238 27818 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:35.429244 27818 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.430747 27779 tablet_bootstrap.cc:654] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.432826 27487 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:35.448127 27487 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:35.504878 27778 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:35.505475 27778 ts_tablet_manager.cc:1403] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.243s	user 0.020s	sys 0.038s
I20260709 12:17:35.509320 27780 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:35.509199 27778 raft_consensus.cc:359] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.510120 27778 raft_consensus.cc:385] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.510139 27780 ts_tablet_manager.cc:1403] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.317s	user 0.053s	sys 0.033s
I20260709 12:17:35.510517 27778 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.511237 27778 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:35.495777 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63478.7us
I20260709 12:17:35.513676 27486 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:35.516124 27780 raft_consensus.cc:359] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.516507 27778 ts_tablet_manager.cc:1434] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:17:35.516752 27780 raft_consensus.cc:385] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.517094 27780 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.517720 27778 tablet_bootstrap.cc:492] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:35.517758 27780 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.520356 27780 ts_tablet_manager.cc:1434] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:17:35.522876 27780 tablet_bootstrap.cc:492] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:35.525493 27779 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:35.526607 27779 ts_tablet_manager.cc:1403] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.158s	user 0.024s	sys 0.013s
I20260709 12:17:35.539860 27780 tablet_bootstrap.cc:654] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.542394 27779 raft_consensus.cc:359] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.543056 27779 raft_consensus.cc:385] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.543380 27779 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.544337 27486 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:35.548686 27778 tablet_bootstrap.cc:654] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.544238 27779 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.557096 27779 ts_tablet_manager.cc:1434] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.002s
I20260709 12:17:35.558048 27779 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:35.566510 27779 tablet_bootstrap.cc:654] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.566167 27484 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:35.570254 27484 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:35.571635 27780 tablet_bootstrap.cc:492] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:35.572130 27780 ts_tablet_manager.cc:1403] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.049s	user 0.016s	sys 0.005s
I20260709 12:17:35.574877 27780 raft_consensus.cc:359] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.576769 27780 raft_consensus.cc:385] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.577059 27780 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.577612 27780 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.579695 27780 ts_tablet_manager.cc:1434] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:17:35.580629 27780 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:35.584685 27778 tablet_bootstrap.cc:492] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:35.585435 27778 ts_tablet_manager.cc:1403] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.068s	user 0.010s	sys 0.008s
I20260709 12:17:35.588910 27830 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:35.589259 27778 raft_consensus.cc:359] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.589948 27778 raft_consensus.cc:385] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.590269 27778 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.591130 27778 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.593326 27780 tablet_bootstrap.cc:654] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.593539 27778 ts_tablet_manager.cc:1434] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:17:35.594508 27778 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:35.601325 27778 tablet_bootstrap.cc:654] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.617972 27828 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:35.618716 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:35.618767 27828 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.623731 27828 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.628242 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:35.628245 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:35.630018 27552 leader_election.cc:343] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:35.631345 27552 leader_election.cc:343] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.632608 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.631961 27828 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.634301 27903 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.636336 27807 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.636809 27807 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.638913 27807 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.640196 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:35.641172 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.623744 27780 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:35.644032 27780 ts_tablet_manager.cc:1403] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.064s	user 0.012s	sys 0.008s
W20260709 12:17:35.644374 27630 leader_election.cc:343] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.647035 27780 raft_consensus.cc:359] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.647918 27780 raft_consensus.cc:385] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.648226 27780 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.653879 27779 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:35.654407 27779 ts_tablet_manager.cc:1403] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.097s	user 0.017s	sys 0.017s
I20260709 12:17:35.657368 27779 raft_consensus.cc:359] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.658121 27779 raft_consensus.cc:385] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.658454 27779 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.659633 27780 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:35.659994 27627 leader_election.cc:343] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.661870 27780 ts_tablet_manager.cc:1434] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260709 12:17:35.660671 27627 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.662858 27780 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:35.663252 27893 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.668948 27779 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.671221 27779 ts_tablet_manager.cc:1434] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260709 12:17:35.671730 27778 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:35.672284 27779 tablet_bootstrap.cc:492] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:35.680729 27778 ts_tablet_manager.cc:1403] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.086s	user 0.013s	sys 0.015s
I20260709 12:17:35.682029 27780 tablet_bootstrap.cc:654] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.685199 27585 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:17:35.685736 27585 pending_rounds.cc:85] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 0
I20260709 12:17:35.685781 27778 raft_consensus.cc:359] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.686291 27585 pending_rounds.cc:107] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:17:35.686600 27778 raft_consensus.cc:385] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.686765 27585 raft_consensus.cc:2889] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:35.686923 27778 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.687912 27778 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.688962 27893 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.689755 27778 ts_tablet_manager.cc:1434] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:17:35.690687 27778 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:35.700757 27903 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.705442 27779 tablet_bootstrap.cc:654] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.709559 27778 tablet_bootstrap.cc:654] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.713752 27733 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:35.716601 27893 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:35.723793 27903 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:35.756176 27818 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.756783 27818 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.759052 27818 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.771627 27828 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.772192 27828 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.789359 27828 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.791745 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.793468 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.794142 27664 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:35.782378 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.795773 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:35.796213 27585 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:35.797315 27893 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:17:35.797847 27552 leader_election.cc:343] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.797312 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:17:35.799772 27552 leader_election.cc:343] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:35.799933 27818 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:35.800246 27552 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:35.800345 27818 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.806329 27903 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:35.806531 27818 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.830551 27780 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:35.832085 27778 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:35.832540 27778 ts_tablet_manager.cc:1403] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.142s	user 0.014s	sys 0.014s
I20260709 12:17:35.831199 27780 ts_tablet_manager.cc:1403] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.169s	user 0.022s	sys 0.026s
I20260709 12:17:35.838584 27779 tablet_bootstrap.cc:492] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:35.839260 27779 ts_tablet_manager.cc:1403] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.167s	user 0.015s	sys 0.012s
I20260709 12:17:35.842332 27780 raft_consensus.cc:359] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.843092 27780 raft_consensus.cc:385] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.843495 27780 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.844254 27780 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.848209 27780 ts_tablet_manager.cc:1434] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 12:17:35.848474 27818 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.849334 27780 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:35.856591 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:35.857985 27589 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.862495 27778 raft_consensus.cc:359] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.863226 27778 raft_consensus.cc:385] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.863552 27778 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.867110 27779 raft_consensus.cc:359] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.867790 27779 raft_consensus.cc:385] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.868080 27779 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.870525 27818 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.876528 27778 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.878664 27778 ts_tablet_manager.cc:1434] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.000s
I20260709 12:17:35.883783 27779 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.886050 27779 ts_tablet_manager.cc:1434] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.001s
I20260709 12:17:35.896231 27780 tablet_bootstrap.cc:654] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.899260 27778 tablet_bootstrap.cc:492] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:35.905768 27589 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:35.906239 27778 tablet_bootstrap.cc:654] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.907946 27779 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:35.913353 27585 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:17:35.913741 27585 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 0
I20260709 12:17:35.913976 27585 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:17:35.914219 27585 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:35.911595 27704 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:35.916879 27953 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:35.917040 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:35.917555 27953 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:35.917707 27659 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.918493 27953 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.920308 27733 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:35.921243 27955 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:35.926426 27664 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:35.928174 27818 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:35.931589 27779 tablet_bootstrap.cc:654] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:35.932161 27659 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:35.935473 27955 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:35.938971 27955 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.939600 27955 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.943341 27780 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:35.948608 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:35.949198 27807 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:35.955892 27780 ts_tablet_manager.cc:1403] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.107s	user 0.031s	sys 0.009s
I20260709 12:17:35.955879 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.972684 27828 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.973169 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:35.973316 27828 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.954974 27955 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:35.982931 27780 raft_consensus.cc:359] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:35.984061 27780 raft_consensus.cc:385] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:35.984436 27780 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:35.987777 27828 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:35.990238 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:35.991063 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:35.991173 27664 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:35.985307 27780 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.000706 27780 ts_tablet_manager.cc:1434] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.000s
I20260709 12:17:36.002043 27780 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:36.014305 27664 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.003577 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:36.017483 27934 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.017922 27934 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:36.017951 27552 leader_election.cc:343] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.018349 27934 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.023178 27485 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:36.056975 27779 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.057615 27779 ts_tablet_manager.cc:1403] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.150s	user 0.052s	sys 0.025s
I20260709 12:17:36.056939 27818 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:36.060724 27903 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.061231 27903 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.062144 27778 tablet_bootstrap.cc:492] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:36.062618 27778 ts_tablet_manager.cc:1403] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.164s	user 0.018s	sys 0.000s
I20260709 12:17:36.063874 27903 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.066455 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.067548 27733 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:36.066885 27778 raft_consensus.cc:359] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.073477 27778 raft_consensus.cc:385] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.073755 27778 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.075361 27778 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.081092 27778 ts_tablet_manager.cc:1434] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.002s
I20260709 12:17:36.083598 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.084774 27903 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.085213 27903 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.089600 27778 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:36.085654 27903 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.090436 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:36.092188 27552 leader_election.cc:343] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.098889 27780 tablet_bootstrap.cc:654] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.102025 27807 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.102568 27807 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.108063 27807 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.108479 27828 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.108986 27828 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.110982 27828 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.112805 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:36.114687 27552 leader_election.cc:343] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.114630 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.116160 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.118312 27955 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.098752 27934 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.125578 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.133158 27934 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.139941 27779 raft_consensus.cc:359] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.140676 27779 raft_consensus.cc:385] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.140973 27779 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.142053 27627 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.143471 27807 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:17:36.143788 27903 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.146000 27903 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:17:36.147599 27552 leader_election.cc:343] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.148103 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.150478 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.150605 27779 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.151842 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:36.152503 27664 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.152980 27779 ts_tablet_manager.cc:1434] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.095s	user 0.001s	sys 0.005s
I20260709 12:17:36.157487 27903 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:36.157806 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.158483 27733 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.159191 27779 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.160028 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:36.165652 27779 tablet_bootstrap.cc:654] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.165993 27733 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:36.158495 27778 tablet_bootstrap.cc:654] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.168357 27664 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:36.170742 27780 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:36.171231 27780 ts_tablet_manager.cc:1403] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.170s	user 0.002s	sys 0.021s
I20260709 12:17:36.172680 27955 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.173344 27955 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.174094 27780 raft_consensus.cc:359] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.174813 27780 raft_consensus.cc:385] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:17:36.157531 27552 leader_election.cc:343] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:36.176394 27552 leader_election.cc:343] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.177241 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.177888 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.178467 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:36.179171 27934 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.175103 27780 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.179354 27903 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.179847 27934 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:36.179925 27903 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:36.180230 27780 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.178567 27585 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:36.180843 27903 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.180876 27934 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.184517 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.185310 27664 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:36.175945 27955 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.186867 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.188093 27955 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.188520 27955 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.188923 27955 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.218003 27778 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:36.223523 27778 ts_tablet_manager.cc:1403] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.134s	user 0.013s	sys 0.000s
I20260709 12:17:36.225942 27780 ts_tablet_manager.cc:1434] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.054s	user 0.009s	sys 0.004s
I20260709 12:17:36.227108 27779 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.227124 27780 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:36.227814 27779 ts_tablet_manager.cc:1403] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.069s	user 0.020s	sys 0.014s
I20260709 12:17:36.234997 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.236459 27778 raft_consensus.cc:359] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.237110 27778 raft_consensus.cc:385] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.237376 27778 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.237996 27778 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.241421 27779 raft_consensus.cc:359] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.242188 27779 raft_consensus.cc:385] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.242493 27779 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.247611 27955 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.249924 27955 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.251273 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.252063 27585 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.255860 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:36.256450 27780 tablet_bootstrap.cc:654] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.256583 27664 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.260661 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.262071 27779 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.264133 27779 ts_tablet_manager.cc:1434] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.036s	user 0.004s	sys 0.002s
I20260709 12:17:36.269804 27779 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.275496 27779 tablet_bootstrap.cc:654] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.281880 27664 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:36.283506 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.284976 27955 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.286664 27955 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:36.287846 27955 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.291011 27778 ts_tablet_manager.cc:1434] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.067s	user 0.007s	sys 0.000s
I20260709 12:17:36.276629 27585 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:36.297994 27780 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:36.298516 27778 tablet_bootstrap.cc:492] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:36.298516 27780 ts_tablet_manager.cc:1403] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.072s	user 0.026s	sys 0.021s
I20260709 12:17:36.298944 27779 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.299445 27779 ts_tablet_manager.cc:1403] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.005s
I20260709 12:17:36.301697 27780 raft_consensus.cc:359] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.301893 27779 raft_consensus.cc:359] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.302570 27779 raft_consensus.cc:385] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.302573 27780 raft_consensus.cc:385] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.303051 27779 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.303261 27780 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.303813 27779 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.306331 27779 ts_tablet_manager.cc:1434] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:17:36.307288 27779 tablet_bootstrap.cc:492] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.316332 27779 tablet_bootstrap.cc:654] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.304164 27780 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.327145 27778 tablet_bootstrap.cc:654] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.327734 27780 ts_tablet_manager.cc:1434] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.002s
I20260709 12:17:36.328907 27780 tablet_bootstrap.cc:492] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:36.331873 27955 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.332636 27955 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.335104 27780 tablet_bootstrap.cc:654] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.337285 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:36.338766 27704 leader_election.cc:343] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.331161 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:36.343467 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:36.345552 27701 leader_election.cc:343] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.346024 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:36.346822 27984 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:36.352357 27955 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.382893 27974 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:36.383780 27974 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.387987 27974 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.387917 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.388870 27733 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:36.388794 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.390080 27585 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:36.391597 27630 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.393081 27974 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:36.393262 27972 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.393476 27974 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:36.393841 27974 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:36.393750 27972 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.396009 27972 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.399936 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:36.401857 27627 leader_election.cc:343] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.402400 27974 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.406040 27779 tablet_bootstrap.cc:492] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.406576 27779 ts_tablet_manager.cc:1403] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.099s	user 0.026s	sys 0.042s
I20260709 12:17:36.406250 27974 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.409436 27779 raft_consensus.cc:359] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.410185 27779 raft_consensus.cc:385] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.410557 27779 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.410218 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.411283 27779 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.412683 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.413625 27779 ts_tablet_manager.cc:1434] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:17:36.414978 27779 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.421584 27779 tablet_bootstrap.cc:654] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.411916 27733 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:36.425787 27955 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.431958 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.432679 27585 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:36.433800 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.436349 27807 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.436794 27807 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.437177 27807 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.437448 27583 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:36.438027 27779 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.438520 27779 ts_tablet_manager.cc:1403] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.024s	user 0.021s	sys 0.000s
I20260709 12:17:36.438640 27630 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.440668 27972 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:36.441704 27972 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:36.442610 27972 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.452934 27779 raft_consensus.cc:359] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.453883 27779 raft_consensus.cc:385] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.454317 27779 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.449664 27807 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.455139 27778 tablet_bootstrap.cc:492] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:36.455673 27778 ts_tablet_manager.cc:1403] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.159s	user 0.014s	sys 0.000s
I20260709 12:17:36.455222 27779 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.457064 27807 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.457787 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.457815 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.458258 27583 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.459438 27779 ts_tablet_manager.cc:1434] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.021s	user 0.010s	sys 0.000s
I20260709 12:17:36.460474 27779 tablet_bootstrap.cc:492] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.463716 27778 raft_consensus.cc:359] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.464339 27778 raft_consensus.cc:385] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.464589 27778 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.465366 27583 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:36.465205 27778 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.466078 27733 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.466646 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:36.467571 27893 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.468004 27893 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:36.468816 27893 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.474898 27733 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:36.480258 27779 tablet_bootstrap.cc:654] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.494031 27778 ts_tablet_manager.cc:1434] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 12:17:36.495867 27893 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.496366 27893 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.498313 27893 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.500041 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:36.501428 27630 leader_election.cc:343] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.503211 27955 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:36.503746 27955 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.503484 27486 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.505143 27486 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.508342 27780 tablet_bootstrap.cc:492] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:36.514745 27780 ts_tablet_manager.cc:1403] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.186s	user 0.035s	sys 0.038s
I20260709 12:17:36.506765 27955 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.514129 27778 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:36.514421 27984 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:36.509127 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.520526 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:36.522153 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.523492 27984 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:36.523849 27984 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:36.524207 27984 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:36.524421 27779 tablet_bootstrap.cc:492] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.524857 27779 ts_tablet_manager.cc:1403] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.065s	user 0.013s	sys 0.002s
I20260709 12:17:36.531997 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.532809 27664 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:36.533625 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.534264 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:36.549588 27934 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.550334 27934 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.551923 27907 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.552562 27907 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.558238 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:36.558892 27934 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.559283 28002 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.559720 28002 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.560066 28002 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.561832 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.562994 27779 raft_consensus.cc:359] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:36.563395 27552 leader_election.cc:343] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.563723 27779 raft_consensus.cc:385] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.563787 27907 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.564038 27779 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.563971 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.564800 27779 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.566664 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.565079 27663 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:36.567251 27587 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:36.567675 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.568665 28002 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.569936 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.571455 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.571847 27953 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.575850 27953 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:36.576702 27630 leader_election.cc:343] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.578342 27955 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:17:36.578689 27955 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.565773 27780 raft_consensus.cc:359] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.579037 27955 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:36.579360 27780 raft_consensus.cc:385] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.579231 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.579759 27780 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.580892 27780 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.581509 27953 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.584957 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.585708 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:36.586653 27701 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:36.587890 27955 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.593940 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.594739 27587 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:36.595898 27779 ts_tablet_manager.cc:1434] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.071s	user 0.010s	sys 0.004s
I20260709 12:17:36.568854 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.583042 27780 ts_tablet_manager.cc:1434] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.068s	user 0.006s	sys 0.001s
I20260709 12:17:36.583691 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.596882 27779 tablet_bootstrap.cc:492] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.596698 27733 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.596609 27984 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.598078 27934 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.598531 27934 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.598784 27984 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.598943 27934 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.595275 27953 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.603518 27955 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.603724 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:36.606714 27780 tablet_bootstrap.cc:492] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:36.606684 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:36.608552 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:36.579202 27778 tablet_bootstrap.cc:654] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.613837 27780 tablet_bootstrap.cc:654] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.616102 27807 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.616626 27807 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:36.617638 27807 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.626142 27903 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.626754 27903 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.629384 27903 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.630307 27807 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.630795 27807 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.632596 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.633790 27807 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.633817 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.634519 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:36.635007 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.635478 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.635813 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:17:36.636857 27552 leader_election.cc:343] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.637203 27807 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.637598 27807 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.637956 27807 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:36.637794 27552 leader_election.cc:343] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.638569 27552 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.639495 27903 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:36.572856 28002 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:17:36.642235 27627 leader_election.cc:343] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.644508 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.651014 27780 tablet_bootstrap.cc:492] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:36.651521 27780 ts_tablet_manager.cc:1403] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.045s	user 0.013s	sys 0.000s
I20260709 12:17:36.663691 27953 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:36.663826 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:36.673039 27934 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.680653 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.683012 27934 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.667287 27587 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:17:36.685146 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:36.685739 27664 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.686388 28005 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:17:36.688555 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:17:36.673982 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78824.1us
I20260709 12:17:36.691825 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.692414 28005 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:36.692457 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:17:36.693326 27552 leader_election.cc:343] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.693989 27663 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:36.694834 27934 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.695202 28005 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.697031 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.698629 27778 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:36.699088 27778 ts_tablet_manager.cc:1403] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.185s	user 0.003s	sys 0.014s
I20260709 12:17:36.699745 27953 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:36.700275 27953 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:36.701618 27486 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.703806 27780 raft_consensus.cc:359] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.704308 27780 raft_consensus.cc:385] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.704530 27780 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.705423 28002 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.705416 27780 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.705756 27953 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.714789 27780 ts_tablet_manager.cc:1434] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.063s	user 0.007s	sys 0.000s
I20260709 12:17:36.711437 28002 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.732479 27780 tablet_bootstrap.cc:492] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:36.722831 27807 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.741204 27807 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.742357 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:36.743286 27972 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:36.743635 27903 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.743937 27972 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:36.746860 27585 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.747501 28002 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.748332 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.751556 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:36.751787 27779 tablet_bootstrap.cc:654] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.758456 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:36.758389 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.760169 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.762229 27972 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:36.764269 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.764958 27583 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:36.766307 27630 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:36.766842 27778 raft_consensus.cc:359] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.767473 27972 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:36.767678 27778 raft_consensus.cc:385] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.767932 27972 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:17:36.768177 27778 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.768426 27972 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:36.769134 27778 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.771869 27659 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.772034 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.774232 27934 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.769833 27627 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.759137 27780 tablet_bootstrap.cc:654] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.774335 27778 ts_tablet_manager.cc:1434] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.071s	user 0.006s	sys 0.000s
I20260709 12:17:36.777680 27778 tablet_bootstrap.cc:492] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:36.785626 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.786618 27737 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:36.788020 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:36.788774 27972 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:36.790794 27972 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:36.775288 27934 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:36.791383 27486 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:36.792483 27934 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.791404 27972 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:36.798725 27972 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:36.799289 27984 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.799834 27984 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.799716 27972 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.807898 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:17:36.807205 27627 leader_election.cc:343] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.838297 27984 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.839118 27893 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:36.839102 27955 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.842170 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:36.846159 27955 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:36.847867 27733 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:36.848686 27955 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:36.855253 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.858600 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.859181 27664 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
W20260709 12:17:36.860495 27701 leader_election.cc:343] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:36.861153 27779 tablet_bootstrap.cc:492] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:36.861603 27779 ts_tablet_manager.cc:1403] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.265s	user 0.015s	sys 0.021s
I20260709 12:17:36.861533 27701 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:36.864243 27485 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:36.881081 27585 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:36.883080 27630 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:36.884347 27893 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:36.885169 27778 tablet_bootstrap.cc:654] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:36.885039 27893 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:17:36.887135 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73220.6us
I20260709 12:17:36.876549 28006 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:17:36.888870 28006 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.887789 27486 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.889356 28006 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:36.889583 27955 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.862445 28002 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.894477 28002 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.886379 27893 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:36.896739 27903 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.897989 27903 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:36.901115 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:36.902012 27585 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:36.909529 27780 tablet_bootstrap.cc:492] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:36.910339 27780 ts_tablet_manager.cc:1403] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.178s	user 0.015s	sys 0.003s
I20260709 12:17:36.916596 27733 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:17:36.917181 27779 raft_consensus.cc:359] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.917923 27779 raft_consensus.cc:385] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.918221 27779 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.919139 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:36.919521 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:36.918915 27779 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.922464 27974 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:36.928375 27779 ts_tablet_manager.cc:1434] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.065s	user 0.007s	sys 0.000s
I20260709 12:17:36.933179 27779 tablet_bootstrap.cc:492] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:36.925293 28018 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.927506 27974 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.939731 28018 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.942775 27984 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.939769 27974 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:36.943257 27984 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.930522 27807 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:17:36.941923 28018 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:36.950183 27984 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:36.955473 28006 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:36.971395 28006 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:36.984990 27972 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:36.985827 27972 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:17:36.987263 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:36.988147 27658 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:36.989220 28026 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:36.989648 28026 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:36.991587 27780 raft_consensus.cc:359] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:36.996531 27780 raft_consensus.cc:385] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:36.997156 27780 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:36.998037 27780 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.004032 27780 ts_tablet_manager.cc:1434] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.093s	user 0.009s	sys 0.000s
I20260709 12:17:37.005311 27780 tablet_bootstrap.cc:492] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:37.005407 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.006232 27585 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:37.010998 28002 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.013229 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:37.017681 27552 leader_election.cc:343] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:37.018497 28021 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:17:37.020361 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.021140 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.021905 27903 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:37.022351 27903 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.022778 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.023134 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:37.026724 27587 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:37.021930 27585 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:37.022697 27903 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.030530 27484 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.991753 28026 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.039475 27984 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:37.050789 27778 tablet_bootstrap.cc:492] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:37.051437 27778 ts_tablet_manager.cc:1403] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.274s	user 0.013s	sys 0.008s
I20260709 12:17:37.054564 27704 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:37.056356 27984 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:37.056742 27984 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.057101 27984 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.058133 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:37.061803 27779 tablet_bootstrap.cc:654] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:37.054203 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115960us
W20260709 12:17:37.069227 27627 leader_election.cc:343] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.049551 27974 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.073552 27780 tablet_bootstrap.cc:654] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.066731 28006 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:37.089700 27484 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:37.098006 27779 tablet_bootstrap.cc:492] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:37.098696 27779 ts_tablet_manager.cc:1403] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.170s	user 0.017s	sys 0.006s
I20260709 12:17:37.100855 27778 raft_consensus.cc:359] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.104646 27778 raft_consensus.cc:385] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.105039 27778 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.094714 28026 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:37.108374 27778 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.112789 27778 ts_tablet_manager.cc:1434] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.061s	user 0.006s	sys 0.001s
I20260709 12:17:37.090863 28006 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.118432 27974 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.119444 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:37.103876 28024 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.123921 27779 raft_consensus.cc:359] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.124752 27779 raft_consensus.cc:385] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.125013 27779 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.126417 27778 tablet_bootstrap.cc:492] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:37.119109 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:37.120664 28006 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:37.104269 27903 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.125625 27779 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.134567 27663 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:37.128542 28024 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.139972 27780 tablet_bootstrap.cc:492] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:37.140482 27780 ts_tablet_manager.cc:1403] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.135s	user 0.023s	sys 0.031s
I20260709 12:17:37.141491 27779 ts_tablet_manager.cc:1434] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.002s
I20260709 12:17:37.142285 27779 tablet_bootstrap.cc:492] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:37.120208 28030 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260709 12:17:37.145031 28024 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.151557 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:37.152235 27583 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.156574 27903 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.157956 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:37.158317 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:37.159089 27733 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:37.158535 27664 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.159664 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:37.160270 27737 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.139616 27984 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.164767 27701 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:37.166378 27953 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:17:37.168614 27583 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:37.169876 27664 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:37.170077 27584 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:37.179278 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:17:37.183550 27627 leader_election.cc:343] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.182977 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:37.192551 27934 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:37.193112 27934 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:37.193989 27934 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.197799 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.197666 27780 raft_consensus.cc:359] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.199365 27972 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:37.199870 27974 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.200448 27974 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.201689 27953 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:37.202720 27974 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.201673 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:37.204336 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:37.199882 27972 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:37.205024 27585 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.205771 27972 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.198390 27780 raft_consensus.cc:385] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.204445 27737 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:37.216712 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.216712 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:37.218940 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.219791 27903 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.220099 28024 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.208479 27780 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.223523 27780 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.227434 27780 ts_tablet_manager.cc:1434] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.087s	user 0.005s	sys 0.003s
I20260709 12:17:37.228327 27778 tablet_bootstrap.cc:654] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.231302 27984 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:37.235152 27659 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:17:37.235600 27659 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 1
I20260709 12:17:37.235838 27659 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:17:37.236095 27659 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:37.239792 27780 tablet_bootstrap.cc:492] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:37.229075 28006 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.255770 27779 tablet_bootstrap.cc:654] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.263075 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:37.263769 27587 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:37.266309 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:37.266883 27664 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:37.245296 27630 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.268529 27485 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:37.274451 27664 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:37.275028 28026 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:17:37.275296 27907 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:37.239290 27893 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:17:37.296700 27587 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:17:37.302313 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:37.304180 27955 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:37.304821 27955 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:37.299672 27585 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
W20260709 12:17:37.339380 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 100130us
I20260709 12:17:37.339846 27704 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:37.283057 28024 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:37.250159 28006 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:37.345889 28006 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:37.306804 27955 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:37.310200 27552 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:17:37.352265 28035 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.009s	sys 0.000s
I20260709 12:17:37.352532 27663 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:37.352918 28006 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:37.352738 28026 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.353096 28030 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.340493 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:37.353847 28006 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.363628 27585 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:17:37.365728 27627 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:17:37.366418 27701 leader_election.cc:343] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.366623 27778 tablet_bootstrap.cc:492] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:37.367875 27778 ts_tablet_manager.cc:1403] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.242s	user 0.015s	sys 0.004s
I20260709 12:17:37.368093 28005 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.343169 27780 tablet_bootstrap.cc:654] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.375398 27778 raft_consensus.cc:359] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.375851 27779 tablet_bootstrap.cc:492] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:37.376087 27778 raft_consensus.cc:385] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.376386 27778 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.377175 27778 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.377988 27585 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:37.338299 27486 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:37.385937 27779 ts_tablet_manager.cc:1403] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.244s	user 0.015s	sys 0.007s
I20260709 12:17:37.407598 27778 ts_tablet_manager.cc:1434] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 12:17:37.411577 27778 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:37.415514 28030 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.421629 28030 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.430891 28018 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.431564 28018 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.406760 27972 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260709 12:17:37.466486 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:37.433787 28018 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.467461 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:37.470635 28005 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:37.516820 27779 raft_consensus.cc:359] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.517582 27779 raft_consensus.cc:385] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.517876 27779 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.518518 27779 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.520402 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:37.521559 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:37.523079 28018 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:37.533056 27779 ts_tablet_manager.cc:1434] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.147s	user 0.010s	sys 0.007s
I20260709 12:17:37.534073 27779 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:37.525754 28018 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.537159 28018 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:37.546073 27552 leader_election.cc:343] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.547456 28005 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.548554 28005 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.549783 28030 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.550570 28005 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:37.553340 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:37.555574 27778 tablet_bootstrap.cc:654] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:37.558387 27627 leader_election.cc:343] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.553344 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:37.583570 27738 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:17:37.585866 27893 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:37.586090 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.586812 27972 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:17:37.587007 27585 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:37.587177 27972 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:37.588089 27972 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:37.589649 27972 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:37.602523 27780 tablet_bootstrap.cc:492] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:37.604141 27780 ts_tablet_manager.cc:1403] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.365s	user 0.018s	sys 0.017s
I20260709 12:17:37.623878 27780 raft_consensus.cc:359] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.631647 27780 raft_consensus.cc:385] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.633247 27780 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.634550 27780 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.658788 27780 ts_tablet_manager.cc:1434] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.054s	user 0.009s	sys 0.000s
I20260709 12:17:37.660531 27780 tablet_bootstrap.cc:492] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:37.666483 27779 tablet_bootstrap.cc:654] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.676050 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:37.676347 27485 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 2, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:37.678748 27907 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.689890 27934 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:17:37.694296 27778 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:37.694886 27778 ts_tablet_manager.cc:1403] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.286s	user 0.027s	sys 0.024s
I20260709 12:17:37.697122 28018 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.700273 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:37.700846 27663 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.707645 28018 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.708608 27907 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:37.709184 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:17:37.711251 27552 leader_election.cc:343] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.711968 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:37.714378 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.716975 27780 tablet_bootstrap.cc:654] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.717563 28018 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:37.718202 28018 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:37.719363 28018 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.763904 27934 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.764410 27934 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.766484 27934 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.768131 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:37.770049 27552 leader_election.cc:343] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.770743 27893 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.771679 27893 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.770776 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:37.773451 27552 leader_election.cc:343] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.774995 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.775063 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.775797 27934 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:37.774566 27893 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:37.785722 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:37.786548 28024 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:17:37.790795 27627 leader_election.cc:343] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.776680 27778 raft_consensus.cc:359] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.795029 27778 raft_consensus.cc:385] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.795380 27778 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.796137 27778 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.799857 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.800844 28030 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:37.803220 27779 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:37.803679 27779 ts_tablet_manager.cc:1403] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.270s	user 0.032s	sys 0.036s
I20260709 12:17:37.803828 27780 tablet_bootstrap.cc:492] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:37.804292 27780 ts_tablet_manager.cc:1403] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.144s	user 0.012s	sys 0.000s
I20260709 12:17:37.805056 27778 ts_tablet_manager.cc:1434] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.110s	user 0.010s	sys 0.000s
I20260709 12:17:37.806109 27778 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:37.806273 27779 raft_consensus.cc:359] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.807010 27779 raft_consensus.cc:385] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.806766 27780 raft_consensus.cc:359] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.807299 27779 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.816093 27779 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.807482 27780 raft_consensus.cc:385] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.818559 27780 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:37.813117 27778 tablet_bootstrap.cc:654] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.814859 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:37.819228 27780 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.818228 27779 ts_tablet_manager.cc:1434] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.014s	user 0.001s	sys 0.003s
I20260709 12:17:37.824488 27779 tablet_bootstrap.cc:492] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:37.825911 27662 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:37.825984 27953 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.826823 27780 ts_tablet_manager.cc:1434] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:17:37.827795 27780 tablet_bootstrap.cc:492] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:37.813627 27485 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:37.835932 27780 tablet_bootstrap.cc:654] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.836215 27953 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.846081 27779 tablet_bootstrap.cc:654] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:37.861615 28030 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:17:37.866755 27630 leader_election.cc:343] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:37.868484 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:37.871201 28030 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:37.874523 27974 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.922060 27485 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:37.925047 27659 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:37.929419 27953 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:37.965204 27778 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:37.965679 27778 ts_tablet_manager.cc:1403] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.160s	user 0.009s	sys 0.040s
I20260709 12:17:37.974536 27780 tablet_bootstrap.cc:492] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:37.975761 27780 ts_tablet_manager.cc:1403] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.148s	user 0.021s	sys 0.038s
I20260709 12:17:37.985857 27778 raft_consensus.cc:359] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:37.986613 27778 raft_consensus.cc:385] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:37.995208 27583 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:37.996984 27955 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:37.986917 27778 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.007881 27778 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.016242 27778 ts_tablet_manager.cc:1434] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.050s	user 0.005s	sys 0.001s
I20260709 12:17:38.017542 27778 tablet_bootstrap.cc:492] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.036413 27779 tablet_bootstrap.cc:492] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:38.037185 27779 ts_tablet_manager.cc:1403] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.213s	user 0.034s	sys 0.017s
I20260709 12:17:38.038208 27780 raft_consensus.cc:359] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.038901 27780 raft_consensus.cc:385] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.039160 27780 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.039824 27780 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.044230 27778 tablet_bootstrap.cc:654] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.063670 27779 raft_consensus.cc:359] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.064505 27779 raft_consensus.cc:385] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.064790 27779 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.065343 27780 ts_tablet_manager.cc:1434] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.089s	user 0.004s	sys 0.002s
I20260709 12:17:38.065467 27779 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.066329 27780 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:38.067687 27779 ts_tablet_manager.cc:1434] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.000s
I20260709 12:17:38.068742 27779 tablet_bootstrap.cc:492] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
W20260709 12:17:38.074425 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:38.079706 27778 tablet_bootstrap.cc:492] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:38.080281 27778 ts_tablet_manager.cc:1403] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.063s	user 0.012s	sys 0.002s
I20260709 12:17:38.080622 27953 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.085405 27953 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.087704 27953 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:38.088948 27780 tablet_bootstrap.cc:654] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.089475 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.090610 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.092316 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:38.093456 27953 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:38.095043 27779 tablet_bootstrap.cc:654] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.092414 27778 raft_consensus.cc:359] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.097867 27780 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:38.098333 27780 ts_tablet_manager.cc:1403] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.032s	user 0.006s	sys 0.005s
I20260709 12:17:38.097517 27778 raft_consensus.cc:385] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.098883 27778 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.099924 27778 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.102901 27778 ts_tablet_manager.cc:1434] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I20260709 12:17:38.104700 27778 tablet_bootstrap.cc:492] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.106083 27780 raft_consensus.cc:359] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.106830 27780 raft_consensus.cc:385] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.107106 27780 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.107822 27780 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.111665 27778 tablet_bootstrap.cc:654] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.120788 27780 ts_tablet_manager.cc:1434] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:17:38.122958 27778 tablet_bootstrap.cc:492] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:38.123457 27778 ts_tablet_manager.cc:1403] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.011s
I20260709 12:17:38.126143 27779 tablet_bootstrap.cc:492] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:38.126681 27779 ts_tablet_manager.cc:1403] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.058s	user 0.007s	sys 0.011s
I20260709 12:17:38.129413 27779 raft_consensus.cc:359] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.130219 27779 raft_consensus.cc:385] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.130561 27779 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.131446 27780 tablet_bootstrap.cc:492] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:38.133472 27779 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.134135 28018 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:38.135769 27779 ts_tablet_manager.cc:1434] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:17:38.136719 27779 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:38.138229 27780 tablet_bootstrap.cc:654] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.126606 27733 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.141403 27778 raft_consensus.cc:359] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.142146 27778 raft_consensus.cc:385] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.142406 27778 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.142995 27778 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.148620 27778 ts_tablet_manager.cc:1434] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20260709 12:17:38.149559 27778 tablet_bootstrap.cc:492] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.152436 27779 tablet_bootstrap.cc:654] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.157788 28018 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.158308 28018 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.160374 28018 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.164570 27778 tablet_bootstrap.cc:654] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.168421 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.169816 27738 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.174062 27779 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
W20260709 12:17:38.174640 27552 leader_election.cc:343] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.174724 27779 ts_tablet_manager.cc:1403] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.001s
I20260709 12:17:38.175874 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.176699 27659 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:38.179842 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.182371 28018 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.187956 28018 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.191546 28018 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.192727 27779 raft_consensus.cc:359] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.193504 27779 raft_consensus.cc:385] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.193801 27779 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.194298 27934 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:38.194486 27779 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.195627 27778 tablet_bootstrap.cc:492] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:38.196081 27778 ts_tablet_manager.cc:1403] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.047s	user 0.009s	sys 0.008s
I20260709 12:17:38.196601 27779 ts_tablet_manager.cc:1434] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260709 12:17:38.197568 27779 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:38.198750 27778 raft_consensus.cc:359] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.199470 27778 raft_consensus.cc:385] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.199747 27778 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.199936 27780 tablet_bootstrap.cc:492] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:38.200428 27780 ts_tablet_manager.cc:1403] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.069s	user 0.014s	sys 0.011s
I20260709 12:17:38.200480 27778 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.202656 27778 ts_tablet_manager.cc:1434] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.004s
I20260709 12:17:38.203480 27778 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.203572 27780 raft_consensus.cc:359] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.204276 27780 raft_consensus.cc:385] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.204550 27780 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.208652 27780 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.210311 27778 tablet_bootstrap.cc:654] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.210690 27780 ts_tablet_manager.cc:1434] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.002s
I20260709 12:17:38.212015 27780 tablet_bootstrap.cc:492] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:38.224102 28018 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.226207 28018 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.228992 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:38.230355 27780 tablet_bootstrap.cc:654] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.230885 27779 tablet_bootstrap.cc:654] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:38.231354 27552 leader_election.cc:343] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.232400 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:38.233261 27659 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.242904 27778 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:38.243444 27778 ts_tablet_manager.cc:1403] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.005s
I20260709 12:17:38.246713 27778 raft_consensus.cc:359] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.254581 27779 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:38.255195 27779 ts_tablet_manager.cc:1403] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.058s	user 0.017s	sys 0.003s
I20260709 12:17:38.255230 27780 tablet_bootstrap.cc:492] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:38.256184 27780 ts_tablet_manager.cc:1403] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.045s	user 0.009s	sys 0.003s
I20260709 12:17:38.257700 27779 raft_consensus.cc:359] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.258419 27779 raft_consensus.cc:385] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.258695 27779 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.259377 27779 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.260517 28030 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.260998 28030 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.261462 27779 ts_tablet_manager.cc:1434] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:17:38.262413 27779 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:38.264031 27778 raft_consensus.cc:385] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.264340 27778 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.264982 27778 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.267004 27778 ts_tablet_manager.cc:1434] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.023s	user 0.004s	sys 0.000s
I20260709 12:17:38.267880 27778 tablet_bootstrap.cc:492] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.273087 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.273785 27779 tablet_bootstrap.cc:654] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:38.274885 27630 leader_election.cc:343] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.245075 27659 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:38.277371 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.278335 28018 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:38.278790 28018 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:38.279074 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.280005 27733 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:38.283174 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:38.284210 27974 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.284576 27974 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.284937 27974 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.292591 27778 tablet_bootstrap.cc:654] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.263104 28030 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.301204 27738 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.279728 28018 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.332593 27779 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:38.333211 27779 ts_tablet_manager.cc:1403] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.071s	user 0.008s	sys 0.003s
I20260709 12:17:38.344251 27780 raft_consensus.cc:359] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.347136 27779 raft_consensus.cc:359] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.339597 27974 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.352543 27779 raft_consensus.cc:385] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.352916 27779 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.354012 27780 raft_consensus.cc:385] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.353581 27779 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.357136 27485 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:38.358187 28018 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.366825 28018 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.372484 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.373898 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:38.374998 27552 leader_election.cc:343] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.362063 27778 tablet_bootstrap.cc:492] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
W20260709 12:17:38.376004 27552 leader_election.cc:343] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.376374 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.376533 27778 ts_tablet_manager.cc:1403] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.109s	user 0.033s	sys 0.013s
I20260709 12:17:38.362769 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:38.377812 27934 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:17:38.378189 27630 leader_election.cc:343] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.378499 28018 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.378927 27779 ts_tablet_manager.cc:1434] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.045s	user 0.008s	sys 0.000s
I20260709 12:17:38.379274 27778 raft_consensus.cc:359] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.379899 27778 raft_consensus.cc:385] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.380067 27778 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.380592 27778 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.382148 27778 ts_tablet_manager.cc:1434] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.003s
I20260709 12:17:38.382990 27778 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.389587 27778 tablet_bootstrap.cc:654] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.390115 27779 tablet_bootstrap.cc:492] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:38.408252 27778 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:38.408649 27778 ts_tablet_manager.cc:1403] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.005s
I20260709 12:17:38.410558 27778 raft_consensus.cc:359] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.411226 27778 raft_consensus.cc:385] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.411305 27780 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.411568 27778 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.412362 27778 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.414889 27974 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.415621 27780 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.416572 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:38.417227 27738 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.422178 27780 ts_tablet_manager.cc:1434] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.166s	user 0.006s	sys 0.001s
I20260709 12:17:38.423219 27780 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:38.424394 27778 ts_tablet_manager.cc:1434] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:17:38.425364 27778 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.431104 27779 tablet_bootstrap.cc:654] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.431636 27738 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:38.434239 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:38.434681 27780 tablet_bootstrap.cc:654] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.436460 28030 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:38.436926 28030 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:38.437762 28030 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.466849 27778 tablet_bootstrap.cc:654] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.515686 27779 tablet_bootstrap.cc:492] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:38.517313 27779 ts_tablet_manager.cc:1403] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.137s	user 0.009s	sys 0.004s
I20260709 12:17:38.520627 27934 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.522089 27934 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.534195 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.536540 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.539776 27974 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.540380 27780 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
W20260709 12:17:38.540318 27552 leader_election.cc:343] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:38.551810 27552 leader_election.cc:343] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.556561 27778 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:38.560742 27778 ts_tablet_manager.cc:1403] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.136s	user 0.015s	sys 0.002s
I20260709 12:17:38.538074 27934 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.552995 27552 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.564666 27780 ts_tablet_manager.cc:1403] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.142s	user 0.030s	sys 0.012s
I20260709 12:17:38.565038 27953 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.567363 28018 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.571674 28018 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.574672 28018 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.576351 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.567251 27953 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.546741 27974 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.566427 27934 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:38.579952 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.581102 27779 raft_consensus.cc:359] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.581804 27974 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.581938 27779 raft_consensus.cc:385] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.582281 27779 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.582667 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.583074 27779 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.583500 27585 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:38.584162 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.584553 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.585347 27583 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:38.585716 27779 ts_tablet_manager.cc:1434] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.068s	user 0.009s	sys 0.003s
I20260709 12:17:38.586148 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:38.587273 27779 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:38.587203 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:38.588012 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:38.588541 27907 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.588857 27974 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.588918 27907 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:38.589232 27552 leader_election.cc:343] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.589363 27974 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.589458 27907 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.589907 27974 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:38.590250 27552 leader_election.cc:343] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.590804 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.593058 27934 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:38.588224 27778 raft_consensus.cc:359] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.594990 27778 raft_consensus.cc:385] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.595494 27778 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.603588 27484 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:38.609390 27778 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.642177 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.645890 27953 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:38.656551 27780 raft_consensus.cc:359] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.657780 27780 raft_consensus.cc:385] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.658229 27780 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.660537 27780 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.662397 27780 ts_tablet_manager.cc:1434] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.097s	user 0.009s	sys 0.000s
I20260709 12:17:38.672518 27779 tablet_bootstrap.cc:654] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.674680 27778 ts_tablet_manager.cc:1434] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.113s	user 0.010s	sys 0.011s
I20260709 12:17:38.687248 27780 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:38.689260 27907 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.699709 27778 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:38.701175 27779 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:38.703438 27779 ts_tablet_manager.cc:1403] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.117s	user 0.024s	sys 0.009s
I20260709 12:17:38.689827 27974 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.702405 27953 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.714987 27659 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:38.704090 27955 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.718235 27955 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.721544 27955 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:38.715909 27953 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.718852 27907 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:38.740587 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:38.741297 27585 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.741204 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.748916 27974 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.753461 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:38.759577 27738 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:17:38.759801 27704 leader_election.cc:343] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.796417 27780 tablet_bootstrap.cc:654] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.768114 27953 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:38.801728 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.794049 28030 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.808965 27587 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:38.766999 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.768507 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:38.808948 28030 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.814999 27779 raft_consensus.cc:359] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.815735 27779 raft_consensus.cc:385] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.815995 27779 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.818660 27779 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.820830 28097 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:38.821328 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51301.1us
I20260709 12:17:38.821446 28097 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.824100 27585 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:38.824920 28097 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:17:38.828819 27701 leader_election.cc:343] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.829241 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:38.830084 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.830847 28044 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:38.820840 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:38.830813 27734 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:38.833814 27658 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:38.835566 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:38.812630 27974 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.838568 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:38.839239 27974 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.840530 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.849310 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.847735 27953 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.852982 27953 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.854126 27903 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.854615 27903 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.854952 27903 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.855145 27974 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.856664 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:38.861675 27907 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:38.862181 27907 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:38.863559 27778 tablet_bootstrap.cc:654] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:38.869839 27779 ts_tablet_manager.cc:1434] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.166s	user 0.013s	sys 0.002s
I20260709 12:17:38.871013 27664 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:38.853451 27953 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.871097 27779 tablet_bootstrap.cc:492] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:38.866274 28093 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.886515 28093 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.888850 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.888923 28093 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:38.881457 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.892922 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.894073 27664 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:38.863269 27907 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.898651 27972 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:17:38.899767 27630 leader_election.cc:343] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:38.902603 27627 leader_election.cc:343] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.903191 27627 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:38.907588 27972 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:38.910888 27780 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:38.911657 27780 ts_tablet_manager.cc:1403] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.225s	user 0.011s	sys 0.014s
I20260709 12:17:38.915534 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.916260 27583 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:38.914500 27780 raft_consensus.cc:359] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.917250 27780 raft_consensus.cc:385] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:38.917526 27780 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:38.921046 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:38.926285 27780 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.908823 27903 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.927572 27733 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.930924 28030 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.937546 27972 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.938098 27972 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.938539 27972 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.937633 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:38.943616 27659 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:38.945636 27780 ts_tablet_manager.cc:1434] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.034s	user 0.003s	sys 0.003s
I20260709 12:17:38.946859 27780 tablet_bootstrap.cc:492] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:38.957417 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.962054 27779 tablet_bootstrap.cc:654] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:38.963582 27627 leader_election.cc:343] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:38.965023 27630 consensus_peers.cc:597] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:38.948729 27972 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.968849 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.976785 28102 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.977327 28102 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.978183 27972 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.979796 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:38.980567 27903 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:17:38.981782 27627 leader_election.cc:343] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:38.982203 28102 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.982404 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:38.983820 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:38.987929 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:38.988394 27585 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.988863 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:38.988696 27583 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:38.990885 28093 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:38.991760 28098 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.989799 27903 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:17:38.992291 28098 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:38.992750 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:38.994562 28103 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:38.995008 28103 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.995389 28103 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:38.995850 28098 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:38.996948 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:38.996755 27484 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:38.997629 27734 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:38.999624 28104 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.001111 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:39.002995 27585 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:39.004621 28104 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.005053 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.006863 28102 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.012547 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.013132 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.009385 28104 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.016712 27658 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.004591 28098 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.019572 28100 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260709 12:17:39.022396 27778 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:39.022853 27778 ts_tablet_manager.cc:1403] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.331s	user 0.009s	sys 0.017s
I20260709 12:17:39.015617 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.020251 28102 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:39.021327 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:17:39.029004 27701 leader_election.cc:343] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.024207 28098 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.031198 28098 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.027170 27583 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:39.036711 27953 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.038966 27704 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:39.038070 27658 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:39.041334 28104 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.047679 28104 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.027235 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.048203 28104 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.028517 28102 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.053730 27737 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:39.051046 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:39.051429 27972 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.055655 27733 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:39.058218 28100 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.057155 28103 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.065121 27664 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:39.067746 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:39.069697 27780 tablet_bootstrap.cc:654] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:39.074237 28106 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:17:39.066215 28100 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:39.077814 27893 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.079226 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.090147 27893 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.090941 27663 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:39.092800 27893 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.093762 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.094731 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:39.095312 27630 leader_election.cc:343] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.095688 28103 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.082274 28098 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.095899 28100 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.096565 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:17:39.096760 27627 leader_election.cc:343] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.097287 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.072604 27953 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.083899 27779 tablet_bootstrap.cc:492] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.083535 27778 raft_consensus.cc:359] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.100030 27778 raft_consensus.cc:385] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.100271 27779 ts_tablet_manager.cc:1403] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.229s	user 0.016s	sys 0.015s
I20260709 12:17:39.100304 27778 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.100670 27584 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:39.100651 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:39.105794 27737 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.104357 27778 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.117246 27778 ts_tablet_manager.cc:1434] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.094s	user 0.001s	sys 0.006s
I20260709 12:17:39.118463 27778 tablet_bootstrap.cc:492] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:39.119693 28104 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.099022 28103 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.123574 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.122131 28104 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.123550 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:39.126420 27584 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.128072 28098 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.128839 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.129105 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:39.129644 27662 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:39.130954 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.132068 28098 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.130857 27779 raft_consensus.cc:359] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.133845 27779 raft_consensus.cc:385] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.134138 27779 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.135239 27779 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:39.136756 27701 leader_election.cc:343] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.165400 27778 tablet_bootstrap.cc:654] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.165400 27484 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:39.178682 27779 ts_tablet_manager.cc:1434] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.078s	user 0.008s	sys 0.000s
I20260709 12:17:39.175472 28097 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:39.184665 28097 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:39.186712 28097 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.190716 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.192379 27659 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:17:39.193966 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:39.194988 28097 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:17:39.195436 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.196442 27738 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:39.205050 27737 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:39.207049 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:39.190382 27584 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:39.209991 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:39.211045 27704 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:39.195696 28097 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:17:39.215770 27955 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.218714 27779 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:39.214479 27907 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.215099 28102 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.222400 27780 tablet_bootstrap.cc:492] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:39.222335 27893 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.222956 27780 ts_tablet_manager.cc:1403] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.276s	user 0.015s	sys 0.006s
I20260709 12:17:39.222923 27893 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.223614 27955 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:39.224578 27955 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.225335 27893 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.215997 28097 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:17:39.236758 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.237478 27583 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:39.237860 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.233906 27485 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:39.239742 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:39.240911 27893 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.241267 27893 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.241616 27893 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:39.242450 27627 leader_election.cc:343] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.242377 27780 raft_consensus.cc:359] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.243278 27780 raft_consensus.cc:385] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:17:39.243980 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:39.250561 27780 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.234440 28102 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:39.256457 28097 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:39.265486 27780 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.271659 28102 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.276481 27893 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.282708 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.283933 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:39.284564 27584 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.278349 27779 tablet_bootstrap.cc:654] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.280607 27780 ts_tablet_manager.cc:1434] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.057s	user 0.008s	sys 0.000s
I20260709 12:17:39.289665 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:39.290455 27733 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:39.291766 28097 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.304378 27584 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:39.305720 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:39.306517 28103 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.307034 28103 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:39.307898 28103 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.293439 27780 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:39.312489 27779 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.312918 27779 ts_tablet_manager.cc:1403] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.096s	user 0.019s	sys 0.000s
I20260709 12:17:39.292469 27486 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:39.315455 27779 raft_consensus.cc:359] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.316089 27779 raft_consensus.cc:385] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.316368 27779 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.293780 27893 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.295442 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:39.316984 27779 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.321074 27780 tablet_bootstrap.cc:654] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:39.321003 27627 leader_election.cc:343] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.322142 27779 ts_tablet_manager.cc:1434] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:17:39.323020 27779 tablet_bootstrap.cc:492] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:39.297873 27663 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:17:39.324965 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.326133 28097 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:17:39.326961 27778 tablet_bootstrap.cc:492] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:39.327447 27778 ts_tablet_manager.cc:1403] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.209s	user 0.013s	sys 0.004s
I20260709 12:17:39.327704 27485 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:39.329650 28103 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.330217 28097 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:39.330089 28103 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.334648 28103 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.336300 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.337057 27584 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:39.331558 28097 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:39.338398 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.338959 27779 tablet_bootstrap.cc:654] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.339784 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.340457 27972 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.348990 27972 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.352340 27779 tablet_bootstrap.cc:492] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.352842 27779 ts_tablet_manager.cc:1403] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.004s
I20260709 12:17:39.353216 27778 raft_consensus.cc:359] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.354041 27778 raft_consensus.cc:385] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.359442 27778 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.356674 27779 raft_consensus.cc:359] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.360559 27779 raft_consensus.cc:385] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.360757 27779 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.360284 27778 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.361358 27779 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.363427 27778 ts_tablet_manager.cc:1434] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.001s
I20260709 12:17:39.363493 27779 ts_tablet_manager.cc:1434] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:17:39.367496 27778 tablet_bootstrap.cc:492] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:39.369789 27486 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:39.373857 27484 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:39.381019 27955 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.381979 27778 tablet_bootstrap.cc:654] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.381603 27955 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.382688 27779 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:39.385070 27780 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:39.385918 27780 ts_tablet_manager.cc:1403] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.097s	user 0.013s	sys 0.000s
I20260709 12:17:39.386849 27955 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.386956 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.388445 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.389997 27584 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:39.390069 27663 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:39.388711 27779 tablet_bootstrap.cc:654] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.391183 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:39.392307 27955 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.392789 27955 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.393181 27955 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.393698 27780 raft_consensus.cc:359] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.394335 27780 raft_consensus.cc:385] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.394583 27780 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.395220 27780 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.397740 27780 ts_tablet_manager.cc:1434] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:17:39.398713 27780 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:39.401944 27972 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.403443 27972 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.402148 27779 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.405294 27779 ts_tablet_manager.cc:1403] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.002s
I20260709 12:17:39.408053 27779 raft_consensus.cc:359] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.408692 27779 raft_consensus.cc:385] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.408893 27780 tablet_bootstrap.cc:654] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.408960 27779 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.409775 27779 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.410495 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.411154 27584 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:39.411995 27779 ts_tablet_manager.cc:1434] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:17:39.412434 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:39.412950 27779 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:39.414009 28103 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.414407 28103 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.414746 28103 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.414958 27778 tablet_bootstrap.cc:492] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:39.415509 27778 ts_tablet_manager.cc:1403] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.048s	user 0.013s	sys 0.002s
I20260709 12:17:39.418354 27778 raft_consensus.cc:359] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.418992 27778 raft_consensus.cc:385] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.420687 27779 tablet_bootstrap.cc:654] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.419251 27778 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.422238 27778 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.423528 28102 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.424013 28102 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.424795 27778 ts_tablet_manager.cc:1434] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:17:39.425782 27778 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:39.426082 28102 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.428179 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:39.429648 27627 leader_election.cc:343] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.429482 28104 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:39.437604 27778 tablet_bootstrap.cc:654] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.437476 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:39.439807 27630 leader_election.cc:343] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.440248 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.441830 28102 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.442162 27780 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:39.442624 27780 ts_tablet_manager.cc:1403] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.044s	user 0.010s	sys 0.003s
I20260709 12:17:39.445266 27780 raft_consensus.cc:359] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.445967 27780 raft_consensus.cc:385] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.446246 27780 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.446851 27780 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.449471 27780 ts_tablet_manager.cc:1434] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:17:39.450452 27780 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:39.457566 28103 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.459950 28103 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.464520 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:39.472323 27737 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.473465 27955 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.475762 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.464524 27972 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.482990 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:39.484087 27584 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.493443 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:39.494082 27585 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.500628 27734 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.503121 27585 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:39.504961 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:39.506542 27953 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.507164 27953 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:39.507601 28093 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.508294 27953 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.512512 27737 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:39.512403 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.515544 27955 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:39.519683 27659 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.524567 27779 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.525274 27779 ts_tablet_manager.cc:1403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.113s	user 0.034s	sys 0.007s
I20260709 12:17:39.531183 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:39.532770 27972 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.533025 27779 raft_consensus.cc:359] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.533808 27972 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:39.533896 27779 raft_consensus.cc:385] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.535022 27779 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.534758 27972 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.535703 27779 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.541787 28103 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.543100 27778 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:39.543139 27893 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:39.543970 27778 ts_tablet_manager.cc:1403] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.118s	user 0.018s	sys 0.000s
I20260709 12:17:39.544447 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:39.548051 27779 ts_tablet_manager.cc:1434] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.001s
I20260709 12:17:39.558173 27780 tablet_bootstrap.cc:654] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.561659 27779 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:39.562026 27584 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:39.564288 27778 raft_consensus.cc:359] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.565136 27778 raft_consensus.cc:385] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.565376 27778 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.574270 27778 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.580003 27484 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:39.584101 27778 ts_tablet_manager.cc:1434] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.004s
I20260709 12:17:39.595788 27778 tablet_bootstrap.cc:492] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:39.596765 27779 tablet_bootstrap.cc:654] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.604830 27486 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:39.618185 27778 tablet_bootstrap.cc:654] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.648515 27780 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:39.649000 27780 ts_tablet_manager.cc:1403] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.199s	user 0.018s	sys 0.031s
I20260709 12:17:39.663966 28104 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:39.666600 27780 raft_consensus.cc:359] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.667412 27780 raft_consensus.cc:385] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.667680 27780 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.668308 27780 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.670768 27780 ts_tablet_manager.cc:1434] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 12:17:39.671772 27780 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:39.677819 28097 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.678843 28018 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.681650 27779 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.682190 27779 ts_tablet_manager.cc:1403] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.121s	user 0.011s	sys 0.000s
I20260709 12:17:39.684288 28018 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.697440 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.697801 28097 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.699461 28018 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.708587 27779 raft_consensus.cc:359] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.709448 27779 raft_consensus.cc:385] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.709717 27779 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.710315 27779 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.711251 28093 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.711716 27780 tablet_bootstrap.cc:654] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.711474 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.717836 28102 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.718528 28102 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.721685 28102 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.722649 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.723636 27779 ts_tablet_manager.cc:1434] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.003s
W20260709 12:17:39.723954 27627 leader_election.cc:343] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.736487 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:39.737883 27630 leader_election.cc:343] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.738173 28097 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.738310 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.738674 28097 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.739337 28102 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.740787 28097 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.743083 27778 tablet_bootstrap.cc:492] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:39.744115 27778 ts_tablet_manager.cc:1403] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.149s	user 0.013s	sys 0.053s
I20260709 12:17:39.745231 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.746083 28100 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.747763 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.748627 27734 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:39.750356 27778 raft_consensus.cc:359] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.753185 27778 raft_consensus.cc:385] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.753871 27778 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.748804 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.754441 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:39.755155 27659 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:39.755716 27934 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.756134 27934 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.756498 27934 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.756868 28093 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.757412 28093 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } }
I20260709 12:17:39.757534 27778 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.762703 28093 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:39.763252 27934 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.767105 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:39.764691 27779 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
W20260709 12:17:39.770707 27627 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:39.780881 27780 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:39.781373 27780 ts_tablet_manager.cc:1403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.110s	user 0.009s	sys 0.002s
I20260709 12:17:39.781550 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.783283 27659 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:17:39.784307 27780 raft_consensus.cc:359] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.784976 27780 raft_consensus.cc:385] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.785259 27780 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.785698 27701 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:39.787015 28104 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:17:39.797248 27934 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.799443 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:39.800045 27734 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.804181 27780 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.810873 27779 tablet_bootstrap.cc:654] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.815554 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.816219 27659 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.824358 27780 ts_tablet_manager.cc:1434] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.002s
I20260709 12:17:39.825407 27780 tablet_bootstrap.cc:492] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:39.831501 27778 ts_tablet_manager.cc:1434] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.087s	user 0.004s	sys 0.003s
I20260709 12:17:39.836892 27972 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.838296 27934 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.838816 27934 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.829233 27734 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:39.840767 27659 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:39.848464 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:39.849429 28097 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.849992 28097 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:39.847141 27934 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.855969 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.856895 27778 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
W20260709 12:17:39.858100 27552 leader_election.cc:343] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.854497 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.850907 28097 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:39.861256 27552 leader_election.cc:343] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.861649 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.855576 28030 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:17:39.862603 27934 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.871690 27934 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.872555 27934 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.882674 27659 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:39.885737 27934 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.886488 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:39.887401 27778 tablet_bootstrap.cc:654] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.887589 27659 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:39.889219 28098 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:39.890188 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:39.894542 27552 leader_election.cc:343] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.896111 27780 tablet_bootstrap.cc:654] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.896677 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.899889 27779 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:39.900382 27779 ts_tablet_manager.cc:1403] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.136s	user 0.017s	sys 0.004s
I20260709 12:17:39.898289 28098 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:39.901506 28098 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.901849 28098 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.905076 27779 raft_consensus.cc:359] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.905805 27779 raft_consensus.cc:385] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.906054 27779 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.906780 27779 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.907473 28102 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.908025 28102 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.910241 28102 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.911237 28098 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.913317 28098 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:39.918993 27972 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.919603 27779 ts_tablet_manager.cc:1434] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260709 12:17:39.915030 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:39.920807 27779 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:39.915632 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:39.920940 27659 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:39.918869 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:39.923687 27630 leader_election.cc:343] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:39.924465 27627 leader_election.cc:343] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.924958 27627 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.925899 27972 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:39.936241 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:17:39.948968 27552 leader_election.cc:343] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:39.951844 28093 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.952261 27733 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:39.953740 27893 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.954448 27934 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:39.955220 27778 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:39.955765 27778 ts_tablet_manager.cc:1403] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.099s	user 0.011s	sys 0.004s
I20260709 12:17:39.956218 27779 tablet_bootstrap.cc:654] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:39.959007 27778 raft_consensus.cc:359] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.964694 27778 raft_consensus.cc:385] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:39.964542 28030 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.965553 27778 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:39.948936 27659 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:39.969266 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:39.968760 27486 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:39.970633 27934 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:39.971115 27934 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:39.972041 27934 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.978482 27778 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:39.996313 28104 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:39.996713 27778 ts_tablet_manager.cc:1434] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.001s
I20260709 12:17:39.998032 27778 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:40.022253 27587 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.086217 27779 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.088946 27779 ts_tablet_manager.cc:1403] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.168s	user 0.033s	sys 0.045s
I20260709 12:17:40.091553 27778 tablet_bootstrap.cc:654] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.088382 27780 tablet_bootstrap.cc:492] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:40.093423 27974 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.094071 27974 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.097079 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.097640 27589 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:40.093852 27780 ts_tablet_manager.cc:1403] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.269s	user 0.011s	sys 0.011s
I20260709 12:17:40.098778 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:40.099571 28030 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.100124 28030 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.091581 27779 raft_consensus.cc:359] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.100952 27779 raft_consensus.cc:385] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.101238 27779 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.101866 27779 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.100530 28030 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.103910 27779 ts_tablet_manager.cc:1434] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:17:40.104787 27779 tablet_bootstrap.cc:492] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:40.107520 27974 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.109469 27780 raft_consensus.cc:359] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.114984 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:40.118930 27780 raft_consensus.cc:385] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.120273 27779 tablet_bootstrap.cc:654] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.120345 27780 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
W20260709 12:17:40.119941 27627 leader_election.cc:343] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.120914 28030 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.124022 28030 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.124253 27778 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.124759 27778 ts_tablet_manager.cc:1403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.127s	user 0.011s	sys 0.016s
I20260709 12:17:40.125944 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:40.124841 27780 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:40.128005 27627 leader_election.cc:343] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.127702 27778 raft_consensus.cc:359] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.129532 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:40.129083 27778 raft_consensus.cc:385] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.130185 27585 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.130734 27778 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.133384 27778 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.135397 27778 ts_tablet_manager.cc:1434] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:17:40.136408 27778 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:40.142297 27780 ts_tablet_manager.cc:1434] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.001s
I20260709 12:17:40.143676 27780 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:40.150276 27780 tablet_bootstrap.cc:654] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.152724 27779 tablet_bootstrap.cc:492] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.153287 27779 ts_tablet_manager.cc:1403] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.049s	user 0.019s	sys 0.001s
I20260709 12:17:40.163765 27585 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:40.164909 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:40.165925 28030 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:40.166392 28030 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:40.167548 28030 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.169250 27955 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.169752 27955 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.170646 27778 tablet_bootstrap.cc:654] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.171758 27955 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.172832 27779 raft_consensus.cc:359] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.173468 27779 raft_consensus.cc:385] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.173724 27779 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.174130 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.174672 27487 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:40.179832 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.181594 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:40.182806 27955 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:40.183498 27779 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.186372 27779 ts_tablet_manager.cc:1434] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20260709 12:17:40.187737 27779 tablet_bootstrap.cc:492] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:40.191291 27778 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.191983 27780 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:40.192390 27733 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.196071 27780 ts_tablet_manager.cc:1403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.053s	user 0.013s	sys 0.008s
I20260709 12:17:40.198742 27780 raft_consensus.cc:359] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.199507 27780 raft_consensus.cc:385] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.199831 27780 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.200557 27780 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.202627 27780 ts_tablet_manager.cc:1434] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:17:40.205341 27780 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:40.203006 27778 ts_tablet_manager.cc:1403] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.067s	user 0.019s	sys 0.007s
I20260709 12:17:40.210788 27778 raft_consensus.cc:359] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.211531 27778 raft_consensus.cc:385] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.212375 27778 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.217700 27780 tablet_bootstrap.cc:654] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.219295 27778 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.221251 27779 tablet_bootstrap.cc:654] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.222276 27778 ts_tablet_manager.cc:1434] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260709 12:17:40.223372 27778 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:40.223243 27484 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:40.285151 27778 tablet_bootstrap.cc:654] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.322973 27779 tablet_bootstrap.cc:492] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.324867 27780 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:40.325435 27780 ts_tablet_manager.cc:1403] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.120s	user 0.022s	sys 0.024s
I20260709 12:17:40.328478 27780 raft_consensus.cc:359] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.329463 27779 ts_tablet_manager.cc:1403] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.142s	user 0.024s	sys 0.023s
I20260709 12:17:40.329113 27780 raft_consensus.cc:385] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.331969 27780 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.334692 27780 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.336787 27780 ts_tablet_manager.cc:1434] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.006s
I20260709 12:17:40.338519 28097 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:17:40.348150 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:40.332101 27779 raft_consensus.cc:359] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.333806 27778 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.356777 27778 ts_tablet_manager.cc:1403] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.134s	user 0.017s	sys 0.018s
I20260709 12:17:40.359359 27780 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:40.361135 27778 raft_consensus.cc:359] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.361950 27778 raft_consensus.cc:385] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.362277 27778 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.363013 27778 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.365062 27778 ts_tablet_manager.cc:1434] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:17:40.366086 27778 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:40.368872 27779 raft_consensus.cc:385] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.369511 27779 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.370039 27780 tablet_bootstrap.cc:654] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.370730 27779 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.372812 27779 ts_tablet_manager.cc:1434] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.043s	user 0.005s	sys 0.000s
I20260709 12:17:40.373729 27779 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:40.384164 28104 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.384712 28104 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.386850 28104 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.387909 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.388733 27662 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:40.388813 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.390265 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:40.390946 28104 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.391463 27779 tablet_bootstrap.cc:654] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.402562 27778 tablet_bootstrap.cc:654] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.402753 27780 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:40.403226 27780 ts_tablet_manager.cc:1403] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.044s	user 0.011s	sys 0.008s
I20260709 12:17:40.406484 27780 raft_consensus.cc:359] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.407158 27780 raft_consensus.cc:385] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.408002 27780 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.408636 27780 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.410629 27780 ts_tablet_manager.cc:1434] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:17:40.411526 27780 tablet_bootstrap.cc:492] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:40.414007 28104 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:40.417966 27778 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.418627 27778 ts_tablet_manager.cc:1403] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.053s	user 0.007s	sys 0.005s
I20260709 12:17:40.421176 27780 tablet_bootstrap.cc:654] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.422135 27778 raft_consensus.cc:359] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.432418 27779 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.432986 27779 ts_tablet_manager.cc:1403] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.059s	user 0.017s	sys 0.010s
I20260709 12:17:40.433101 28099 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.433634 28099 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.435954 27779 raft_consensus.cc:359] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.436901 27779 raft_consensus.cc:385] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.437194 27779 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.437958 27779 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.423064 27778 raft_consensus.cc:385] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.439227 27778 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.440160 27779 ts_tablet_manager.cc:1434] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:17:40.439900 27778 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.441113 27779 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:40.441928 27778 ts_tablet_manager.cc:1434] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:17:40.442929 27778 tablet_bootstrap.cc:492] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:40.448750 27779 tablet_bootstrap.cc:654] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.448750 27778 tablet_bootstrap.cc:654] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.460525 28093 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.460805 27779 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.461316 27779 ts_tablet_manager.cc:1403] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.007s
I20260709 12:17:40.461059 28093 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.464341 27779 raft_consensus.cc:359] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.465229 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.465858 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:40.465269 27779 raft_consensus.cc:385] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.466779 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.466979 27779 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.467465 27585 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:40.468026 27778 tablet_bootstrap.cc:492] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.467675 27779 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.468271 27974 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.468449 27778 ts_tablet_manager.cc:1403] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.004s
I20260709 12:17:40.468784 27974 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.469920 27779 ts_tablet_manager.cc:1434] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:17:40.470880 27779 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:40.471127 27778 raft_consensus.cc:359] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.471793 27778 raft_consensus.cc:385] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.467691 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:40.472129 27778 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.472065 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.472892 28104 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.472997 27585 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:40.473276 28104 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.473603 28104 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.474357 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:40.475433 27972 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.475772 27972 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.476095 27972 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.477617 28099 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.477950 27779 tablet_bootstrap.cc:654] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.480479 27934 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:40.483848 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:40.483673 27778 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.484454 27738 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:40.485352 28093 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.487365 27778 ts_tablet_manager.cc:1434] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:17:40.488526 27778 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:40.489629 27779 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.490125 27779 ts_tablet_manager.cc:1403] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.008s
I20260709 12:17:40.492838 27779 raft_consensus.cc:359] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.492133 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.493503 27779 raft_consensus.cc:385] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.493773 27663 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:40.493798 27779 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.494378 27778 tablet_bootstrap.cc:654] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.498812 27972 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.501551 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:40.502100 27587 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.504220 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:40.507457 28097 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.507828 28097 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.508199 28097 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.509171 27587 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:40.510288 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:40.510969 27893 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:40.517616 27955 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.517994 27903 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:40.518127 27955 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.520174 27955 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.521554 27778 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.522151 27778 ts_tablet_manager.cc:1403] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.034s	user 0.014s	sys 0.008s
I20260709 12:17:40.525431 27778 raft_consensus.cc:359] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.526219 27778 raft_consensus.cc:385] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.526542 27778 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.527400 27778 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.529891 27778 ts_tablet_manager.cc:1434] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:17:40.530956 27778 tablet_bootstrap.cc:492] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
W20260709 12:17:40.534201 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65647.3us
I20260709 12:17:40.537395 27778 tablet_bootstrap.cc:654] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.541668 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.542426 27585 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:40.552171 27893 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:40.553032 27893 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.558851 28181 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.559394 28181 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.566869 27974 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.583788 27779 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.589740 28097 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.597919 28097 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.598910 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:40.599155 27779 ts_tablet_manager.cc:1434] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.109s	user 0.011s	sys 0.002s
I20260709 12:17:40.599551 27737 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.600605 27779 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:40.601207 27972 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.617163 28181 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.619290 27779 tablet_bootstrap.cc:654] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:40.622182 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:40.625737 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.625883 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:40.503213 28104 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.639129 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:40.640103 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:40.627475 28093 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.646907 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.641197 28183 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.650827 27955 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.651490 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:40.651574 27955 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.652097 27663 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.654275 28104 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:17:40.654541 27627 leader_election.cc:343] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.644469 27903 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:40.655664 27903 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:40.655714 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:40.656572 27903 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.644661 28097 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.660023 28097 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.662598 28097 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.663476 28102 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:40.665205 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:40.665994 27733 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
W20260709 12:17:40.669219 27627 leader_election.cc:343] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.670723 27735 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.670513 27934 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:40.666177 28183 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:40.674907 27630 leader_election.cc:343] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.646412 28093 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.677806 28093 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.680435 27955 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.680881 27663 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:40.700672 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.701525 27663 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:40.692619 27731 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:40.706841 28183 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:40.710165 27778 tablet_bootstrap.cc:492] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:40.710775 27778 ts_tablet_manager.cc:1403] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.180s	user 0.037s	sys 0.054s
I20260709 12:17:40.713049 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:40.715085 27630 leader_election.cc:343] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.714958 27778 raft_consensus.cc:359] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.715842 27778 raft_consensus.cc:385] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:40.716154 27778 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:40.718035 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:40.718680 27663 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.725826 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:40.718292 27778 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:40.732492 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64309.2us
I20260709 12:17:40.734236 27701 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:17:40.732664 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82674.1us
I20260709 12:17:40.738384 27778 ts_tablet_manager.cc:1434] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260709 12:17:40.743106 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:40.743731 27587 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.744549 27663 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:40.756840 27587 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:17:40.763298 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:40.732383 28099 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.742537 28093 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:40.767455 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.768731 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.769549 27663 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:40.771286 27955 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:40.771951 27955 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:40.772934 27955 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.770382 27487 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:40.784123 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:40.785023 27662 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:40.791663 27778 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
W20260709 12:17:40.793980 27704 leader_election.cc:343] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.767453 28099 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.795063 27627 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:40.795648 27484 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:40.790473 28102 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.798607 28099 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.806445 27779 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:40.807363 27779 ts_tablet_manager.cc:1403] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.207s	user 0.020s	sys 0.012s
I20260709 12:17:40.811370 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:40.812372 27701 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:40.824024 27955 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.824651 27955 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.826375 27955 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.825470 28195 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.830197 28195 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.830732 28195 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:40.798040 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:40.849041 27955 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.802299 28181 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:40.859666 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:17:40.866720 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52067.4us
W20260709 12:17:40.870169 27704 leader_election.cc:343] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.871295 27955 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:40.803038 27972 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20260709 12:17:40.859046 28181 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.876487 27778 tablet_bootstrap.cc:654] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:40.877314 27704 leader_election.cc:343] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:40.837055 28102 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:40.880973 28181 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:40.982303 28193 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:17:40.910953 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.002259 27780 tablet_bootstrap.cc:492] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:41.002833 27780 ts_tablet_manager.cc:1403] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.592s	user 0.036s	sys 0.098s
W20260709 12:17:41.003082 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 190840us
I20260709 12:17:40.990595 27779 raft_consensus.cc:359] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.004271 27779 raft_consensus.cc:385] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.004593 27779 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.011068 27779 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.014294 27734 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:41.021813 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:41.023056 27663 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:41.027346 28190 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.242s	user 0.006s	sys 0.005s
I20260709 12:17:41.027698 28099 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.028558 27780 raft_consensus.cc:359] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.029471 27780 raft_consensus.cc:385] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.029837 27780 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.030105 28099 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.030807 27780 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.033003 27780 ts_tablet_manager.cc:1434] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 12:17:41.034340 27780 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:41.052012 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:41.064282 27630 leader_election.cc:343] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:41.071837 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184763us
I20260709 12:17:41.047457 28030 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.109252 27663 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
W20260709 12:17:41.124943 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82009.1us
I20260709 12:17:41.126752 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:41.120417 28102 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.114655 28030 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.160943 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:41.161800 27736 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.162977 28030 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.138276 27974 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:17:41.169013 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67869.8us
I20260709 12:17:41.197988 27779 ts_tablet_manager.cc:1434] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.390s	user 0.020s	sys 0.020s
I20260709 12:17:41.209079 28195 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.209939 27780 tablet_bootstrap.cc:654] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:41.211493 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74260.9us
I20260709 12:17:41.221248 27955 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.221789 27955 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.233225 27955 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:41.254469 27778 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:41.254938 27778 ts_tablet_manager.cc:1403] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.496s	user 0.017s	sys 0.015s
I20260709 12:17:41.257468 27778 raft_consensus.cc:359] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.258190 27778 raft_consensus.cc:385] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.258400 27778 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.253589 28219 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.268756 28195 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:41.259140 27778 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.266330 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:41.239307 28181 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.277030 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.268687 28219 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:41.278461 27627 leader_election.cc:343] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:41.278833 28181 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.282317 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.283249 27738 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 0.
I20260709 12:17:41.284583 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:17:41.285205 27630 leader_election.cc:343] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:41.285372 27738 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:41.285727 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:41.290693 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:41.290935 28219 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.291092 27778 ts_tablet_manager.cc:1434] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.000s
I20260709 12:17:41.314670 28217 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.323542 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:41.304797 28181 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:41.328130 27732 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.332531 28181 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:41.341063 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:17:41.345136 27630 leader_election.cc:343] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:41.348098 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:41.349169 28181 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.350708 28030 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.359069 27779 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:41.362851 27780 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:41.363386 27780 ts_tablet_manager.cc:1403] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.329s	user 0.017s	sys 0.026s
I20260709 12:17:41.366398 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:41.367236 27581 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:41.368626 27736 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:41.369637 27780 raft_consensus.cc:359] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.370189 27780 raft_consensus.cc:385] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.370436 27780 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.371091 27780 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.372880 27780 ts_tablet_manager.cc:1434] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:17:41.373801 27780 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:41.387909 27780 tablet_bootstrap.cc:654] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:41.388760 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:41.397420 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:41.401808 28199 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:41.406594 28220 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:41.410100 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.411864 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:41.415196 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:41.415971 27664 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:17:41.423307 27704 leader_election.cc:343] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:41.414561 28220 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:41.425136 28199 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:17:41.448567 28199 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.442233 28208 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 12:17:41.461275 27778 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:41.462332 27779 tablet_bootstrap.cc:654] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:41.464677 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:41.437812 28181 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.466873 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:41.467464 27662 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.467783 27972 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.468155 28181 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.468253 27972 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.476972 27701 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:41.442194 28097 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260709 12:17:41.478945 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:41.480093 28201 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.482815 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:41.483798 28093 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.480437 27729 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:17:41.487474 27630 leader_election.cc:343] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:41.502904 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:41.503702 27659 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:41.492049 28104 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:17:41.508654 27972 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.511905 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.471004 28220 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.525430 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:41.536777 27630 leader_election.cc:343] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:17:41.544420 27627 leader_election.cc:343] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:41.544931 27627 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:41.550410 28206 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:17:41.549197 27974 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.549089 28103 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:17:41.532461 27662 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:41.555070 27779 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:41.555823 27779 ts_tablet_manager.cc:1403] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.197s	user 0.015s	sys 0.003s
I20260709 12:17:41.567257 27778 tablet_bootstrap.cc:654] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:41.570559 28201 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:41.578025 27729 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:41.579986 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:17:41.580891 28103 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:41.581532 28103 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:41.583948 28103 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:41.613991 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:41.626111 28097 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.627467 27780 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:41.629151 27780 ts_tablet_manager.cc:1403] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.256s	user 0.018s	sys 0.009s
I20260709 12:17:41.628647 28097 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.638789 28097 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.640554 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:41.641683 27664 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:41.711858 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:41.720752 27779 raft_consensus.cc:359] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:41.730592 28233 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:17:41.733692 27779 raft_consensus.cc:385] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.734050 27779 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
W20260709 12:17:41.731416 28230 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:17:41.735368 27779 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:17:41.736209 28231 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:17:41.739210 27780 raft_consensus.cc:359] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.762334 27780 raft_consensus.cc:385] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.762641 27780 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.721699 28097 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.741075 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.771607 27780 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.789951 27780 ts_tablet_manager.cc:1434] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.160s	user 0.009s	sys 0.000s
I20260709 12:17:41.791926 27779 ts_tablet_manager.cc:1434] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.236s	user 0.011s	sys 0.010s
I20260709 12:17:41.803606 27779 tablet_bootstrap.cc:492] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:41.799741 27732 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:41.813603 28097 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.814816 27778 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:41.815469 27778 ts_tablet_manager.cc:1403] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.354s	user 0.028s	sys 0.021s
I20260709 12:17:41.854224 27778 raft_consensus.cc:359] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.854856 27778 raft_consensus.cc:385] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.855046 27778 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
W20260709 12:17:41.862084 28238 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.237s	user 0.004s	sys 0.000s
I20260709 12:17:41.870388 27780 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:41.871553 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:41.880678 27903 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.898553 27778 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.900292 27903 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:41.900900 27778 ts_tablet_manager.cc:1434] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.085s	user 0.004s	sys 0.005s
I20260709 12:17:41.902540 27778 tablet_bootstrap.cc:492] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:41.904814 27779 tablet_bootstrap.cc:654] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
W20260709 12:17:41.907346 28239 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.098s	user 0.001s	sys 0.001s
I20260709 12:17:41.918814 27780 tablet_bootstrap.cc:654] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:41.931702 27778 tablet_bootstrap.cc:654] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:41.939656 27485 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:41.941694 27779 tablet_bootstrap.cc:492] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:41.942180 27779 ts_tablet_manager.cc:1403] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.142s	user 0.023s	sys 0.014s
I20260709 12:17:41.942131 27485 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:41.945302 27780 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:41.950248 27780 ts_tablet_manager.cc:1403] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.084s	user 0.003s	sys 0.011s
I20260709 12:17:41.957404 27779 raft_consensus.cc:359] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.960778 27779 raft_consensus.cc:385] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.961166 27779 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.963581 27779 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.964551 28228 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:41.966552 27779 ts_tablet_manager.cc:1434] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.004s
I20260709 12:17:41.967739 27779 tablet_bootstrap.cc:492] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Bootstrap starting.
I20260709 12:17:41.969687 27780 raft_consensus.cc:359] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.970331 27780 raft_consensus.cc:385] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:41.970585 27780 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:41.971241 27780 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.975456 27780 ts_tablet_manager.cc:1434] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 12:17:41.976642 27778 tablet_bootstrap.cc:492] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:41.977231 27778 ts_tablet_manager.cc:1403] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.075s	user 0.008s	sys 0.013s
I20260709 12:17:41.977890 27780 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:41.981496 27779 tablet_bootstrap.cc:654] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:41.987942 28201 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:41.989099 28215 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.990223 28215 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:41.992820 28215 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:41.995119 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:41.996383 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:41.997133 27736 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:17:41.998800 27552 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.000075 28220 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:41.986824 27778 raft_consensus.cc:359] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.012174 27778 raft_consensus.cc:385] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.012054 27484 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:42.021875 28220 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:42.022902 27780 tablet_bootstrap.cc:654] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.012814 27778 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.032306 27934 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.032825 27934 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.034533 27934 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.037835 27893 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:42.042963 27487 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:42.056733 28201 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.057317 28201 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.060273 27934 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:42.061754 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.062657 27662 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.065224 27779 tablet_bootstrap.cc:492] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: No bootstrap required, opened a new log
I20260709 12:17:42.065706 28201 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:42.067131 27779 ts_tablet_manager.cc:1403] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent bootstrapping tablet: real 0.098s	user 0.010s	sys 0.031s
I20260709 12:17:42.067540 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.068243 27658 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:42.069451 27701 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:42.071365 28228 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.071740 28228 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.072098 28228 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.072290 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:42.073060 27934 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.073381 27934 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.073676 27934 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.073918 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:42.074921 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.079105 27778 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.081380 27778 ts_tablet_manager.cc:1434] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.104s	user 0.005s	sys 0.000s
I20260709 12:17:42.081681 27934 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.082394 27778 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:42.083448 27934 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.083679 27581 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:42.088447 28030 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:42.089167 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.089831 27658 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.070573 27779 raft_consensus.cc:359] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.092854 27779 raft_consensus.cc:385] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.093154 27779 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.093792 27779 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.095221 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:42.096089 27779 ts_tablet_manager.cc:1434] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260709 12:17:42.092226 28228 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.105439 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:42.106231 27581 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:42.107708 27736 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:42.110726 28228 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:42.112905 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.114817 27701 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:42.115967 27658 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.117807 28199 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:42.120800 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.121582 27934 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:42.122239 27934 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:42.123265 27934 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.114817 28220 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:17:42.128207 27778 tablet_bootstrap.cc:654] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.164462 28203 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.164953 28203 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.168000 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:42.169716 27630 leader_election.cc:343] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:42.172530 28203 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.173993 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:17:42.175230 27627 leader_election.cc:343] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:42.175661 27627 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.176671 28203 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:42.191262 27780 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:42.192950 27780 ts_tablet_manager.cc:1403] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.215s	user 0.029s	sys 0.008s
I20260709 12:17:42.195794 27780 raft_consensus.cc:359] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.196628 27780 raft_consensus.cc:385] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.196954 27780 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.197633 27780 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.200009 27780 ts_tablet_manager.cc:1434] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:17:42.201833 27780 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:42.207902 27780 tablet_bootstrap.cc:654] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.212184 27780 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:42.212587 27780 ts_tablet_manager.cc:1403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:17:42.214594 27780 raft_consensus.cc:359] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.215098 27780 raft_consensus.cc:385] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.215353 27780 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.215268 27487 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:42.216329 27780 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.217417 27778 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:42.217888 27778 ts_tablet_manager.cc:1403] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.136s	user 0.012s	sys 0.004s
I20260709 12:17:42.218523 27780 ts_tablet_manager.cc:1434] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:17:42.219537 27780 tablet_bootstrap.cc:492] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:42.220706 27778 raft_consensus.cc:359] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.221530 27778 raft_consensus.cc:385] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.221870 27778 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.222760 27778 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.228726 27780 tablet_bootstrap.cc:654] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.231178 27778 ts_tablet_manager.cc:1434] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:17:42.232319 27778 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:42.244268 28203 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.244896 28203 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.247797 27778 tablet_bootstrap.cc:654] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.251714 28203 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.253922 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.255610 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:17:42.256500 27627 leader_election.cc:343] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:42.261727 27780 tablet_bootstrap.cc:492] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:42.262251 27780 ts_tablet_manager.cc:1403] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.043s	user 0.013s	sys 0.020s
W20260709 12:17:42.263518 27630 leader_election.cc:343] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:42.264024 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.265447 27893 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:42.265319 27780 raft_consensus.cc:359] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.266068 27780 raft_consensus.cc:385] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.266364 27780 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.268260 27780 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.270483 27780 ts_tablet_manager.cc:1434] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:17:42.272502 27778 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:42.272979 27778 ts_tablet_manager.cc:1403] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.041s	user 0.006s	sys 0.006s
I20260709 12:17:42.274891 27780 tablet_bootstrap.cc:492] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Bootstrap starting.
I20260709 12:17:42.284121 27780 tablet_bootstrap.cc:654] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.288766 27778 raft_consensus.cc:359] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.293057 27778 raft_consensus.cc:385] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.293471 27778 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.294550 27778 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.297147 27778 ts_tablet_manager.cc:1434] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.001s
I20260709 12:17:42.297515 27780 tablet_bootstrap.cc:492] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: No bootstrap required, opened a new log
I20260709 12:17:42.298022 27780 ts_tablet_manager.cc:1403] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.010s
I20260709 12:17:42.302088 27780 raft_consensus.cc:359] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.302713 27780 raft_consensus.cc:385] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.302949 27780 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.304615 27780 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.306032 27780 ts_tablet_manager.cc:1434] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20260709 12:17:42.309289 27778 tablet_bootstrap.cc:492] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Bootstrap starting.
I20260709 12:17:42.316220 27778 tablet_bootstrap.cc:654] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Neither blocks nor log segments found. Creating new log.
I20260709 12:17:42.342949 28098 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.343549 28098 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.345820 28098 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.347235 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.347525 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.347903 27658 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
W20260709 12:17:42.349120 27552 leader_election.cc:343] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:17:42.349337 27778 tablet_bootstrap.cc:492] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: No bootstrap required, opened a new log
I20260709 12:17:42.349798 27778 ts_tablet_manager.cc:1403] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent bootstrapping tablet: real 0.041s	user 0.013s	sys 0.021s
I20260709 12:17:42.351070 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.351933 28098 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.352317 28098 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.352238 27778 raft_consensus.cc:359] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.352841 28098 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.352983 27778 raft_consensus.cc:385] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:17:42.353328 27778 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Initialized, Role: FOLLOWER
I20260709 12:17:42.354075 27778 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.356086 27778 ts_tablet_manager.cc:1434] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:17:42.358649 27934 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.359091 27934 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.361050 27934 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.362438 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.363106 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.363919 28199 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.364435 28199 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.365355 28203 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.365473 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:42.365880 28203 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.367215 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:42.368520 27584 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:42.369427 27934 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.369810 27934 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.369606 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.370174 27934 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.370373 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:42.370604 27737 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.371942 28228 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.372335 28228 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.372695 28228 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.373700 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:42.374925 28203 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.368865 28199 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:42.377168 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.381834 28228 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.382905 27737 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:42.383546 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.387557 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 0.
I20260709 12:17:42.382505 27934 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.401523 27934 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.405047 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:42.405659 27737 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 1: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:17:42.383836 28098 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.404106 27584 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:42.408592 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:42.409518 28098 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.403230 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.413663 27658 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.414647 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:42.416200 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.416481 27736 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.416805 27660 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.422026 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.423226 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.423712 27660 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.425249 27736 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.424929 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:42.426844 27934 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:42.427290 28220 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:42.427443 27934 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:42.427842 28220 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:42.428443 27934 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.428609 28220 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.443425 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.444543 27658 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 1: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:17:42.448248 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:17:42.448975 28199 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:17:42.449694 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:17:42.450810 27893 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:17:42.459251 28228 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 1 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:42.470868 27484 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:42.488814 27484 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:42.516496 27934 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.518013 27934 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.521785 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.522390 27659 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.523523 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:42.523993 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.524205 27903 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.524616 27903 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.524669 27736 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.524987 27903 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.528650 27934 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.532586 27903 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.534514 27903 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.535331 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.535952 27660 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.535975 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:42.536695 27736 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.545966 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.547116 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:42.547528 27736 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.547878 28098 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:42.548482 28098 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:42.549306 28098 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.568187 27484 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:42.584954 28098 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:42.588774 28098 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.590111 28098 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.592373 28098 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.592994 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:17:42.593645 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.595928 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:42.596628 27736 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 0.
I20260709 12:17:42.602711 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:42.603503 28099 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:17:42.603924 28099 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.604326 28099 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.613588 28099 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.615796 28099 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:42.621352 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:17:42.622008 27664 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.621980 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:42.627990 27736 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:17:42.632664 27664 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.633790 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:42.634907 28099 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:17:42.641242 27736 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:17:42.642400 28099 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:17:42.643507 28099 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:42.648552 27903 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:42.673552 27484 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:17:42.687688 27535 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:17:42.816359 28099 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:42.833971 28220 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:42.888532 27903 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:42.913285 28098 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:43.011526 28098 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:17:43.035694 27903 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:43.051574 28098 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:43.087921 27934 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:43.093573 28103 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.094107 28103 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:43.096295 28103 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:43.100232 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:17:43.100929 27584 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:43.104482 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:17:43.105154 27737 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:17:43.105384 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:17:43.107036 28030 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:17:43.107424 28030 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.107766 28030 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:43.119540 28030 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:43.122190 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:17:43.122790 27584 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:43.127246 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:17:43.127885 27737 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:17:43.131441 28030 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:43.135524 27737 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:43.136651 27627 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:17:43.137645 28030 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:17:43.138140 28030 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:17:43.140004 28030 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:17:43.145826 27584 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:17:43.173635 27484 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 0 to 2, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:43.521993 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:17:43.532797 28283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:43.549806 28284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:17:43.550849 28286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:17:43.553709 26408 server_base.cc:1061] running on GCE node
I20260709 12:17:43.555648 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:17:43.555997 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:17:43.556244 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599463556225 us; error 0 us; skew 500 ppm
I20260709 12:17:43.557796 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:17:43.564661 26408 webserver.cc:533] Webserver started at http://127.25.202.4:43799/ using document root <none> and password file <none>
I20260709 12:17:43.565369 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:17:43.565663 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:17:43.566140 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:17:43.568092 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-3-root/instance:
uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
format_stamp: "Formatted at 2026-07-09 12:17:43 on dist-test-slave-m57n"
I20260709 12:17:43.576823 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:17:43.582446 28291 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:43.586256 26408 fs_manager.cc:730] Time spent opening block manager: real 0.007s	user 0.003s	sys 0.001s
I20260709 12:17:43.586613 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-3-root
uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
format_stamp: "Formatted at 2026-07-09 12:17:43 on dist-test-slave-m57n"
I20260709 12:17:43.586937 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:17:43.595947 27737 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:43.598083 28103 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:17:43.601008 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:17:43.602762 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:17:43.606470 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:17:43.617185 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:17:43.617513 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:43.617830 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:17:43.618063 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:17:43.646924 27584 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:17:43.662756 27893 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:17:43.948575 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.4:39321
I20260709 12:17:43.948737 28361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.4:39321 every 8 connection(s)
I20260709 12:17:43.954608 26408 auto_leader_rebalancer-test.cc:439] add a tserver: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:43.971727 28362 heartbeater.cc:344] Connected to a master server at 127.25.202.62:43715
I20260709 12:17:43.972272 28362 heartbeater.cc:461] Registering TS with master...
I20260709 12:17:43.973261 28362 heartbeater.cc:507] Master 127.25.202.62:43715 requested a full tablet report, sending...
I20260709 12:17:43.976164 27487 ts_manager.cc:194] Registered new tserver with Master: 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:17:43.978067 27487 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45922
I20260709 12:17:44.974586 27664 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:44.981300 28362 heartbeater.cc:499] Master 127.25.202.62:43715 was elected leader, sending a full tablet report...
I20260709 12:17:44.985539 27737 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:44.988155 28030 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:44.989078 27584 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:44.992692 28030 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:17:45.010591 27628 consensus_peers.cc:597] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27d7343bc8174a5f89f4652b4401c3ec. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:45.012018 28030 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.014353 27588 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.044761 27487 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:45.053786 27737 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.087513 27732 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.103672 27664 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:45.106235 27588 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:17:45.107028 28367 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:45.113531 28367 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:17:45.140086 27702 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6491d84db64f4d48b3b3d5be8aec385b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:45.145968 28368 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.151357 27588 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.203675 27660 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.215860 27484 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:45.227407 27736 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:17:45.228616 27628 consensus_peers.cc:597] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f392cf87198499d9c064d1ec232d3d7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:45.229162 28030 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:45.229727 27664 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.234400 27581 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:45.236034 28030 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:45.282135 28103 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.304474 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:45.327788 27736 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.333509 27581 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:17:45.359341 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75015.8us
I20260709 12:17:45.398531 28391 ts_tablet_manager.cc:933] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:45.398334 27581 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.418313 28391 tablet_copy_client.cc:323] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:17:45.439081 27736 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:45.441345 27659 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:45.443360 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 27d7343bc8174a5f89f4652b4401c3ec from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:17:45.444027 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 27d7343bc8174a5f89f4652b4401c3ec from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-27d7343bc8174a5f89f4652b4401c3ec
I20260709 12:17:45.444067 28371 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:45.446810 28371 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:45.454789 27674 tablet_copy_source_session.cc:215] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:45.461254 28391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d7343bc8174a5f89f4652b4401c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:45.470531 28098 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.473830 27736 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.490566 27484 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.507107 28391 tablet_copy_client.cc:806] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:45.507932 28391 tablet_copy_client.cc:670] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:45.487870 27659 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:45.525256 28391 tablet_copy_client.cc:538] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:17:45.525862 27553 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd03dfdd894243c3af75d463b6913332. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:45.534361 28391 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:45.623077 28391 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:45.624168 28391 tablet_bootstrap.cc:492] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:45.626510 28391 ts_tablet_manager.cc:1403] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.091s	user 0.035s	sys 0.020s
I20260709 12:17:45.629587 28391 raft_consensus.cc:359] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.630723 28391 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:45.631471 28391 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.672129 28391 ts_tablet_manager.cc:1434] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.045s	user 0.014s	sys 0.020s
I20260709 12:17:45.676185 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-27d7343bc8174a5f89f4652b4401c3ec received from {username='slave'} at 127.0.0.1:36354
I20260709 12:17:45.676715 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-27d7343bc8174a5f89f4652b4401c3ec on tablet 27d7343bc8174a5f89f4652b4401c3ec with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:45.683185 28400 ts_tablet_manager.cc:933] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:45.700081 28400 tablet_copy_client.cc:323] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:17:45.731261 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 6491d84db64f4d48b3b3d5be8aec385b from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:17:45.731956 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 6491d84db64f4d48b3b3d5be8aec385b from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-6491d84db64f4d48b3b3d5be8aec385b
I20260709 12:17:45.742642 27748 tablet_copy_source_session.cc:215] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:45.748673 28400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491d84db64f4d48b3b3d5be8aec385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:45.769220 28400 tablet_copy_client.cc:806] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:45.769949 28400 tablet_copy_client.cc:670] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:45.775560 28400 tablet_copy_client.cc:538] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:45.782739 28400 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:45.831065 28400 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:45.832173 28400 tablet_bootstrap.cc:492] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:45.833250 28400 ts_tablet_manager.cc:1403] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.051s	user 0.017s	sys 0.021s
I20260709 12:17:45.833925 28391 ts_tablet_manager.cc:933] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:45.835803 28391 tablet_copy_client.cc:323] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:17:45.836614 28400 raft_consensus.cc:359] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.837751 28400 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:45.837860 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 4f392cf87198499d9c064d1ec232d3d7 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:17:45.838725 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 4f392cf87198499d9c064d1ec232d3d7 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-4f392cf87198499d9c064d1ec232d3d7
I20260709 12:17:45.838860 28400 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:45.848217 28400 ts_tablet_manager.cc:1434] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.004s
I20260709 12:17:45.850610 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-6491d84db64f4d48b3b3d5be8aec385b received from {username='slave'} at 127.0.0.1:47076
I20260709 12:17:45.851099 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-6491d84db64f4d48b3b3d5be8aec385b on tablet 6491d84db64f4d48b3b3d5be8aec385b with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:45.855866 27674 tablet_copy_source_session.cc:215] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:45.859699 28391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f392cf87198499d9c064d1ec232d3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:45.929733 28391 tablet_copy_client.cc:806] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:45.930456 28391 tablet_copy_client.cc:670] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:45.934478 28391 tablet_copy_client.cc:538] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:45.938275 28400 ts_tablet_manager.cc:933] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:45.943971 28391 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:45.948086 28400 tablet_copy_client.cc:323] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:17:45.960243 28337 raft_consensus.cc:1217] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:46.039505 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet fd03dfdd894243c3af75d463b6913332 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:17:46.040123 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet fd03dfdd894243c3af75d463b6913332 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-fd03dfdd894243c3af75d463b6913332
I20260709 12:17:46.045735 28391 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:46.046804 28391 tablet_bootstrap.cc:492] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:46.047539 28391 ts_tablet_manager.cc:1403] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.104s	user 0.023s	sys 0.046s
I20260709 12:17:46.050825 28391 raft_consensus.cc:359] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:46.051750 28391 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:46.052444 28391 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:46.055405 27599 tablet_copy_source_session.cc:215] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:46.061167 28400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd03dfdd894243c3af75d463b6913332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:46.063776 28391 ts_tablet_manager.cc:1434] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.003s
I20260709 12:17:46.066466 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-4f392cf87198499d9c064d1ec232d3d7 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:17:46.068621 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-4f392cf87198499d9c064d1ec232d3d7 on tablet 4f392cf87198499d9c064d1ec232d3d7 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:46.072129 28337 raft_consensus.cc:1217] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 12:17:46.099820 28400 tablet_copy_client.cc:806] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:46.100397 28400 tablet_copy_client.cc:670] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:46.110050 28400 tablet_copy_client.cc:538] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:46.120580 28400 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:46.150007 28400 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:46.151444 28400 tablet_bootstrap.cc:492] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:46.156272 28400 ts_tablet_manager.cc:1403] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.036s	user 0.014s	sys 0.012s
I20260709 12:17:46.159456 28400 raft_consensus.cc:359] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:46.177351 28400 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:46.178479 28400 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:46.183777 28400 ts_tablet_manager.cc:1434] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 12:17:46.192019 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-fd03dfdd894243c3af75d463b6913332 received from {username='slave'} at 127.0.0.1:56296
I20260709 12:17:46.192536 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-fd03dfdd894243c3af75d463b6913332 on tablet fd03dfdd894243c3af75d463b6913332 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:46.229451 28337 raft_consensus.cc:1217] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:17:46.496882 28337 raft_consensus.cc:1217] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:46.537149 28409 raft_consensus.cc:1064] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:46.544277 28409 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:46.552204 28398 raft_consensus.cc:1064] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:46.559919 28398 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:46.566349 28337 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:17:46.568852 27581 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:17:46.570555 28367 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:46.572942 28367 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:46.579572 27659 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:17:46.583500 28368 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:46.585685 27584 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:46.586212 27732 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:46.593876 28335 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:46.595490 28398 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:46.601725 28411 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:46.606026 28398 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:17:46.650060 28420 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:46.711998 28417 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:17:46.722530 28337 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.740625 28411 raft_consensus.cc:1064] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:46.742707 27487 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:46.741446 27660 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:46.759829 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103492us
I20260709 12:17:46.747983 28411 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:46.762405 28425 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.765427 27737 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.781735 27587 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.791008 27581 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.833132 28336 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.837833 27588 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:17:46.842816 27737 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:17:46.846056 28337 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:17:46.848456 28425 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:46.850268 28426 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:46.851791 28425 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:46.888760 28411 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.889681 27737 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.891359 27486 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:46.915719 27664 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:46.923998 28336 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:46.923615 27581 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.925781 28103 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:46.925836 27737 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:46.930234 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:46.933002 28337 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.941419 27734 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:46.941840 28425 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.948673 28336 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:17:46.950660 28367 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:17:46.953861 27664 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:46.958587 28335 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.975559 27581 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:17:46.976219 28430 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:46.978288 28430 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:46.988011 27581 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:17:46.987891 27732 raft_consensus.cc:2955] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.989944 28429 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:17:46.996812 28409 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:46.998953 27588 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.011574 28334 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:17:47.016775 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 27d7343bc8174a5f89f4652b4401c3ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:47.017592 27484 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.020853 28103 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:47.034152 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 27d7343bc8174a5f89f4652b4401c3ec with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:17:47.038489 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6491d84db64f4d48b3b3d5be8aec385b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:47.039063 28426 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.044667 28334 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.051195 28337 raft_consensus.cc:2955] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.052119 27581 raft_consensus.cc:2955] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.067852 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4f392cf87198499d9c064d1ec232d3d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:47.074414 27484 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.099121 27934 raft_consensus.cc:1064] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:47.101338 27934 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:47.104333 28439 tablet_replica.cc:333] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:17:47.109519 27485 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:47.111595 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f392cf87198499d9c064d1ec232d3d7 with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34456
I20260709 12:17:47.135666 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet 6491d84db64f4d48b3b3d5be8aec385b with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49990
I20260709 12:17:47.141958 28334 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:47.148056 28440 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:17:47.148826 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45677.3us
I20260709 12:17:47.157056 27664 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:47.157372 27732 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:47.159401 28442 tablet_replica.cc:333] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:17:47.161429 28098 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:47.169699 28439 raft_consensus.cc:2243] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:47.182026 28439 raft_consensus.cc:2272] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:47.188848 28439 ts_tablet_manager.cc:1916] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:47.200477 28442 raft_consensus.cc:2243] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:47.166908 28371 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:17:47.207203 28442 raft_consensus.cc:2272] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:47.209124 28444 tablet_replica.cc:333] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:17:47.291215 28442 ts_tablet_manager.cc:1916] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:47.312836 28444 raft_consensus.cc:2243] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:47.313758 28444 raft_consensus.cc:2272] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:47.316996 28444 ts_tablet_manager.cc:1916] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:47.378804 28371 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.381069 28334 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.392320 27664 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.408641 27737 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.413255 28439 ts_tablet_manager.cc:1929] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:47.414108 28439 log.cc:1199] T 27d7343bc8174a5f89f4652b4401c3ec P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/27d7343bc8174a5f89f4652b4401c3ec
I20260709 12:17:47.426321 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 27d7343bc8174a5f89f4652b4401c3ec (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:47.410948 27485 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:47.443403 28442 ts_tablet_manager.cc:1929] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:17:47.443936 28442 log.cc:1199] T 4f392cf87198499d9c064d1ec232d3d7 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/4f392cf87198499d9c064d1ec232d3d7
I20260709 12:17:47.445580 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 4f392cf87198499d9c064d1ec232d3d7 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:47.454838 27587 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:47.472117 27737 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:47.473759 28445 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:17:47.481443 28334 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:47.491235 28371 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:47.501237 28444 ts_tablet_manager.cc:1929] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260709 12:17:47.502502 28444 log.cc:1199] T 6491d84db64f4d48b3b3d5be8aec385b P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/6491d84db64f4d48b3b3d5be8aec385b
I20260709 12:17:47.504552 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet 6491d84db64f4d48b3b3d5be8aec385b (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:47.505875 28443 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.512583 28334 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.512920 27737 raft_consensus.cc:2955] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.521186 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet fd03dfdd894243c3af75d463b6913332 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:47.533329 27485 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:47.551533 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet fd03dfdd894243c3af75d463b6913332 with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49990
I20260709 12:17:47.552570 28444 tablet_replica.cc:333] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:17:47.553588 28444 raft_consensus.cc:2243] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:47.554598 28444 raft_consensus.cc:2272] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:47.563263 28444 ts_tablet_manager.cc:1916] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:47.585963 28444 ts_tablet_manager.cc:1929] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:47.586413 28444 log.cc:1199] T fd03dfdd894243c3af75d463b6913332 P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/fd03dfdd894243c3af75d463b6913332
I20260709 12:17:47.587915 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet fd03dfdd894243c3af75d463b6913332 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:48.857434 27587 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:17:48.881194 27553 consensus_peers.cc:597] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b881b4e55d543f9ade316642ea1b7e3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:48.882073 27734 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:48.882269 27658 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:48.884357 28453 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:48.886056 28452 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:48.901152 28440 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:48.902403 27737 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:48.926707 27484 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:48.929644 27737 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:48.932101 27658 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:48.954211 27658 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:48.956466 28368 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:17:48.959826 27702 consensus_peers.cc:597] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c3943cf144748fca8d2892751757c1a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:48.964244 27587 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:48.966207 28458 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:48.990324 27587 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:48.982352 28368 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.052048 27664 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.067807 27587 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.083400 27487 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:17:49.123008 27553 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb215cc72c384ec8b9421c81d953589b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:49.137763 27660 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:49.139182 28452 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:49.139593 27737 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:49.140874 28440 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:49.147898 28452 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.149958 27660 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.155812 27734 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.168915 27587 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.174763 27485 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:17:49.185065 27553 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85b7862879764c33b1db83d8596e3e87. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:49.185369 27660 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:17:49.186406 27736 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:17:49.186689 28440 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:49.189448 28440 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:17:49.208338 28453 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.210886 27660 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.211961 27736 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.229974 27485 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 4, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:49.444003 28475 ts_tablet_manager.cc:933] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:49.463858 28475 tablet_copy_client.cc:323] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:17:49.466109 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet 3b881b4e55d543f9ade316642ea1b7e3 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:17:49.466718 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet 3b881b4e55d543f9ade316642ea1b7e3 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-3b881b4e55d543f9ade316642ea1b7e3
I20260709 12:17:49.476812 27599 tablet_copy_source_session.cc:215] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:49.480616 28475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b881b4e55d543f9ade316642ea1b7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:49.504863 28475 tablet_copy_client.cc:806] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:49.505499 28475 tablet_copy_client.cc:670] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:49.505520 28477 ts_tablet_manager.cc:933] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:49.509675 28475 tablet_copy_client.cc:538] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:49.519237 28475 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:49.526392 28477 tablet_copy_client.cc:323] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:17:49.528643 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 1c3943cf144748fca8d2892751757c1a from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:17:49.529229 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 1c3943cf144748fca8d2892751757c1a from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-1c3943cf144748fca8d2892751757c1a
I20260709 12:17:49.542634 27748 tablet_copy_source_session.cc:215] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:49.548418 28479 ts_tablet_manager.cc:933] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:49.563999 28479 tablet_copy_client.cc:323] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:17:49.566049 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet eb215cc72c384ec8b9421c81d953589b from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:17:49.566577 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet eb215cc72c384ec8b9421c81d953589b from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-eb215cc72c384ec8b9421c81d953589b
I20260709 12:17:49.583520 28477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3943cf144748fca8d2892751757c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:49.588223 27599 tablet_copy_source_session.cc:215] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:49.592116 28479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb215cc72c384ec8b9421c81d953589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:49.630930 28475 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:49.631968 28475 tablet_bootstrap.cc:492] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:49.632704 28475 ts_tablet_manager.cc:1403] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.114s	user 0.024s	sys 0.020s
I20260709 12:17:49.635836 28475 raft_consensus.cc:359] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.636709 28475 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:49.637378 28475 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.644591 28479 tablet_copy_client.cc:806] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:49.645197 28479 tablet_copy_client.cc:670] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:49.649731 28479 tablet_copy_client.cc:538] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:49.662868 28477 tablet_copy_client.cc:806] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:49.663503 28477 tablet_copy_client.cc:670] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:49.663810 28479 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:49.665012 28475 ts_tablet_manager.cc:1434] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.004s
I20260709 12:17:49.666738 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-3b881b4e55d543f9ade316642ea1b7e3 received from {username='slave'} at 127.0.0.1:56296
I20260709 12:17:49.667204 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-3b881b4e55d543f9ade316642ea1b7e3 on tablet 3b881b4e55d543f9ade316642ea1b7e3 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:49.693766 28479 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:49.694748 28479 tablet_bootstrap.cc:492] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:49.695519 28479 ts_tablet_manager.cc:1403] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.017s
I20260709 12:17:49.698647 28479 raft_consensus.cc:359] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.699657 28479 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:49.700344 28479 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.703052 28479 ts_tablet_manager.cc:1434] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:17:49.704766 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-eb215cc72c384ec8b9421c81d953589b received from {username='slave'} at 127.0.0.1:56296
I20260709 12:17:49.705173 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-eb215cc72c384ec8b9421c81d953589b on tablet eb215cc72c384ec8b9421c81d953589b with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:49.710325 28475 ts_tablet_manager.cc:933] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:49.741432 28477 tablet_copy_client.cc:538] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:49.751248 28477 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:49.780475 28475 tablet_copy_client.cc:323] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:17:49.782178 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet 85b7862879764c33b1db83d8596e3e87 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:17:49.782759 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet 85b7862879764c33b1db83d8596e3e87 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-85b7862879764c33b1db83d8596e3e87
I20260709 12:17:49.790779 27599 tablet_copy_source_session.cc:215] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:49.794093 28475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b7862879764c33b1db83d8596e3e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:49.831941 28477 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:49.832916 28477 tablet_bootstrap.cc:492] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:49.833818 28477 ts_tablet_manager.cc:1403] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.083s	user 0.026s	sys 0.012s
I20260709 12:17:49.836891 28477 raft_consensus.cc:359] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.837761 28477 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:49.838413 28477 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.840790 28477 ts_tablet_manager.cc:1434] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:17:49.842938 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-1c3943cf144748fca8d2892751757c1a received from {username='slave'} at 127.0.0.1:47076
I20260709 12:17:49.843395 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-1c3943cf144748fca8d2892751757c1a on tablet 1c3943cf144748fca8d2892751757c1a with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:49.851915 28475 tablet_copy_client.cc:806] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:49.853919 28475 tablet_copy_client.cc:670] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:49.871353 28475 tablet_copy_client.cc:538] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:49.878969 28334 raft_consensus.cc:1217] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:49.890355 28475 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
W20260709 12:17:49.933962 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44681.9us
I20260709 12:17:49.939033 28475 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:49.943825 28475 tablet_bootstrap.cc:492] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:49.944722 28475 ts_tablet_manager.cc:1403] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.056s	user 0.030s	sys 0.016s
I20260709 12:17:49.948719 28475 raft_consensus.cc:359] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.950192 28475 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:49.951023 28475 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:49.962615 28337 raft_consensus.cc:1217] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:49.970675 28475 ts_tablet_manager.cc:1434] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.004s
I20260709 12:17:49.972790 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-85b7862879764c33b1db83d8596e3e87 received from {username='slave'} at 127.0.0.1:56296
I20260709 12:17:49.973277 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-85b7862879764c33b1db83d8596e3e87 on tablet 85b7862879764c33b1db83d8596e3e87 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:50.036801 28337 raft_consensus.cc:1217] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:50.122138 28337 raft_consensus.cc:1217] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260709 12:17:50.418495 28455 raft_consensus.cc:1064] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:50.423570 28455 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:50.448412 28337 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.449934 28455 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:50.460022 27734 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.462821 27660 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.474212 28489 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:50.478617 28489 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:50.511090 28455 raft_consensus.cc:1064] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:50.513826 28455 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:50.540731 28452 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.574314 28458 raft_consensus.cc:1064] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:50.573638 27660 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.576825 28458 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:50.580505 27484 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:50.588889 27737 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.607486 28334 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.612468 28337 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.615157 27734 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.622459 28497 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:50.624806 28497 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:50.627224 27660 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.648149 28489 raft_consensus.cc:1064] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:50.651191 28489 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:50.657655 28497 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:50.683558 27581 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:50.709461 27664 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:17:50.711836 28489 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:17:50.715018 28337 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:17:50.717298 28455 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:17:50.717926 27737 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:17:50.720103 28496 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:17:50.764554 28496 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.770020 28335 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:50.769237 27664 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.771974 28495 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:17:50.790236 28497 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.792121 28335 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.818243 27658 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:50.820911 28455 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:50.827562 27658 raft_consensus.cc:2955] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:50.834486 28504 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
I20260709 12:17:50.849020 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3b881b4e55d543f9ade316642ea1b7e3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:17:50.860313 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 162821us
I20260709 12:17:50.867580 28453 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.871258 27587 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.883425 28337 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:50.894181 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54143.5us
I20260709 12:17:50.908617 27737 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.917387 28334 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:17:50.919504 28498 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:17:50.920936 27659 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:50.922595 28508 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:50.922917 28336 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.946920 28508 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:50.949302 27658 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:50.961997 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:50.972559 27486 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:50.981660 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84092.7us
I20260709 12:17:50.996007 28458 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:51.010711 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b881b4e55d543f9ade316642ea1b7e3 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:34456
I20260709 12:17:51.064988 27736 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.071169 27487 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 4 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.111570 28520 tablet_replica.cc:333] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:17:51.112918 28520 raft_consensus.cc:2243] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:51.101526 28511 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.152329 27582 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:51.152544 27581 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:51.157213 27658 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.166029 28520 raft_consensus.cc:2272] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:51.166841 27737 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:17:51.166580 28337 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.169499 28518 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:17:51.179284 28453 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.186550 28334 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:17:51.188162 28335 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:51.191220 28496 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:17:51.191609 28489 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:51.193812 27588 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.205994 27486 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:51.208209 27737 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.222618 28337 raft_consensus.cc:2955] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.236526 28514 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.238548 27658 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:51.238111 28337 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.241410 28496 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:51.242091 28520 ts_tablet_manager.cc:1916] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:51.258227 27485 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 5 to 6, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.247340 27658 raft_consensus.cc:2955] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.266809 27736 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:51.272784 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet 85b7862879764c33b1db83d8596e3e87 with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49990
I20260709 12:17:51.270176 27487 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:51.274290 28335 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:51.277755 27660 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:51.279697 28511 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:51.281579 28511 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:51.291620 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet eb215cc72c384ec8b9421c81d953589b with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34456
I20260709 12:17:51.297111 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 85b7862879764c33b1db83d8596e3e87 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:51.297950 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet eb215cc72c384ec8b9421c81d953589b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:51.298900 28515 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.303696 28522 tablet_replica.cc:333] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:17:51.304955 28522 raft_consensus.cc:2243] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:51.307462 28337 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.310575 28522 raft_consensus.cc:2272] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:51.307960 27664 raft_consensus.cc:2955] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:51.319916 28522 ts_tablet_manager.cc:1916] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:51.354986 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1c3943cf144748fca8d2892751757c1a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:51.364398 28522 ts_tablet_manager.cc:1929] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 12:17:51.364948 28522 log.cc:1199] T 85b7862879764c33b1db83d8596e3e87 P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/85b7862879764c33b1db83d8596e3e87
I20260709 12:17:51.366813 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet 85b7862879764c33b1db83d8596e3e87 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:51.379469 27486 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:51.408589 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c3943cf144748fca8d2892751757c1a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:39844
I20260709 12:17:51.415257 28520 ts_tablet_manager.cc:1929] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:51.415792 28520 log.cc:1199] T 3b881b4e55d543f9ade316642ea1b7e3 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/3b881b4e55d543f9ade316642ea1b7e3
I20260709 12:17:51.417580 28520 tablet_replica.cc:333] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:17:51.418681 28520 raft_consensus.cc:2243] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:51.419943 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 3b881b4e55d543f9ade316642ea1b7e3 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:51.425945 28524 tablet_replica.cc:333] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:17:51.427709 28520 raft_consensus.cc:2272] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:51.431903 28520 ts_tablet_manager.cc:1916] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:51.433987 28524 raft_consensus.cc:2243] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:51.439954 28524 raft_consensus.cc:2272] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:51.447417 28524 ts_tablet_manager.cc:1916] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:51.452237 28520 ts_tablet_manager.cc:1929] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:51.452833 28520 log.cc:1199] T eb215cc72c384ec8b9421c81d953589b P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/eb215cc72c384ec8b9421c81d953589b
I20260709 12:17:51.454582 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet eb215cc72c384ec8b9421c81d953589b (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:51.472196 28524 ts_tablet_manager.cc:1929] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:51.472823 28524 log.cc:1199] T 1c3943cf144748fca8d2892751757c1a P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/1c3943cf144748fca8d2892751757c1a
I20260709 12:17:51.475026 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 1c3943cf144748fca8d2892751757c1a (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:52.487985 27737 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:52.494645 27664 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.494720 27587 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:17:52.495023 27702 consensus_peers.cc:597] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 723b2024e4414796ba50ece654596156. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:52.496215 28515 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:52.497251 28511 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:52.504886 28511 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.506716 27587 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.508458 27658 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.522576 27664 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:52.528650 27484 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:17:52.542984 27628 consensus_peers.cc:597] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74b289dab6a94832ad11d8ac1e30d802. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:52.544535 27736 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.546486 28526 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:52.546545 27587 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.550083 28487 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:52.564437 28411 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.565897 27736 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.589071 27588 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.601025 27664 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:52.602083 27486 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:17:52.618636 27628 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4811ee81866e469abca31f212c08ace2. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:52.619577 27588 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.621264 28545 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:52.625113 27734 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.628208 28411 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:52.637928 28545 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.640643 27588 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.640436 27734 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.652863 27736 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:52.658990 27485 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:17:52.684932 27702 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42e7d195f3f545b98a8dedb26c479c9c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:52.687515 27587 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.687566 27664 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:52.690026 28515 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:52.691943 28511 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:52.700237 28454 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.701785 27664 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.719013 27485 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:52.709002 27587 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:17:52.898278 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800100620 after lost signal to thread 26411
W20260709 12:17:52.899358 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080017f580 after lost signal to thread 27521
W20260709 12:17:52.900506 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800195da0 after lost signal to thread 27613
W20260709 12:17:52.901428 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800100840 after lost signal to thread 27688
W20260709 12:17:52.902436 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800284440 after lost signal to thread 27762
W20260709 12:17:52.903520 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08002fc6e0 after lost signal to thread 28361
W20260709 12:17:52.903968 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800315be0 after lost signal to thread 28501
W20260709 12:17:52.904322 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800323ea0 after lost signal to thread 28516
I20260709 12:17:52.966629 28552 ts_tablet_manager.cc:933] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:52.987767 28552 tablet_copy_client.cc:323] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:17:52.989590 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 74b289dab6a94832ad11d8ac1e30d802 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:17:52.990124 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 74b289dab6a94832ad11d8ac1e30d802 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-74b289dab6a94832ad11d8ac1e30d802
I20260709 12:17:52.996822 27674 tablet_copy_source_session.cc:215] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:52.999859 28552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b289dab6a94832ad11d8ac1e30d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:53.021353 28552 tablet_copy_client.cc:806] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:53.022007 28552 tablet_copy_client.cc:670] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:53.025961 28552 tablet_copy_client.cc:538] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:53.035315 28552 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:53.072335 28555 ts_tablet_manager.cc:933] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:53.085845 28552 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:53.095842 28555 tablet_copy_client.cc:323] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:17:53.097923 28552 tablet_bootstrap.cc:492] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:53.098160 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 723b2024e4414796ba50ece654596156 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:17:53.098755 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 723b2024e4414796ba50ece654596156 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-723b2024e4414796ba50ece654596156
I20260709 12:17:53.109923 27748 tablet_copy_source_session.cc:215] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:53.098764 28552 ts_tablet_manager.cc:1403] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.064s	user 0.020s	sys 0.012s
I20260709 12:17:53.113759 28555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723b2024e4414796ba50ece654596156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:53.114866 28552 raft_consensus.cc:359] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.115830 28552 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:53.117189 28552 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.125010 28556 ts_tablet_manager.cc:933] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:53.130168 28552 ts_tablet_manager.cc:1434] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 12:17:53.132079 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-74b289dab6a94832ad11d8ac1e30d802 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:17:53.132557 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-74b289dab6a94832ad11d8ac1e30d802 on tablet 74b289dab6a94832ad11d8ac1e30d802 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:53.140247 28555 tablet_copy_client.cc:806] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:53.140878 28555 tablet_copy_client.cc:670] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:53.161376 28555 tablet_copy_client.cc:538] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:53.161510 28552 ts_tablet_manager.cc:933] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:53.171111 28555 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:53.170981 28552 tablet_copy_client.cc:323] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:17:53.172683 28556 tablet_copy_client.cc:323] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:17:53.174291 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 4811ee81866e469abca31f212c08ace2 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:17:53.174785 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 4811ee81866e469abca31f212c08ace2 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-4811ee81866e469abca31f212c08ace2
I20260709 12:17:53.183086 27674 tablet_copy_source_session.cc:215] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:53.187139 28556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4811ee81866e469abca31f212c08ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:53.190100 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 42e7d195f3f545b98a8dedb26c479c9c from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:17:53.190701 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 42e7d195f3f545b98a8dedb26c479c9c from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-42e7d195f3f545b98a8dedb26c479c9c
I20260709 12:17:53.204367 27748 tablet_copy_source_session.cc:215] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:53.218348 28552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e7d195f3f545b98a8dedb26c479c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:53.219125 28555 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:53.220822 28555 tablet_bootstrap.cc:492] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:53.221501 28555 ts_tablet_manager.cc:1403] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.051s	user 0.020s	sys 0.004s
I20260709 12:17:53.244483 28555 raft_consensus.cc:359] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.245378 28555 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:53.246019 28555 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.248968 28555 ts_tablet_manager.cc:1434] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.027s	user 0.010s	sys 0.001s
I20260709 12:17:53.253975 28556 tablet_copy_client.cc:806] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:53.254529 28556 tablet_copy_client.cc:670] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:53.258162 28556 tablet_copy_client.cc:538] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:17:53.278131 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47624.7us
I20260709 12:17:53.282023 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-723b2024e4414796ba50ece654596156 received from {username='slave'} at 127.0.0.1:47076
I20260709 12:17:53.282517 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-723b2024e4414796ba50ece654596156 on tablet 723b2024e4414796ba50ece654596156 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:53.290067 28552 tablet_copy_client.cc:806] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:53.290689 28552 tablet_copy_client.cc:670] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:53.295038 28556 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:53.308569 28552 tablet_copy_client.cc:538] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:53.318285 28552 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:53.355500 28556 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:53.356992 28556 tablet_bootstrap.cc:492] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:53.357862 28556 ts_tablet_manager.cc:1403] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.063s	user 0.043s	sys 0.017s
I20260709 12:17:53.360982 28556 raft_consensus.cc:359] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.361841 28556 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:53.362509 28556 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.365039 28556 ts_tablet_manager.cc:1434] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:17:53.365491 28552 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:53.366870 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-4811ee81866e469abca31f212c08ace2 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:17:53.367421 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-4811ee81866e469abca31f212c08ace2 on tablet 4811ee81866e469abca31f212c08ace2 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:53.366967 28552 tablet_bootstrap.cc:492] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:53.370440 28552 ts_tablet_manager.cc:1403] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.052s	user 0.013s	sys 0.018s
I20260709 12:17:53.374084 28552 raft_consensus.cc:359] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.374889 28552 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:53.375524 28552 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:53.377777 28552 ts_tablet_manager.cc:1434] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:17:53.379037 28335 raft_consensus.cc:1217] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:53.379727 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-42e7d195f3f545b98a8dedb26c479c9c received from {username='slave'} at 127.0.0.1:47076
I20260709 12:17:53.380177 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-42e7d195f3f545b98a8dedb26c479c9c on tablet 42e7d195f3f545b98a8dedb26c479c9c with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:53.626546 28335 raft_consensus.cc:1217] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:53.677152 28335 raft_consensus.cc:1217] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:53.693536 28335 raft_consensus.cc:1217] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:53.811064 28566 raft_consensus.cc:1064] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:53.818156 28566 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:53.872895 28335 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:53.874790 28545 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:53.877039 27736 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:53.877724 27587 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:53.878948 28526 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:53.881479 28526 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:53.917860 28545 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:53.921844 27732 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:53.929453 27587 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:53.940523 28335 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:53.947863 27486 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:53.995245 27659 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:54.003858 28335 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:54.010358 28545 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:54.016237 27732 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:54.019409 28545 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:17:54.026578 28566 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.029140 28335 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.030396 27736 raft_consensus.cc:2955] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.048215 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 74b289dab6a94832ad11d8ac1e30d802 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:54.057474 27485 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.084448 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 74b289dab6a94832ad11d8ac1e30d802 with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:17:54.100482 28575 tablet_replica.cc:333] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:17:54.100924 28511 raft_consensus.cc:1064] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:54.108439 28575 raft_consensus.cc:2243] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:54.113544 28575 raft_consensus.cc:2272] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:54.109807 28511 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:54.117573 28575 ts_tablet_manager.cc:1916] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:54.179371 28335 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.183939 28545 raft_consensus.cc:1064] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:54.188697 28545 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:54.196867 28575 ts_tablet_manager.cc:1929] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:54.201574 28575 log.cc:1199] T 74b289dab6a94832ad11d8ac1e30d802 P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/74b289dab6a94832ad11d8ac1e30d802
I20260709 12:17:54.209323 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 74b289dab6a94832ad11d8ac1e30d802 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
W20260709 12:17:54.219825 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59414.5us
I20260709 12:17:54.230362 28515 raft_consensus.cc:1064] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:54.233855 27659 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.236824 28515 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:54.245884 28583 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:17:54.249091 28586 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:17:54.309007 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73197.5us
I20260709 12:17:54.315194 27587 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.316994 28411 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:54.320536 27588 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.333214 27736 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.345665 28584 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:17:54.416052 28337 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.343848 28545 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 12:17:54.436512 28599 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:54.440016 27664 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.455413 28515 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:17:54.461673 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 118367us
I20260709 12:17:54.493497 27588 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:17:54.504093 28587 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:17:54.504312 28584 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:54.519625 28335 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:54.556324 28581 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:17:54.587136 28601 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:17:54.522087 28332 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.640470 27736 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.671732 28578 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.654963 28526 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.747648 27485 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.752056 27587 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:54.755514 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65264.1us
I20260709 12:17:54.757941 27659 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:54.776404 28605 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 12:17:54.786947 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 326083us
I20260709 12:17:54.799170 27664 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.794271 28598 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.908531 28335 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.913000 27486 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.904698 27584 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.900424 27589 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:54.996685 28337 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.099623 27487 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.126173 27734 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:55.132097 27736 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:55.139734 28335 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:55.146183 28335 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:55.147558 28581 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:55.153443 28581 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:55.162195 27589 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:55.161796 27664 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:55.163801 28581 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:55.165205 28578 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.168809 27662 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:55.172504 28584 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:55.188776 28335 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:17:55.229246 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55406.9us
I20260709 12:17:55.238986 28597 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.243683 27589 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.245210 28337 raft_consensus.cc:2955] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.307114 27486 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.323035 27584 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:55.329856 28337 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:55.330008 28566 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:55.344053 28611 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:17:55.339322 27664 raft_consensus.cc:2955] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.356293 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet 42e7d195f3f545b98a8dedb26c479c9c with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49990
I20260709 12:17:55.358117 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 723b2024e4414796ba50ece654596156 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:55.360545 28411 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.371578 27589 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.376060 27484 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.388254 28337 raft_consensus.cc:2955] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:55.422307 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 723b2024e4414796ba50ece654596156 with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:17:55.435240 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 42e7d195f3f545b98a8dedb26c479c9c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:55.438409 28617 tablet_replica.cc:333] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:17:55.441450 28617 raft_consensus.cc:2243] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:55.443068 28617 raft_consensus.cc:2272] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:55.447765 28617 ts_tablet_manager.cc:1916] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:55.453375 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4811ee81866e469abca31f212c08ace2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:55.468518 28618 tablet_replica.cc:333] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:17:55.469750 28618 raft_consensus.cc:2243] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:55.470510 28618 raft_consensus.cc:2272] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:55.478358 28618 ts_tablet_manager.cc:1916] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:55.479501 27486 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:55.503562 28617 ts_tablet_manager.cc:1929] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:55.504030 28617 log.cc:1199] T 42e7d195f3f545b98a8dedb26c479c9c P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/42e7d195f3f545b98a8dedb26c479c9c
I20260709 12:17:55.508802 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet 42e7d195f3f545b98a8dedb26c479c9c (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:55.518090 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 4811ee81866e469abca31f212c08ace2 with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34456
I20260709 12:17:55.521095 28618 ts_tablet_manager.cc:1929] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:55.521642 28618 log.cc:1199] T 723b2024e4414796ba50ece654596156 P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/723b2024e4414796ba50ece654596156
I20260709 12:17:55.523814 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 723b2024e4414796ba50ece654596156 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:55.536029 28619 tablet_replica.cc:333] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:17:55.537273 28619 raft_consensus.cc:2243] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:55.538092 28619 raft_consensus.cc:2272] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:55.542367 28619 ts_tablet_manager.cc:1916] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:55.565781 28619 ts_tablet_manager.cc:1929] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:55.566294 28619 log.cc:1199] T 4811ee81866e469abca31f212c08ace2 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/4811ee81866e469abca31f212c08ace2
I20260709 12:17:55.567999 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 4811ee81866e469abca31f212c08ace2 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:56.603175 27664 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:17:56.611033 27628 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e634fbff5fb24e2fa133b95599b0db1f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:56.611673 27589 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:56.611673 27736 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:56.613262 28545 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:56.616108 28566 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:56.625376 28545 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.628650 27589 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.633680 27736 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.649745 27485 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.659531 27736 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:56.679476 27589 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:56.679913 27664 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:56.681130 28597 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:56.688838 28631 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:17:56.689855 27702 consensus_peers.cc:597] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 456e2eac35d647a3bf9eeb45a3d7596f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:56.706418 28597 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.711874 27581 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.723600 27662 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.736989 27486 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.746600 27581 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:56.756237 27736 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:56.756623 27662 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:56.758013 28518 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:56.759465 28518 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:17:56.760703 27553 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5357eb349b4b4934a5b0d36f90884fc6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:56.769937 28625 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.770978 27736 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.779791 27662 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.786993 27484 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.787279 27660 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:17:56.796651 27628 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7efb19c4fe844c2ea043bd3638faae4e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:56.797948 27736 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:56.798640 27581 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:17:56.801517 28626 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:56.802412 28545 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:17:56.826208 28626 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.837877 27581 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.840607 27736 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:17:56.843997 27486 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:57.091991 28643 ts_tablet_manager.cc:933] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:57.094717 28643 tablet_copy_client.cc:323] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:17:57.096524 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet e634fbff5fb24e2fa133b95599b0db1f from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:17:57.097054 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet e634fbff5fb24e2fa133b95599b0db1f from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-e634fbff5fb24e2fa133b95599b0db1f
I20260709 12:17:57.109119 27674 tablet_copy_source_session.cc:215] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:57.112622 28643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e634fbff5fb24e2fa133b95599b0db1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:57.128887 28643 tablet_copy_client.cc:806] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:57.129488 28643 tablet_copy_client.cc:670] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:57.133039 28643 tablet_copy_client.cc:538] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:57.140486 28643 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:57.165287 28643 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:57.166208 28643 tablet_bootstrap.cc:492] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:57.166877 28643 ts_tablet_manager.cc:1403] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.027s	user 0.022s	sys 0.001s
I20260709 12:17:57.169865 28643 raft_consensus.cc:359] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.170884 28643 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:57.171939 28643 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.176152 28643 ts_tablet_manager.cc:1434] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.009s	user 0.012s	sys 0.000s
I20260709 12:17:57.179189 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-e634fbff5fb24e2fa133b95599b0db1f received from {username='slave'} at 127.0.0.1:36354
I20260709 12:17:57.179670 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-e634fbff5fb24e2fa133b95599b0db1f on tablet e634fbff5fb24e2fa133b95599b0db1f with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:57.200383 28643 ts_tablet_manager.cc:933] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:17:57.202178 28643 tablet_copy_client.cc:323] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:17:57.203774 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 7efb19c4fe844c2ea043bd3638faae4e from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:17:57.204260 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 7efb19c4fe844c2ea043bd3638faae4e from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-7efb19c4fe844c2ea043bd3638faae4e
I20260709 12:17:57.210389 27674 tablet_copy_source_session.cc:215] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:57.217573 28643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efb19c4fe844c2ea043bd3638faae4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:57.233301 28648 ts_tablet_manager.cc:933] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:17:57.235229 28648 tablet_copy_client.cc:323] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:17:57.235337 28643 tablet_copy_client.cc:806] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:57.236115 28643 tablet_copy_client.cc:670] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:57.237027 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet 5357eb349b4b4934a5b0d36f90884fc6 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:17:57.237555 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet 5357eb349b4b4934a5b0d36f90884fc6 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-5357eb349b4b4934a5b0d36f90884fc6
I20260709 12:17:57.240480 28643 tablet_copy_client.cc:538] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:57.247618 27599 tablet_copy_source_session.cc:215] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:57.250296 28643 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:57.250340 28648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5357eb349b4b4934a5b0d36f90884fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:57.271428 28648 tablet_copy_client.cc:806] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:57.272120 28648 tablet_copy_client.cc:670] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:57.283157 28648 tablet_copy_client.cc:538] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:57.288050 28643 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:57.289122 28643 tablet_bootstrap.cc:492] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:57.289819 28643 ts_tablet_manager.cc:1403] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.040s	user 0.022s	sys 0.017s
I20260709 12:17:57.291675 28648 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:57.293005 28643 raft_consensus.cc:359] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.293903 28643 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:57.294601 28643 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.299582 28643 ts_tablet_manager.cc:1434] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:17:57.322623 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-7efb19c4fe844c2ea043bd3638faae4e received from {username='slave'} at 127.0.0.1:36354
I20260709 12:17:57.323186 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-7efb19c4fe844c2ea043bd3638faae4e on tablet 7efb19c4fe844c2ea043bd3638faae4e with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:57.328308 28651 ts_tablet_manager.cc:933] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:17:57.332825 28648 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:57.333783 28648 tablet_bootstrap.cc:492] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:57.334517 28648 ts_tablet_manager.cc:1403] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.043s	user 0.023s	sys 0.000s
I20260709 12:17:57.337739 28648 raft_consensus.cc:359] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.338284 28651 tablet_copy_client.cc:323] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:17:57.338608 28648 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:57.339300 28648 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.343098 28648 ts_tablet_manager.cc:1434] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260709 12:17:57.343009 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 456e2eac35d647a3bf9eeb45a3d7596f from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:17:57.343870 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 456e2eac35d647a3bf9eeb45a3d7596f from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-456e2eac35d647a3bf9eeb45a3d7596f
I20260709 12:17:57.350835 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-5357eb349b4b4934a5b0d36f90884fc6 received from {username='slave'} at 127.0.0.1:56296
I20260709 12:17:57.351300 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-5357eb349b4b4934a5b0d36f90884fc6 on tablet 5357eb349b4b4934a5b0d36f90884fc6 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:57.358237 27748 tablet_copy_source_session.cc:215] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:57.361599 28651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456e2eac35d647a3bf9eeb45a3d7596f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:57.378438 28651 tablet_copy_client.cc:806] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:57.379093 28651 tablet_copy_client.cc:670] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:57.382845 28651 tablet_copy_client.cc:538] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:57.392383 28651 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:17:57.413779 28651 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:17:57.414901 28651 tablet_bootstrap.cc:492] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:17:57.415788 28651 ts_tablet_manager.cc:1403] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.004s
I20260709 12:17:57.419060 28651 raft_consensus.cc:359] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.419914 28651 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:17:57.420562 28651 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:17:57.423040 28651 ts_tablet_manager.cc:1434] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20260709 12:17:57.426009 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-456e2eac35d647a3bf9eeb45a3d7596f received from {username='slave'} at 127.0.0.1:47076
I20260709 12:17:57.426565 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-456e2eac35d647a3bf9eeb45a3d7596f on tablet 456e2eac35d647a3bf9eeb45a3d7596f with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:17:57.583628 28335 raft_consensus.cc:1217] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:57.713364 28335 raft_consensus.cc:1217] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:57.805922 28335 raft_consensus.cc:1217] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:17:57.824467 28335 raft_consensus.cc:1217] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:17:58.117581 28628 raft_consensus.cc:1064] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:58.122219 28628 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.141058 28335 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.141942 27584 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.142753 27732 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.143181 28628 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.144807 28411 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.147686 28626 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:58.193562 28518 raft_consensus.cc:1064] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:58.194792 28656 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.201738 27732 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.200723 28518 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.215601 28337 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.228832 28335 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.230418 28625 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.226603 27485 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:58.232661 27662 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.232782 27732 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.235863 28625 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.238492 27584 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.240594 28518 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.253641 28597 raft_consensus.cc:1064] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:58.268182 28625 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.269870 27662 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.274580 28597 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.318271 27732 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.332134 28335 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.334192 28636 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:17:58.344619 27662 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.346232 27584 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:17:58.346279 28636 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:17:58.349202 28598 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.372130 28337 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.386348 28597 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.387993 27662 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.400312 28337 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.414460 27487 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.435180 27584 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.449378 28656 raft_consensus.cc:1064] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:17:58.453794 28656 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.470512 27662 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.477617 27589 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:58.479429 28337 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:58.480233 28626 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:58.481853 28628 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:58.483940 27484 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:58.489212 28332 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:17:58.493575 27732 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:17:58.497365 28411 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.496294 27581 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:17:58.501014 27582 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.502965 28656 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:17:58.503916 28626 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.510070 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e634fbff5fb24e2fa133b95599b0db1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:58.509040 28337 raft_consensus.cc:2955] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.515100 27584 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.516042 28628 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:17:58.522542 27732 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:58.523114 28335 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:58.525125 28672 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:17:58.531095 28518 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:58.543136 28621 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.545996 28335 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.558668 27736 raft_consensus.cc:2955] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.560626 28411 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.565896 27734 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.573082 27589 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.582990 28335 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.590327 27662 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:58.588687 27484 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:58.591741 28636 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:58.603556 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5357eb349b4b4934a5b0d36f90884fc6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:58.608217 27732 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.615748 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet e634fbff5fb24e2fa133b95599b0db1f with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34456
I20260709 12:17:58.614908 28597 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.621704 28332 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:17:58.626343 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 456e2eac35d647a3bf9eeb45a3d7596f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:58.628407 28598 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:58.627170 27660 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.624306 27487 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.631259 27487 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.633347 28335 raft_consensus.cc:2955] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.643690 28680 tablet_replica.cc:333] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:17:58.645200 28680 raft_consensus.cc:2243] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:17:58.646061 27471 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 456e2eac35d647a3bf9eeb45a3d7596f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20260709 12:17:58.646050 28680 raft_consensus.cc:2272] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:58.656914 28680 ts_tablet_manager.cc:1916] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:58.682989 27482 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:58.684417 27662 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:17:58.693245 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet 5357eb349b4b4934a5b0d36f90884fc6 with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49990
I20260709 12:17:58.706856 28335 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:17:58.707983 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 456e2eac35d647a3bf9eeb45a3d7596f with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:17:58.709781 28656 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:17:58.723801 28680 ts_tablet_manager.cc:1929] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:58.729586 28680 log.cc:1199] T e634fbff5fb24e2fa133b95599b0db1f P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/e634fbff5fb24e2fa133b95599b0db1f
I20260709 12:17:58.731900 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet e634fbff5fb24e2fa133b95599b0db1f (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:58.743175 28656 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.744632 28335 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.749450 27589 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:17:58.751106 28676 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:17:58.709967 28682 tablet_replica.cc:333] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:17:58.765355 27582 raft_consensus.cc:2955] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.805234 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7efb19c4fe844c2ea043bd3638faae4e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:58.767297 28682 raft_consensus.cc:2243] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:58.831915 28682 raft_consensus.cc:2272] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:58.771888 28683 tablet_replica.cc:333] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:17:58.849498 28682 ts_tablet_manager.cc:1916] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:58.862985 27482 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:17:58.904560 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 7efb19c4fe844c2ea043bd3638faae4e with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34456
I20260709 12:17:58.907090 28680 tablet_replica.cc:333] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:17:58.910256 28680 raft_consensus.cc:2243] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:58.918051 28680 raft_consensus.cc:2272] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:58.927001 28680 ts_tablet_manager.cc:1916] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:58.938712 28682 ts_tablet_manager.cc:1929] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:58.939584 28682 log.cc:1199] T 5357eb349b4b4934a5b0d36f90884fc6 P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/5357eb349b4b4934a5b0d36f90884fc6
I20260709 12:17:58.942544 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet 5357eb349b4b4934a5b0d36f90884fc6 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:58.947643 28680 ts_tablet_manager.cc:1929] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:17:58.948253 28680 log.cc:1199] T 7efb19c4fe844c2ea043bd3638faae4e P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/7efb19c4fe844c2ea043bd3638faae4e
I20260709 12:17:58.950898 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 7efb19c4fe844c2ea043bd3638faae4e (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:58.957376 28683 raft_consensus.cc:2243] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:58.958950 28683 raft_consensus.cc:2272] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:58.964084 28683 ts_tablet_manager.cc:1916] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:58.982133 28683 ts_tablet_manager.cc:1929] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:17:58.982550 28683 log.cc:1199] T 456e2eac35d647a3bf9eeb45a3d7596f P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/456e2eac35d647a3bf9eeb45a3d7596f
I20260709 12:17:58.985265 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 456e2eac35d647a3bf9eeb45a3d7596f (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:17:59.983723 27582 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:17:59.991434 27553 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1ffc143bb914f039c5b8bb615692aa3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:59.991655 27664 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:59.992305 27732 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:17:59.993038 28621 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:17:59.994094 28518 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:00.003865 28621 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.005528 27732 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.018702 27664 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.026719 27662 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.028577 27482 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.041730 27582 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:00.043463 27732 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:00.043638 28656 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:18:00.045080 27628 consensus_peers.cc:597] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62a7e798fd5e41e0bb2d9f5c7dab1adc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:00.046149 28685 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:00.060858 28678 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.063367 27584 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.083178 27734 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.097151 27732 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.105178 27485 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:18:00.127151 27702 consensus_peers.cc:597] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73b2d9da2f874f958d24ff32c2097c0a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:00.127662 27660 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:00.129288 28598 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:00.131110 27584 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:00.134598 28636 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:00.154852 28695 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.158792 27660 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.160605 27584 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.178799 27485 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.187211 27589 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:18:00.203989 27553 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbdcf17dc981450f989b93d262e9a5d0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:00.204936 27732 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:00.206056 27660 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:00.207135 28684 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:00.208257 28621 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:00.220074 28518 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.226471 27660 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.228473 27732 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:00.240403 27482 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:00.439610 28706 ts_tablet_manager.cc:933] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:00.441883 28706 tablet_copy_client.cc:323] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:18:00.444106 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:18:00.444772 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-62a7e798fd5e41e0bb2d9f5c7dab1adc
I20260709 12:18:00.455168 27674 tablet_copy_source_session.cc:215] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:00.459810 28706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:00.475795 28706 tablet_copy_client.cc:806] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:00.476485 28706 tablet_copy_client.cc:670] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:00.482810 28706 tablet_copy_client.cc:538] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:00.494204 28706 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:00.524391 28706 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:00.525240 28706 tablet_bootstrap.cc:492] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:00.525946 28706 ts_tablet_manager.cc:1403] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.032s	user 0.021s	sys 0.007s
I20260709 12:18:00.529363 28706 raft_consensus.cc:359] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.530177 28706 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:00.530781 28706 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.534245 28706 ts_tablet_manager.cc:1434] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.003s
I20260709 12:18:00.538597 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-62a7e798fd5e41e0bb2d9f5c7dab1adc received from {username='slave'} at 127.0.0.1:36354
I20260709 12:18:00.539137 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-62a7e798fd5e41e0bb2d9f5c7dab1adc on tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:00.554680 28706 ts_tablet_manager.cc:933] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:00.556411 28706 tablet_copy_client.cc:323] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:18:00.558270 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet e1ffc143bb914f039c5b8bb615692aa3 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:18:00.558770 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet e1ffc143bb914f039c5b8bb615692aa3 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-e1ffc143bb914f039c5b8bb615692aa3
I20260709 12:18:00.571821 27599 tablet_copy_source_session.cc:215] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:00.574887 28711 ts_tablet_manager.cc:933] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:00.575734 28706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ffc143bb914f039c5b8bb615692aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:00.586671 28711 tablet_copy_client.cc:323] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:18:00.589291 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 73b2d9da2f874f958d24ff32c2097c0a from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:18:00.589912 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 73b2d9da2f874f958d24ff32c2097c0a from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-73b2d9da2f874f958d24ff32c2097c0a
I20260709 12:18:00.598739 28706 tablet_copy_client.cc:806] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:00.598905 27748 tablet_copy_source_session.cc:215] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:00.599418 28706 tablet_copy_client.cc:670] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:00.604092 28706 tablet_copy_client.cc:538] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:00.605329 28711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b2d9da2f874f958d24ff32c2097c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:00.613313 28706 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:00.622005 28711 tablet_copy_client.cc:806] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:00.622561 28711 tablet_copy_client.cc:670] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:00.628489 28711 tablet_copy_client.cc:538] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:00.637914 28711 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:00.651364 28706 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:00.652347 28706 tablet_bootstrap.cc:492] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:00.653096 28706 ts_tablet_manager.cc:1403] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.017s
I20260709 12:18:00.656556 28706 raft_consensus.cc:359] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.657454 28706 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:00.658167 28706 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.664304 28706 ts_tablet_manager.cc:1434] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:18:00.665516 28711 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:00.666182 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-e1ffc143bb914f039c5b8bb615692aa3 received from {username='slave'} at 127.0.0.1:56296
I20260709 12:18:00.666648 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-e1ffc143bb914f039c5b8bb615692aa3 on tablet e1ffc143bb914f039c5b8bb615692aa3 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:00.668365 28711 tablet_bootstrap.cc:492] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:00.669260 28711 ts_tablet_manager.cc:1403] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.032s	user 0.024s	sys 0.004s
I20260709 12:18:00.672640 28711 raft_consensus.cc:359] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.673498 28711 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:00.674209 28711 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.676985 28711 ts_tablet_manager.cc:1434] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:18:00.680498 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-73b2d9da2f874f958d24ff32c2097c0a received from {username='slave'} at 127.0.0.1:47076
I20260709 12:18:00.681005 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-73b2d9da2f874f958d24ff32c2097c0a on tablet 73b2d9da2f874f958d24ff32c2097c0a with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:00.813781 28711 ts_tablet_manager.cc:933] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:00.815696 28711 tablet_copy_client.cc:323] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:18:00.817231 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet dbdcf17dc981450f989b93d262e9a5d0 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:18:00.817685 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet dbdcf17dc981450f989b93d262e9a5d0 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-dbdcf17dc981450f989b93d262e9a5d0
I20260709 12:18:00.826465 27599 tablet_copy_source_session.cc:215] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:00.831898 28711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdcf17dc981450f989b93d262e9a5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:00.850128 28711 tablet_copy_client.cc:806] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:00.850710 28711 tablet_copy_client.cc:670] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:00.854930 28711 tablet_copy_client.cc:538] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:00.865483 28711 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:00.885671 28711 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:00.886667 28711 tablet_bootstrap.cc:492] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:00.887267 28711 ts_tablet_manager.cc:1403] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.022s	user 0.020s	sys 0.000s
I20260709 12:18:00.890053 28711 raft_consensus.cc:359] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.890882 28711 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:00.891556 28711 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:00.893182 28711 ts_tablet_manager.cc:1434] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:18:00.899626 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-dbdcf17dc981450f989b93d262e9a5d0 received from {username='slave'} at 127.0.0.1:56296
I20260709 12:18:00.900068 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-dbdcf17dc981450f989b93d262e9a5d0 on tablet dbdcf17dc981450f989b93d262e9a5d0 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:01.044903 28335 raft_consensus.cc:1217] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:01.051496 28335 raft_consensus.cc:1217] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:01.064950 28335 raft_consensus.cc:1217] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:01.291771 28335 raft_consensus.cc:1217] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:01.444080 28636 raft_consensus.cc:1064] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:01.447772 28636 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.454332 27664 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.454331 28335 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.456101 27589 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.456110 28700 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:01.457197 28695 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:01.460821 28700 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:01.490376 28718 raft_consensus.cc:1064] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:01.493741 28718 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.498589 28636 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.511953 27589 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.535455 27482 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.542160 27732 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.546918 28656 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:01.558645 28332 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.561686 27589 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.560240 27664 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.585131 28335 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.595917 28723 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:01.609472 28685 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:01.630647 28723 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.636948 28332 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.640080 27589 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.657593 27732 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.679445 28684 raft_consensus.cc:1064] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:01.681574 27485 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.682004 28684 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.700007 27732 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.710623 28332 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:01.713908 27659 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:01.715914 28636 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:01.728216 28700 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:01.732738 27659 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.732889 28726 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.737169 27664 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.730464 28335 raft_consensus.cc:2955] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.748267 28332 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:01.752331 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 73b2d9da2f874f958d24ff32c2097c0a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:01.754215 28337 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.749650 27734 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:01.755973 27736 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.756322 27659 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.757987 28518 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:01.758500 28685 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:01.761878 28717 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:01.763016 28723 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:01.767395 28717 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:01.785373 28656 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.790438 27482 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:01.801092 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:01.804101 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 73b2d9da2f874f958d24ff32c2097c0a with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:18:01.826745 28740 tablet_replica.cc:333] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:18:01.832484 27485 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:01.833202 28736 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.836347 28740 raft_consensus.cc:2243] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:01.837419 28740 raft_consensus.cc:2272] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:01.839291 27736 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.834941 28337 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.843937 27737 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.858148 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:18:01.876029 28335 raft_consensus.cc:2955] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.900403 28740 ts_tablet_manager.cc:1916] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:01.911041 27485 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.914489 28717 raft_consensus.cc:1064] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:01.914851 27664 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:01.917054 28717 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.945231 27584 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:01.955561 28335 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:01.957841 28736 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:01.965617 27736 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:01.988786 28684 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:01.998267 27664 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:01.998365 27735 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:02.004930 28337 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:02.005149 28740 ts_tablet_manager.cc:1929] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:18:02.006803 28684 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.005667 28740 log.cc:1199] T 73b2d9da2f874f958d24ff32c2097c0a P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/73b2d9da2f874f958d24ff32c2097c0a
I20260709 12:18:02.011723 28740 tablet_replica.cc:333] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:18:02.016825 28742 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:02.017359 28337 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.022581 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 73b2d9da2f874f958d24ff32c2097c0a (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:18:02.023371 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e1ffc143bb914f039c5b8bb615692aa3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:02.023882 28740 raft_consensus.cc:2243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:02.024502 28518 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:02.027041 28518 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:02.030238 27735 raft_consensus.cc:2955] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.036083 27482 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.052059 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet e1ffc143bb914f039c5b8bb615692aa3 with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49990
I20260709 12:18:02.063127 28740 raft_consensus.cc:2272] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:02.066417 28745 tablet_replica.cc:333] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:18:02.067663 28745 raft_consensus.cc:2243] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:02.068475 28745 raft_consensus.cc:2272] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:02.073653 28745 ts_tablet_manager.cc:1916] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:02.075497 28740 ts_tablet_manager.cc:1916] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:02.106066 28745 ts_tablet_manager.cc:1929] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:18:02.106663 28745 log.cc:1199] T e1ffc143bb914f039c5b8bb615692aa3 P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/e1ffc143bb914f039c5b8bb615692aa3
I20260709 12:18:02.105170 28743 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.113945 28740 ts_tablet_manager.cc:1929] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:18:02.115736 28740 log.cc:1199] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/62a7e798fd5e41e0bb2d9f5c7dab1adc
I20260709 12:18:02.115787 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet e1ffc143bb914f039c5b8bb615692aa3 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:18:02.118429 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 62a7e798fd5e41e0bb2d9f5c7dab1adc (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:18:02.118342 28335 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.123366 27735 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.131039 27485 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:02.137892 27664 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.178416 27584 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:02.184803 28335 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:02.186408 28743 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:02.187992 27664 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:02.189669 28736 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:02.194887 28518 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.201273 28335 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.206292 27664 raft_consensus.cc:2955] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:02.212953 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet dbdcf17dc981450f989b93d262e9a5d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:02.221936 27482 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 3 to 4, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:02.247701 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet dbdcf17dc981450f989b93d262e9a5d0 with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34456
I20260709 12:18:02.249965 28748 tablet_replica.cc:333] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:18:02.251170 28748 raft_consensus.cc:2243] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:02.251871 28748 raft_consensus.cc:2272] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:02.255745 28748 ts_tablet_manager.cc:1916] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:02.271768 28748 ts_tablet_manager.cc:1929] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:18:02.272157 28748 log.cc:1199] T dbdcf17dc981450f989b93d262e9a5d0 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/dbdcf17dc981450f989b93d262e9a5d0
I20260709 12:18:02.273694 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet dbdcf17dc981450f989b93d262e9a5d0 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:18:03.328353 27589 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:18:03.335057 27553 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5865d310fddd49789e9de0a6a7bb260a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:03.335078 27735 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:03.335285 27659 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:03.336741 28751 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:03.337821 28743 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:03.345782 28751 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.347548 27659 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.350832 27735 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.359378 27664 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:03.371815 27487 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:18:03.394024 27628 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22597fcae204491f84839e680c083ecc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:03.397719 27732 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:03.397719 27589 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:03.400058 28685 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:03.400949 28656 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:03.433118 28685 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.435479 27732 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.436345 27584 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.450098 27732 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:03.452080 27487 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:03.467157 27659 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:18:03.467720 27702 consensus_peers.cc:597] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 310b1b37298244c0bd5bbe2b0c7328d7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:03.469071 28726 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:03.470536 27584 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:18:03.471807 28726 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:03.491899 28726 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.494114 27659 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.495153 27584 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.507072 27659 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:03.513439 27485 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 3, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.517704 27732 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:18:03.518327 27628 consensus_peers.cc:597] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81e6b2dabf8f4e6cb549ca250430e568. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:03.519433 28656 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:03.520758 27584 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:03.522694 28656 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:03.533957 28685 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.537690 27735 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.543002 27584 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:03.553258 27487 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:04.420534 28792 ts_tablet_manager.cc:933] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:04.465092 28795 ts_tablet_manager.cc:933] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:04.466706 28793 ts_tablet_manager.cc:933] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:04.473985 28795 tablet_copy_client.cc:323] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:18:04.476292 28793 tablet_copy_client.cc:323] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:18:04.480370 28796 ts_tablet_manager.cc:933] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:04.480207 28792 tablet_copy_client.cc:323] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
I20260709 12:18:04.482072 27599 tablet_copy_service.cc:140] P 3f90188458da46f6ba138d5f82c55b69: Received BeginTabletCopySession request for tablet 5865d310fddd49789e9de0a6a7bb260a from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:56296)
I20260709 12:18:04.482594 27599 tablet_copy_service.cc:161] P 3f90188458da46f6ba138d5f82c55b69: Beginning new tablet copy session on tablet 5865d310fddd49789e9de0a6a7bb260a from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:56296: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-5865d310fddd49789e9de0a6a7bb260a
I20260709 12:18:04.490479 27599 tablet_copy_source_session.cc:215] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:04.494911 28792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5865d310fddd49789e9de0a6a7bb260a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:04.498905 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:04.522159 28796 tablet_copy_client.cc:323] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:18:04.522940 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 81e6b2dabf8f4e6cb549ca250430e568 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:18:04.523576 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 81e6b2dabf8f4e6cb549ca250430e568 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-81e6b2dabf8f4e6cb549ca250430e568
I20260709 12:18:04.528101 28792 tablet_copy_client.cc:806] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:04.528659 28792 tablet_copy_client.cc:670] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:04.531389 27674 tablet_copy_source_session.cc:215] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:04.538077 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 310b1b37298244c0bd5bbe2b0c7328d7 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:18:04.538703 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 310b1b37298244c0bd5bbe2b0c7328d7 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-310b1b37298244c0bd5bbe2b0c7328d7
I20260709 12:18:04.541533 28792 tablet_copy_client.cc:538] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:04.551890 28792 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:04.555606 28793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e6b2dabf8f4e6cb549ca250430e568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:04.555969 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 22597fcae204491f84839e680c083ecc from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:18:04.556592 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 22597fcae204491f84839e680c083ecc from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-22597fcae204491f84839e680c083ecc
I20260709 12:18:04.564541 27674 tablet_copy_source_session.cc:215] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:04.565584 27748 tablet_copy_source_session.cc:215] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:04.567907 28796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22597fcae204491f84839e680c083ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:04.591933 28795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 310b1b37298244c0bd5bbe2b0c7328d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:04.648624 28796 tablet_copy_client.cc:806] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:04.649384 28796 tablet_copy_client.cc:670] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:04.653543 28796 tablet_copy_client.cc:538] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:04.668947 28793 tablet_copy_client.cc:806] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:04.669746 28793 tablet_copy_client.cc:670] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:04.672107 28795 tablet_copy_client.cc:806] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:04.672758 28795 tablet_copy_client.cc:670] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:04.673653 28793 tablet_copy_client.cc:538] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:04.678135 28795 tablet_copy_client.cc:538] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:04.689750 28796 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:04.694798 28792 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:04.697330 28792 tablet_bootstrap.cc:492] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:04.699213 28792 ts_tablet_manager.cc:1403] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.148s	user 0.055s	sys 0.065s
I20260709 12:18:04.704669 28792 raft_consensus.cc:359] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:04.710660 28792 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:04.711536 28792 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:04.713156 27659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:04.713904 27659 raft_consensus.cc:606] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:04.723963 28795 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:04.762495 28793 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:04.804839 28723 raft_consensus.cc:993] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:18:04.805430 28773 raft_consensus.cc:1081] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:18:04.806952 28792 ts_tablet_manager.cc:1434] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.106s	user 0.011s	sys 0.025s
I20260709 12:18:04.808825 27599 tablet_copy_service.cc:342] P 3f90188458da46f6ba138d5f82c55b69: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-5865d310fddd49789e9de0a6a7bb260a received from {username='slave'} at 127.0.0.1:56296
I20260709 12:18:04.808924 27588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:18:04.809371 27599 tablet_copy_service.cc:434] P 3f90188458da46f6ba138d5f82c55b69: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-5865d310fddd49789e9de0a6a7bb260a on tablet 5865d310fddd49789e9de0a6a7bb260a with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:04.809638 27588 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:04.810024 27588 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:04.874262 27588 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:04.893692 27588 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:04.901624 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:04.927963 27735 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:04.902940 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:04.929527 27662 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:04.929883 27662 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:04.934278 27662 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:04.963606 27662 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:04.994009 27735 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:05.060340 27662 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:05.065403 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:05.067953 28806 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:05.068477 28806 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:05.069379 28806 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:05.117014 28336 raft_consensus.cc:1217] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:05.124104 28795 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:05.125586 28795 tablet_bootstrap.cc:492] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:05.126595 28795 ts_tablet_manager.cc:1403] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.404s	user 0.019s	sys 0.018s
I20260709 12:18:05.164647 28795 raft_consensus.cc:359] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:05.168279 28795 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
W20260709 12:18:05.186450 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 137240us
I20260709 12:18:05.194700 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:05.195358 27735 raft_consensus.cc:606] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:05.187610 28795 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:05.233963 27487 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:05.258210 28796 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:05.265367 28795 ts_tablet_manager.cc:1434] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.138s	user 0.011s	sys 0.000s
I20260709 12:18:05.269040 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-310b1b37298244c0bd5bbe2b0c7328d7 received from {username='slave'} at 127.0.0.1:47076
I20260709 12:18:05.269536 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-310b1b37298244c0bd5bbe2b0c7328d7 on tablet 310b1b37298244c0bd5bbe2b0c7328d7 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:05.259224 28796 tablet_bootstrap.cc:492] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:05.278918 28796 ts_tablet_manager.cc:1403] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.589s	user 0.026s	sys 0.017s
I20260709 12:18:05.288182 27589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:05.288829 27589 raft_consensus.cc:606] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:05.289208 28793 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:18:05.289358 28796 raft_consensus.cc:359] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:05.290156 28793 tablet_bootstrap.cc:492] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:05.290167 28796 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:05.291044 28796 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:05.305015 27588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:05.305661 27588 raft_consensus.cc:606] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:05.306563 28796 ts_tablet_manager.cc:1434] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:18:05.313668 27589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:05.314260 27589 raft_consensus.cc:606] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:05.315658 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-22597fcae204491f84839e680c083ecc received from {username='slave'} at 127.0.0.1:36354
I20260709 12:18:05.316100 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-22597fcae204491f84839e680c083ecc on tablet 22597fcae204491f84839e680c083ecc with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:05.319033 28793 ts_tablet_manager.cc:1403] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.557s	user 0.025s	sys 0.020s
I20260709 12:18:05.331266 28793 raft_consensus.cc:359] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:05.332202 28793 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:05.332885 28793 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:05.336477 27588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:05.337430 27588 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:05.337610 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002ed4a0 after lost signal to thread 26411
W20260709 12:18:05.338845 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002edb80 after lost signal to thread 27521
W20260709 12:18:05.339967 28358 debug-util.cc:398] Leaking SignalData structure 0x7b0800333ba0 after lost signal to thread 27613
W20260709 12:18:05.340958 28358 debug-util.cc:398] Leaking SignalData structure 0x7b080023c140 after lost signal to thread 27688
W20260709 12:18:05.341951 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb760 after lost signal to thread 27762
W20260709 12:18:05.343149 28358 debug-util.cc:398] Leaking SignalData structure 0x7b080024b440 after lost signal to thread 28361
I20260709 12:18:05.344028 28793 ts_tablet_manager.cc:1434] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.024s	user 0.002s	sys 0.005s
I20260709 12:18:05.348086 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:18:05.351043 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:05.356784 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:05.362571 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-81e6b2dabf8f4e6cb549ca250430e568 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:18:05.363040 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-81e6b2dabf8f4e6cb549ca250430e568 on tablet 81e6b2dabf8f4e6cb549ca250430e568 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:05.370540 28333 raft_consensus.cc:1217] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:05.387665 28808 raft_consensus.cc:1064] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:05.390100 28808 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.407430 27659 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:05.408772 28812 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:05.410197 27734 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:05.412393 28333 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:05.414364 28812 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:05.415861 28774 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:18:05.483336 28828 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:18:05.494830 28774 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.497984 28332 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.507447 27659 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.523972 27482 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.543268 28778 raft_consensus.cc:993] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.543741 28778 raft_consensus.cc:1081] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Signalling peer 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.543138 27734 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.546808 28332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:58264
I20260709 12:18:05.548030 28332 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:05.548475 28332 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.558158 28332 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.562855 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:05.563555 27589 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:05.563858 27589 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:05.564544 27589 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.565696 27589 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.589159 27589 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:05.586926 27588 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.590912 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:05.600696 27734 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.601564 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:05.602484 28804 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:05.599802 28332 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:05.607767 28333 raft_consensus.cc:1217] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:18:05.610770 27735 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:05.611456 28336 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:05.622113 28808 consensus_queue.cc:1050] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:05.624135 28754 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:18:05.635613 28809 raft_consensus.cc:993] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:18:05.636125 28809 raft_consensus.cc:1081] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Signalling peer ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:18:05.638023 28778 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.639472 28336 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.641214 27734 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:05.644188 27738 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:34464
I20260709 12:18:05.644794 27738 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:05.645123 27738 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.643961 27659 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:05.675657 27738 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:05.675470 28804 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:05.680292 28804 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.683477 28806 raft_consensus.cc:993] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.684088 28806 raft_consensus.cc:1081] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Signalling peer 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.687177 27738 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:05.686403 27734 raft_consensus.cc:2955] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.688196 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:05.688763 27659 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.664350 28812 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:18:05.691648 28332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:58264
I20260709 12:18:05.692273 28332 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:05.692657 28332 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:05.705571 27735 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:05.706892 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:05.707595 27588 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:05.707985 27588 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:05.708950 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:05.709980 27664 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:05.708930 27588 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:05.711222 28754 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:05.711603 27588 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.717484 28332 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.728408 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5865d310fddd49789e9de0a6a7bb260a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:05.737169 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:05.738260 28820 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:05.740640 28820 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:05.741546 28820 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:05.749574 27588 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:05.748061 27482 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 1 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:05.752530 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:05.753522 27588 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:05.754071 27588 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:05.750101 28332 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:05.761642 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:05.762264 27738 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:05.765439 27588 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.766510 28810 raft_consensus.cc:993] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.771356 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:05.771370 28810 raft_consensus.cc:1081] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Signalling peer 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.770352 27588 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:05.774991 28336 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:58252
I20260709 12:18:05.775056 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:05.775887 28336 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:05.776367 28336 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.778893 28833 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:05.779556 28833 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:05.780550 28833 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.791626 27588 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:05.796085 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet 5865d310fddd49789e9de0a6a7bb260a with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49990
I20260709 12:18:05.850219 28754 raft_consensus.cc:993] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.850793 28754 raft_consensus.cc:1081] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Signalling peer 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:05.852373 27485 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:05.856670 28790 raft_consensus.cc:1064] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:05.865217 28790 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.867628 28333 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:58264
I20260709 12:18:05.868263 28333 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:05.873098 28333 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.885079 28839 tablet_replica.cc:333] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:18:05.887087 28333 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.942839 28333 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:05.889927 27588 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:05.943555 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:05.890491 28332 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:05.944154 27589 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:05.944476 27589 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:05.945166 27589 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.946152 27589 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.895599 28336 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.948892 28336 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:05.919461 27738 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:05.933267 27482 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 2 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:05.950942 27482 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 3 to 4, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 4 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:05.953040 28723 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:05.955106 28818 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:05.956176 28336 raft_consensus.cc:1217] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:18:05.957506 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:05.958060 27662 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.963023 28839 raft_consensus.cc:2243] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:05.964540 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:05.964784 28839 raft_consensus.cc:2272] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:05.965231 27659 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.973511 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:05.974171 27734 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:05.974447 27734 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:05.974860 27662 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:05.978339 28723 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:05.979504 28839 ts_tablet_manager.cc:1916] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:05.980641 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:05.985998 28804 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:05.986553 28804 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:05.975250 27734 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.988452 27659 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:05.988629 27734 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:05.993917 27589 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:05.997789 28804 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:05.999346 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:06.001101 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:06.008718 28833 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:06.009231 28833 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:06.010133 28833 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:06.062825 28839 ts_tablet_manager.cc:1929] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:18:06.070678 28838 raft_consensus.cc:1064] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:06.064522 28839 log.cc:1199] T 5865d310fddd49789e9de0a6a7bb260a P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/5865d310fddd49789e9de0a6a7bb260a
I20260709 12:18:06.073181 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet 5865d310fddd49789e9de0a6a7bb260a (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:18:06.073830 28838 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:06.096624 28777 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.109200 27738 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:18:06.131198 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47556.9us
I20260709 12:18:06.137667 28336 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.152107 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 2 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.157733 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 2, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 2 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.172091 27483 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.160573 27588 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.189718 27589 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:06.190469 27738 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:06.191124 28833 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:06.192382 28804 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:06.198845 28333 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:18:06.203282 27584 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:18:06.203460 27581 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:06.204411 28814 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:06.206136 27659 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:18:06.207912 28787 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:06.209661 28810 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:06.210969 28820 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:06.215781 27584 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:18:06.238833 28804 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:06.272634 27662 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:06.310281 28850 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:18:06.370349 27658 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:18:06.395624 28843 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:18:06.396684 28852 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.001s	sys 0.001s
I20260709 12:18:06.536293 28838 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:06.560585 28820 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.581269 28334 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:06.599888 27662 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:06.601718 27659 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:06.611665 27587 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:06.612615 28864 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:06.615018 27482 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:06.632194 28860 raft_consensus.cc:1064] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:06.634852 28860 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:06.639775 28833 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:06.640066 28832 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:06.661563 28863 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:18:06.655458 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 270866us
I20260709 12:18:06.669546 28336 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.694357 28804 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:06.703930 27664 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.706905 27581 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:06.747184 27734 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:18:07.120929 27738 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:18:07.123926 27734 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:07.149269 28804 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:07.150045 28832 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:07.157747 28334 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:07.572758 28855 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:07.573872 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.062s	sys 0.097s
W20260709 12:18:07.575951 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.062s	sys 0.097s
W20260709 12:18:07.577062 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 829371us
W20260709 12:18:07.587719 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.044s	sys 0.161s
W20260709 12:18:07.588300 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.044s	sys 0.161s
W20260709 12:18:07.588723 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 881004us
I20260709 12:18:07.573377 28855 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:07.574810 28860 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s	user 0.133s	sys 0.152s
I20260709 12:18:07.574669 28777 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:07.677904 28860 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.036s	user 0.133s	sys 0.153s
I20260709 12:18:07.688086 28860 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:07.723495 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1189) took 1025 ms. Trace:
I20260709 12:18:07.724035 27662 rpcz_store.cc:276] 0709 12:18:06.697721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.698043 (+   322us) service_pool.cc:224] Handling call
0709 12:18:06.698500 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:06.698998 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:06.699023 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:06.702173 (+  3150us) log.cc:844] Serialized 26 byte log entry
0709 12:18:06.736140 (+ 33967us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:06.736193 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:06.736264 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:07.723125 (+986861us) raft_consensus.cc:1685] finished
0709 12:18:07.723168 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:07.723376 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":234,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":2152,"thread_start_us":25098,"threads_started":1,"wal-append.queue_time_us":963679}
I20260709 12:18:07.726284 28820 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:07.727001 28820 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:07.729674 28820 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:07.732831 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:07.688817 28860 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:07.740813 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.074s	sys 0.116s
W20260709 12:18:07.741199 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.074s	sys 0.116s
I20260709 12:18:07.680464 28777 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:07.747573 27664 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.016s	sys 0.018s
W20260709 12:18:07.749146 27664 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.016s	sys 0.018s
I20260709 12:18:07.751019 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1191) took 1048 ms. Trace:
I20260709 12:18:07.766808 28860 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:07.791270 28860 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:07.761070 28838 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.004s	sys 0.031s
W20260709 12:18:07.808787 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.012s	sys 0.012s
W20260709 12:18:07.833983 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.002s	sys 0.031s
I20260709 12:18:07.805050 28860 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:07.876309 27735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.013s	sys 0.022s
W20260709 12:18:07.871659 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 217189us
I20260709 12:18:07.753661 27664 rpcz_store.cc:276] 0709 12:18:06.702813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.703390 (+   577us) service_pool.cc:224] Handling call
0709 12:18:06.703632 (+   242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:06.703813 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:18:07.080593 (+376780us) log.cc:844] Serialized 14 byte log entry
0709 12:18:07.750506 (+669913us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:07.750563 (+    57us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:07.750671 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:07.750778 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:07.750945 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37959,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"thread_start_us":669515,"threads_started":1}
W20260709 12:18:07.864866 28833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.037s	sys 0.067s
W20260709 12:18:07.902055 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.005s	sys 0.048s
W20260709 12:18:07.903388 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.003s	sys 0.031s
W20260709 12:18:07.903828 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.013s	sys 0.022s
W20260709 12:18:07.911633 28833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.040s	sys 0.075s
I20260709 12:18:07.911695 28893 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:07.918993 28893 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:07.903018 28838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.004s	sys 0.031s
W20260709 12:18:07.939603 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.006s	sys 0.049s
W20260709 12:18:07.951463 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 91296.5us
W20260709 12:18:07.903506 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.013s	sys 0.012s
W20260709 12:18:07.903857 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44011.4us
W20260709 12:18:07.906635 27581 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.192s	user 0.019s	sys 0.010s
I20260709 12:18:07.971768 28893 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:07.918473 28882 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:07.972887 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64852.5us
I20260709 12:18:07.973209 28882 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:07.973701 27482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 225) took 1318 ms. Trace:
I20260709 12:18:07.974129 27482 rpcz_store.cc:276] 0709 12:18:06.655574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.656695 (+  1121us) service_pool.cc:224] Handling call
0709 12:18:07.973589 (+1316894us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5,"reported_tablets":1}
I20260709 12:18:07.976660 28860 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:07.977577 28882 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:08.001065 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83890.9us
I20260709 12:18:08.007720 28899 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.008417 28899 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:08.011765 28899 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:07.906517 28878 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:08.022275 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57724.5us
I20260709 12:18:07.905342 28855 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:07.906210 28777 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.005235 28860 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:08.028925 28889 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.120s	user 0.007s	sys 0.006s
I20260709 12:18:08.029390 28814 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.030124 28820 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.024384 28892 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.066782 28893 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.060808 28882 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260709 12:18:08.059592 28860 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.025872 28878 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.079304 28893 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:08.085681 28887 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.118s	user 0.009s	sys 0.019s
I20260709 12:18:08.086354 28893 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:08.085846 27587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.354s	user 0.168s	sys 0.223s
W20260709 12:18:08.089170 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.381s	user 0.169s	sys 0.224s
I20260709 12:18:08.065529 28820 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.064522 28814 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.101718 28886 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.113230 28814 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.104609 28900 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.102056 28893 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.105902 28894 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.075248 28892 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.103420 28782 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:08.107707 28880 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.200s	user 0.002s	sys 0.005s
I20260709 12:18:08.108078 28896 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.093582 28838 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.117353 28896 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:08.120612 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 92613.5us
I20260709 12:18:08.114049 28900 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.114895 28893 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.117195 28782 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.113356 28886 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.115103 28894 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:08.129060 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 176745us
I20260709 12:18:08.136798 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 34) took 1434 ms. Trace:
I20260709 12:18:08.137315 27587 rpcz_store.cc:276] 0709 12:18:06.702726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.702841 (+   115us) service_pool.cc:224] Handling call
0709 12:18:06.703118 (+   277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:06.703478 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:06.703504 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:06.707564 (+  4060us) log.cc:844] Serialized 26 byte log entry
0709 12:18:08.089676 (+1382112us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:08.089722 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.089790 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:08.106177 (+ 16387us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:08.135273 (+ 29096us) raft_consensus.cc:1685] finished
0709 12:18:08.135305 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.136695 (+  1390us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2639,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":195450752,"thread_start_us":1381809,"threads_started":1,"wal-append.queue_time_us":1096555}
I20260709 12:18:08.134625 28893 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.121433 28838 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.171689 28900 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.166380 28820 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.509702 28896 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.510982 28892 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.513919 28838 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.515446 28886 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.507776 28860 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.532675 28812 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.564813 28812 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.567245 28812 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.570209 28812 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.578529 28812 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.545801 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1169) took 1839 ms. Trace:
I20260709 12:18:08.590085 27581 rpcz_store.cc:276] 0709 12:18:06.706112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.706257 (+   145us) service_pool.cc:224] Handling call
0709 12:18:06.706598 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:06.706792 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:18:07.965222 (+1258430us) log.cc:844] Serialized 14 byte log entry
0709 12:18:08.545419 (+580197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:08.545475 (+    56us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:08.545521 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.545599 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.545744 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":535039,"raft.queue_time_us":550340,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":140,"spinlock_wait_cycles":1896192,"thread_start_us":44657,"threads_started":2,"wal-append.queue_time_us":26158}
I20260709 12:18:08.563675 28900 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.592439 28900 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.594676 28900 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.573765 28906 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.583039 28890 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.595582 28906 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.596055 28890 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.598001 28906 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.598549 28890 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.599383 28890 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.601018 28890 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.540679 28909 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.613761 28881 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.582270 28902 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.583654 28901 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.595165 28900 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.577596 28864 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.601892 28787 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.544636 28908 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:08.618362 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83912.5us
I20260709 12:18:08.622129 28787 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.613976 28909 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.614403 28881 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.624781 28900 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.617522 28902 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.699556 28910 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.774616 28913 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.775260 28913 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.777639 28913 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.764871 28905 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.617547 28901 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.767892 28890 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.763931 28908 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.621891 28864 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.798905 28915 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.799727 28915 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.803769 28915 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.804369 28917 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.793150 28890 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.809945 28787 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.810552 28787 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.811106 28787 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.811621 28919 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.813300 28787 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.778568 28910 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.816002 28812 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.823053 28902 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.823148 28916 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.794158 28905 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.826972 28812 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.815451 28890 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.823951 28916 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.831832 28908 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.840265 28908 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.840560 28915 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.840965 28908 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.841181 28915 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.812368 28919 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.830379 28901 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.844878 28900 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.844751 28913 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.848397 28910 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.848565 28913 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.848994 28910 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:08.846704 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.842s	user 0.026s	sys 0.038s
I20260709 12:18:08.820411 28911 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:08.851850 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.027s	sys 0.038s
I20260709 12:18:08.809144 28917 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.852959 28811 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.857301 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 22597fcae204491f84839e680c083ecc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:08.843290 28908 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.843691 28915 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.868098 28917 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.857475 28811 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.830773 28812 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.870400 28914 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:08.873485 28893 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:08.853394 28864 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.873958 28909 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.875361 28890 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.863179 28911 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.878089 28881 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.880692 28905 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.853546 28910 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.895223 28811 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.907838 28906 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.901398 28913 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.919456 28909 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.904899 28893 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.924645 28864 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.920369 28906 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.929646 28812 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.929530 28864 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.896637 28914 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.920991 28905 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.920176 28894 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:08.924086 28909 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:08.936110 28914 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.937083 28909 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.937139 28893 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.938019 28864 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.938158 28906 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.938612 28905 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.933772 28911 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:08.931550 28885 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:08.932590 28919 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.939903 28885 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.940531 28910 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.942880 28885 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.943432 28905 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:08.950635 28874 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.043s	user 0.001s	sys 0.005s
I20260709 12:18:08.994059 28812 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.994306 28913 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.995982 28812 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.998626 28811 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.998848 28917 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.995862 28913 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.000093 28812 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:08.999818 28917 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:08.990386 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.002638 28913 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.004529 28811 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.003275 28917 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.008239 28811 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.008419 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.026873 28812 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.027761 28812 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.029592 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.031087 28911 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.031932 28910 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.033655 28910 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.047613 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.049044 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.052073 28812 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.052903 28812 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:09.053182 27586 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:09.053858 28812 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.055727 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.055924 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.056649 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.058411 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.059937 28913 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.060874 28913 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.063448 28913 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.064826 28913 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.065611 28913 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.068331 28913 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.034662 28911 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.071451 28913 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.072260 28913 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:09.074712 28910 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.080593 28911 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.083794 28894 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.089874 28894 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.093456 28894 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.084923 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.086160 28890 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:09.102783 28890 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.106320 28890 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.087357 28814 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.107349 28814 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.110160 28814 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.110930 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:09.113119 28814 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.114007 28814 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.116845 28814 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.119280 28890 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.120205 28890 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.123299 28890 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.124939 28782 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.123876 28890 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.130909 28890 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.135646 28878 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.136677 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.138233 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.143788 28804 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:08.505877 28832 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.144724 28804 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.145093 28832 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.148029 28893 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.149055 28893 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:09.150606 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.119s	user 0.031s	sys 0.030s
W20260709 12:18:09.151032 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.123s	user 0.031s	sys 0.030s
I20260709 12:18:09.156028 28832 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.157189 28833 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.160230 28833 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.165254 28833 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.166180 28833 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:09.158808 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 1.13473e+06us
I20260709 12:18:09.167382 28878 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.174525 28833 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.175233 28878 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.177024 28833 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.177340 28878 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.178329 28891 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.179198 28891 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.182155 28891 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.182945 28892 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:09.183633 28892 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.184623 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.187409 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.188309 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:09.190968 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.173652 28890 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.192566 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.193140 28885 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.193866 28913 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.193804 28885 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.196074 28832 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.196825 28832 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.201725 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.204313 28864 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.205349 28864 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.208218 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.203697 28905 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.209353 27586 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:09.209507 28905 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.211527 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.213454 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.214140 27583 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:09.216563 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.218428 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.218736 28810 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.209542 28812 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.219456 28810 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.152865 28893 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.223662 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.199262 28832 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.227910 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.230088 28804 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.230937 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.232730 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.235345 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.235736 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.236948 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.237668 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.237895 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.238299 27734 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.238603 27583 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:09.239537 28864 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.221236 28894 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.243916 28894 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.244509 28810 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.245702 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.249756 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.250375 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.250876 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.252168 28892 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.252808 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.221616 28833 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.259510 28833 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:09.264113 28884 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.356s	user 0.001s	sys 0.005s
I20260709 12:18:09.272989 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.273639 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.274675 28893 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.271903 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.276139 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:09.274765 28916 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.263305 28833 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.275954 28893 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.278096 28894 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.279371 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.227957 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.281468 27729 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:09.282352 28893 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.287529 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.288414 27729 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.286069 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.292428 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.293542 27730 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.292331 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.302062 28859 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:09.302340 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.303788 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.304605 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.298614 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.303293 28859 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.301950 28928 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.348567 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.348989 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.317929 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.321239 28782 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.321141 28864 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.321907 28906 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.323395 28860 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.324043 28882 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.332372 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.333628 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.335361 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:09.338281 28885 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.343822 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.349346 28928 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.348289 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.348695 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.349838 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.344614 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.366539 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:09.369674 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.368858 28885 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.372442 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.373047 28885 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.373921 28899 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.375663 27729 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.376052 28885 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.377221 28859 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.382247 28833 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.383141 28833 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.383513 28782 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.383980 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.385883 28833 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.386059 28782 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.386641 28833 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.387964 28804 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.388885 28804 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:07.904021 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:09.390136 28864 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.392884 28864 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.393157 28926 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.393833 28882 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.394137 28926 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.395840 28332 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:18:09.397442 28926 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.397572 28882 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:09.387138 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52003.4us
I20260709 12:18:09.365654 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.383219 27483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 118) took 2699 ms. Trace:
I20260709 12:18:09.406047 28893 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.383497 28832 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.406697 28893 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.407626 28832 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.408481 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.409673 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.409756 28893 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.410570 28832 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.411770 28860 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.413646 28893 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.414104 28893 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.414525 28893 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.418009 28905 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.419026 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.419185 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.420141 28334 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:09.421669 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.423121 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.404763 27483 rpcz_store.cc:276] 0709 12:18:06.683958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.687430 (+  3472us) service_pool.cc:224] Handling call
0709 12:18:09.382926 (+2695496us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2539059,"reported_tablets":1}
I20260709 12:18:09.424029 28333 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:09.425175 28925 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.425766 28925 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.426311 28925 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.426929 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.374948 28906 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.414891 28860 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.402787 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.427664 28334 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:09.431679 28804 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.453708 27487 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 232) took 2788 ms. Trace:
I20260709 12:18:09.454181 27487 rpcz_store.cc:276] 0709 12:18:06.665428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.665684 (+   256us) service_pool.cc:224] Handling call
0709 12:18:09.453476 (+2787792us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2672995,"reported_tablets":1}
I20260709 12:18:09.432049 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.455811 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.456074 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.433003 27737 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.434506 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.458161 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.435657 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.436080 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.436612 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.435674 27485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 217) took 1594 ms. Trace:
I20260709 12:18:09.461073 27485 rpcz_store.cc:276] 0709 12:18:07.841676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:07.911075 (+ 69399us) service_pool.cc:224] Handling call
0709 12:18:08.557872 (+646797us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a350f3a 00007f0ea0106e9c 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc 00007f0e9e6c5392 00007f0e9868348c 00007f0e986828c6
0709 12:18:09.435481 (+877609us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":669448,"reported_tablets":3,"spinlock_wait_cycles":80851712}
I20260709 12:18:09.436682 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.438040 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:09.462606 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.333s	user 0.007s	sys 0.019s
I20260709 12:18:09.438935 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.439142 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:09.463155 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.333s	user 0.007s	sys 0.019s
I20260709 12:18:09.442262 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.446094 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.435686 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.449594 27731 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.450459 28906 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.434443 28928 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.434698 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.474749 28894 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.475374 28894 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.475457 28809 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.475914 28894 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.475984 28809 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.476534 28809 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.472575 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.472724 27581 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:09.492314 28893 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:09.494661 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.36521e+06us
I20260709 12:18:09.495426 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.495662 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.497686 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.487955 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.508440 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.460289 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.501502 28918 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:09.512333 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.513796 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.514669 27658 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.451061 28833 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.518177 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.517417 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.521112 28337 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:09.522219 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.522473 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:09.522631 27610 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3820 after lost signal to thread 26411
I20260709 12:18:09.523092 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:09.523525 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:09.524067 27610 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9c80 after lost signal to thread 27521
I20260709 12:18:09.524248 27580 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
W20260709 12:18:09.525751 27610 debug-util.cc:398] Leaking SignalData structure 0x7b0800356ca0 after lost signal to thread 27613
I20260709 12:18:09.526333 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:09.526965 27610 debug-util.cc:398] Leaking SignalData structure 0x7b0800356400 after lost signal to thread 27688
I20260709 12:18:09.529343 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.510214 28918 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:09.520577 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.531092 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.533722 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:09.516489 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106948us
I20260709 12:18:09.536847 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.541487 27658 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:09.542706 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.543465 27664 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.544683 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.532824 28918 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:09.543064 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.549548 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:09.550697 28912 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.553506 28335 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:09.559715 28814 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.560253 28814 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:09.560616 28814 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.561571 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.562876 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.562704 27730 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.564178 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.564599 27485 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 1 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:09.567495 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.568746 28912 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.569211 28912 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.568293 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.568420 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.574494 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.576503 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:09.572732 27610 debug-util.cc:398] Leaking SignalData structure 0x7b0800292f80 after lost signal to thread 27762
I20260709 12:18:09.582557 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.588022 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.592675 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.589845 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.595520 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.592605 28331 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:09.597684 28899 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.598292 28899 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.598799 28899 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.599022 28814 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.607908 28814 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:09.608628 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.503s (SENT) [suppressed 3 similar messages]
I20260709 12:18:09.612895 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.615635 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.619684 27658 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:09.619804 28855 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.620398 28855 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.620920 28855 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.625835 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:09.626578 28331 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.628553 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:09.632570 28893 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.637616 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.639760 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.640177 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.640407 28336 raft_consensus.cc:1813] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Handling vote request from an unknown peer 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:09.641010 28336 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 2: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:09.641547 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.642206 28337 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
W20260709 12:18:09.647825 27610 debug-util.cc:398] Leaking SignalData structure 0x7b0800344900 after lost signal to thread 28361
I20260709 12:18:09.653734 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:09.648294 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.657248 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.601679 28925 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.621399 28809 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:09.662422 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49008.5us
I20260709 12:18:09.663699 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.667737 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.668277 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.668449 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.669414 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.675838 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.677354 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.678464 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.679750 28893 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.680045 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:09.680119 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 104264us
I20260709 12:18:09.681260 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.682301 28893 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.684633 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.686344 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.655097 28904 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.693259 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:09.691424 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.710839 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.712961 28833 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.714072 28812 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.714532 28812 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.714947 28812 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:18:09.719993 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111529us
I20260709 12:18:09.715762 28814 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:09.736137 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.737377 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.738137 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.727216 28913 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.738729 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.746054 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.746950 28809 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.747932 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.749100 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:09.749588 28912 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.744714 28913 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.750851 28913 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.732364 28894 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:09.758930 27732 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:09.766675 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.768163 28777 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.758246 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.769323 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.772017 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.772953 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:09.773442 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.773959 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.774092 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:09.774577 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.775182 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.775871 27729 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:09.776829 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.773341 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.780123 28925 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.783803 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.752043 28916 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:09.787571 28916 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.787984 28916 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:09.776028 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.771699 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.752099 28912 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:09.779975 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.805401 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.807312 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.810312 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.810999 28332 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:09.813350 27586 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:18:09.818634 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87387.3us
W20260709 12:18:09.820571 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110604us
I20260709 12:18:09.824920 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:09.826316 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.830216 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:09.831655 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.804961 27733 raft_consensus.cc:2955] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:09.789031 28899 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.818491 28931 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.824843 28890 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:18:09.838402 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.848650 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:09.849080 28855 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.852974 27738 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:09.853982 28899 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:09.855154 28894 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:09.860000 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.877511 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.860425 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.861070 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.861169 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.862315 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.863059 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.863374 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.866500 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.859915 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.871016 28912 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:09.872653 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:09.886328 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:09.893435 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:09.896350 27737 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:09.897156 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.900194 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:09.900827 28337 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.903228 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.903841 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.906692 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.907878 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.908771 28910 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.908917 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.909365 28910 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.909821 28910 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:09.910104 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.910959 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.911461 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.912016 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.913259 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.878002 28913 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:09.915140 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.917972 27736 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:09.919168 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:09.919883 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47319.5us
I20260709 12:18:09.895860 28331 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:09.921283 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:09.906883 28777 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:09.896459 27657 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:09.896807 28899 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:18:09.908820 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.932176 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.923036 28809 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:09.937379 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:09.941241 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.941397 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.942351 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.942698 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:09.942778 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:09.910840 28864 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 12:18:09.949568 28918 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:09.950197 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:09.939551 28809 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:09.957816 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.958160 28881 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:09.965746 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.966087 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.966490 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.967021 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:09.967701 28937 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:18:09.956180 28777 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.946897 28940 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:09.967706 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.982896 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.963804 28809 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:09.987955 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.977216 27586 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:09.956967 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:09.981699 28777 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:09.984724 28940 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:09.977419 28881 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:10.003998 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:10.006457 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58852.7us
I20260709 12:18:10.007656 28833 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:10.008080 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.008786 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:10.009320 28333 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.011569 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:10.012748 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:10.015720 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.999085 28855 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:10.026480 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97125.6us
I20260709 12:18:10.029160 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:10.002221 28777 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.035364 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:10.014353 28940 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.013911 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:09.998824 28918 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:10.034976 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:10.045380 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:10.043035 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.013828 28913 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.045969 27582 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.049682 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:10.049577 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:10.040191 28331 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.051280 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.055207 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:10.057986 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.059485 28929 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:10.059938 28929 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:10.060295 28929 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:10.061743 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.063436 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:10.063727 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.063867 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.065537 28329 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.065864 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.045267 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:10.068235 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.069550 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.067365 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.064911 28881 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.056882 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.048238 27656 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:10.054462 28855 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:10.094058 28855 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:10.084357 28931 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.085758 28936 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.076040 28913 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.106506 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:10.086524 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.085541 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:18:10.117674 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68172.8us
W20260709 12:18:10.118975 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62880.7us
I20260709 12:18:10.102201 28936 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:10.103211 28913 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:10.103098 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.121161 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.121284 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:10.086246 28916 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.103205 28925 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:18:10.130388 28910 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.132273 28910 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:10.128996 28814 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:10.135936 28814 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:10.107496 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.115140 28838 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:10.136983 28814 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:10.137611 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.139676 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:10.140328 28331 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.120905 28936 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.124189 28913 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.142731 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:10.146983 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.147526 28833 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.148852 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.150117 28924 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:10.150218 28916 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:10.150118 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.161656 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.141507 28918 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.121178 28812 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.184152 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:10.193315 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 52253.9us
I20260709 12:18:10.193885 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:10.166755 28893 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.195456 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:10.169947 28926 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:10.180125 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.197805 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:10.198801 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.174976 28912 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.199870 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:10.199944 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:10.204696 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.220124 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:10.174855 28940 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.202973 28924 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.222925 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.203370 28912 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:10.216598 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:10.173331 28855 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.231304 28853 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.233399 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.243023 27585 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20260709 12:18:10.235405 27729 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.244149 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.244251 28924 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.244383 28777 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.245278 28949 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.245664 28949 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:10.246050 28949 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.247241 27582 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.249164 27582 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.250043 27585 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:10.256656 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.269892 28929 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.270987 28954 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:10.271747 28853 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.283599 28954 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:10.284163 28954 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:10.271967 28951 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:18:10.274618 28913 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.289723 28913 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:10.281409 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58571.6us
I20260709 12:18:10.229566 28912 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.295084 28929 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:10.297057 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 211692us
I20260709 12:18:10.300590 28777 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:10.282090 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 141549us
I20260709 12:18:10.314591 27585 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:10.316349 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:10.318409 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71170.7us
I20260709 12:18:10.319454 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.243166 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.312332 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:10.303782 28918 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:10.302771 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:10.332499 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.329178 28944 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.343739 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.348284 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:10.349231 28931 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:10.349821 28931 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:10.350450 28912 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.331763 28918 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:10.360410 27582 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.360539 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:10.361545 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:10.361634 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.362949 27584 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.365267 28337 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.334267 28924 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.382184 28809 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.340703 28960 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.345417 28812 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.345269 28944 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:10.395758 28333 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:10.350692 28931 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:10.343569 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100466us
I20260709 12:18:10.396132 28957 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 12:18:10.397185 28912 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.399926 27583 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:10.384406 28940 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.357407 28918 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:10.395923 28809 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:10.404879 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:10.400301 28777 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.401022 28960 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.424772 28948 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:10.424932 28963 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.402509 28944 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.425274 28948 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:10.425817 28948 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:18:10.428395 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 61554.5us
I20260709 12:18:10.316155 28332 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:10.403843 28925 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.417630 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.431418 27661 raft_consensus.cc:606] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:10.435820 28964 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:10.436389 28963 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:10.438514 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:10.438778 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:10.448777 27585 raft_consensus.cc:2955] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:10.461026 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:10.427350 28777 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:10.409839 28926 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:18:10.441006 28961 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:18:10.451970 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:10.444792 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98339.6us
I20260709 12:18:10.472667 28334 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.473747 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 310b1b37298244c0bd5bbe2b0c7328d7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:18:10.460811 28893 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:18:10.476333 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66737.1us
I20260709 12:18:10.466609 28777 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.404798 28936 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:10.482357 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96835.2us
I20260709 12:18:10.480635 28890 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:18:10.497712 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138361us
I20260709 12:18:10.499507 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.500262 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.496922 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.503469 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:10.504269 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 142666us
I20260709 12:18:10.478236 28913 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:18:10.508117 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.508013 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.512790 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.487692 28954 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.513173 28899 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:10.602831 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86719.1us
I20260709 12:18:10.604187 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:10.627593 28969 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:18:10.628396 28890 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:10.629526 28890 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.518729 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:10.636171 28972 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.118s
I20260709 12:18:10.635165 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.504880 28961 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
I20260709 12:18:10.609190 28918 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.664880 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:10.677613 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 175578us
W20260709 12:18:10.679314 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 196919us
I20260709 12:18:10.652951 28913 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:10.681460 28809 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:10.723585 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.734176 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.737566 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.676769 28970 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
W20260709 12:18:10.773449 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 267711us
I20260709 12:18:10.774282 28970 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:10.774698 28970 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:10.775923 28968 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:10.775823 28970 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.777189 28970 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.777336 28950 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:10.777786 28950 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:10.816864 27583 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:10.744122 27580 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:10.738651 28812 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.732053 28904 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.822734 27585 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:18:10.822640 27583 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:10.779392 28913 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:10.830714 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93253us
I20260709 12:18:10.832002 27582 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:10.806807 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 303654us
I20260709 12:18:10.838198 27584 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.838240 27581 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.834396 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.839047 27588 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.840366 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.842075 27587 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.817559 28936 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.841581 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.841602 27583 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.846671 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.848042 27583 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.823303 28964 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:18:10.859711 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.828249 28950 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.865851 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.854635 28904 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.869453 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:10.871502 28812 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.873049 28912 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.875301 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.857424 28957 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:10.896155 28957 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:10.897583 28957 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:10.899817 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.865561 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:10.901546 28949 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.655s	user 0.012s	sys 0.000s
I20260709 12:18:10.901878 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.853662 28913 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.905650 28952 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.899008 28957 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:10.905463 28810 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.909286 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.883517 28936 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.867276 28954 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:10.852595 28892 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:18:10.910144 28913 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:10.916635 28925 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.916911 28918 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.917454 28925 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.856467 28979 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.233s
I20260709 12:18:10.919231 28925 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:10.919612 28925 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:10.920547 28925 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.921793 28925 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:10.924162 28337 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:10.924540 28331 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.784s	user 0.000s	sys 0.013s
I20260709 12:18:10.924968 28331 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:10.925510 28833 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.928242 28333 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.929419 28777 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:10.930828 28944 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.505s	user 0.007s	sys 0.003s
W20260709 12:18:10.933363 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:10.934739 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.931305 28944 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.938231 28966 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:10.939157 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.902246 28949 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.939692 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:10.948946 28810 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.950255 28949 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:10.950469 28336 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:10.952329 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.955891 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.956957 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.963418 28974 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.963794 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.977490 28967 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:10.979301 28981 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:10.980162 28981 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:10.981235 28981 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:10.987382 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.926784 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.991044 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.982142 27583 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:10.992194 28777 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:10.995152 28949 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:18:10.996206 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72556.7us
W20260709 12:18:10.997555 28948 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.571s	user 0.006s	sys 0.005s
I20260709 12:18:11.000947 28948 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:10.984225 28966 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.004235 27552 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:11.008848 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:10.991185 28967 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.012812 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:11.014114 28974 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.015657 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:11.017956 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:11.022058 28810 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.023969 28959 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.024590 28853 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:11.027765 28853 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.009598 28949 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.028954 28853 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.030565 28853 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:11.025030 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.025766 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.027189 28931 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:11.008208 28966 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.037413 28944 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.051493 28948 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.021291 28974 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.046582 28946 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.073690 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.075932 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:11.076856 28972 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.079749 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.100518 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.101073 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:11.101054 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.012s	sys 0.009s
I20260709 12:18:11.098179 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.107901 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:11.119266 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.012s	sys 0.009s
W20260709 12:18:11.119903 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 612337us
I20260709 12:18:11.115203 28838 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:11.120641 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.119781 28952 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.121881 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.123502 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:11.125460 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 286984us
I20260709 12:18:11.127403 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.115823 28972 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.133067 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.134202 27485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 218) took 1576 ms. Trace:
I20260709 12:18:11.123118 28952 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.138108 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.120893 28946 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.128295 28918 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.143970 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.130872 28944 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.133862 28957 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.126641 28838 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.141332 28809 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:18:11.190618 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57638us
I20260709 12:18:11.191838 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:11.138679 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 297162us
W20260709 12:18:11.193437 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49810.9us
I20260709 12:18:11.154539 28964 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.197162 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:11.210271 27659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:11.196765 28918 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.216487 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.207841 28929 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.218636 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:11.219280 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.134712 27485 rpcz_store.cc:276] 0709 12:18:09.558246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.558407 (+   161us) service_pool.cc:224] Handling call
0709 12:18:11.134103 (+1575696us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:09.674093 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:09.675040 (+   947us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:09.675106 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:18:09.675156 (+    50us) tablet.cc:662] Decoding operations
0709 12:18:09.675721 (+   565us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:09.675841 (+   120us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:09.675888 (+    47us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:18:09.676250 (+   362us) tablet.cc:701] Row locks acquired
0709 12:18:09.676310 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:18:09.676517 (+   207us) write_op.cc:270] Start()
0709 12:18:09.676715 (+   198us) write_op.cc:276] Timestamp: P: 1783599489676433 usec, L: 0
0709 12:18:09.676769 (+    54us) op_driver.cc:348] REPLICATION: starting
0709 12:18:09.677763 (+   994us) log.cc:844] Serialized 533 byte log entry
0709 12:18:10.101366 (+423603us) op_driver.cc:464] REPLICATION: finished
0709 12:18:10.211205 (+109839us) write_op.cc:301] APPLY: starting
0709 12:18:10.221047 (+  9842us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:10.230645 (+  9598us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:10.280924 (+ 50279us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:10.301839 (+ 20915us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:10.302107 (+   268us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:18:10.302182 (+    75us) write_op.cc:312] APPLY: finished
0709 12:18:10.344923 (+ 42741us) log.cc:844] Serialized 23 byte log entry
0709 12:18:10.392208 (+ 47285us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:10.402219 (+ 10011us) write_op.cc:454] Released schema lock
0709 12:18:10.410406 (+  8187us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:10.458493 (+ 48087us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:10.458530 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3383,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":71354,"apply.run_cpu_time_us":3325,"apply.run_wall_time_us":245713,"num_ops":1,"prepare.queue_time_us":86974,"prepare.run_cpu_time_us":40323,"prepare.run_wall_time_us":90248,"raft.queue_time_us":112499,"raft.run_cpu_time_us":16662,"raft.run_wall_time_us":91037,"replication_time_us":407758,"thread_start_us":296639,"threads_started":4,"wal-append.queue_time_us":193736}]]}
I20260709 12:18:11.210264 28970 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
W20260709 12:18:11.226506 28332 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.730s	user 0.012s	sys 0.001s
I20260709 12:18:11.230209 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:11.217960 28777 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260709 12:18:11.239382 27661 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.217720 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:11.236222 28964 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.236289 28949 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:11.282902 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 68392.3us
I20260709 12:18:11.238096 28929 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.219730 28954 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:18:11.217815 28983 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260709 12:18:11.217557 28968 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:18:11.286912 28925 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.217993 28972 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.217165 28946 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.307817 28963 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.308346 28963 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:11.310909 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92354.9us
W20260709 12:18:11.290939 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76896.6us
I20260709 12:18:11.317668 28964 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.318131 28964 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:11.326638 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:11.331192 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:11.311906 28971 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.333456 27584 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.309340 28963 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.337196 27576 tablet_service.cc:1558] Processing DeleteTablet for tablet 22597fcae204491f84839e680c083ecc with delete_type TABLET_DATA_TOMBSTONED (TS 3f90188458da46f6ba138d5f82c55b69 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39844
I20260709 12:18:11.337962 28963 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.327891 28946 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.334012 28838 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:11.339718 28972 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:18:11.373977 28980 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.101s	user 0.010s	sys 0.000s
I20260709 12:18:11.391985 27661 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
W20260709 12:18:11.393088 28904 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.500s	user 0.010s	sys 0.002s
I20260709 12:18:11.393651 28904 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.382108 28950 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.319056 28964 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:11.404623 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 185395us
I20260709 12:18:11.405484 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.397215 28331 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.365484 28975 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:18:11.413684 27584 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.397135 28960 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:18:11.415591 28984 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:11.425738 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.416690 28954 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.430701 27627 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.433250 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.434784 28987 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.423837 28964 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.437615 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.443794 28995 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20260709 12:18:11.445659 28940 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260709 12:18:11.468730 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 107427us
I20260709 12:18:11.433001 28983 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.469772 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:11.468336 28777 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.472280 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.486114 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.445091 28960 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.496503 28993 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.468000 28989 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:11.505267 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.471540 28777 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.469132 28970 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.509994 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:11.512517 27588 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:11.474805 28995 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:11.510593 28998 tablet_replica.cc:333] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:18:11.508912 28983 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.508894 28853 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.522392 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.518967 28995 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.526053 28995 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.525910 28962 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:18:11.521826 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.521086 28329 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:11.519008 27655 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:11.532392 27656 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:11.528709 28954 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.544436 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.548169 27662 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.581869 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 227) took 1464 ms. Trace:
I20260709 12:18:11.544112 28987 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:18:11.570611 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.520884 28989 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.582533 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.590801 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:11.600461 28957 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.600956 28957 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.603152 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 261) took 1583 ms. Trace:
I20260709 12:18:11.603415 28978 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.603720 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.603685 28334 rpcz_store.cc:276] 0709 12:18:10.019856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.020772 (+   916us) service_pool.cc:224] Handling call
0709 12:18:10.403677 (+382905us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:10.489695 (+ 86018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:10.955654 (+465959us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b54003ed0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:11.602909 (+647255us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54003ed0d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:11.603081 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":931705600}
I20260709 12:18:11.605226 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.603896 28978 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:11.616215 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:11.618005 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:11.583570 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.629086 27659 raft_consensus.cc:606] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:11.629686 27659 raft_consensus.cc:614] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:11.632714 27657 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
W20260709 12:18:11.634451 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 128926us
I20260709 12:18:11.616166 28995 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.642905 28924 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:11.678795 27656 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.683670 27662 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20260709 12:18:11.684199 27662 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 2
I20260709 12:18:11.684499 27662 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260709 12:18:11.684864 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.685819 28864 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.686412 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.687493 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:11.688395 28334 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:11.688567 28894 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:11.656395 28924 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.652887 28978 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.692028 28864 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:11.691087 28330 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.684939 27662 raft_consensus.cc:2917] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20260709 12:18:11.693353 27662 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:11.582342 28332 rpcz_store.cc:276] 0709 12:18:10.117316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.118627 (+  1311us) service_pool.cc:224] Handling call
0709 12:18:10.228242 (+109615us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:10.255534 (+ 27292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:10.303294 (+ 47760us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:11.239651 (+936357us) log.cc:844] Serialized 14 byte log entry
0709 12:18:11.539432 (+299781us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:11.547016 (+  7584us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:11.547931 (+   915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:11.552340 (+  4409us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:11.581780 (+ 29440us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":423,"raft.queue_time_us":1599,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7837,"spinlock_wait_cycles":2750208,"thread_start_us":284248,"threads_started":1,"wal-append.queue_time_us":286285}
I20260709 12:18:11.694368 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:11.691728 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1095) took 1185 ms. Trace:
I20260709 12:18:11.700987 28994 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:11.740161 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.740221 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.741110 27735 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.743026 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.749637 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.750391 27736 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.743808 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.734640 28950 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.737092 28963 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:11.768846 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.773432 28963 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.775219 28970 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.775624 28970 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.776410 27736 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:11.740568 28994 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:11.772805 28924 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.778782 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.769001 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.776551 28970 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.775658 28963 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.783349 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.780575 27738 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.780628 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:11.781606 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:11.784446 27583 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:11.771162 28995 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.784344 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:11.799620 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:11.779553 28981 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.021s	sys 0.038s
I20260709 12:18:11.782020 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 263) took 1352 ms. Trace:
I20260709 12:18:11.701778 27657 rpcz_store.cc:276] 0709 12:18:10.497963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.501522 (+  3559us) service_pool.cc:224] Handling call
0709 12:18:11.518877 (+1017355us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:11.682972 (+164095us) spinlock_profiling.cc:243] Waited 99.8 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:11.683236 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":236615936}
I20260709 12:18:11.780364 28950 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:11.803378 28981 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.024s	sys 0.045s
I20260709 12:18:11.802837 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.807106 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:11.784349 28954 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:18:11.814258 27735 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:11.803247 28924 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:11.828621 28924 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.829475 27738 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:11.830914 28924 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.831503 28961 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.831936 28961 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.779735 28994 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:11.832600 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:11.832846 28961 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.834262 28961 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.834363 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:11.838313 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:11.838665 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:11.838933 28994 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.839496 28960 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:11.839694 28925 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.839985 28960 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.840265 28925 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.819005 27656 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:11.840960 28960 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.822664 28986 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:18:11.826748 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.822702 28330 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20260709 12:18:11.814826 27586 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.847679 28330 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 2
I20260709 12:18:11.847970 28330 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260709 12:18:11.844619 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.845570 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:11.848268 28330 raft_consensus.cc:2917] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20260709 12:18:11.848788 28330 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:11.820660 28950 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.842499 28925 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.824635 28333 rpcz_store.cc:276] 0709 12:18:10.428308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.434357 (+  6049us) service_pool.cc:224] Handling call
0709 12:18:10.928027 (+493670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:10.928717 (+   690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:11.763453 (+834736us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54006108d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:11.780323 (+ 16870us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134334720}
I20260709 12:18:11.847232 28987 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.848829 28993 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.845834 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:11.852537 27735 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:11.853850 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:11.855739 28950 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:11.854887 28994 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:11.856135 28950 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.856576 28950 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.845634 28968 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:11.859650 28968 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:11.860466 28968 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.861833 28968 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.861967 28987 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:11.862731 28970 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.869727 28986 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.870463 28986 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.878031 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:11.879194 27585 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:11.900061 27735 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:11.903057 28986 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:11.903820 27587 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:11.903959 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.853691 28958 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 12:18:11.910190 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:11.916124 27586 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:11.917168 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:11.918421 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:11.920712 28943 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:11.920809 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:11.877581 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:11.922636 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.922762 27738 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:11.814127 28777 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.924592 28986 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:11.921679 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:11.933992 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:11.934069 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:11.928895 28916 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:11.947995 28943 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:11.935808 28986 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:11.952471 28986 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.947456 29009 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:11.961903 29009 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.962400 29009 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:11.967758 28925 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:11.968327 28925 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:11.968834 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:11.970467 28925 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:11.932053 28892 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:18:11.985219 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:11.986120 27662 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:11.981812 28943 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.989694 28975 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:11.989683 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:11.992429 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:12.004848 27662 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:11.945909 28950 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:11.983525 29013 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.014781 28924 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:18:12.025552 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 105342us
I20260709 12:18:11.990736 28975 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.029414 27657 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.033913 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:12.034595 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 113876us
I20260709 12:18:12.036664 27664 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.040868 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:12.041808 28899 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:12.042364 28899 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.043344 28899 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.044494 28899 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:12.024685 28924 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:12.048491 28916 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.051783 28950 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.052256 28916 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.053755 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:12.054368 27738 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.054615 28330 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:12.056321 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.031136 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:12.059993 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:12.066423 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.067996 28958 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.061739 28959 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:12.075448 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:12.076074 27662 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.081832 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:12.089210 27659 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:12.093330 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.090513 28995 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:12.113683 28918 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.115161 28925 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:12.091068 28959 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.115501 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.121449 29016 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:12.080446 28957 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.128113 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.130504 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.115671 28925 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:12.156641 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.158366 29015 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:12.123898 28995 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:12.091284 28924 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.126926 28918 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:12.149045 28929 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.166025 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.174018 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.161439 28925 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:12.170393 28957 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.170992 28971 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:18:12.181341 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.155882 29009 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.184185 29009 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.171907 28929 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:12.172497 29015 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.187700 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.179371 29017 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.188177 28981 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:18:12.192126 28971 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.186000 28924 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:12.200824 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:12.212761 27584 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.196369 28959 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:12.239264 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165526us
W20260709 12:18:12.215281 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 198242us
W20260709 12:18:12.242427 27701 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.553s (SENT)
I20260709 12:18:12.246949 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.208257 28929 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.247443 29014 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:12.206589 28995 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.239563 27586 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:12.264189 27701 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.744s (SENT)
W20260709 12:18:12.265199 29018 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:18:12.264921 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.268395 28894 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.259676 29004 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.256309 28943 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:18:12.245850 29015 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.271037 28925 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.283711 28925 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.286419 28994 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:12.299796 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.331238 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.335726 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.290578 29014 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.291271 28916 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:18:12.297328 28986 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.355595 28332 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.337967 28994 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:12.386962 28916 consensus_queue.cc:1243] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:12.361529 28904 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.368510 27661 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.337534 29004 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.387430 28894 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:12.342501 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51895.1us
I20260709 12:18:12.364142 28978 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.395416 28994 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:12.399184 28904 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.369103 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.411867 28994 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.412388 28925 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:18:12.412570 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52295.4us
I20260709 12:18:12.414127 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.414990 28925 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:12.421384 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.368919 29014 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.421574 28975 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:12.423051 28989 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:12.454313 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 112499us
I20260709 12:18:12.419489 28936 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:18:12.456890 28986 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.454569 27580 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.464053 28812 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.464461 28812 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.465378 28812 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.471637 28984 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:12.472631 28998 raft_consensus.cc:2243] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:12.472910 28984 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.473361 28984 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.474289 28984 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.476125 28984 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.477355 28971 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.477806 28971 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.484449 28989 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:12.485363 28988 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:12.486385 28988 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.486745 28988 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.489500 27482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 233) took 2921 ms. Trace:
I20260709 12:18:12.487586 28988 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.491372 28988 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.494776 28998 raft_consensus.cc:2272] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:12.429883 28904 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.495783 28996 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:12.496567 28996 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:12.497447 28904 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.489951 27482 rpcz_store.cc:276] 0709 12:18:09.568126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.568291 (+   165us) service_pool.cc:224] Handling call
0709 12:18:12.489322 (+2921031us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":78208}
I20260709 12:18:12.500332 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:12.500941 27659 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:12.502514 28924 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.503026 28924 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.503297 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:12.506319 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1249) took 1017 ms. Trace:
I20260709 12:18:12.507046 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:12.510056 28998 ts_tablet_manager.cc:1916] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:12.506752 27584 rpcz_store.cc:276] 0709 12:18:11.489152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.490538 (+  1386us) service_pool.cc:224] Handling call
0709 12:18:12.206355 (+715817us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:12.206541 (+   186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:12.219708 (+ 13167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.506080 (+286372us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:12.506258 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23077504}
I20260709 12:18:12.478658 28971 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:12.524082 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.528940 27659 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:12.530916 27584 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.531625 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:12.534603 29032 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.173s	user 0.002s	sys 0.000s
I20260709 12:18:12.515383 28777 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.529983 27664 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
W20260709 12:18:12.598438 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:12.607564 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.485775 28812 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.616832 27658 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:12.619490 28986 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:12.620385 29033 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:18:12.620090 28986 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.622326 28986 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:12.622409 28924 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.547297 28957 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.630136 28957 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.638391 28332 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.654817 28335 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:12.658895 28924 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.659646 28924 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.662377 28924 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.536309 29000 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20260709 12:18:12.664295 28946 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.680603 28946 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.683177 28946 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:12.664680 28853 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:12.683746 27738 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.683998 28853 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.665906 28988 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:18:12.663914 29029 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.257s	user 0.005s	sys 0.005s
I20260709 12:18:12.662997 28964 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.696933 28964 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.632263 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:12.668224 28943 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:12.665140 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:12.667470 28820 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:12.700080 28943 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.666358 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:12.669674 28948 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.688585 27735 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.702535 28943 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.702574 28948 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.700227 27587 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:12.704249 28948 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.705526 28948 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.688544 28994 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.708909 27732 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.709475 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.711508 28994 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:12.684904 28929 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.692503 27582 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.693113 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:12.693305 28853 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.694380 27487 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 227) took 3091 ms. Trace:
I20260709 12:18:12.694512 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:12.721880 27730 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.687988 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.696347 27483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 119) took 3090 ms. Trace:
I20260709 12:18:12.723656 28943 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:18:12.723835 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46124.4us
I20260709 12:18:12.723446 27483 rpcz_store.cc:276] 0709 12:18:09.606480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.606668 (+   188us) service_pool.cc:224] Handling call
0709 12:18:09.624519 (+ 17851us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b54003c9a28. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea053a331 00007f0ea05353fc 00007f0ea0106e84 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc 00007f0e9e6c5392 00007f0e9868348c 00007f0e986828c6
0709 12:18:12.696280 (+3071761us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":486653,"reported_tablets":3,"spinlock_wait_cycles":8060800}
I20260709 12:18:12.726032 28943 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.726441 28943 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:12.727634 28943 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.700289 28964 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:12.721903 27487 rpcz_store.cc:276] 0709 12:18:09.603196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.603433 (+   237us) service_pool.cc:224] Handling call
0709 12:18:12.694324 (+3090891us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1103379,"reported_tablets":3}
I20260709 12:18:12.728914 28943 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.740679 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:12.742601 28964 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.743366 28964 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.746313 28924 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:12.746933 28924 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.747483 28924 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:12.757074 29004 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.762831 28957 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.762980 28984 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.764109 28984 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.764674 27738 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:12.764959 28984 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.766105 28984 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.769783 29017 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:18:12.770916 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75818.1us
I20260709 12:18:12.773420 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:12.780099 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:12.790035 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:12.791759 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87559.6us
I20260709 12:18:12.792768 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:12.767611 28971 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:18:12.798617 29030 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.376s	user 0.000s	sys 0.007s
I20260709 12:18:12.771123 28964 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.810665 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.798849 28957 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:12.799382 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:12.818517 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46456.8us
I20260709 12:18:12.803251 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:12.822887 27660 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:12.816160 28957 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.822883 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:12.846978 27661 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:12.828801 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:12.847728 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:12.848409 28986 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:12.847769 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:12.848919 28986 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:12.849330 27659 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:12.850314 28998 ts_tablet_manager.cc:1929] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:18:12.831699 28855 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.835109 27581 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:12.837087 27656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:12.842231 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:12.843295 28892 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:12.843986 28988 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.844995 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:12.849958 28986 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.830039 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:12.874193 28328 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:12.844172 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:12.854911 27735 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.877303 28952 raft_consensus.cc:1064] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:12.876660 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.878185 28925 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.872605 28332 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:18:12.871893 27656 raft_consensus.cc:606] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.880486 27656 raft_consensus.cc:614] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:12.882475 28952 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.872274 28988 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.852689 28957 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:12.887665 28988 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.888765 28988 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.875546 28892 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:12.902258 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.901851 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.902819 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.903860 27738 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:12.856019 28981 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260709 12:18:12.906265 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.909327 27738 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:12.874162 28998 log.cc:1199] T 22597fcae204491f84839e680c083ecc P 3f90188458da46f6ba138d5f82c55b69: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/wals/22597fcae204491f84839e680c083ecc
I20260709 12:18:12.868534 28894 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:18:12.907773 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.907995 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.897248 28892 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:12.914425 28335 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:12.916028 28855 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:12.918215 28892 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.919476 28892 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.905707 28943 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.907366 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:12.922921 28950 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.923524 28950 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.900180 28924 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.931847 28924 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:12.932955 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:12.933230 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.933629 27584 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:12.933828 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:12.933974 28978 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:12.934556 27730 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:12.904579 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:12.935837 29036 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:12.936378 29036 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.925822 28950 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:12.935287 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.949096 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.950977 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.932827 28924 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:12.980224 28924 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.980607 28924 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:12.982004 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:12.983150 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:12.983731 27660 raft_consensus.cc:606] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:12.984058 27660 raft_consensus.cc:614] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:12.985006 28978 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.985502 28978 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:12.988262 28994 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:12.988667 28994 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:12.989014 28994 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:12.994910 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.996024 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:12.996909 28853 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:12.997274 28853 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.997598 28853 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:13.001371 27588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:13.001698 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:13.001984 27588 raft_consensus.cc:606] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:13.002743 28950 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:13.003208 28950 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.003608 28950 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:13.004832 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:18:13.006518 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:13.006572 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:13.007444 27586 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:12.939411 29036 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:13.010818 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:13.003629 28978 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:13.017359 27733 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:13.018141 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:13.020267 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:13.041918 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:13.039942 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:13.043047 27733 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:18:13.043879 27582 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:13.044478 27584 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:13.038077 28981 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:18:13.047188 28940 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.047732 28940 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.016161 27731 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:13.049916 28940 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:13.035507 28964 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:13.052649 28894 raft_consensus.cc:993] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:18:13.052951 28964 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.053202 28894 raft_consensus.cc:1081] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:18:13.053399 28964 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:13.054273 28924 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.054536 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:13.055361 28333 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:13.056257 28929 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:13.056777 28929 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.058926 28929 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:13.057451 27588 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:13.061491 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:13.062727 27733 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:13.064307 28995 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:13.064839 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:13.065483 27663 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:18:13.064320 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.049329 28994 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:12.919175 28998 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39844 (request call id 74) took 1600 ms. Trace:
I20260709 12:18:13.073259 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:13.065599 27660 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:13.064749 28995 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.093962 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:13.096676 27736 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:13.100104 27589 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:18:13.529328 29034 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:13.307523 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.336448 28924 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:13.326051 27655 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:50014
I20260709 12:18:13.076869 27730 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:13.533970 29036 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:13.129818 27735 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:13.100612 29015 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:13.539042 28964 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:13.247253 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116566us
I20260709 12:18:13.535672 28333 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:18:13.335477 28853 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.246803 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:13.579841 27585 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:13.239658 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:13.100199 28995 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:13.129580 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:13.541133 28913 raft_consensus.cc:993] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
W20260709 12:18:13.595755 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 506685us
I20260709 12:18:13.596709 28986 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:13.597241 28986 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.529877 29034 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.397567 28910 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.545728 29015 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.457671 28929 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.084492 28998 rpcz_store.cc:276] 0709 12:18:11.318675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.336886 (+ 18211us) service_pool.cc:224] Handling call
0709 12:18:11.429544 (+ 92658us) ts_tablet_manager.cc:1151] Deleting tablet 22597fcae204491f84839e680c083ecc
0709 12:18:11.444517 (+ 14973us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:18:12.829994 (+1385477us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:12.850170 (+ 20176us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:12.919073 (+ 68903us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":964720,"tablet-delete.queue_time_us":76638,"thread_start_us":45656,"threads_started":1}
I20260709 12:18:13.548415 27587 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:18:13.227280 29013 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.308s
I20260709 12:18:13.553897 27658 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:18:13.555544 27735 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:13.572840 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:13.573568 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:13.129670 28952 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:13.576102 28812 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.549187 28820 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:13.170082 28958 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:13.286360 28936 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.319s
I20260709 12:18:13.588171 28328 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:13.557348 27655 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:13.589495 29051 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.659147 28777 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:13.592062 29050 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:13.594165 29044 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.277s	user 0.003s	sys 0.005s
I20260709 12:18:13.600756 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:13.541045 27661 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.042s	sys 0.027s
I20260709 12:18:13.618407 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:13.634641 27663 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:13.653487 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 80742.9us
I20260709 12:18:13.655714 27582 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:13.693431 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1366) took 1161 ms. Trace:
I20260709 12:18:13.548935 29036 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.658818 28991 raft_consensus.cc:1081] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Signalling peer 4aa1523b9af84c3db99ba394e2b9aa9a to start an election
I20260709 12:18:13.599260 28986 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:13.672727 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.591598 29052 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.659036 28820 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.659226 28890 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:13.693820 27583 rpcz_store.cc:276] 0709 12:18:12.521319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.521493 (+   174us) service_pool.cc:224] Handling call
0709 12:18:13.682351 (+1160858us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54000bd6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:13.682463 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1118814080}
I20260709 12:18:13.676146 27655 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:13.682988 28924 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:13.655890 28910 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.659968 28952 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.720872 28910 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:13.721457 28952 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:13.708884 29036 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:13.710233 27580 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:18:13.691745 27661 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.042s	sys 0.027s
I20260709 12:18:13.654945 29054 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260709 12:18:13.689252 29051 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:13.716238 27729 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.659s	user 0.007s	sys 0.003s
I20260709 12:18:13.669461 28812 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.717124 28853 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:13.717718 28957 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.830s	user 0.010s	sys 0.000s
I20260709 12:18:13.717932 28995 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.661674 28929 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.712332 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:13.674369 28958 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.693079 29050 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:13.743634 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56945.1us
W20260709 12:18:13.787928 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 205526us
I20260709 12:18:13.706207 28943 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.704385 28972 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:13.789126 29055 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:18:13.794477 28950 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.790s	user 0.008s	sys 0.003s
I20260709 12:18:13.741070 28924 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:13.947337 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:13.739197 28986 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:13.744587 28964 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:13.799431 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:13.766811 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67532.6us
I20260709 12:18:13.787957 29013 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:13.892390 28890 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:13.978133 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:13.950273 27729 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:13.969043 28853 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:13.968873 29040 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:18:14.027921 27733 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
W20260709 12:18:14.028504 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46953.8us
I20260709 12:18:14.030452 27702 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:14.039513 27702 leader_election.cc:403] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:13.956956 29063 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:13.772717 28855 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.204s
I20260709 12:18:13.957507 29057 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:13.980211 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:13.960112 28910 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260709 12:18:13.966372 29054 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:13.955029 28957 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:13.976207 29000 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.055125 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1187) took 1039 ms. Trace:
W20260709 12:18:14.042109 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71610.4us
I20260709 12:18:14.057461 27704 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:13.970732 28913 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:13.978736 29058 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.271s	user 0.002s	sys 0.002s
I20260709 12:18:13.810462 29061 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.182s
I20260709 12:18:13.705610 29052 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.956947 28972 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.961177 28943 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.070133 28913 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.072252 28943 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.048223 28958 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.073841 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.048305 28924 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.049463 28929 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.049521 29056 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:14.050487 29051 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.050690 28812 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.030944 27657 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:13.982208 28986 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.053020 27655 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.971201 28950 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:13.938305 29062 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.231s
I20260709 12:18:14.055701 27738 rpcz_store.cc:276] 0709 12:18:13.015242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.016457 (+  1215us) service_pool.cc:224] Handling call
0709 12:18:13.016786 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:13.017110 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:13.017158 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:13.035443 (+ 18285us) log.cc:844] Serialized 26 byte log entry
0709 12:18:13.559470 (+524027us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:13.559508 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:13.559571 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:14.054869 (+495298us) raft_consensus.cc:1685] finished
0709 12:18:14.054899 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:14.055055 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1454,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":19058,"thread_start_us":388354,"threads_started":1,"wal-append.queue_time_us":588098}
I20260709 12:18:14.039296 28964 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.123970 27655 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.124620 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:50014 (request call id 1288) took 1058 ms. Trace:
I20260709 12:18:14.124888 27664 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:14.053002 29054 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.125121 27655 rpcz_store.cc:276] 0709 12:18:13.065797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.325825 (+260028us) service_pool.cc:224] Handling call
0709 12:18:14.124521 (+798696us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:14.125825 27659 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:14.124313 28950 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.126471 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.055543 29000 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:14.127190 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68824.6us
I20260709 12:18:14.115487 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:14.117022 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 136892us
I20260709 12:18:14.118057 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:14.127293 27655 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.134690 27588 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:14.039088 29015 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.124362 28964 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.135720 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.136442 27655 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:14.137192 28964 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.128115 27582 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:14.042174 28853 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.108513 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:14.187000 27584 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:14.031706 29013 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.163406 28952 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260709 12:18:14.201932 29013 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.203742 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1278) took 1248 ms. Trace:
I20260709 12:18:14.204358 27662 rpcz_store.cc:276] 0709 12:18:12.946691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.946806 (+   115us) service_pool.cc:224] Handling call
0709 12:18:12.947176 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:12.948511 (+  1335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:12.948540 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:13.580439 (+631899us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:13.580906 (+   467us) log.cc:844] Serialized 26 byte log entry
0709 12:18:13.657415 (+ 76509us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:13.657501 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:13.658510 (+  1009us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:13.771894 (+113384us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:14.194903 (+423009us) raft_consensus.cc:1685] finished
0709 12:18:14.194938 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:14.195145 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5124096,"thread_start_us":68348,"threads_started":1,"wal-append.queue_time_us":95683}
I20260709 12:18:14.140472 28329 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:14.137751 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.129256 29000 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:14.235891 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107716us
I20260709 12:18:14.192159 28910 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.133728 28994 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:14.131718 28820 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:14.043268 29063 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.233325 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.193439 29015 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.193755 28950 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.198010 28964 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.138659 28929 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:14.127602 29054 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.239859 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:14.290858 27662 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4.
I20260709 12:18:14.237011 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:14.230455 29073 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.062469 29057 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.238085 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1284) took 1288 ms. Trace:
I20260709 12:18:14.240612 29014 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.230623 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.240938 27655 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:14.308372 29052 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.308971 27656 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:14.310125 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.298919 28972 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.287525 28913 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.214051 28890 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.213749 29051 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.337337 29072 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.295698 29015 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.272087 29000 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.140156 29070 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.178s
I20260709 12:18:14.322751 29073 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.328641 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:14.271962 28910 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.298070 28950 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.341624 27657 raft_consensus.cc:2368] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
W20260709 12:18:14.299391 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62518.5us
I20260709 12:18:14.300653 28964 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.365242 29067 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:14.338028 29072 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.365792 29067 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:14.366236 29067 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:14.366349 28964 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.326479 28994 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.342530 29075 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.329303 29050 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.328786 29054 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.344318 28957 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.342554 29057 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.323014 29014 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.342232 29034 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.344475 28925 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:14.343787 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 260) took 1369 ms. Trace:
I20260709 12:18:14.375571 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.382664 27733 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:14.377099 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.383827 27738 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.387749 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.388398 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:14.389761 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.391070 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.382452 28335 rpcz_store.cc:276] 0709 12:18:12.974390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.974649 (+   259us) service_pool.cc:224] Handling call
0709 12:18:12.975246 (+   597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:13.022280 (+ 47034us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:13.036592 (+ 14312us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:13.037060 (+   468us) log.cc:844] Serialized 26 byte log entry
0709 12:18:14.341848 (+1304788us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b540039fba0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a2bf6b8 00007f0e9a34892b 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c
0709 12:18:14.341935 (+    87us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:14.342003 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.342125 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:14.342176 (+    51us) raft_consensus.cc:1685] finished
0709 12:18:14.342192 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:14.343694 (+  1502us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":128,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":115,"spinlock_wait_cycles":223786880,"thread_start_us":430317,"threads_started":1,"wal-append.queue_time_us":474585}
I20260709 12:18:14.378218 28988 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.380012 29013 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.367452 29000 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.340998 29051 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:14.380604 28853 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.370590 28994 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.414636 27734 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.416034 29015 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.306857 27661 rpcz_store.cc:276] 0709 12:18:12.948445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.984102 (+ 35657us) service_pool.cc:224] Handling call
0709 12:18:12.984419 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:12.984635 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:13.770077 (+785442us) log.cc:844] Serialized 14 byte log entry
0709 12:18:14.162936 (+392859us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:14.193563 (+ 30627us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:14.196582 (+  3019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.214654 (+ 18072us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:14.236920 (+ 22266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23310,"prepare.queue_time_us":587126,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":43,"raft.queue_time_us":268482,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":43976,"thread_start_us":1079852,"threads_started":2}
I20260709 12:18:14.416728 28994 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:14.378793 29054 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.402448 29057 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:14.408915 28988 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.419497 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.391686 29075 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.444046 29075 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.444875 27657 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.416074 28913 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.450449 29014 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.409829 29013 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.455107 28988 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:14.438161 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.21147e+06us
I20260709 12:18:14.447855 29072 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:14.458992 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 59) took 1647 ms (client timeout 2023 ms). Trace:
I20260709 12:18:14.435299 29057 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.448758 28913 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:14.467692 27729 rpcz_store.cc:269] 0709 12:18:12.808566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.814397 (+  5831us) service_pool.cc:224] Handling call
0709 12:18:13.010642 (+196245us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223e9c 00007f0e97223a5b 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:14.455666 (+1445024us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:14.455792 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20245248}
I20260709 12:18:14.505765 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.488579 29072 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.519912 29072 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.457672 29063 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.522149 29072 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.523155 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.520254 28950 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.555995 28913 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.493649 28777 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.472204 29054 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.556921 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.556798 28777 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.557690 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:14.474160 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.555544 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.505833 29051 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.569953 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.571532 29065 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.571552 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.506304 29057 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.579859 27664 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:14.572299 28777 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.572942 27657 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:14.573275 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1189) took 1505 ms. Trace:
I20260709 12:18:14.573933 29054 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.574012 28971 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.475425 28910 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.576617 27660 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.523536 28967 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.588001 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:14.594367 27664 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:14.563182 29013 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.590010 27474 catalog_manager.cc:5028] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): tablet 22597fcae204491f84839e680c083ecc (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:18:14.597421 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.586808 27731 rpcz_store.cc:276] 0709 12:18:13.067717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.077466 (+  9749us) service_pool.cc:224] Handling call
0709 12:18:14.522984 (+1445518us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:14.573206 (+ 50222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71613696}
I20260709 12:18:14.595608 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.595940 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:14.583978 28989 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.558354 29072 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.649313 27702 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:14.598542 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.649659 27702 leader_election.cc:403] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:14.641983 28995 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:14.642539 29089 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.662642 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:14.645164 28986 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:14.634634 29057 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.653223 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1145) took 1638 ms. Trace:
I20260709 12:18:14.592519 29054 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.635412 29062 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.650653 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1153) took 1570 ms. Trace:
I20260709 12:18:14.650583 27738 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:14.651764 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.649930 27735 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
W20260709 12:18:14.634975 27474 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48290.2us
I20260709 12:18:14.654654 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.654541 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.670423 27664 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:14.734198 27736 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:14.686802 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.663482 28335 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:14.735056 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:14.591081 28971 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.688654 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1287) took 1669 ms. Trace:
I20260709 12:18:14.713126 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.736721 27659 rpcz_store.cc:276] 0709 12:18:13.014335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.017079 (+  2744us) service_pool.cc:224] Handling call
0709 12:18:13.049627 (+ 32548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:14.126268 (+1076641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.652987 (+526719us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001e50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:14.653163 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2474453760}
I20260709 12:18:14.653362 29051 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.739251 27658 rpcz_store.cc:276] 0709 12:18:13.019620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.049009 (+ 29389us) service_pool.cc:224] Handling call
0709 12:18:13.553565 (+504556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:13.652328 (+ 98763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.688447 (+1036119us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400345150. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:14.688594 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1159601920}
I20260709 12:18:14.742429 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.742412 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:14.744927 27734 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:14.746008 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:14.733716 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:14.594391 29000 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:14.750909 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:14.595162 28910 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.735826 27663 rpcz_store.cc:276] 0709 12:18:13.080154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.596751 (+516597us) service_pool.cc:224] Handling call
0709 12:18:13.597054 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:13.693551 (+ 96497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.650446 (+956895us) spinlock_profiling.cc:243] Waited 37.3 ms on lock 0x7b54001e4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:14.650600 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":85725440}
I20260709 12:18:14.689365 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1378) took 1744 ms. Trace:
I20260709 12:18:14.686733 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.758220 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:14.679797 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:14.655993 28948 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.742132 29086 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.773494 27738 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:14.689546 29016 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.684190 28925 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.663118 29072 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.680377 28986 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.687134 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.797940 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.808629 28332 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:14.686493 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.799505 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:14.662274 28989 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.801892 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:14.647179 29067 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.591533 29065 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.738013 28978 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.679914 29057 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.799137 29056 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:14.828215 27663 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:14.804494 29000 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:14.805930 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.804850 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:14.754581 29051 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.742543 29062 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.831055 27660 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:14.824276 28899 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:14.804050 28940 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:14.829280 29017 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:14.824954 28777 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:14.803718 28910 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.853708 29062 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:14.673381 29089 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.854528 28899 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:14.805699 28948 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.797158 27580 rpcz_store.cc:276] 0709 12:18:12.944882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.946048 (+  1166us) service_pool.cc:224] Handling call
0709 12:18:13.692239 (+746191us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:13.739203 (+ 46964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.689150 (+949947us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54000bd6d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:14.689281 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1574992256}
I20260709 12:18:14.599114 28967 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.861511 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:14.831305 29056 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.837623 29000 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:14.815939 28925 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.875963 28328 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:14.822273 28986 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.865262 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:14.848598 27731 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:14.849611 29017 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:14.831315 28978 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.808785 29086 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.872705 29081 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:14.884286 29056 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:14.886344 29000 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:14.820361 29016 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.864436 28777 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.897177 29017 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:14.910439 28924 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.043s	sys 0.036s
W20260709 12:18:14.953471 28924 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.043s	sys 0.036s
I20260709 12:18:14.894485 27731 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:14.864482 28940 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:14.962725 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:14.980335 28332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:58264
I20260709 12:18:14.926203 29054 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.042809 29100 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.046269 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.047987 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:15.045431 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.048882 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:15.044162 29097 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.049443 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.922822 28331 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:15.057039 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.058280 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.058486 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.059999 27582 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.060039 28332 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:15.060577 28332 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:14.924103 28329 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:18:15.059099 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.062114 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:15.062048 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:14.969769 29089 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.093240 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.095171 28948 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.095757 28948 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.096232 28948 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:15.097770 29069 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.098338 29069 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.099421 29065 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.099197 28332 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:14.980911 28950 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.062826 29097 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.100591 29069 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.100235 28950 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.063206 29085 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.101142 29069 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.101498 29085 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.101648 29069 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.102332 29097 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.102782 28950 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.103474 29087 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.103786 29085 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.963789 28899 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.104041 29087 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.098289 28948 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.106076 29087 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.106796 29054 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:15.107308 29054 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:15.107486 28967 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.108201 29054 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.109704 27582 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:14.972631 29095 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.967965 29072 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:14.988435 28940 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.110863 29095 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.111555 29057 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.112885 29067 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.114545 29016 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.115003 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.115756 27585 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:14.964039 28333 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:15.010213 29034 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.121150 28335 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:15.041102 28978 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.046860 28971 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:15.127435 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 312846us
I20260709 12:18:15.131989 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.129904 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.133117 28335 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:15.134712 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:15.136860 27701 leader_election.cc:400] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:15.137332 27701 leader_election.cc:403] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:15.128496 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:15.129063 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 307519us
I20260709 12:18:15.139274 27585 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:15.046272 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119578us
I20260709 12:18:15.144153 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.150838 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.153729 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.154388 27582 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:18:15.155006 27552 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4.
I20260709 12:18:15.155435 27552 leader_election.cc:403] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:15.158806 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.161795 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:15.155001 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:15.149538 29055 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:18:15.149235 28890 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:15.149737 29057 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:15.167369 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:15.166194 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.149914 29016 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.163908 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:15.171856 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:15.060539 29100 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.192200 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.137414 29034 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.204608 28335 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:15.167062 29065 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.207477 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:15.209143 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.210397 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:15.170228 28890 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.169113 29069 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.175478 29087 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.228472 29087 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.228948 29087 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.229540 28984 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.204444 29085 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.230100 28984 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.231413 29034 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.212728 28988 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.231971 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.232179 28988 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:15.232632 28988 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.232998 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.233366 29100 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.233426 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.227721 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.222898 29067 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.238533 29067 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.238902 29067 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.239022 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.240267 28984 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.240346 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.224849 28890 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:15.169889 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.175567 29016 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.223716 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.245975 27583 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:15.168102 28940 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4.
I20260709 12:18:15.225209 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:15.222263 29062 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.227667 29099 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.128695 27731 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.241717 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:15.250746 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.206722 28910 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.250748 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.212183 29065 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.243651 29086 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.256925 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.222349 28995 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.174467 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.249323 28943 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:15.223683 28925 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.230100 29085 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.264878 29085 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.265465 28995 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.212534 28994 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:18:15.268115 28995 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.260402 27731 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.251438 29099 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.247426 29016 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.247563 29051 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.275835 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.282068 27663 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:15.277842 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.283677 27660 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.285364 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.256940 29065 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.289324 27663 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:15.290791 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.292032 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.292883 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.293217 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.293545 27657 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:15.294181 27659 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:15.294626 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.251085 29062 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.295797 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.263131 29017 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.293296 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.295202 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.269698 28777 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.305378 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.306084 28337 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:15.282130 29099 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.307020 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.308970 28337 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:15.292212 28925 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:15.317657 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.320641 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.320662 29017 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.323459 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.303035 29056 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.310676 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.329126 27580 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:15.317070 28925 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:15.329879 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.330281 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.325099 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:15.330718 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.328305 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.316085 28936 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.327210 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.318787 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.324342 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:15.311836 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147213us
I20260709 12:18:15.315819 29100 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.307911 27731 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:15.336556 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.338740 28337 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4.
I20260709 12:18:15.339105 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.341507 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.342082 27734 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:15.341658 29081 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.340893 28994 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:15.346377 27589 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:15.346184 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.346903 28936 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.347359 29100 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.348423 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 265) took 1662 ms. Trace:
I20260709 12:18:15.323911 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.371577 28334 rpcz_store.cc:276] 0709 12:18:13.685990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.692254 (+  6264us) service_pool.cc:224] Handling call
0709 12:18:15.347760 (+1655506us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:15.347905 (+   145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:15.348073 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:15.348099 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:15.348138 (+    39us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:15.348178 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.348237 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:15.348350 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15399680}
I20260709 12:18:15.357347 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.355584 28899 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.373675 28899 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.359782 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:15.380160 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1159) took 1378 ms. Trace:
I20260709 12:18:15.358439 28971 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:18:15.361495 28925 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.380595 27664 rpcz_store.cc:276] 0709 12:18:14.001630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.747145 (+745515us) service_pool.cc:224] Handling call
0709 12:18:14.747561 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:14.754072 (+  6511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:14.756039 (+  1967us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:14.805167 (+ 49128us) log.cc:844] Serialized 26 byte log entry
0709 12:18:15.168573 (+363406us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:15.168613 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.168685 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:15.379952 (+211267us) raft_consensus.cc:1685] finished
0709 12:18:15.379971 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:15.380101 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6177,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":40453,"thread_start_us":361912,"threads_started":1,"wal-append.queue_time_us":455616}
I20260709 12:18:15.345541 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.365355 28936 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.365417 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.368336 27584 raft_consensus.cc:1813] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Handling vote request from an unknown peer 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.387647 27584 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:18:15.358964 29017 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:15.357101 28994 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.369657 29100 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.370638 29000 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.358747 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.374126 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.362448 29062 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.354785 29081 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.343976 28943 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:15.348470 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.382633 29101 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:15.380442 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:15.344293 28948 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:18:15.385910 29078 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:15.389734 28971 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:15.395727 28899 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.396049 28329 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:15.390264 28994 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.396564 27589 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:15.408115 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.418644 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:15.434434 27738 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:15.440932 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.393203 27586 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:15.417163 29056 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.415614 28943 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.415812 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.444672 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.401434 29067 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.440358 29017 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.471628 29017 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.457031 27659 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:15.472054 29017 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.457564 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.430334 28936 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:15.472797 28936 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.473420 28936 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.414388 29099 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.474617 27583 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.425021 29078 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.482719 28890 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.483536 29078 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.485117 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.469761 29073 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.470367 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1205) took 1093 ms. Trace:
I20260709 12:18:15.471284 27587 raft_consensus.cc:2368] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:15.472322 29067 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.430630 28971 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:15.435184 29101 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:15.476158 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.477331 29099 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.480921 29000 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.406338 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:15.487438 27731 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:15.492286 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.465890 28943 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:15.591449 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.592856 28894 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.593401 28894 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.594237 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.490330 28936 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.597301 28936 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.597932 28925 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:15.599473 28894 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.600096 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.491394 29064 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:15.484926 29065 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.490447 29079 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.588893 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.591517 29000 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.591558 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.436048 29087 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.600471 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:15.601342 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 212490us
I20260709 12:18:15.588967 28334 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:15.586194 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:15.604110 27664 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:15.610080 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.613842 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.614696 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:15.615907 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.602424 29056 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:15.601956 29016 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.621263 28894 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.627462 28894 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.627897 28894 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.630839 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1168) took 1031 ms. Trace:
I20260709 12:18:15.631247 27662 rpcz_store.cc:276] 0709 12:18:14.599130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.742006 (+142876us) service_pool.cc:224] Handling call
0709 12:18:15.630667 (+888661us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400229554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:15.630778 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277324032}
I20260709 12:18:15.633428 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.634590 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.637223 28335 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:15.638267 29101 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.639457 29087 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.640257 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.641359 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.641450 28890 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.642445 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.643452 29081 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.643612 29056 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:15.621299 29036 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.644950 29036 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.645339 29036 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:15.258679 28967 consensus_queue.cc:1175] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 3 status { last_received { term: 2 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:18:15.647691 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.610461 29000 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.648253 27662 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:15.604799 29085 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.650362 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.605242 29064 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:15.655684 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.657362 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:15.659986 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.662773 27553 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4.
I20260709 12:18:15.661993 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.664916 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.665812 27585 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.608901 28988 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.648893 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.668922 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.669500 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.611893 29079 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.670351 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.649654 27580 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:15.637509 28924 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:18:15.677556 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:15.691628 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.551s (SENT) [suppressed 7 similar messages]
I20260709 12:18:15.691450 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.665452 29078 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.620004 28971 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.748790 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.749029 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:15.749500 27659 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:15.672387 28890 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:15.490732 27730 rpcz_store.cc:276] 0709 12:18:14.376653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.377479 (+   826us) service_pool.cc:224] Handling call
0709 12:18:14.386586 (+  9107us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:14.411336 (+ 24750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:14.411379 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:14.411951 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 12:18:15.255770 (+843819us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b180017f0e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:18:15.328003 (+ 72233us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:15.330608 (+  2605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.363599 (+ 32991us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:15.464949 (+101350us) raft_consensus.cc:1685] finished
0709 12:18:15.465362 (+   413us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:15.470278 (+  4916us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":80,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":3624,"spinlock_wait_cycles":4937984,"thread_start_us":57317,"threads_started":1,"wal-append.queue_time_us":976027}
I20260709 12:18:15.756346 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.759554 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:15.760279 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.695583 28904 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:15.665258 28943 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.763800 27588 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.676311 29062 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:15.764536 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1217) took 1101 ms. Trace:
I20260709 12:18:15.765112 29014 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:15.765338 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.766235 27588 raft_consensus.cc:2368] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:15.766331 29099 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:15.766835 29099 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:15.768061 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.767990 29099 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.768802 27735 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:15.769208 29099 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.769776 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.770659 27588 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.771967 27588 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:15.769327 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:15.773375 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81773.8us
I20260709 12:18:15.773726 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:15.775302 28986 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.776108 29014 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.776851 27583 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:15.776775 29014 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.778360 27583 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:15.779093 29014 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.670888 28899 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:15.670807 28984 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:15.784569 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.764998 27737 rpcz_store.cc:276] 0709 12:18:14.663552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.663685 (+   133us) service_pool.cc:224] Handling call
0709 12:18:15.243980 (+580295us) spinlock_profiling.cc:243] Waited 9.66 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:15.244231 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:15.299991 (+ 55760us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:15.300033 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:15.300076 (+    43us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:15.300112 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.764283 (+464171us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b540044e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:15.764334 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:15.764475 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141792000}
I20260709 12:18:15.652174 29017 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:15.739755 29064 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.802865 29017 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.803836 27589 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.804049 29065 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.736078 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.808007 27586 raft_consensus.cc:2368] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:15.788900 29056 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.811563 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.790820 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.746501 29000 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.745003 29079 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.791632 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.792538 28994 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.741180 29016 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.737591 29015 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.794625 28943 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.822944 27729 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:15.823227 28994 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.823096 29015 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.824301 28894 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.795037 29085 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.796065 28890 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:18:15.796847 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 119361us
I20260709 12:18:15.797358 29062 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:15.797700 28988 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.797816 28989 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.694713 27731 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:15.850415 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.851780 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.853056 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.854362 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.800173 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:15.858065 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1155) took 1878 ms. Trace:
I20260709 12:18:15.800772 28984 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:15.860250 28984 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.810093 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.693457 29075 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:15.813148 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.863888 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.865105 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.865710 27736 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:15.866598 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.814251 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.815541 27582 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.872283 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.873102 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.873571 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.874526 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.789834 28952 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:15.875540 28952 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.875630 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:15.876184 28952 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.876418 28334 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:15.877908 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:15.783382 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:15.879644 28328 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.882126 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1345) took 1083 ms. Trace:
I20260709 12:18:15.787431 28899 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:15.882694 27729 rpcz_store.cc:276] 0709 12:18:14.798964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.348186 (+549222us) service_pool.cc:224] Handling call
0709 12:18:15.881875 (+533689us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54000a55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:15.882055 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":929045888}
I20260709 12:18:15.883569 28899 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.884526 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:15.885249 27729 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.828878 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.887243 27664 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.889786 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.826959 29081 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.829392 28904 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.838631 28995 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.839692 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.891558 27659 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.891379 28904 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.892458 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.892930 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.893689 28904 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.893965 27587 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:15.841809 29078 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.819072 29066 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.843581 28894 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:15.671203 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:15.895962 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.847229 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:15.867449 27730 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:15.868270 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.870627 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:15.874750 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:15.931032 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.893797 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:15.848889 29073 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:15.893702 27588 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.898569 28332 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.890738 27737 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:15.900815 27664 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:15.901793 27582 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:15.906819 27659 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:15.855154 27731 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:15.890717 29081 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:15.928861 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.931198 27584 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.928936 27732 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:15.932638 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.815889 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.948170 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.935150 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:15.948321 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132492us
I20260709 12:18:15.949645 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.936163 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:15.950598 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.951187 27662 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.894874 29066 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.955734 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1357) took 1011 ms. Trace:
I20260709 12:18:15.956274 27737 rpcz_store.cc:276] 0709 12:18:14.959251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.889466 (+930215us) service_pool.cc:224] Handling call
0709 12:18:15.955663 (+ 66197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:15.957912 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.959481 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:15.961503 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:15.844585 29015 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:15.963696 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:15.964010 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:15.970192 27737 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.943149 29071 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:15.975922 29071 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:15.976596 29066 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:15.977159 27586 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:15.978137 29071 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:15.978885 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.971822 28943 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:15.979700 28943 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:15.980068 28943 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:15.980365 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 286) took 1305 ms. Trace:
I20260709 12:18:15.982596 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:15.945182 29072 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:15.963382 28936 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.858543 27656 rpcz_store.cc:276] 0709 12:18:13.980793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.365428 (+384635us) service_pool.cc:224] Handling call
0709 12:18:14.365766 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:14.455955 (+ 90189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:14.457394 (+  1439us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:14.463826 (+  6432us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:14.466465 (+  2639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.857815 (+1391350us) spinlock_profiling.cc:243] Waited 84.9 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:15.857865 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:15.858003 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195374976}
I20260709 12:18:15.957015 28336 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.960193 29016 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:15.993922 28936 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:16.007555 29016 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.007704 28936 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.008009 29016 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:15.996232 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.011961 27736 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3.
I20260709 12:18:16.012908 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.014097 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.014694 27736 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:15.992329 29015 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:15.963347 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.000530 28899 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:15.968379 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1349) took 1127 ms. Trace:
I20260709 12:18:16.003018 27588 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:15.995424 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.050170 27732 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:16.027719 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:15.963853 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1339) took 1224 ms. Trace:
I20260709 12:18:15.980829 28329 rpcz_store.cc:276] 0709 12:18:14.675032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.679422 (+  4390us) service_pool.cc:224] Handling call
0709 12:18:15.980161 (+1300739us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400611054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:15.980283 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254144256}
I20260709 12:18:15.932195 28995 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.049430 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:15.981634 29073 raft_consensus.cc:2764] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:18:16.015233 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:15.968515 29099 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.020282 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:16.101560 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81406.5us
I20260709 12:18:16.051277 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.048146 28913 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.883122 29036 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.088263 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:16.069676 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:16.065078 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.106841 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.107440 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:16.104559 29059 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:16.057673 29085 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.108439 29059 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.093322 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:15.982044 29071 raft_consensus.cc:2764] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:18:16.015444 29072 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.113309 27733 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:16.103597 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.070657 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.075919 27661 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:16.069027 29095 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:16.186345 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93149.3us
I20260709 12:18:16.188217 27704 leader_election.cc:400] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:16.190152 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:16.181143 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 68947.6us
I20260709 12:18:16.053146 27730 rpcz_store.cc:276] 0709 12:18:14.739804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.791302 (+1051498us) service_pool.cc:224] Handling call
0709 12:18:15.960864 (+169562us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.064157 28328 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.202310 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.049350 27731 rpcz_store.cc:276] 0709 12:18:14.836570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.854097 (+1017527us) service_pool.cc:224] Handling call
0709 12:18:15.963742 (+109645us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.114745 28958 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.203085 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.205158 27704 leader_election.cc:400] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:16.105950 28913 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.066680 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:16.099677 28936 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.110119 28988 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:16.203840 29098 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.261675 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.262953 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:16.110040 28890 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.246795 27704 leader_election.cc:403] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:18:16.206524 27661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1166) took 1828 ms (client timeout 1735 ms). Trace:
I20260709 12:18:16.288360 27588 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:16.253515 27586 raft_consensus.cc:2368] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:16.206691 29056 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:18:16.070634 28952 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.287009 29124 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
I20260709 12:18:16.109949 29059 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.230072 28967 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:16.265897 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.292608 29056 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:16.293067 29056 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.294030 29056 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.295676 29056 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.296279 28913 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:16.296986 29016 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.297936 28936 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.259634 28943 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.298961 29016 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.300340 28943 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.300711 28952 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.289489 29095 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:18:16.283360 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 78310.1us
I20260709 12:18:16.301350 29095 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:16.302014 29095 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.302841 29124 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:16.307439 27587 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
W20260709 12:18:16.315065 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 467875us
I20260709 12:18:16.315974 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.319057 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.320011 27732 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.048848 27584 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:16.316816 29071 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.325199 27585 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:18:16.321936 27704 leader_election.cc:400] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3.
W20260709 12:18:16.288009 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 221419us
I20260709 12:18:16.288303 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.334775 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.258713 28988 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.331192 29081 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.333415 29125 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:16.340402 29125 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.342505 29125 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.327356 29078 raft_consensus.cc:2764] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:18:16.288748 29085 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:16.303861 28967 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.343055 29071 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.343510 28967 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.344542 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:16.321177 29098 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:18:16.345831 29073 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:16.346406 29073 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:16.347359 29073 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.348976 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:16.337296 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 226327us
I20260709 12:18:16.339277 29072 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.351962 28292 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3.
I20260709 12:18:16.352385 28292 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:16.354485 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.356257 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:16.321512 29069 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:16.333182 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.360219 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:16.350319 28988 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.382580 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:16.288180 27661 rpcz_store.cc:269] 0709 12:18:14.376470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.417368 (+ 40898us) service_pool.cc:224] Handling call
0709 12:18:15.946416 (+1529048us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:16.204506 (+258090us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5767936}
I20260709 12:18:16.383803 29125 raft_consensus.cc:2764] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:18:16.379729 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:16.384589 29123 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.385007 27662 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:16.359504 29102 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:16.385643 29102 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:16.383870 27737 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.385710 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.359287 29060 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:16.386114 27583 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.062s	sys 0.104s
W20260709 12:18:16.389756 27583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.062s	sys 0.104s
I20260709 12:18:16.390213 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.386853 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.391456 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:16.392241 27737 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:16.359529 29085 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:16.393199 29125 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.398480 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:16.381400 29072 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.410791 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.412727 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:16.414227 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:16.415875 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.410003 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.385998 29081 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.393436 29071 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.387306 28899 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.432086 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:16.457370 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.402076 28995 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:16.471045 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.022s	sys 0.036s
I20260709 12:18:16.436007 29102 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:16.471879 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.022s	sys 0.036s
I20260709 12:18:16.397691 29066 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:16.471261 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1348) took 1644 ms (client timeout 1695 ms). Trace:
W20260709 12:18:16.477103 27734 rpcz_store.cc:269] 0709 12:18:14.826678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.061692 (+235014us) service_pool.cc:224] Handling call
0709 12:18:16.471024 (+1409332us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54002e86d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:16.471179 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":529200000}
W20260709 12:18:16.478652 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 630065us
I20260709 12:18:16.407490 29062 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 12:18:16.397370 29098 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:18:16.418871 28984 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:16.387398 29069 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:16.459066 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.042s	sys 0.047s
I20260709 12:18:16.485019 27734 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:16.490424 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1344) took 1737 ms. Trace:
I20260709 12:18:16.440131 29125 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:18:16.488246 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132059us
I20260709 12:18:16.490892 27734 rpcz_store.cc:276] 0709 12:18:14.753344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.484364 (+1731020us) service_pool.cc:224] Handling call
0709 12:18:16.484773 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.490073 (+  5300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.490352 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.497608 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1358) took 1530 ms. Trace:
I20260709 12:18:16.390775 28890 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.498749 27734 rpcz_store.cc:276] 0709 12:18:14.967472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.494976 (+1527504us) service_pool.cc:224] Handling call
0709 12:18:16.496751 (+  1775us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.496928 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.496965 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:16.497009 (+    44us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:16.497298 (+   289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.497383 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.497551 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:16.499889 29096 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.294s	user 0.002s	sys 0.003s
I20260709 12:18:16.479925 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.504150 29085 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:16.508870 28336 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.000s
I20260709 12:18:16.509279 28336 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:16.488703 29102 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.437821 29081 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.504465 29060 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:16.511793 29062 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:16.512231 29062 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.511929 29060 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.513079 29062 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.514353 29060 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.515244 29098 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:16.504495 29125 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:16.515125 29095 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.515786 29098 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.516394 29125 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.516775 29098 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.517858 29095 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.485946 29094 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:16.519057 29094 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.519554 29094 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.519954 28984 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.521018 28984 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:16.522295 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.524917 27586 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:16.525329 29081 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:16.489796 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.022s	user 0.042s	sys 0.047s
I20260709 12:18:16.530503 27582 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:16.549392 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.525370 28958 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:16.486724 29066 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.541059 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:16.529663 28986 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3.
I20260709 12:18:16.564651 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.566015 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:16.567194 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:16.567955 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1359) took 1526 ms (client timeout 1916 ms). Trace:
I20260709 12:18:16.569334 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1308) took 1234 ms. Trace:
I20260709 12:18:16.573912 27581 rpcz_store.cc:276] 0709 12:18:15.334562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.334669 (+   107us) service_pool.cc:224] Handling call
0709 12:18:15.341434 (+  6765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:15.367720 (+ 26286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:15.367771 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:15.367819 (+    48us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:15.367856 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.569102 (+1201246us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:16.569135 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.569271 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49142016}
W20260709 12:18:16.575901 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184499us
I20260709 12:18:16.580857 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:16.581800 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:16.582906 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:16.568347 27733 rpcz_store.cc:269] 0709 12:18:15.041675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.759927 (+718252us) service_pool.cc:224] Handling call
0709 12:18:16.567761 (+807834us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54000a6ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:16.567892 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":799560960}
I20260709 12:18:16.588326 27733 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:16.590293 27733 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:16.538194 29098 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.554986 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:16.558132 29124 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:16.555517 28958 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.594254 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:16.534875 29081 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 12:18:16.576428 28812 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.594959 29064 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.613667 27630 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:16.601315 29056 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.613842 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:16.555234 29004 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:18:16.592929 29097 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.618173 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:16.595125 28894 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.614465 29056 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.607216 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:16.621819 27730 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:16.593214 29015 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.622782 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:16.604063 28995 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:16.600497 29071 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:16.605324 29123 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.638530 29140 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.643100 29140 raft_consensus.cc:2764] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260709 12:18:16.647073 29118 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.532s	user 0.001s	sys 0.000s
I20260709 12:18:16.650084 29069 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.651852 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1334) took 1004 ms. Trace:
I20260709 12:18:16.652244 27585 rpcz_store.cc:276] 0709 12:18:15.647904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.802592 (+154688us) service_pool.cc:224] Handling call
0709 12:18:16.324835 (+522243us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.328504 (+  3669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.651647 (+323143us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54001888d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:16.651790 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1199697664}
I20260709 12:18:16.628831 28913 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:16.628597 28894 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.623179 29016 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:16.603839 28967 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:16.652971 29126 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:18:16.666838 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:16.629299 28995 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.614008 29064 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.652070 29071 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.678458 28894 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.665699 28913 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:16.659243 28943 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:16.680143 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.680325 29056 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.680606 29066 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.663430 29059 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.684609 28971 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:16.698752 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.715567 27732 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.720467 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.722044 27732 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:18:16.716789 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 98693.8us
W20260709 12:18:16.714599 29054 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.257s	user 0.009s	sys 0.018s
I20260709 12:18:16.668164 29069 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:16.711210 29071 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.723794 27732 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:16.726857 29054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.010s	sys 0.019s
I20260709 12:18:16.686121 28995 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.736866 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:16.717990 28994 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:16.733454 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:16.726186 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:18:16.718784 28943 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.730028 28913 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.731101 29069 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.749215 29059 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.749567 29069 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.726711 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.753530 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:16.754901 29129 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:18:16.758157 29064 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:16.729166 29124 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:16.767495 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.762565 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.760685 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1363) took 1515 ms. Trace:
I20260709 12:18:16.768775 27662 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3.
I20260709 12:18:16.763713 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.729558 27732 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:16.765738 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.768734 27734 rpcz_store.cc:276] 0709 12:18:15.245526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.500109 (+1254583us) service_pool.cc:224] Handling call
0709 12:18:16.500485 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.520116 (+ 19631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.520150 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:16.520643 (+   493us) log.cc:844] Serialized 26 byte log entry
0709 12:18:16.664033 (+143390us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:16.664069 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.664131 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:16.760418 (+ 96287us) raft_consensus.cc:1685] finished
0709 12:18:16.760445 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.760613 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19214,"raft.queue_time_us":22730,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":142973,"threads_started":1,"wal-append.queue_time_us":155979}
I20260709 12:18:16.767113 29124 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.770041 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.781919 27584 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:18:16.737013 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1314) took 1452 ms. Trace:
I20260709 12:18:16.783035 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.771247 27662 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.771963 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:16.769999 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.771162 27732 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:16.780870 27583 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:16.760386 28995 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.783696 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:16.779253 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:16.785108 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.785704 27664 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.784174 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1335) took 2045 ms. Trace:
I20260709 12:18:16.786154 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.786592 27737 rpcz_store.cc:276] 0709 12:18:14.738657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.783435 (+2044778us) service_pool.cc:224] Handling call
0709 12:18:16.783705 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.783866 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.783890 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:16.783927 (+    37us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:16.783954 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.784002 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.784119 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.788028 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.785627 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.787179 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.790411 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:16.792040 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.792440 27662 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:16.795042 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1351) took 1932 ms. Trace:
I20260709 12:18:16.794129 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.796897 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.795506 27736 rpcz_store.cc:276] 0709 12:18:14.862853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.426219 (+1563366us) service_pool.cc:224] Handling call
0709 12:18:16.771820 (+345601us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:16.794974 (+ 23154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6573824}
I20260709 12:18:16.855098 27582 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:16.750463 29137 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:18:16.855029 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56834.6us
I20260709 12:18:16.860342 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:16.856114 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:16.858536 27664 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.861567 27658 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:16.861058 27732 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:16.797266 29102 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:18:16.867374 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71978.7us
I20260709 12:18:16.870229 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.870293 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.870939 27737 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.872082 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.872746 27658 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.874456 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.875504 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:16.877229 27661 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:16.882898 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.885147 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.887706 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:16.892537 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.893194 27662 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.859499 29137 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:16.867064 29057 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.904202 27657 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:16.905860 27662 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:16.899904 29137 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.907506 28943 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:16.909438 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.910248 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.911588 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.912292 27662 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.906745 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:16.917795 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1347) took 2084 ms (client timeout 2080 ms). Trace:
W20260709 12:18:16.918398 27738 rpcz_store.cc:269] 0709 12:18:14.816679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.373887 (+557208us) service_pool.cc:224] Handling call
0709 12:18:16.901097 (+1527210us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:16.901194 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94076160}
I20260709 12:18:16.864315 28812 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:18:16.860666 29015 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20260709 12:18:16.925360 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64952.8us
I20260709 12:18:16.929059 29102 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:16.916965 27657 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:16.935534 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.951001 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.958001 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.958601 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.960860 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:16.872023 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:16.962194 27662 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:16.894541 29066 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260709 12:18:16.921514 29057 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.894383 29085 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.054s
I20260709 12:18:16.966892 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:16.959089 27660 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:16.967588 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:16.948698 29069 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:16.924990 28904 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
W20260709 12:18:16.971400 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 105695us
I20260709 12:18:16.974377 29127 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:16.974785 28295 leader_election.cc:400] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:16.974961 29127 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:16.975157 28295 leader_election.cc:403] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:16.975961 29127 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.977126 27662 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:16.978073 29060 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:16.978778 29060 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:18:16.979583 29153 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:16.979949 29153 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:16.980350 28995 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:16.981292 27658 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:18:16.980996 29153 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:16.982296 29153 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.947948 28967 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:16.985540 27581 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:16.985791 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:16.986480 28330 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:16.989219 29139 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:16.989781 29139 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:18:16.994374 29132 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.280s	user 0.000s	sys 0.003s
I20260709 12:18:16.993968 29102 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.009894 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:17.010600 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
W20260709 12:18:17.011879 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:17.011847 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 137485us
I20260709 12:18:17.009931 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:17.011113 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 288) took 1231 ms. Trace:
W20260709 12:18:17.005101 29141 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:18:16.979233 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:17.014757 28335 rpcz_store.cc:276] 0709 12:18:15.779987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.781161 (+  1174us) service_pool.cc:224] Handling call
0709 12:18:16.543891 (+762730us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:17.009758 (+465867us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223e9c 00007f0e97223a5b 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:17.011057 (+  1299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17313024}
I20260709 12:18:16.995831 29097 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:17.000116 28943 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:17.033452 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58428.9us
I20260709 12:18:17.042200 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.000402 29069 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:16.990859 29139 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.033540 29097 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:17.035893 27657 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.055624 28335 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:17.035499 28971 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:17.049571 27588 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:17.049806 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.050084 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:17.051002 28995 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:17.051707 29145 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:18:17.052121 29071 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.052805 29149 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:17.053053 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.046690 29072 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:18:16.795450 27655 rpcz_store.cc:276] 0709 12:18:15.284246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.288234 (+  3988us) service_pool.cc:224] Handling call
0709 12:18:15.304043 (+ 15809us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:15.304421 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:15.304447 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:15.463821 (+159374us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4c001c39c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98614a20 00007f0e98625a29 00007f0e985eb644 00007f0e98612f2d 00007f0e98612b93 00007f0e9a34318b 00007f0e9a343cb5 00007f0e9a345004 00007f0e9a344729 00007f0e9a349d96 00007f0e9a33eadb
0709 12:18:15.468021 (+  4200us) log.cc:844] Serialized 556 byte log entry
0709 12:18:16.070757 (+602736us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e9723e0ac 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b
0709 12:18:16.538123 (+467366us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:16.541986 (+  3863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.556013 (+ 14027us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:16.725274 (+169261us) raft_consensus.cc:1685] finished
0709 12:18:16.726627 (+  1353us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.730737 (+  4110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81714304,"thread_start_us":1063445,"threads_started":1,"wal-append.queue_time_us":1079146}
I20260709 12:18:17.057725 29087 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:17.038230 28943 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.036788 29064 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:18:17.075165 27580 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:17.075258 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:17.047672 28812 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:17.069118 27584 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:17.066404 29094 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.541s	user 0.007s	sys 0.003s
I20260709 12:18:17.060216 28971 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:17.079458 28812 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:17.085613 28812 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.070586 29149 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:17.086680 29149 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:17.087651 29131 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.373s	user 0.000s	sys 0.002s
I20260709 12:18:17.083957 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.090174 27584 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:17.077363 29015 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:17.076040 29130 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:17.092559 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.095717 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.093968 29097 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.097059 28971 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.098920 29137 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.116014 27663 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:17.116914 29137 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.076494 27655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22597fcae204491f84839e680c083ecc"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:17.068950 28995 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.117763 27655 raft_consensus.cc:606] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:17.118171 27655 raft_consensus.cc:614] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:17.089162 29057 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.103979 29064 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:17.105491 29069 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:17.100576 29015 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:17.119990 29015 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.108700 29071 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.125402 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.089905 29094 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.127491 28337 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:17.092768 29127 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:17.109539 28988 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.713s	user 0.009s	sys 0.000s
W20260709 12:18:17.128688 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57497.5us
I20260709 12:18:17.086156 29142 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:18:17.168563 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:17.320333 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.080142 29156 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:17.321239 27655 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:18:17.322360 27655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:17.322520 28936 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:17.323112 27655 raft_consensus.cc:606] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:17.323149 28936 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.310268 29142 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:17.324931 29142 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.325961 29142 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.325842 28936 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.316221 27657 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:17.318038 29071 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:17.318975 28943 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:17.323776 27655 raft_consensus.cc:614] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:17.315186 28988 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:17.327033 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 201716us
I20260709 12:18:17.326224 28330 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:17.340122 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.367789 28336 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:17.367535 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:17.368762 27589 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:17.330425 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.128527 29069 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:17.344282 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:17.346002 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:17.348493 27587 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:17.349045 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:17.352211 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1244) took 1473 ms. Trace:
I20260709 12:18:17.329511 28971 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:17.396903 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66587.3us
I20260709 12:18:17.418648 27733 rpcz_store.cc:276] 0709 12:18:15.879026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.633981 (+754955us) service_pool.cc:224] Handling call
0709 12:18:16.634265 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.670702 (+ 36437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.670749 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:16.670813 (+    64us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:16.670859 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.351919 (+681060us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:17.351978 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:17.352137 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83227008}
W20260709 12:18:17.420819 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71888us
I20260709 12:18:17.376430 27663 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:17.397356 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1346) took 2595 ms (client timeout 2167 ms). Trace:
W20260709 12:18:17.422688 27735 rpcz_store.cc:269] 0709 12:18:14.802301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.790150 (+987849us) service_pool.cc:224] Handling call
0709 12:18:17.384762 (+1594612us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54000a5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:17.397244 (+ 12482us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1734508416}
I20260709 12:18:17.423000 27663 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:17.338804 28936 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.424958 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.396291 29127 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.340224 29097 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:17.097823 29159 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.326s
I20260709 12:18:17.426964 29065 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.432174 27730 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:17.429703 28894 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:17.432610 29065 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.424052 29059 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.437114 29069 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:17.437567 28936 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.428032 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.440214 28995 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.440742 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.433190 28894 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:17.441586 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:17.443562 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:17.444469 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:17.454499 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84318.3us
I20260709 12:18:17.455682 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.456815 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.451957 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:17.461460 28292 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3.
I20260709 12:18:17.461690 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:17.339629 29071 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:17.467340 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.435190 29065 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:17.373872 29060 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
W20260709 12:18:17.473423 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.460237 28995 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:17.428532 29128 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:18:17.474304 29165 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:17.467406 29087 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:17.474651 29127 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:17.465308 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:17.475384 28936 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.483702 29054 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:17.452251 29094 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:17.477582 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.525s	user 0.032s	sys 0.022s
W20260709 12:18:17.505450 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.553s	user 0.032s	sys 0.022s
W20260709 12:18:17.506268 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.57108e+06us
I20260709 12:18:17.507225 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:17.478608 28984 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:17.478215 29149 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.508647 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.479950 28988 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.480899 29015 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.444224 28943 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:17.481348 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:17.398576 29095 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:18:17.482055 27586 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:18:17.453276 28894 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.452904 29059 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.488689 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:17.516731 29064 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:17.486227 28913 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:17.488166 29078 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.498409 28995 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:17.500452 29127 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.563285 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.538973 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:17.504572 29054 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:17.475530 28986 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:17.567400 27735 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:17.515728 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:17.516237 28984 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:17.546772 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90015.1us
I20260709 12:18:17.547765 29085 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:17.557252 29015 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:17.545349 27655 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:17.547626 29064 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:17.521520 29123 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.560153 29071 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:17.561311 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98215.4us
I20260709 12:18:17.564275 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:17.564886 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.569615 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:17.569846 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:17.578860 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.546821 27656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:17.585850 27656 raft_consensus.cc:606] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:17.586369 27656 raft_consensus.cc:614] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:18:17.588893 29133 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.875s	user 0.010s	sys 0.017s
I20260709 12:18:17.589577 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:17.589774 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:17.590162 27735 raft_consensus.cc:606] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:17.590396 28335 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:17.586750 28995 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.594192 29054 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.595503 29054 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.597770 27589 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:18:17.599660 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.601893 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:17.602844 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1237) took 1727 ms. Trace:
I20260709 12:18:17.603426 27584 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:17.604440 27586 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:18:17.606361 29166 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:18:17.607826 29060 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:17.608355 29060 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:17.609475 29060 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.612314 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:17.561862 29102 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.612912 27664 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.574668 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.560456 29072 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:18:17.565726 29130 raft_consensus.cc:2764] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:18:17.576929 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.611112 29060 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.546343 28943 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:17.573016 28984 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.581650 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.581773 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.583066 29071 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:17.653883 29102 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.624626 29149 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:17.659783 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:17.660310 28943 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.654721 29097 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.661111 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:17.661535 29097 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.662261 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.576469 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.663580 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.663844 29097 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:17.666033 29128 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.575371 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.618820 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:17.620069 29172 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:17.620355 29015 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:17.621194 28988 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:17.621531 27586 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:17.676481 27586 raft_consensus.cc:606] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:17.667130 27729 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:17.676908 27586 raft_consensus.cc:614] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:17.677198 29130 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.621744 29078 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:17.603262 27730 rpcz_store.cc:276] 0709 12:18:15.875674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.633252 (+757578us) service_pool.cc:224] Handling call
0709 12:18:16.633652 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:17.432643 (+798991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.602610 (+169967us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b54002e9ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:17.602783 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1827530496}
I20260709 12:18:17.574218 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:17.680496 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:17.681321 27657 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:17.558166 28958 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.628854 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:17.630605 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:17.683637 29169 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:18:17.654192 29123 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:17.687686 29123 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.688187 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:17.642071 27664 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:17.578258 29064 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:17.654183 27737 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:17.566752 29127 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.611881 29065 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:18:17.669523 27584 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:18:17.671906 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53203.1us
I20260709 12:18:17.674072 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:17.674496 29000 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.674716 29036 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:17.692153 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:17.695628 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:17.620414 28994 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:18:17.702682 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:17.717976 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:17.734874 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 119849us
I20260709 12:18:17.689860 29123 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:17.735242 27664 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:17.741418 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:17.742352 27738 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
W20260709 12:18:17.707867 29151 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.774s	user 0.004s	sys 0.010s
I20260709 12:18:17.747998 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.745817 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:17.748806 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.750250 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.735654 28330 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:17.731937 29097 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:17.747366 29165 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:17.739740 27656 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:17.741577 27664 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:17.781827 27664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:17.782338 27664 raft_consensus.cc:606] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:17.782624 27664 raft_consensus.cc:614] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:17.756414 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.734220 29130 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:17.791746 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 210164us
I20260709 12:18:17.797224 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.800522 27664 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:17.740516 29121 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:17.802057 27659 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:17.858882 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:18:17.861220 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:17.862231 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:18:17.864318 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67174.6us
I20260709 12:18:17.778582 28986 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:17.865334 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:17.784446 29175 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:17.741325 29172 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:17.866765 29163 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:18:17.902361 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:17.790377 29165 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.796111 29094 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20260709 12:18:17.869882 29142 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.901750 28986 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.957830 27660 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:17.888832 29072 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.944679 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1188) took 1170 ms. Trace:
I20260709 12:18:17.960242 27658 rpcz_store.cc:276] 0709 12:18:16.773731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.982971 (+209240us) service_pool.cc:224] Handling call
0709 12:18:17.345770 (+362799us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:17.374576 (+ 28806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.944463 (+569887us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54001e55d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:17.944612 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":833554816}
I20260709 12:18:17.952481 29071 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:17.905282 29172 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:17.802057 29130 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:17.971546 29178 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:18:17.971390 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:17.972370 28329 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:17.974012 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:17.974546 29054 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:17.969467 29142 raft_consensus.cc:2764] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:18:17.975956 29004 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:17.976553 29004 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:18:17.979547 29132 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:18:17.981231 29127 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:17.936474 29121 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.982025 29121 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:17.984412 29165 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:17.963188 29174 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:18:17.978046 29004 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.001152 29142 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:18.002425 29172 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:17.978843 28984 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.004017 29130 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:18.004513 29176 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:18:17.985931 29102 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:18.008603 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63657.1us
W20260709 12:18:18.051986 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.999449 28952 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:17.991539 28894 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.055778 28952 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.057488 28894 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.058535 28952 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.051007 28988 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:18.059754 28988 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.062419 28988 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.063540 27734 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.009234 29142 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:18.064635 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.071019 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:18.072700 29142 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.078794 29071 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:18.098502 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.086524 29180 raft_consensus.cc:2764] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:18:18.138652 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.139721 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.140369 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.144261 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.143071 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:18.148731 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 392375us
I20260709 12:18:18.121896 29121 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.154122 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.152644 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:18.157727 27735 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:18.158318 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.160106 27732 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:18.161546 27732 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:18.162792 27663 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:18.163206 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.165416 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.170171 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.156358 29146 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.186440 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.187103 27659 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:18.179014 28984 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.192796 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.193454 27659 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:18.194192 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.196403 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.197649 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.201932 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 279) took 3216 ms. Trace:
I20260709 12:18:18.202550 28337 rpcz_store.cc:276] 0709 12:18:14.985401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.385983 (+400582us) service_pool.cc:224] Handling call
0709 12:18:15.773994 (+388011us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:16.545449 (+771455us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223e9c 00007f0e97223a5b 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:16.781584 (+236135us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:16.781765 (+   181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:17.127952 (+346187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.201539 (+1073587us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54005a2654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:18.201798 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":831898624}
I20260709 12:18:18.204726 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.206104 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.206861 27662 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:18.207892 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.208374 29085 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.208909 29085 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.209100 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:18.210728 29064 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.504s	user 0.002s	sys 0.007s
I20260709 12:18:18.207017 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.211961 27664 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:18.213274 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.214020 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.190038 29192 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.185994 29146 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.214455 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:18.232781 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.233631 27659 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:18.234156 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.235112 28971 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.235555 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.235639 28971 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.236588 29097 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:18.237188 28958 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.237879 28971 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.237612 29097 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.211162 29064 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:18.239544 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 265596us
W20260709 12:18:18.239907 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 491122us
I20260709 12:18:18.241326 29015 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.241932 29015 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.243736 28812 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:18.243927 29064 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.244635 29087 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:18.245445 28971 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:18.244313 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.248010 27584 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:18.248198 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.249318 27582 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:18.249586 28996 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:18.250037 28996 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:18.250998 28996 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.251765 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.252058 28996 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.252544 27581 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:18.253911 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.254127 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.255434 27659 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:18.259837 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.261009 27587 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:18.244372 29015 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.263635 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.264511 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:18.214752 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.216539 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.220952 28332 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260709 12:18:18.268668 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87516.5us
I20260709 12:18:18.223548 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:18.230489 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:18.216725 27734 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:18.220659 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.268047 27659 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:18.244247 28812 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.274387 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:18.274473 27662 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:18.275550 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:18.276525 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:18.276988 27662 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:18.270517 28334 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:18.278122 27662 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:18.278682 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58149.6us
I20260709 12:18:18.279433 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.282707 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.283808 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.284591 29130 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.284955 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.280709 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.286216 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:18.286809 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:18.286981 27662 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:18.287616 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:18.223397 27657 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.541s	user 0.007s	sys 0.006s
I20260709 12:18:18.288040 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:18.221860 28331 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:18.252449 29187 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 12:18:18.290249 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.293749 29152 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:18:18.290532 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:18.272243 29149 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.291879 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.308207 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.309237 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.317721 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.293691 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.294884 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.306768 29059 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:18.287799 29072 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.296092 28336 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:18.294338 27657 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.222384 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1338) took 3482 ms. Trace:
I20260709 12:18:18.297808 29128 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:18.221616 29121 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.229212 29071 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.331455 29121 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.223223 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:18.226003 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:18.225304 29085 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.333588 29121 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.317355 29149 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.335408 29085 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.312812 28967 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:18.288251 28995 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.335988 29149 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.215106 29060 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.336113 28967 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.336621 29060 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.217577 29146 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.288519 27659 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:18.320271 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.295300 29130 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:18.273896 29064 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:18:18.323277 29194 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.323267 29173 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:18:18.214999 29192 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.307719 29094 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.312350 29172 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.301172 29102 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:18:18.329589 27656 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:18.321537 29059 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:18.305516 28989 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260709 12:18:18.348773 29152 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.371356 29060 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.330626 29193 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:18.348014 29071 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.372133 29193 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.372643 29193 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.320040 28943 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:18.286545 28812 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:18.291040 29153 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.381184 27661 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:18.381489 29153 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.382186 29192 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.382607 28335 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:18.345239 29146 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.343040 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.349542 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.328902 29128 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.383692 29128 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.383910 29153 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.384267 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.321262 29069 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
W20260709 12:18:18.356159 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65696.3us
I20260709 12:18:18.361793 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:18.362465 29187 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.358197 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.351082 28986 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.340406 29130 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:18.377478 28943 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.347227 29072 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:18.313638 29122 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.386045 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.390137 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.312747 28971 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:18.327097 27731 rpcz_store.cc:276] 0709 12:18:14.739527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.487354 (+1747827us) service_pool.cc:224] Handling call
0709 12:18:16.536777 (+ 49423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.747470 (+210693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.221885 (+1474415us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540017f2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:18.222099 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406504704}
I20260709 12:18:18.409298 29060 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:18.400444 28967 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.403863 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.421339 27657 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:18.421427 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.405706 29130 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.399652 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:18.398348 29192 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.398106 29146 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.411019 27659 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:18.406771 28943 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.414032 29153 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.404541 28958 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.414145 28971 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.425290 27737 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:18.439633 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.426651 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.371393 29152 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.406137 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.406853 29072 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:18.424194 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.432958 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.431617 29153 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.438022 28971 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.442077 27659 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:18.422735 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.435067 28812 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.447686 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.428328 29194 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.427879 28967 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:18.449782 29072 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.448505 28958 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.477417 28812 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.466631 29153 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.425361 29122 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.472409 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.469949 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.481619 27738 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:18.483450 29153 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.484112 29153 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.485256 29122 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.486135 27580 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 12:18:18.486631 29153 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.487404 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:18.490120 28971 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.493324 28943 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.496731 29152 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.493738 28971 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.482908 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.474551 28996 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:18.479235 28958 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.467504 29128 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.490356 28988 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.505803 29192 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:18:18.519134 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 203191us
W20260709 12:18:18.521314 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.508358 28986 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.517912 29121 raft_consensus.cc:993] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:18:18.507923 28996 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:18.505450 28952 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:18:18.525857 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.526669 28996 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.530202 28812 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:18.532866 28812 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:18.533775 28812 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.534927 29146 raft_consensus.cc:1081] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:18:18.517098 28958 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.536568 28336 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:18.537184 28958 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.540441 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1190) took 1766 ms. Trace:
I20260709 12:18:18.540935 27660 rpcz_store.cc:276] 0709 12:18:16.774008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.008055 (+234047us) service_pool.cc:224] Handling call
0709 12:18:17.696811 (+688756us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:17.958255 (+261444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.304960 (+346705us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54000e56d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:18.540250 (+235290us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54000e56d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:18.540381 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2182822016}
I20260709 12:18:18.541695 28943 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.542660 29128 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.515390 28971 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.545243 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.547663 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:18.538702 29204 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:18.555606 29156 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:18.555759 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.556186 29156 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:18.521166 29192 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.558779 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.560842 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.562273 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.563035 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:18.565516 28329 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:18.516100 29122 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:18:18.567538 29102 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.568324 29128 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.557523 29173 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:18:18.564008 29192 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:18.583366 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.572698 28943 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:18:18.588358 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 294710us
I20260709 12:18:18.590736 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.592439 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.574972 29204 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.559309 28971 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.576259 29193 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.600407 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:18.602382 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.595403 29152 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:18.588989 28996 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.598621 29097 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
W20260709 12:18:18.611964 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.595049 29071 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:18:18.620195 29054 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.623952 28971 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.628563 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.632362 29062 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.635167 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:18.615005 29152 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:18.643947 29097 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:18.655766 29097 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:18.655827 29078 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:18.644500 29204 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.656447 29078 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:18.656277 29097 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.662981 29152 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.663954 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:18.629856 29187 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:18.666031 29187 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.666442 29187 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.628067 29149 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:18:18.645938 29087 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:18.636062 29153 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:18:18.647162 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.673693 27658 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.674278 29087 consensus_queue.cc:1243] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:18:18.674362 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.649174 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.675302 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.675984 27656 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:18.676225 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.609211 29016 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:18:18.678869 27581 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:18.607776 29072 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.652760 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.680663 27659 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:18.653498 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.682368 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:18.653746 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 297) took 1166 ms. Trace:
I20260709 12:18:18.636636 28971 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:18.654456 28337 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:18.655191 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.613634 29130 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.666927 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:18.686683 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.687439 27734 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.687691 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.653920 28996 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:18.670497 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.654520 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.693444 29130 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.682479 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.695740 29153 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:18.695868 29175 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:18.696558 29175 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:18.696918 29175 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.697100 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:18.683106 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.654095 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.684566 27737 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
W20260709 12:18:18.698011 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97674.4us
I20260709 12:18:18.668030 29071 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:18:18.689716 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:18.691560 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.700512 27734 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.675158 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:18.693742 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:18.701470 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.701833 27733 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.702015 27660 raft_consensus.cc:2376] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2. Re-sending same reply.
W20260709 12:18:18.695410 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.698352 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.683924 28330 rpcz_store.cc:276] 0709 12:18:17.487283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.488966 (+  1683us) service_pool.cc:224] Handling call
0709 12:18:18.653560 (+1164594us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54006bc6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:18.653686 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370227968}
I20260709 12:18:18.709899 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.704231 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.651404 28943 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 12:18:18.704599 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:18.695859 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.711987 27589 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:18.698107 28971 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:18.715412 29130 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:18.715755 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.717415 27583 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:18.717375 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:18.729584 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:18.729693 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.730268 27581 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.715569 28989 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:18.727660 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.701057 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.728735 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.732987 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.733799 27583 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:18.733963 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.720677 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:18.722837 27584 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4.
I20260709 12:18:18.735018 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.705081 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:18.722996 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.736264 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.737792 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:18.738480 27583 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:18.740551 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:18.741026 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:18.686069 29066 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:18:18.664990 29097 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.727455 29056 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:18.727679 27661 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:18.722486 29193 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:18.735550 28989 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.764758 27587 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:18.768023 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:18.768884 27736 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.726253 28812 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:18:18.777628 27658 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.779711 28943 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:18.781301 27660 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:18:18.723374 29156 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:18.765615 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:18.751127 29056 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.736241 27731 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.739950 27656 raft_consensus.cc:2376] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3. Re-sending same reply.
I20260709 12:18:18.781443 28943 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.798715 28943 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:18.782043 29130 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:18.742965 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.753268 29057 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:18.751358 29071 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:18.786620 27658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:50014
I20260709 12:18:18.770757 29087 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:18.766144 28989 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.804702 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.731146 29214 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
W20260709 12:18:18.806680 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63828.8us
I20260709 12:18:18.808166 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.796880 27734 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:18.803562 27658 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:18.812657 27658 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.812634 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:18.788424 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.794322 29056 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.740216 29015 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260709 12:18:18.751747 29000 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 12:18:18.822939 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:18.816118 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:18.842950 27658 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.807036 28892 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.837746 29036 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.859571 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:18.855389 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 108779us
I20260709 12:18:18.858246 27658 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.860318 27704 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4.
I20260709 12:18:18.862056 27736 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:18.862807 27704 leader_election.cc:403] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:18:18.862860 27736 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:18.837642 27729 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:18.875277 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.876076 27658 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.813951 29057 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:18.863787 27736 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.877219 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:18.879519 27658 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:18.879287 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:18.880493 27737 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:18.803759 29193 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:18:18.880543 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.887123 27658 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:18.883028 28334 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:18:18.850178 28988 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:18.885910 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.852007 28996 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:18:18.847465 29130 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:18.902029 27736 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.848894 27657 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:18.873134 28330 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:18.861225 28967 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:18.889989 27586 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:18.847273 29066 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:18:18.913549 27658 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:18.928877 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:18.914171 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 177966us
I20260709 12:18:18.814191 28971 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:18.890174 28892 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4.
I20260709 12:18:18.888404 29072 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:18.985816 27581 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
W20260709 12:18:18.988328 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54013.4us
W20260709 12:18:18.990486 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 104700us
W20260709 12:18:18.991573 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64991.2us
I20260709 12:18:18.994019 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:19.023599 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.911088 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:18.890022 29057 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.001572 29193 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:18:19.031674 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:18.908661 29130 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.010102 29146 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:18.888763 29192 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.053966 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:19.029470 29215 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:19.025833 28988 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:19.025631 29036 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:19.033006 29097 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:19.045783 27736 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:19.035037 29121 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.056830 28330 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:19.074388 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:19.075685 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:19.076607 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:19.077121 27589 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:19.078158 27587 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:19.041395 27661 raft_consensus.cc:2376] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3. Re-sending same reply.
I20260709 12:18:19.044356 29015 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:19.073482 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.066368 29036 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:19.062074 28988 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.100548 27585 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:18:19.070572 29215 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:19.064116 29128 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:18:19.086763 29015 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:19.097465 29036 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.081452 29121 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.123112 29015 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:19.152575 28899 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:19.153327 28899 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.138041 29186 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260709 12:18:19.162122 29192 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.163381 28988 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.164780 28988 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.167030 29173 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:19.167547 29173 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:19.142148 29204 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:19.168620 29204 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:19.168466 29173 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.169597 29173 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.169507 29204 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.170936 29204 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:19.174026 29183 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.307s	user 0.003s	sys 0.000s
I20260709 12:18:19.174284 28899 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.131055 29224 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260709 12:18:19.175498 28899 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.175757 29224 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:19.176198 29224 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:19.137861 29190 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.888s	user 0.002s	sys 0.002s
I20260709 12:18:19.176991 29224 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.202735 29121 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:19.204515 29059 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.205660 29059 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:19.207309 29056 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.211239 28943 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.210810 29015 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.207842 29224 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.213856 29004 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:19.142692 29062 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.510s	user 0.007s	sys 0.003s
I20260709 12:18:19.134707 29215 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.183076 29146 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:19.222615 29175 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.525s	user 0.006s	sys 0.004s
I20260709 12:18:19.236227 29156 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:19.184252 29130 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.246316 29142 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:19.253243 29175 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.226080 29078 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.255112 29078 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:19.256027 29175 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:19.239933 29187 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.573s	user 0.010s	sys 0.003s
I20260709 12:18:19.237785 29087 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:19.260056 29087 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.259887 29187 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.262151 29187 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.243729 29143 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:19.263571 29143 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.264756 29143 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.266810 29143 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.247095 29209 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:19.267935 29209 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:19.269155 29209 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.270852 29209 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:18:19.331555 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.335398 27487 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:19.339573 27583 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:19.339664 27581 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:18:19.341765 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.342337 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 85) took 1727 ms. Trace:
W20260709 12:18:19.345498 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.343065 27655 rpcz_store.cc:276] 0709 12:18:17.615079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.622124 (+  7045us) service_pool.cc:224] Handling call
0709 12:18:17.622480 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:17.673183 (+ 50703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:17.693467 (+ 20284us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:17.784915 (+ 91448us) log.cc:844] Serialized 26 byte log entry
0709 12:18:18.098642 (+313727us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:18.099267 (+   625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.102165 (+  2898us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:18.902623 (+800458us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:19.341272 (+438649us) raft_consensus.cc:1685] finished
0709 12:18:19.341342 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:19.342002 (+   660us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32217,"raft.queue_time_us":45742,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":4214,"spinlock_wait_cycles":11378944,"thread_start_us":284623,"threads_started":1,"wal-append.queue_time_us":318340}
I20260709 12:18:19.339489 27586 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:19.349954 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.362006 27487 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3), now has a pending config: VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1), VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3), VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:18:19.368681 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:19.369508 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.369889 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.376149 27630 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:19.376674 27630 leader_election.cc:403] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:19.377992 27580 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:19.386861 27662 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:19.390589 27663 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:19.380388 27583 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:19.381065 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:19.399983 27663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1342) took 1242 ms (client timeout 1555 ms). Trace:
W20260709 12:18:19.400511 27663 rpcz_store.cc:269] 0709 12:18:18.157279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.163516 (+  6237us) service_pool.cc:224] Handling call
0709 12:18:18.248045 (+ 84529us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:19.399093 (+1151048us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:19.399302 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2550531712}
I20260709 12:18:19.383401 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:19.401854 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.402128 27586 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:19.398042 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.400820 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:19.405534 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:19.380403 27487 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:19.416255 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.416725 27736 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:19.417358 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:19.418983 27737 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:19.420043 28332 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.213869 28943 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.423403 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 299) took 1206 ms. Trace:
I20260709 12:18:19.432332 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:19.433947 28952 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:19.436625 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:19.455750 28337 rpcz_store.cc:276] 0709 12:18:18.216897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.217016 (+   119us) service_pool.cc:224] Handling call
0709 12:18:18.654142 (+437126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:18.684146 (+ 30004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:19.422558 (+738412us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54006bc6d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:19.423063 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1004579200}
I20260709 12:18:19.477336 29146 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:19.478083 28333 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:19.382404 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:19.488437 29233 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:19.489180 29233 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.468662 29056 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:19.508906 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:19.510008 27729 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.511554 29097 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:19.512506 29097 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.417384 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
W20260709 12:18:19.478207 28989 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.002s
I20260709 12:18:19.226418 29015 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:19.225468 29065 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.490902 29233 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.536032 29233 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:19.515756 27487 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:19.479943 29107 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:18:19.545289 27586 raft_consensus.cc:2368] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:19.550021 27582 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:19.484157 28952 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.515528 29181 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 12:18:19.558547 29214 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:19.572576 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:19.574553 27737 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:18:19.577281 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.005s	sys 0.022s
W20260709 12:18:19.577678 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.006s	sys 0.022s
I20260709 12:18:19.584830 28943 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.585547 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:19.585453 28943 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.586493 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:19.587508 29072 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:19.587987 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:19.588892 27660 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.589211 29064 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.589936 29064 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.588416 29072 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.592449 29072 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.592532 29156 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 12:18:19.592449 29064 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:19.594498 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.011s	sys 0.060s
W20260709 12:18:19.594902 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.011s	sys 0.060s
I20260709 12:18:19.597553 29186 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.558583 29071 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.540710 28989 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.614338 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.618537 29209 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:19.614583 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:19.600700 27487 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:19.624708 27660 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:19.626809 29153 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.627985 29153 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.628253 29065 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.629819 29234 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.619966 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:19.568347 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50747.4us
I20260709 12:18:19.609508 29214 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:19.662869 28332 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:19.660687 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:19.632949 27487 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:19.631059 29153 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.625125 29130 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:19.668908 27583 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:19.671331 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57442.7us
I20260709 12:18:19.679095 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:19.680239 29187 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
I20260709 12:18:19.682116 29204 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.118s
I20260709 12:18:19.683777 29156 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.684567 29156 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.685511 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.686398 28295 leader_election.cc:400] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:19.686765 28295 leader_election.cc:403] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:18:19.687547 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.667465 28943 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.691978 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:19.701825 29186 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:19.630578 29234 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.694025 27487 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:19.663625 29071 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.709012 27487 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 3, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:19.712384 28952 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:19.713634 29097 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:19.716092 29057 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.014s	sys 0.032s
W20260709 12:18:19.716472 29137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.963s	user 0.017s	sys 0.017s
I20260709 12:18:19.717257 29229 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:19.718578 29071 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.723111 27730 raft_consensus.cc:2368] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:19.720638 27738 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:19.731181 27734 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:19.737874 29062 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.738513 29062 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.739228 28943 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:19.740619 29214 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.741297 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:19.742767 29094 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:19.743244 28971 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.743752 28334 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:18:19.744244 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.724066 29165 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.109s
I20260709 12:18:19.744717 29065 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:19.747653 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.748217 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:19.748730 29209 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:19.749127 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.734561 29130 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.722851 29156 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:19.735136 29137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.076s	user 0.018s	sys 0.017s
W20260709 12:18:19.735770 29057 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.015s	sys 0.033s
I20260709 12:18:19.750869 27729 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:19.727775 29064 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.751526 29234 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:19.754649 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.760025 29181 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:19.766498 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
W20260709 12:18:19.729997 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43901.2us
I20260709 12:18:19.730798 29229 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:19.739581 29072 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:19.774359 29202 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:18:19.805250 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.774515 28943 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.775105 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:19.767819 29209 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:19.776629 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.790380 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.778281 29130 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:19.778653 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:19.779445 29078 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.778653 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:19.780220 28971 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.781925 28292 leader_election.cc:400] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4.
I20260709 12:18:19.782298 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:19.784911 29232 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:19.782323 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107481us
I20260709 12:18:19.775962 29156 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:19.804729 28330 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:19.765857 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 4 to 5, VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) evicted. New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:19.763638 29087 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.823715 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:19.820472 29156 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:19.828715 28958 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 12:18:19.815289 28952 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:19.830422 28952 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.832764 28952 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:19.835465 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:19.836238 28333 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:19.836910 28971 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.837548 29064 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.837796 29078 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:18:19.839960 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61472.7us
I20260709 12:18:19.839793 27487 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:19.841622 28943 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.842502 28892 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:19.843186 28892 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.843690 28988 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.843406 29072 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.844290 28988 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.844349 28892 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.845824 29072 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:19.846407 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67930.2us
I20260709 12:18:19.846613 28988 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:19.848218 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:19.849648 28996 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:19.850188 28996 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.851222 28996 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.824154 29097 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:19.859812 27729 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:18:19.860320 27729 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 1
I20260709 12:18:19.860641 27729 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:18:19.861038 27729 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:19.862058 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:19.864089 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82495.3us
W20260709 12:18:19.866173 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91265.8us
I20260709 12:18:19.866987 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:19.826874 27730 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:19.868147 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 311) took 1147 ms. Trace:
I20260709 12:18:19.868772 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:19.870193 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:19.870690 29036 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.719s	user 0.007s	sys 0.005s
I20260709 12:18:19.871517 29097 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:19.872354 29097 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:19.872830 29097 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:19.873427 29156 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.873867 29097 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.874881 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1389) took 1170 ms. Trace:
I20260709 12:18:19.806347 28995 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:18:19.889652 28952 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:19.841568 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:19.900518 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59099.6us
I20260709 12:18:19.866953 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:19.868806 28332 rpcz_store.cc:276] 0709 12:18:18.720304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.720449 (+   145us) service_pool.cc:224] Handling call
0709 12:18:19.867821 (+1147372us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54003705d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:19.868028 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1574678400}
I20260709 12:18:19.878356 29181 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:19.878472 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.879266 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.879563 29078 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.882351 29139 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:19.884307 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1289) took 1198 ms. Trace:
I20260709 12:18:19.885170 29087 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.886732 29016 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:19.887223 29232 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:19.887036 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:19.875406 27582 rpcz_store.cc:276] 0709 12:18:18.704475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.705164 (+   689us) service_pool.cc:224] Handling call
0709 12:18:18.705523 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.567089 (+861566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:19.874537 (+307448us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:19.874784 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1941144960}
I20260709 12:18:19.827733 29229 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.890679 29127 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:19.892714 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1376) took 1184 ms. Trace:
I20260709 12:18:19.893821 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:19.896019 27580 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:19.897033 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:19.898461 27661 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:19.840101 29064 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:19.909140 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.903623 27738 rpcz_store.cc:276] 0709 12:18:18.686247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.688352 (+  2105us) service_pool.cc:224] Handling call
0709 12:18:18.688679 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.733459 (+1044780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:19.884004 (+150545us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:19.884223 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2168043776}
I20260709 12:18:19.919943 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:19.927090 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.902621 28952 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:19.915186 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.908564 29016 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:19.913404 29065 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:19.939224 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:19.940276 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:19.871132 29036 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:19.919075 28989 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:19.942032 27732 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:19.913745 29127 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:19.941294 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:19.951782 28332 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.940984 29065 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:19.943964 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72796.7us
I20260709 12:18:19.930392 28988 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:19.919487 27586 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:19.940891 29016 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.937664 29181 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:19.965781 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 100551us
I20260709 12:18:19.967386 27734 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:19.938596 27729 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:19.954097 29065 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:19.944433 29064 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.006160 27733 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:19.965797 27730 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
W20260709 12:18:19.976400 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.004683 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:19.972532 29130 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.970665 29233 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:20.007139 29072 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.000842 28958 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.033262 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:19.979693 28988 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:20.010210 29234 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:20.033859 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.037262 29207 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.038218 29107 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.038820 29107 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:20.041535 29240 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.321s	user 0.001s	sys 0.001s
I20260709 12:18:20.041958 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:20.042536 29233 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:20.043198 29214 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:20.043748 29214 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:20.044147 29214 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:20.044274 29233 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.048833 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.049721 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
W20260709 12:18:20.053562 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 116784us
I20260709 12:18:20.056269 29234 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.058621 29234 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.060122 29036 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.061779 28958 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.029263 29165 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.064069 28958 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.063825 29165 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.064858 29130 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.065814 27662 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:20.066465 29165 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:20.067304 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54762.6us
I20260709 12:18:20.068387 29214 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.075888 29181 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:20.077692 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62404.5us
I20260709 12:18:20.078542 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.078807 27662 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:20.079253 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.080543 29087 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:20.081024 29078 raft_consensus.cc:2764] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260709 12:18:20.081235 29087 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:20.082455 27581 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:20.083696 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.085158 29172 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.085886 27660 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:20.082882 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.090674 27658 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
W20260709 12:18:20.093218 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.087347 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:20.080477 27580 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:20.131510 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53073.3us
I20260709 12:18:20.132915 27584 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:20.133404 27584 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:20.129758 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:20.138306 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51039.5us
I20260709 12:18:20.116335 29165 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:20.178905 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:20.159183 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 309068us
I20260709 12:18:20.034384 29064 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.085810 29172 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.037994 29239 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.239s
I20260709 12:18:20.038641 29188 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.125s
I20260709 12:18:20.164742 29165 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:20.192883 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.192992 28958 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:20.223677 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 189851us
I20260709 12:18:20.226104 28332 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:20.196141 29107 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.164147 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:20.194435 28971 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:20.082476 29087 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.211395 27584 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.216228 29153 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:20.237931 29180 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:20.230850 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:20.244858 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86585.1us
I20260709 12:18:20.247409 27660 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.250734 29173 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:20.251520 29173 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.229562 27586 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:20.261384 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:20.262445 29094 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:20.262986 29094 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.266480 29165 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.267684 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.271859 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:20.254053 29180 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.276932 29180 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.278945 29165 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:20.241056 29232 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.257166 28958 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.281579 29204 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.282173 29204 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.246148 29097 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:20.296093 27662 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:20.317924 27628 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:20.283923 29094 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:20.268460 28971 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.348587 29232 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:20.330614 29153 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:18:20.373206 28994 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:20.373330 29153 consensus_queue.cc:1243] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:20.331362 29097 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.373900 28994 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:20.374954 27735 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:20.375880 29097 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.377846 28958 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:20.372709 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51309.2us
I20260709 12:18:20.380173 29172 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.389542 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:20.392176 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.403496 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.405176 27662 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:20.400405 29036 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.408034 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:20.379491 29173 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.320456 28988 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:20.428824 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:20.435835 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 174623us
I20260709 12:18:20.406097 27660 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
W20260709 12:18:20.442489 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:20.406643 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 269887us
W20260709 12:18:20.446446 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 209888us
W20260709 12:18:20.436115 28330 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.002s
I20260709 12:18:20.452889 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:19.924384 27656 rpcz_store.cc:276] 0709 12:18:18.708341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.835873 (+127532us) service_pool.cc:224] Handling call
0709 12:18:18.880671 (+ 44798us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:18.933309 (+ 52638us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:18.985503 (+ 52194us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:19.058457 (+ 72954us) log.cc:844] Serialized 26 byte log entry
0709 12:18:19.579243 (+520786us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:19.617482 (+ 38239us) spinlock_profiling.cc:243] Waited 8.39 ms on lock 0x7b18003f9280. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:19.664520 (+ 47038us) log.cc:844] Serialized 14 byte log entry
0709 12:18:19.892137 (+227617us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b54000666a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c3733 00007f0e9a348dbf 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c
0709 12:18:19.892199 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:19.892331 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:19.892378 (+    47us) raft_consensus.cc:1685] finished
0709 12:18:19.892392 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:19.892529 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39227,"prepare.queue_time_us":125401,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":6839,"raft.queue_time_us":201608,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":31757,"spinlock_wait_cycles":75322496,"thread_start_us":573610,"threads_started":2,"wal-append.queue_time_us":501920}
I20260709 12:18:20.374982 28994 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:20.450033 29156 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.576s	user 0.009s	sys 0.004s
I20260709 12:18:20.463510 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.463352 29156 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:20.466224 29253 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.151s	user 0.001s	sys 0.001s
I20260709 12:18:20.467301 29064 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.466174 29156 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.468892 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.470623 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.454706 28330 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:20.472167 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 311) took 1014 ms. Trace:
I20260709 12:18:20.472570 28330 rpcz_store.cc:276] 0709 12:18:19.469946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.740861 (+270915us) service_pool.cc:224] Handling call
0709 12:18:20.472078 (+731217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:20.440557 29180 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.475142 29130 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.475306 29266 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.476192 29180 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:20.476615 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.458412 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.478259 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.478338 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.478858 29231 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.479242 27659 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:20.479390 29231 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.480963 29204 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.481786 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.482514 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:20.478536 28971 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.485795 28333 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:18:20.492147 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:20.492981 27583 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.463192 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.408119 29036 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.494735 27662 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:20.501724 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:20.503791 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:20.506949 29065 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.511595 27586 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:20.512369 29036 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.475862 29266 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.497483 29072 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:20.518275 28328 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:18:20.500478 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:20.506996 29094 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.520381 29270 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:20.520601 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:20.521973 27627 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:20.522475 29094 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.523205 29094 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.523792 29094 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.524129 27658 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:20.524288 29204 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:20.525920 29204 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:18:20.526031 29016 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.518s	user 0.000s	sys 0.009s
I20260709 12:18:20.526314 29204 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.526763 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:20.526862 29204 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:20.526477 29016 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.520856 29270 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:20.572562 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.575416 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.534554 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.532553 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.611135 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.547290 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:20.539402 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:20.543474 29097 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:20.566421 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.498416 29130 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:20.542737 29156 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:20.536535 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.537240 29214 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.527096 27589 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:20.534291 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:20.532076 28943 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.536780 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:20.537328 28995 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:18:20.619823 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72646.9us
I20260709 12:18:20.575809 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:20.623895 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87192.8us
I20260709 12:18:20.626080 27738 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.623872 27736 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:20.637619 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:20.623049 29078 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:20.639202 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:20.638967 29078 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.639994 29266 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.622857 29139 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:20.574198 29065 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.609264 29270 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:20.557173 29165 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:18:20.640559 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.557080 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.649036 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.643076 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.644470 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.653542 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:20.645358 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 106070us
I20260709 12:18:20.648473 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:20.665375 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:20.646083 29269 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:18:20.645565 28971 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:20.642737 29232 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:20.647074 29094 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.649279 29159 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.645732 29181 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:20.649448 27656 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:20.644129 29156 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:20.641474 29078 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.685281 27585 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:20.643445 28943 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.641607 29139 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.644410 29064 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.666613 29130 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.686299 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.671099 29065 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.685359 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:20.704267 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:20.716826 29081 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:20.717387 29081 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.717841 29081 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.697228 29232 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:20.716409 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.721550 27732 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:20.723542 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.728080 28995 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.698571 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:20.714035 29233 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.730814 28328 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:20.696398 28971 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:20.730756 29233 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.732116 28971 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.733266 28943 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.734237 29232 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.735409 27583 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:20.745043 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.744837 29064 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.745251 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.745764 27656 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:20.745950 27662 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:20.746265 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:20.746860 27731 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:20.712141 29156 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.695524 28989 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:18:20.712390 29139 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:20.771116 27729 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:20.773020 29127 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.774621 29127 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:20.771236 29241 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.775683 29241 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.751592 28995 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:18:20.749888 29128 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:20.777740 29128 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:20.773265 29204 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:20.747049 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95221.3us
I20260709 12:18:20.720166 29094 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:20.779410 28996 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.787463 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:20.734911 29233 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.749621 29280 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:20.747711 29016 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.747143 29064 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.802597 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:20.805194 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:20.805636 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:20.808820 29057 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.809265 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:20.810050 29057 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.803815 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.811470 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:20.723944 29130 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.814193 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:20.814509 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:20.807127 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 114643us
I20260709 12:18:20.751448 29036 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:18:20.808094 28989 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:20.819449 29057 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.779043 29128 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.820750 28994 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.822171 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:20.822460 29204 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.725975 29159 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:20.823392 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.798169 29094 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:20.836855 29094 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.715957 29181 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.800529 28995 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:20.823098 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106773us
I20260709 12:18:20.845963 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:20.748790 29209 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.835911 29066 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:18:20.836232 28332 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:20.868966 29066 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:20.869513 29066 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:20.840433 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.870735 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:20.872157 27628 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:20.872268 29271 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:20.827906 29016 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:20.872939 29271 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.873144 29016 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:20.873394 29271 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.817066 29233 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:20.800444 28996 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.850270 28988 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:20.830754 28958 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:20.851116 29159 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.837028 29128 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.852988 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:20.853353 29065 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.853763 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:20.836246 29241 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.857921 29130 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.860548 29165 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:20.898663 29165 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.900225 27729 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:20.900123 29094 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.830994 28994 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.904214 28994 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:20.904304 29271 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.867468 28995 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.909096 29094 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.872537 27628 leader_election.cc:403] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:18:20.841641 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:20.878541 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1297) took 1472 ms. Trace:
I20260709 12:18:20.791646 29270 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:20.879293 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140799us
I20260709 12:18:20.889123 27738 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:20.891666 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.874368 29016 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:20.895104 28996 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.896719 29181 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:20.897779 29229 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:20.885802 29204 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:20.889757 29209 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:20.913790 28995 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:20.870738 29066 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.918187 29165 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:20.962177 29280 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:20.964354 27656 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:20.922196 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:20.923023 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1236) took 1059 ms. Trace:
I20260709 12:18:20.866123 29081 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:20.937763 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:20.970633 27663 rpcz_store.cc:276] 0709 12:18:19.863206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.880288 (+ 17082us) service_pool.cc:224] Handling call
0709 12:18:19.880680 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:19.881206 (+   526us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:19.881236 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:19.881782 (+   546us) log.cc:844] Serialized 26 byte log entry
0709 12:18:20.015585 (+133803us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:20.015626 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:20.015695 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:20.921099 (+905404us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b24002ac248. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:18:20.922708 (+  1609us) raft_consensus.cc:1685] finished
0709 12:18:20.922728 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:20.922962 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8704,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":177,"spinlock_wait_cycles":58213632,"thread_start_us":133411,"threads_started":1,"wal-append.queue_time_us":348622}
W20260709 12:18:20.939240 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67295.1us
I20260709 12:18:20.911505 27733 rpcz_store.cc:276] 0709 12:18:19.406641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.771794 (+365153us) service_pool.cc:224] Handling call
0709 12:18:19.775459 (+  3665us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:20.007120 (+231661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:20.878191 (+871071us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b540017e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:20.878462 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":468440832}
I20260709 12:18:20.956471 29153 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:20.957135 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 128170us
I20260709 12:18:20.943189 29180 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:20.970665 28332 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:18:20.968763 29270 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:20.998611 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 169149us
I20260709 12:18:21.028584 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:20.997833 27659 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:21.032415 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:21.002262 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.037075 29036 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:21.026849 29066 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.048805 27738 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:21.048563 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:21.037075 29204 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.055966 29204 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:21.058449 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.058591 29204 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.059444 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:20.988220 29139 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.060557 27657 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260709 12:18:21.043345 29156 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.044970 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.046160 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.046039 29036 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.034406 28994 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:18:20.976050 29288 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20260709 12:18:21.066982 28996 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:21.086386 29062 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.085681 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.098445 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.099448 27587 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:21.105558 29232 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:21.085655 29036 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.106518 29232 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.109495 29232 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.110951 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.111735 29159 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:21.113590 29159 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.113761 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.114462 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:21.114188 29159 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.085333 29204 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.115869 27663 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:21.115429 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.121657 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.116082 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:21.102932 28958 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:21.098470 28952 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.128322 29159 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.130541 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.136590 27658 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:21.135392 28335 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:21.140501 28332 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:21.145292 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.087688 28330 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:21.124377 29127 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:21.129530 29270 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:21.155019 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.158493 27658 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:21.144971 29280 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.160949 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.166713 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:21.166632 29280 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.169019 29280 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
W20260709 12:18:21.173619 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 85) took 2976 ms (client timeout 1501 ms). Trace:
I20260709 12:18:21.152415 28943 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:21.174548 29270 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:21.154158 29127 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.163852 29299 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.175655 29270 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.126233 29204 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.175904 29299 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:21.165124 29297 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:18:21.174137 27735 rpcz_store.cc:269] 0709 12:18:18.197273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.197397 (+   124us) service_pool.cc:224] Handling call
0709 12:18:21.173059 (+2975662us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54002a2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:21.173491 (+   432us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5005443840}
I20260709 12:18:21.179744 29081 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.180296 29231 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.181288 28958 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.182122 28328 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:21.183126 29139 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.174669 29165 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.156454 29159 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:21.175674 29127 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.186801 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:21.189525 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.190351 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.190670 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.191098 29062 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:21.191679 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:21.209005 27736 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.192399 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.192526 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:21.209934 27734 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.193395 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:21.198419 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:21.210676 27658 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.207142 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.213088 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.212915 29062 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.200311 28943 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:21.207026 29159 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.219718 28952 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.223286 29165 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.226984 29062 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.205463 28958 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.228873 28958 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.228892 29299 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.216974 29016 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:21.221009 29271 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.221138 27664 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:21.221405 27730 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:21.221715 29159 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.241415 27659 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:21.246765 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.250761 27587 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:21.242437 29288 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 12:18:21.265192 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.266268 27589 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:21.268040 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.268980 27587 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.268170 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.272897 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.273824 27659 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:21.275763 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.276391 27582 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.282505 29087 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:21.283010 29087 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:21.285210 27584 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:21.285974 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.267824 29288 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:18:21.308988 27586 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:21.286540 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.310177 29232 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:21.310643 27580 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:21.310894 29232 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:21.311769 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63169.6us
I20260709 12:18:21.313158 29232 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.313795 29173 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:21.315223 29173 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:21.316118 29173 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:21.316473 29173 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:21.315246 27482 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:21.317328 29173 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.318492 29173 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.287707 28943 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.319741 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.320402 27730 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:21.321406 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.255561 29097 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:18:21.293954 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:21.299729 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.303705 29127 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.287844 29204 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.314097 27732 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:21.257324 29016 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:18:21.322736 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.364200 27734 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:21.365131 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:21.305112 29087 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.236462 29036 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.328019 27736 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:21.336129 27658 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:21.353268 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.327803 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
W20260709 12:18:21.484310 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.522826 27583 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:21.523494 27738 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:21.496447 27584 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:18:21.486275 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.506904 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.527160 27662 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:21.377348 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:21.386425 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63768.9us
I20260709 12:18:21.524212 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1469) took 1437 ms. Trace:
I20260709 12:18:21.365355 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.365132 27587 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:21.366663 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:21.529035 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 151804us
W20260709 12:18:21.521353 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.529632 27584 rpcz_store.cc:276] 0709 12:18:20.087450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:20.087560 (+   110us) service_pool.cc:224] Handling call
0709 12:18:20.104204 (+ 16644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:21.284547 (+1180343us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b58000a2ed0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:18:21.523970 (+239423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:21.524157 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344452864}
W20260709 12:18:21.423974 29187 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.027s	user 0.005s	sys 0.008s
I20260709 12:18:21.532179 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:21.539743 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.544297 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.547683 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.422895 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.455888 29139 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:21.386205 29301 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:21.386468 29128 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:21.543483 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.364224 29165 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:21.423866 29299 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:21.555521 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:21.375792 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81934.7us
I20260709 12:18:21.539177 29081 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:21.418552 29127 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.540407 29288 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:18:21.547006 29187 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.153s	user 0.007s	sys 0.010s
I20260709 12:18:21.532020 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.560127 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.564024 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.617759 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:21.566207 29288 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.565197 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.561611 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:21.565147 29232 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:21.568063 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.625011 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:21.562809 28995 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:21.625293 27730 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:21.558318 29128 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.639657 29016 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:21.639987 29128 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.640322 29016 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:21.640563 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:21.557009 29180 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.604266 29293 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.641847 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.604349 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.642264 29293 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.555449 29207 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:21.554910 28994 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:21.643076 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.644032 29207 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:21.385895 29159 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:21.619280 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75932.2us
I20260709 12:18:21.644403 29072 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:21.644789 29293 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.644950 29072 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.645380 29072 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.645982 29159 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.646831 27729 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:21.622324 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60803us
I20260709 12:18:21.648382 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.648943 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.649711 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.649718 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.650503 27734 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:21.651218 28996 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:21.651664 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.641458 29057 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:21.652822 29057 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.653215 29057 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:21.654399 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.655252 29036 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.654824 27730 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:21.656452 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.656630 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.641423 29180 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.658476 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.567485 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.554365 29139 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.602730 28330 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
W20260709 12:18:21.629966 29302 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.258s	user 0.002s	sys 0.000s
I20260709 12:18:21.564020 29299 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.624545 29288 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.641423 29016 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.642398 29181 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:21.557703 29239 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:21.643740 28994 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.618517 27658 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.563417 29081 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.580816 29087 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:21.661324 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:21.662760 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:21.717063 27736 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:21.664052 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:21.602329 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.666712 29139 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.671720 27584 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:21.616652 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:21.672498 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.639273 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:21.563691 29165 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.679239 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.730718 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:21.656167 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.691113 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.558456 29301 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.743595 27662 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:18:21.690645 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.697466 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.698985 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.670740 29209 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:21.746229 27582 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:21.710274 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.670701 29180 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.714548 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:21.713949 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.671857 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.652837 28996 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:18:21.725705 27732 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.638407 28995 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.711469 29081 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.676507 29065 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
W20260709 12:18:21.747537 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144084us
I20260709 12:18:21.758942 29280 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:21.759446 29280 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.759835 29280 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:21.721446 28943 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.761422 28943 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.763518 28943 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.746304 29209 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:21.721000 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.748703 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:21.767690 27732 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:21.768882 27732 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:21.749071 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 200352us
I20260709 12:18:21.747784 27588 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:21.740541 27738 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:21.770467 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:21.771245 27732 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 12:18:21.752830 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62292.5us
I20260709 12:18:21.758512 29207 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.712306 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:21.742513 29165 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:21.729691 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.001s (SENT) [suppressed 10 similar messages]
I20260709 12:18:21.670302 29232 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.765753 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.769022 27583 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:21.726867 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:21.773545 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.744846 29301 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.689718 29128 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.707887 29181 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.791606 29128 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.791781 29181 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.792842 28337 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:21.788767 27730 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.794773 27663 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:21.795995 29209 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.797317 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.798058 28994 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.798623 28994 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:21.799751 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.800117 29299 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:21.800686 29299 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:21.801025 29299 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.801355 29299 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.770125 28958 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:21.808657 28958 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.809088 28958 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:21.767087 29231 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:21.731262 29239 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.812940 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.814632 27582 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:21.814857 29239 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.789547 29066 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:21.816146 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.818336 27584 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:21.821961 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.823779 27658 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:21.775764 29127 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:21.826628 27658 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:21.799108 28994 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:21.828840 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:21.794601 29057 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.839093 27589 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:21.840200 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.840876 27584 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:21.842960 29127 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:21.825306 29066 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.844064 29127 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.845638 29066 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:21.848701 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:21.846489 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.849817 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:21.835427 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 193640us
I20260709 12:18:21.860240 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.861204 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.865232 27734 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:21.865324 27584 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:21.836678 29156 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.873692 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.875295 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:21.876178 27658 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:21.848992 29180 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:21.881505 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.843029 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:21.828464 29239 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 12:18:21.882983 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:21.865789 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.869060 29204 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:21.843197 29081 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.899919 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:21.900785 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:21.926854 27583 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:21.927373 27733 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:21.927942 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:21.880285 29156 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.896113 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:21.897557 28995 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.931008 29156 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.887751 29181 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 12:18:21.924980 29311 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.925298 28952 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:21.938824 27655 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20260709 12:18:21.951861 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:21.887739 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:21.955349 27553 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:21.977933 27553 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:21.957679 29232 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:21.933416 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:21.954463 29305 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.596s	user 0.002s	sys 0.000s
I20260709 12:18:21.966617 27735 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:21.902777 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:22.000615 27735 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:22.003172 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34456
I20260709 12:18:22.004768 27735 raft_consensus.cc:606] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Received request to transfer leadership
I20260709 12:18:22.004933 27659 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:22.005256 27735 raft_consensus.cc:614] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:22.006019 29128 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:22.007648 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:22.008213 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.008982 27662 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.013955 28958 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:21.935948 28996 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:18:21.993173 29311 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:21.992120 29279 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:22.022895 27662 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:21.990928 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63053.2us
I20260709 12:18:21.999431 27658 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:22.034958 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 133248us
I20260709 12:18:21.977425 27655 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:22.045400 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:22.045686 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 316012us
I20260709 12:18:22.046224 27662 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:22.047631 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:21.998534 28995 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:22.023435 29279 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:22.058753 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:22.022663 29311 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.067778 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.069655 27663 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:21.931919 28994 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.018847 29280 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.073851 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:22.079316 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 217540us
I20260709 12:18:22.082531 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.024812 29288 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.054133 29128 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:18:22.041991 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 153093us
I20260709 12:18:22.040632 29315 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:22.055442 27657 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:22.108806 27660 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:22.111928 27487 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 228) took 6308 ms. Trace:
I20260709 12:18:22.116981 27658 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:22.088784 29285 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:22.078977 29312 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:22.064893 29231 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:22.015264 29299 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.124920 28330 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8.
I20260709 12:18:22.066313 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:22.152727 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70316.2us
I20260709 12:18:22.152865 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:22.153795 27737 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:18:22.155185 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:22.155506 29315 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:22.156387 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.156548 29209 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:22.157280 29315 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:22.157817 27628 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:22.158133 27628 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:18:22.117713 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 257528us
I20260709 12:18:22.140123 29311 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:18:22.159907 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.095218 29153 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:22.076385 29156 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:18:22.162346 29156 consensus_queue.cc:1243] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:22.151268 29271 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:22.162894 29156 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:22.124244 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.085485 29072 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.103010 27580 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:22.112396 27487 rpcz_store.cc:276] 0709 12:18:15.803710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.804637 (+   927us) service_pool.cc:224] Handling call
0709 12:18:22.111834 (+6307197us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:20.253682 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:20.416595 (+162913us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:20.434029 (+ 17434us) write_op.cc:435] Acquired schema lock
0709 12:18:20.443406 (+  9377us) tablet.cc:662] Decoding operations
0709 12:18:20.482550 (+ 39144us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:20.482653 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:20.482682 (+    29us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:18:20.483673 (+   991us) tablet.cc:701] Row locks acquired
0709 12:18:20.483725 (+    52us) write_op.cc:260] PREPARE: finished
0709 12:18:20.483888 (+   163us) write_op.cc:270] Start()
0709 12:18:20.484061 (+   173us) write_op.cc:276] Timestamp: P: 1783599500483830 usec, L: 0
0709 12:18:20.484100 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 12:18:20.484845 (+   745us) log.cc:844] Serialized 4283 byte log entry
0709 12:18:20.964472 (+479627us) op_driver.cc:464] REPLICATION: finished
0709 12:18:21.072928 (+108456us) write_op.cc:301] APPLY: starting
0709 12:18:21.073034 (+   106us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:21.073299 (+   265us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:21.086606 (+ 13307us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:21.184079 (+ 97473us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:21.184370 (+   291us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0709 12:18:21.184438 (+    68us) write_op.cc:312] APPLY: finished
0709 12:18:21.184778 (+   340us) log.cc:844] Serialized 77 byte log entry
0709 12:18:21.184974 (+   196us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:21.185098 (+   124us) write_op.cc:454] Released schema lock
0709 12:18:21.185160 (+    62us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:21.185562 (+   402us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:21.185596 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":48275,"reported_tablets":11,"spinlock_wait_cycles":15488,"child_traces":[["op",{"apply.queue_time_us":108105,"apply.run_cpu_time_us":4924,"apply.run_wall_time_us":112613,"num_ops":10,"prepare.queue_time_us":194803,"prepare.run_cpu_time_us":35592,"prepare.run_wall_time_us":437604,"raft.queue_time_us":456775,"raft.run_cpu_time_us":16523,"raft.run_wall_time_us":68157,"replication_time_us":480153,"thread_start_us":694964,"threads_started":5,"wal-append.queue_time_us":227571}]]}
I20260709 12:18:22.131438 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.172013 27658 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:22.080376 28995 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.109839 29128 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:22.109579 29065 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:18:22.139539 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.144104 28329 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:22.128017 29285 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.144869 29279 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.145933 29299 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.148864 29181 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:22.102360 28989 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:18:22.120658 28328 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:22.150533 29280 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:22.157284 29209 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:22.164300 29288 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.165427 27583 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.164783 29156 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.154995 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:22.090152 29057 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.167883 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.091725 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:22.115545 29139 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.212632 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:22.220396 27662 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:22.157829 29315 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.172631 29204 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.199172 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1409) took 1004 ms. Trace:
I20260709 12:18:22.200831 29232 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:22.214974 29285 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:22.223529 27731 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:22.288564 27732 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:22.219990 29127 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:22.265260 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.223841 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.273454 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.252465 27730 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:18:22.257169 29229 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:22.191154 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:22.248024 29209 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.252452 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:22.276118 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184483us
I20260709 12:18:22.272969 29036 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:22.259685 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.271807 29153 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:22.305392 27737 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:22.247879 29181 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.279505 29081 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:22.286273 29057 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.316987 27658 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
W20260709 12:18:22.307461 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152509us
I20260709 12:18:22.297724 29065 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.302402 29328 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:22.302836 29209 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:22.328315 27651 tablet_service.cc:1558] Processing DeleteTablet for tablet 310b1b37298244c0bd5bbe2b0c7328d7 with delete_type TABLET_DATA_TOMBSTONED (TS 9c09719959a54e8eb66d2041dd9c3113 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49990
I20260709 12:18:22.328236 28995 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:22.303006 29036 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:22.300361 29127 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.347177 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:22.325551 29062 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:22.312608 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 160594us
I20260709 12:18:22.301499 29087 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:18:22.315485 29311 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:22.382258 29311 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:22.383694 29311 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.280421 27729 rpcz_store.cc:276] 0709 12:18:21.189082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.189211 (+   129us) service_pool.cc:224] Handling call
0709 12:18:22.151228 (+962017us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:22.193855 (+ 42627us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1007529344}
I20260709 12:18:22.329988 29328 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.388454 29328 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:22.389231 29310 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.571s	user 0.006s	sys 0.000s
I20260709 12:18:22.287786 29128 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.393335 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.358093 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:22.322310 29153 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.394847 29299 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.395543 29153 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.364631 28335 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:22.321321 27656 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 3
W20260709 12:18:22.398128 27655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1249) took 2503 ms (client timeout 2001 ms). Trace:
W20260709 12:18:22.398615 27655 rpcz_store.cc:269] 0709 12:18:19.894688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.896731 (+  2043us) service_pool.cc:224] Handling call
0709 12:18:22.397875 (+2501144us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:22.398017 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3302557568}
I20260709 12:18:22.380993 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.402504 29328 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.305928 29229 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.404003 29270 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:22.405105 29328 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.405426 29229 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.394532 27664 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:18:22.399768 27659 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:22.400441 29299 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.409509 29299 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.412896 29072 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.415313 29181 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:22.415802 28958 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.416635 29139 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:22.421600 29337 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:18:22.424243 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:22.424597 29127 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.424995 28330 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.427588 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:22.329233 29315 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.447466 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.450713 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.451987 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.457973 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.458654 27589 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 3
W20260709 12:18:22.465438 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132618us
I20260709 12:18:22.326112 29057 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.475459 27702 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8.
I20260709 12:18:22.476022 27702 leader_election.cc:403] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:22.485989 27589 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:22.330755 29065 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.339152 28943 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.098s
I20260709 12:18:22.495234 29350 tablet_replica.cc:333] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:18:22.495481 29348 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:22.496078 27656 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
W20260709 12:18:22.485965 29336 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:18:22.312896 29156 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.487958 29207 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:18:22.508900 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206524us
I20260709 12:18:22.507783 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:22.493047 28994 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8.
I20260709 12:18:22.506367 28943 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:22.511158 28943 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.514971 28943 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.515141 28988 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:22.501219 29036 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.549049 29107 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.549700 29107 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.551455 28988 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:22.561153 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.561980 27580 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:22.562218 27729 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:22.513489 29207 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:22.567417 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:22.567991 29207 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.570329 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:22.574044 29204 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
W20260709 12:18:22.574391 29331 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.313s	user 0.002s	sys 0.000s
W20260709 12:18:22.575664 27535 auto_rebalancer.cc:893] Removing replace marker failed: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:18:22.578461 27535 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
I20260709 12:18:22.582064 28994 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:22.582746 28994 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.583249 28994 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.568876 27702 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:22.578879 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:18:22.591620 29107 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:22.572602 29270 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:18:22.553416 28988 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.597026 29349 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:22.597604 29349 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.601317 29271 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:22.602054 29271 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:22.605830 29360 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:22.606530 29360 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:22.611860 29360 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:22.618983 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152326us
I20260709 12:18:22.615885 29209 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.602674 29349 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.625957 29365 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.637019 29365 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.633683 29349 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.636795 29016 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
W20260709 12:18:22.643597 29335 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.356s	user 0.002s	sys 0.002s
I20260709 12:18:22.644546 29209 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:22.646351 29366 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:22.646869 29366 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.647809 29366 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.648473 29315 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.649343 29366 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:22.652151 29285 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.654381 29285 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.654305 29288 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:22.654716 29232 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.656647 27583 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:22.626943 28971 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:18:22.661062 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.637327 27657 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:22.666180 29128 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:22.655228 29232 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.663537 29362 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:18:22.679286 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:22.679692 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 170065us
I20260709 12:18:22.693444 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.694859 29293 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.679937 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:22.684064 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:22.696257 28330 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:22.686753 27656 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:22.687669 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.697911 27660 raft_consensus.cc:2410] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:18:22.685477 29232 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.695432 29293 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.700241 29293 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.700534 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.701308 27734 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:22.702471 29187 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.705354 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.706492 27729 raft_consensus.cc:2368] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:22.707757 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.708920 27730 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:22.711509 29062 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:22.712296 29062 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.715042 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.715696 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:22.723531 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.724992 27734 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:22.728595 29325 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.437s	user 0.003s	sys 0.000s
I20260709 12:18:22.730567 29107 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:22.730928 29187 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:22.757079 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:22.759600 29347 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:22.760035 29347 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.760370 29347 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:22.763314 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:22.763618 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:22.764302 27729 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:22.764415 27736 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:22.765590 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.766333 27736 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:22.768131 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.731545 29057 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.731375 29107 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.781193 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.733198 28994 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.734292 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1320) took 1186 ms. Trace:
I20260709 12:18:22.799989 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.736003 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1328) took 1027 ms. Trace:
I20260709 12:18:22.741073 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.743671 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:22.808844 27730 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:22.743916 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.744383 29209 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:22.812341 29209 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.816218 29209 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.746168 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:22.746497 28995 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:22.819517 28995 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.821694 28995 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.748560 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.822420 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:22.746101 29270 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:18:22.773550 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.765684 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:22.774904 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147200us
I20260709 12:18:22.733378 29062 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.824270 27733 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:22.825402 27733 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:22.826831 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.827510 27588 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:22.781960 27587 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:22.806108 27732 rpcz_store.cc:276] 0709 12:18:21.708792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.772317 (+ 63525us) service_pool.cc:224] Handling call
0709 12:18:21.825335 (+ 53018us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:22.289028 (+463693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:22.735722 (+446694us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:22.735935 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1064728320}
I20260709 12:18:22.828553 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.782058 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.783805 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:22.829101 27660 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:22.829227 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.829483 27736 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:22.830405 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.800503 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.831883 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.832774 27662 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:22.833654 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.802085 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.803905 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:22.838639 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.839171 27589 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260709 12:18:22.840698 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.840862 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.841464 27660 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:22.842840 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96788.6us
I20260709 12:18:22.844308 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.849598 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.774483 29362 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:18:22.853353 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.856091 27663 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:22.856367 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.794308 29065 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:22.853325 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:22.858403 27732 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:22.853432 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.854589 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:22.857232 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.805660 27731 rpcz_store.cc:276] 0709 12:18:21.551502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.657682 (+106180us) service_pool.cc:224] Handling call
0709 12:18:22.166281 (+508599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:22.295720 (+129439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:22.734037 (+438317us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54001d42d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:22.734221 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1167624448}
I20260709 12:18:22.862224 27732 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:22.865160 29350 raft_consensus.cc:2243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:22.859167 27659 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:22.866286 29350 raft_consensus.cc:2272] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:22.866567 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.867467 27659 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:22.868076 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:22.868757 28994 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.870712 29107 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:22.868319 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.873739 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.874750 27655 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:22.875811 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.876884 27655 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:18:22.877040 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:22.878525 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.879591 27655 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:22.871824 29204 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:22.880435 29204 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:22.880936 29204 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.881953 28330 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:22.882917 29279 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:22.884074 27661 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260709 12:18:22.860762 27662 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:22.885288 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.886442 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.886981 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.893445 27588 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:22.843544 29214 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:22.889326 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:22.911548 27658 raft_consensus.cc:2376] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5. Re-sending same reply.
I20260709 12:18:22.895043 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.887007 27661 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:18:22.896633 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.899091 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.924746 27664 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:22.891681 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:22.901927 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57697.8us
I20260709 12:18:22.902820 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:22.926015 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.902001 28958 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:22.838099 29181 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:22.906966 29347 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.908305 29365 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:22.909442 29356 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:18:22.875873 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:22.853483 29285 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:22.911228 29214 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.913534 27585 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:22.906991 27662 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:22.959594 27662 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:22.892947 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.900314 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:22.927111 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:22.928041 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:22.903200 27627 leader_election.cc:400] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:22.962431 27627 leader_election.cc:403] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:22.930076 27660 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
W20260709 12:18:22.963579 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63354.5us
I20260709 12:18:22.963874 27660 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:18:22.931222 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46426.4us
I20260709 12:18:22.933310 27732 raft_consensus.cc:2368] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:22.933892 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:22.967425 27588 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:22.935128 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45653.3us
I20260709 12:18:22.842830 29271 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.958580 27585 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:22.932566 28958 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:22.944839 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45897us
W20260709 12:18:22.982667 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.951113 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.984339 27588 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:22.944484 27664 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:22.975591 27587 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:22.983192 27658 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:22.909512 29204 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:22.993173 29204 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.993999 29369 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:22.916502 27655 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:22.969892 27586 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:18:22.972008 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68939.4us
I20260709 12:18:22.991009 29271 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:22.952828 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:22.995539 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:23.052120 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.999374 27735 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:23.044867 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93876.7us
I20260709 12:18:23.052476 27581 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.043538 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:23.053831 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.958312 29349 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:23.070550 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.060176 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:22.935824 27656 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:23.029477 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:23.143399 29204 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.945386 29181 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.042117 29350 ts_tablet_manager.cc:1916] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:22.985074 29232 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:23.020116 27585 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:22.985064 29288 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:23.020486 29285 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:22.985656 28958 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:23.215154 27585 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.194331 27588 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:23.036911 29214 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:23.060734 29359 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.037200 27729 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:23.185758 29315 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:23.307164 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 247076us
W20260709 12:18:23.267107 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 298427us
I20260709 12:18:23.209465 29271 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:18:23.271661 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 228909us
I20260709 12:18:23.255378 29285 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:23.300223 27655 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:18:23.325275 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 281027us
W20260709 12:18:23.306844 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 254821us
I20260709 12:18:23.196151 28330 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
W20260709 12:18:23.267045 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 301711us
I20260709 12:18:23.310985 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.212602 29128 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.101s
I20260709 12:18:23.326733 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.210104 29204 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:23.331868 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:23.335913 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.336591 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:23.334477 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.340073 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.314563 29081 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:23.321908 29062 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.325949 29285 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:23.315714 29159 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:18:23.335927 29315 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.336302 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:23.342509 29344 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:18:23.347186 29365 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:18:23.383069 29360 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:23.372876 29087 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:23.384274 29087 consensus_queue.cc:1243] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:23.368619 28994 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.385028 28994 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:23.393242 29340 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.086s	user 0.003s	sys 0.003s
W20260709 12:18:23.387034 29374 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.120s	user 0.002s	sys 0.000s
W20260709 12:18:23.376943 29368 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
W20260709 12:18:23.398260 29377 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:18:23.402266 29285 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.369244 29347 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:23.382633 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48274.2us
I20260709 12:18:23.415934 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.417241 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.417593 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.395114 29204 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:23.348219 29062 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.340339 29214 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:23.343303 29271 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20260709 12:18:23.374548 28995 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:23.379880 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:23.403296 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.422703 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.428867 29107 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.345768 29081 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.431001 29285 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:23.411350 28994 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:23.345353 29349 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:23.449030 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:23.457243 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.465509 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:23.425299 28989 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:18:23.467531 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50381us
I20260709 12:18:23.440835 27656 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:23.471248 28335 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:18:23.468742 27733 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:23.467422 28995 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:23.502403 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53486.3us
I20260709 12:18:23.507550 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.509091 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.510864 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:23.419837 29365 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260709 12:18:23.488960 29036 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.468402 29366 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:23.503600 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 167081us
I20260709 12:18:23.425046 29357 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20260709 12:18:23.443791 29128 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 12:18:23.488612 29299 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:23.513820 28995 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.530432 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.514688 29360 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:23.504738 29349 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:23.539178 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:23.516607 29107 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:23.530377 28994 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:18:23.568293 29107 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.540407 29288 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:23.570539 29107 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:23.571465 29288 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:23.574150 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:23.566773 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:23.576079 27733 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:23.537739 29078 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:23.566829 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.529790 29066 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:23.529558 29057 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:23.536851 29231 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.516486 29087 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
W20260709 12:18:23.592533 29378 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:18:23.593957 29270 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:23.534017 29315 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
W20260709 12:18:23.598361 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.536937 29348 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 12:18:23.540073 29360 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:23.566991 29343 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:23.619961 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:23.583475 29078 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:23.623399 27729 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:23.583781 29065 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:23.599707 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:23.303776 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:23.639640 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:23.635229 27482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 219) took 4266 ms. Trace:
I20260709 12:18:23.673063 27589 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:18:23.672911 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73318.3us
I20260709 12:18:23.659058 27584 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:23.665454 27588 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:18:23.674636 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.670492 28943 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:23.671411 29380 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:18:23.684919 29370 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.685483 29370 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:23.687660 29370 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:23.689189 29139 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:23.692166 29232 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:23.692646 29232 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:23.693022 29232 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.694525 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:23.695602 27589 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:23.669292 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:23.697901 29239 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:23.699276 29239 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:23.699703 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:23.701725 29081 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:23.702296 29301 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:23.702637 29391 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:23.703994 29299 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:23.704452 29299 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:23.705308 29299 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.705679 29349 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.631145 29359 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.661675 29360 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:23.709933 27581 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.656s	user 0.008s	sys 0.000s
I20260709 12:18:23.710284 27581 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:23.710697 29312 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:23.712342 29299 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:23.676201 27585 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:18:23.715760 28971 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 12:18:23.716588 29156 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.718864 29285 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:23.719391 29285 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.720252 29065 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:23.720732 29065 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:23.721032 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46463.2us
I20260709 12:18:23.721479 29285 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:23.722422 27627 leader_election.cc:400] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:23.722874 27627 leader_election.cc:403] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:23.671846 29078 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.670843 27483 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:23.633648 29393 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.724411 29078 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:23.724745 27483 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 2, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:23.728185 29383 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:18:23.680159 27661 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.749s	user 0.007s	sys 0.003s
I20260709 12:18:23.668191 29087 consensus_queue.cc:1243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:23.682406 29214 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:23.723294 28958 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.733562 27483 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 3 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:23.731506 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:23.743786 29393 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:23.744400 29393 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:23.745328 29393 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.752408 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:23.753846 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:23.742276 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 224850us
I20260709 12:18:23.767457 29301 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:23.774024 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.779284 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:23.780088 27588 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:23.722491 29062 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:23.746608 29360 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:23.739600 27661 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:23.672784 27482 rpcz_store.cc:276] 0709 12:18:19.368933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.369291 (+   358us) service_pool.cc:224] Handling call
0709 12:18:23.635147 (+4265856us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:21.631827 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:21.632445 (+   618us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:21.632483 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:18:21.632502 (+    19us) tablet.cc:662] Decoding operations
0709 12:18:21.632878 (+   376us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:21.632938 (+    60us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:21.632966 (+    28us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:18:21.633167 (+   201us) tablet.cc:701] Row locks acquired
0709 12:18:21.633185 (+    18us) write_op.cc:260] PREPARE: finished
0709 12:18:21.633346 (+   161us) write_op.cc:270] Start()
0709 12:18:21.633479 (+   133us) write_op.cc:276] Timestamp: P: 1783599501633299 usec, L: 0
0709 12:18:21.633500 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:18:21.634186 (+   686us) log.cc:844] Serialized 538 byte log entry
0709 12:18:21.935834 (+301648us) op_driver.cc:464] REPLICATION: finished
0709 12:18:22.261364 (+325530us) write_op.cc:301] APPLY: starting
0709 12:18:22.261451 (+    87us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:22.261556 (+   105us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:22.261739 (+   183us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:22.262011 (+   272us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:22.262220 (+   209us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:18:22.262286 (+    66us) write_op.cc:312] APPLY: finished
0709 12:18:22.262513 (+   227us) log.cc:844] Serialized 23 byte log entry
0709 12:18:22.262703 (+   190us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:22.262784 (+    81us) write_op.cc:454] Released schema lock
0709 12:18:22.262819 (+    35us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:22.263142 (+   323us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:22.263172 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":121542,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":321602,"apply.run_cpu_time_us":1766,"apply.run_wall_time_us":1766,"num_ops":1,"prepare.queue_time_us":312229,"prepare.run_cpu_time_us":2734,"prepare.run_wall_time_us":3155,"replication_time_us":292063,"thread_start_us":506270,"threads_started":2}]]}
I20260709 12:18:23.807021 29359 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4.
I20260709 12:18:23.768141 29301 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:23.812124 29301 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.778812 29239 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.782691 29209 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:23.783483 29383 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:18:23.843286 27588 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:23.823035 29204 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:23.823297 29187 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:23.881418 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.783120 29343 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:18:23.894695 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1331) took 1066 ms. Trace:
I20260709 12:18:23.895310 27735 rpcz_store.cc:276] 0709 12:18:22.827964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.840530 (+ 12566us) service_pool.cc:224] Handling call
0709 12:18:22.999161 (+158631us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:23.058565 (+ 59404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:23.894404 (+835839us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54000a73d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:23.894611 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364020992}
I20260709 12:18:23.824996 29279 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
W20260709 12:18:23.946822 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.884752 27580 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:23.834563 28995 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.882352 29347 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.890157 29204 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.913158 29187 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:23.951748 28988 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:23.991048 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211361us
I20260709 12:18:23.992419 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:24.008003 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:23.989243 29369 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.013337 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:23.967880 29347 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:24.026096 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 244703us
I20260709 12:18:24.024770 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:24.012871 29311 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.031558 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:24.003109 29065 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.001734 28988 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.011965 29232 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.054819 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:24.030400 29285 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:24.067443 27735 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:24.067440 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.069242 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:24.070611 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.071997 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:24.072400 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:24.049715 29036 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.531s	user 0.003s	sys 0.007s
I20260709 12:18:24.018927 29369 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.075374 29370 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:24.079101 27734 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:24.078812 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 261762us
I20260709 12:18:24.049726 28971 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:24.083400 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:24.159559 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76267.3us
W20260709 12:18:24.171316 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 92941.6us
I20260709 12:18:24.173503 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:24.052156 29349 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260709 12:18:24.080260 29288 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:24.084290 29081 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.083277 29370 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:24.064174 29087 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20260709 12:18:24.176795 27738 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:24.183352 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:24.183495 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.184688 27582 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:24.053131 29311 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.160743 27580 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:18:24.085059 29357 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20260709 12:18:24.171591 29360 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:24.181485 29062 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:18:24.180476 29386 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:18:24.180248 29396 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:18:24.173933 29204 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.176169 28952 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:24.193643 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:24.187469 29370 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:24.239735 27587 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:24.200760 29153 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:24.199414 29057 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:24.249424 29347 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:24.192113 29408 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:18:24.259017 29413 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:18:24.260481 29403 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:18:24.238281 28995 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.188674 29081 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.244417 29301 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:18:24.273257 29420 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:18:24.274551 29349 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:24.267547 27735 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:24.275650 29369 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.273312 28943 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.246356 29153 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:24.256479 29187 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.269440 29271 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.270136 29057 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:24.278056 29408 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:24.281869 29301 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:24.306295 29392 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.306326 29153 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.306814 27661 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:24.306998 29392 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.307736 29153 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:24.311491 29409 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:24.312076 29409 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:24.317407 29405 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:18:24.319582 29402 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:24.297698 29347 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.332060 29301 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.332685 29081 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:24.292265 28943 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.339514 27662 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:24.342768 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:24.344269 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.332825 28958 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:24.345793 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:24.346114 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.346515 27655 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:24.346720 27659 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:24.347579 29350 ts_tablet_manager.cc:1929] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:18:24.351055 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.339668 29409 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.352442 29350 log.cc:1199] T 310b1b37298244c0bd5bbe2b0c7328d7 P 9c09719959a54e8eb66d2041dd9c3113: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/wals/310b1b37298244c0bd5bbe2b0c7328d7
W20260709 12:18:24.352754 29414 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:18:24.353860 29350 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49990 (request call id 74) took 2066 ms. Trace:
I20260709 12:18:24.355436 27658 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260709 12:18:24.361089 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.364017 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.364902 27735 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:24.367758 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.370332 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1517) took 1503 ms. Trace:
I20260709 12:18:24.354255 29350 rpcz_store.cc:276] 0709 12:18:22.294498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.327951 (+ 33453us) service_pool.cc:224] Handling call
0709 12:18:22.494757 (+166806us) ts_tablet_manager.cc:1151] Deleting tablet 310b1b37298244c0bd5bbe2b0c7328d7
0709 12:18:22.494866 (+   109us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:18:23.705598 (+1210732us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:24.347444 (+641846us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:24.353755 (+  6311us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":301871,"tablet-delete.queue_time_us":145342,"thread_start_us":80146,"threads_started":1}
I20260709 12:18:24.305099 29271 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.373409 28958 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:24.370836 27583 rpcz_store.cc:276] 0709 12:18:22.866575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.969764 (+103189us) service_pool.cc:224] Handling call
0709 12:18:22.970020 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:22.985916 (+ 15896us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:18:23.061123 (+ 75207us) log.cc:844] Serialized 14 byte log entry
0709 12:18:23.061590 (+   467us) log.cc:844] Serialized 14 byte log entry
0709 12:18:23.375391 (+313801us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b18000a74a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:23.615589 (+240198us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b18000a74a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e99efed70 00007f0e99ef2293 00007f0e99ee7cea 00007f0e99ee823a 00007f0e9a352a2a 00007f0e9a3620bf 00007f0e9a362038
0709 12:18:23.667257 (+ 51668us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:23.667349 (+    92us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:23.667391 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.370051 (+702660us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b54000bd6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:24.370110 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:24.370261 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":177340,"prepare.queue_time_us":389789,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":158952,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":10862,"spinlock_wait_cycles":171646464}
I20260709 12:18:24.377530 28958 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:24.380816 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.023s	sys 0.032s
I20260709 12:18:24.355475 29419 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:24.382578 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.023s	sys 0.033s
W20260709 12:18:24.384240 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 565642us
I20260709 12:18:24.335311 29347 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.372857 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.320253 29402 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:24.391103 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124266us
I20260709 12:18:24.385699 29419 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.386082 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.403578 27659 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:24.334980 29408 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.389012 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:24.408974 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.409832 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.412600 27735 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:24.401618 29315 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.416610 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.414417 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:24.416432 29425 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 12:18:24.419502 29016 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:24.390996 29416 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:18:24.419932 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:24.420725 29402 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:24.422708 27659 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:24.419973 29016 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:24.396660 29383 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:18:24.424275 29016 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:24.423517 29315 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.408774 29311 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.430045 29315 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.430658 29419 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.431071 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.439108 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1512) took 1667 ms. Trace:
I20260709 12:18:24.460026 28943 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.441422 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:24.423050 29418 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:18:24.445591 28988 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:24.446276 29370 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:24.463146 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 83886us
I20260709 12:18:24.389371 29057 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.461820 27582 rpcz_store.cc:276] 0709 12:18:22.772034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.804744 (+ 32710us) service_pool.cc:224] Handling call
0709 12:18:22.805117 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.188422 (+1383305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.438809 (+250387us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:24.439026 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3171791232}
I20260709 12:18:24.440317 29271 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.449265 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.487643 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.462960 27731 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:24.503659 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:24.503810 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.512894 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:24.514496 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:24.525305 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:24.525786 29370 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.526052 27657 raft_consensus.cc:2376] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4. Re-sending same reply.
I20260709 12:18:24.473424 27655 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:24.530377 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.531787 27471 catalog_manager.cc:5028] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): tablet 310b1b37298244c0bd5bbe2b0c7328d7 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
W20260709 12:18:24.533761 29384 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.034s	user 0.020s	sys 0.038s
I20260709 12:18:24.482492 29231 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.502491 28995 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.535729 29231 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.536212 28995 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.538406 29231 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:24.540369 29416 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.541049 29416 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.546123 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:24.546852 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.547640 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:24.555193 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.527091 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.488626 29036 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:24.560899 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.546145 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.562479 29416 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.483836 29419 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:24.538813 29062 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:18:24.552953 28952 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:24.569489 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.510408 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:24.551309 29311 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.543398 29081 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:18:24.620136 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.556627 28988 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:24.558437 29391 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:24.539026 28995 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:24.622982 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:24.557888 29360 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.053s
I20260709 12:18:24.667276 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:24.625231 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:24.659066 29319 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:24.570622 27657 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:24.572728 29288 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:24.705828 28332 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:24.588925 27586 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
W20260709 12:18:24.686401 29382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.152s	user 0.026s	sys 0.031s
I20260709 12:18:24.703079 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.570230 29036 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.658444 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.617646 29428 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.666648 29231 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:24.614152 28952 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:24.699560 29312 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:24.672724 29187 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:24.671988 29271 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:24.662067 27730 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:24.632031 29359 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.661907 29311 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.632366 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:24.604314 29419 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.706029 29066 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:24.705089 28943 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:24.720204 29382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.192s	user 0.027s	sys 0.031s
I20260709 12:18:24.712486 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.735898 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.714280 29430 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.699852 29370 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:18:24.718696 28995 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:24.746945 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81954.8us
I20260709 12:18:24.751792 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:24.725005 28952 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:24.823396 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:24.750471 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:24.826972 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.832404 28995 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.835693 27662 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:24.832816 29430 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.750336 27731 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:24.737408 29429 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.874241 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.875095 27662 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:24.724087 29428 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.833237 28334 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:24.816048 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.815791 29066 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:24.738212 29187 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.880829 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.883795 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:24.745730 29359 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.777268 28943 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.836644 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:24.888720 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:24.891270 27662 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:24.831880 29409 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:18:24.937543 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:24.829421 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:24.892931 27655 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:24.938380 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 390811us
I20260709 12:18:24.942155 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:24.834935 29288 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.935566 29360 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:24.937258 28994 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.936887 29301 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:24.945792 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:24.948957 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:24.972998 29382 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:24.959714 29433 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:24.936977 29429 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:24.989142 29301 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:25.045091 29434 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260709 12:18:24.992002 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211968us
I20260709 12:18:25.002992 29433 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:25.033191 29299 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:25.053328 27736 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:25.026321 29036 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:25.058640 29311 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.000620 29360 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:24.996013 28994 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.059490 28988 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:25.045290 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.065853 29444 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.066860 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:25.066500 29444 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.026927 29382 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.056792 29204 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.059381 29419 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:25.057837 29433 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:25.060171 27730 raft_consensus.cc:606] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:25.060276 29365 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:25.189249 29437 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:18:25.025616 29066 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.064117 29393 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.071751 29299 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:25.190814 29365 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:25.493294 29448 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.530s	user 0.001s	sys 0.000s
I20260709 12:18:25.493999 29365 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.494513 29447 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:25.495087 29447 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.498035 29447 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:25.078778 29036 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:25.245797 29016 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.821s	user 0.008s	sys 0.004s
W20260709 12:18:25.192312 29426 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.361s	user 0.001s	sys 0.001s
W20260709 12:18:25.594764 29441 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.184s	user 0.000s	sys 0.000s
I20260709 12:18:25.594501 29016 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.596051 29036 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.596719 29016 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:25.368031 29435 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.382s	user 0.000s	sys 0.002s
I20260709 12:18:25.184688 27655 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260709 12:18:25.076829 27657 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:25.190632 27730 raft_consensus.cc:614] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:25.597975 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1446) took 1526 ms. Trace:
I20260709 12:18:25.070861 28988 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.580689 29311 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:25.622185 28334 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.742s	user 0.010s	sys 0.001s
W20260709 12:18:25.600450 29445 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.559s	user 0.000s	sys 0.000s
W20260709 12:18:25.601061 28952 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.776s	user 0.009s	sys 0.001s
I20260709 12:18:25.185052 29204 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.485543 29450 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:25.396332 29419 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:25.611208 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1474) took 2622 ms. Trace:
I20260709 12:18:25.607510 28943 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.622579 28334 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
W20260709 12:18:25.623255 27660 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.045s	sys 0.089s
I20260709 12:18:25.621297 27733 rpcz_store.cc:276] 0709 12:18:24.071993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.072863 (+   870us) service_pool.cc:224] Handling call
0709 12:18:24.127911 (+ 55048us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:24.128346 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.128389 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:24.157449 (+ 29060us) log.cc:844] Serialized 52 byte log entry
0709 12:18:24.666261 (+508812us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:24.666303 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.666375 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.597678 (+931303us) raft_consensus.cc:1685] finished
0709 12:18:25.597708 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.597901 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30936,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":6182,"thread_start_us":476603,"threads_started":1,"wal-append.queue_time_us":541898}
W20260709 12:18:25.608902 29449 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.418s	user 0.001s	sys 0.000s
I20260709 12:18:25.612178 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1466) took 1121 ms. Trace:
I20260709 12:18:25.632866 27587 rpcz_store.cc:276] 0709 12:18:22.983601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:23.390318 (+406717us) service_pool.cc:224] Handling call
0709 12:18:23.390692 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.240843 (+850151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.605950 (+1365107us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:25.606209 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1889714944}
W20260709 12:18:25.675799 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.022s	sys 0.032s
I20260709 12:18:25.630254 28952 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.681082 27585 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:25.672230 27660 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.054s	sys 0.089s
I20260709 12:18:25.624689 27661 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:25.584693 29393 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:25.677040 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.022s	sys 0.045s
I20260709 12:18:25.668954 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1294) took 1163 ms. Trace:
W20260709 12:18:25.699440 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.015s	sys 0.019s
W20260709 12:18:25.695544 29437 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.013s	sys 0.037s
I20260709 12:18:25.702456 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 115) took 1028 ms. Trace:
W20260709 12:18:25.713340 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.015s	sys 0.019s
I20260709 12:18:25.630900 29311 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.746999 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1292) took 1319 ms. Trace:
I20260709 12:18:25.747920 27659 rpcz_store.cc:276] 0709 12:18:24.428213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.474758 (+ 46545us) service_pool.cc:224] Handling call
0709 12:18:24.504555 (+ 29797us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.511216 (+  6661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.511263 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.551783 (+ 40520us) log.cc:844] Serialized 26 byte log entry
0709 12:18:24.728798 (+177015us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:25.042490 (+313692us) log.cc:844] Serialized 14 byte log entry
0709 12:18:25.043562 (+  1072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.043654 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.746786 (+703132us) raft_consensus.cc:1685] finished
0709 12:18:25.746806 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.746942 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13836,"prepare.queue_time_us":156475,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":3429,"raft.queue_time_us":7927,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":25416,"thread_start_us":410319,"threads_started":2,"wal-append.queue_time_us":393641}
I20260709 12:18:25.750172 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.750751 27659 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:25.752506 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1299) took 1245 ms. Trace:
I20260709 12:18:25.752918 27659 rpcz_store.cc:276] 0709 12:18:24.506709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.751530 (+1244821us) service_pool.cc:224] Handling call
0709 12:18:25.751892 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.752112 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.752149 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:25.752207 (+    58us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:25.752250 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.752320 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.752445 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.667683 29419 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.638634 29450 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.705619 29416 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.675310 27657 rpcz_store.cc:276] 0709 12:18:24.490621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.970445 (+479824us) service_pool.cc:224] Handling call
0709 12:18:25.612109 (+641664us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.712539 29429 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:25.735283 29442 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.650s	user 0.001s	sys 0.000s
W20260709 12:18:25.745571 29440 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:18:25.754917 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1300) took 1248 ms. Trace:
I20260709 12:18:25.742586 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1295) took 1213 ms. Trace:
I20260709 12:18:25.771152 27661 rpcz_store.cc:276] 0709 12:18:24.505552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.064057 (+558505us) service_pool.cc:224] Handling call
0709 12:18:25.584733 (+520676us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.699314 (+114581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.718810 (+ 19496us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.768390 27659 rpcz_store.cc:276] 0709 12:18:24.506980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.754012 (+1247032us) service_pool.cc:224] Handling call
0709 12:18:25.754351 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.754550 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.754587 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:25.754636 (+    49us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:25.754678 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.754744 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.754862 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.677969 29299 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.734787 27588 rpcz_store.cc:276] 0709 12:18:24.674255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.891852 (+217597us) service_pool.cc:224] Handling call
0709 12:18:24.937608 (+ 45756us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.946019 (+  8411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.946064 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.955134 (+  9070us) log.cc:844] Serialized 26 byte log entry
0709 12:18:25.336230 (+381096us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:25.336285 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.336390 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.702237 (+365847us) raft_consensus.cc:1685] finished
0709 12:18:25.702262 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.702395 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2592,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":6854,"thread_start_us":304560,"threads_started":1,"wal-append.queue_time_us":696054}
I20260709 12:18:25.774861 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1302) took 1256 ms. Trace:
I20260709 12:18:25.766319 29382 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.771941 29065 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:25.777134 29389 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:25.777374 29065 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.777967 29065 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.775902 27659 rpcz_store.cc:276] 0709 12:18:24.518335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.773468 (+1255133us) service_pool.cc:224] Handling call
0709 12:18:25.773848 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.774062 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.774100 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:25.774167 (+    67us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:25.774212 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.774320 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.774802 (+   482us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.780818 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 363) took 1044 ms. Trace:
I20260709 12:18:25.781664 28337 rpcz_store.cc:276] 0709 12:18:24.736800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.736935 (+   135us) service_pool.cc:224] Handling call
0709 12:18:24.737223 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.815515 (+ 78292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.815558 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:24.815605 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:24.815635 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.780477 (+964842us) spinlock_profiling.cc:243] Waited 77.6 ms on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:25.780542 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.780716 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178449152}
I20260709 12:18:25.698707 27655 rpcz_store.cc:276] 0709 12:18:24.505231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.986339 (+481108us) service_pool.cc:224] Handling call
0709 12:18:25.060994 (+ 74655us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:25.612482 (+551488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.637850 (+ 25368us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.785259 27659 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:25.790985 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 332) took 1166 ms. Trace:
I20260709 12:18:25.791905 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 122) took 1402 ms. Trace:
I20260709 12:18:25.776005 29311 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.796623 27658 rpcz_store.cc:276] 0709 12:18:24.389683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.391730 (+  2047us) service_pool.cc:224] Handling call
0709 12:18:24.392212 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.485366 (+ 93154us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:24.660831 (+175465us) spinlock_profiling.cc:243] Waited 4.99 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:18:24.739559 (+ 78728us) log.cc:844] Serialized 14 byte log entry
0709 12:18:24.938101 (+198542us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.941330 (+  3229us) log.cc:844] Serialized 26 byte log entry
0709 12:18:24.941490 (+   160us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:24.941522 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.941597 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.784370 (+842773us) raft_consensus.cc:1685] finished
0709 12:18:25.791573 (+  7203us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.791812 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77921,"prepare.queue_time_us":339901,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":96887,"raft.run_cpu_time_us":2294,"raft.run_wall_time_us":176019,"spinlock_wait_cycles":11479808,"thread_start_us":398992,"threads_started":2,"wal-append.queue_time_us":316917}
I20260709 12:18:25.809708 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:25.811784 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1305) took 1245 ms. Trace:
W20260709 12:18:25.802541 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.083s	sys 0.150s
W20260709 12:18:25.814086 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.084s	sys 0.151s
I20260709 12:18:25.814368 27657 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:25.814630 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 747776us
I20260709 12:18:25.815070 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1303) took 1296 ms. Trace:
I20260709 12:18:25.815485 27657 rpcz_store.cc:276] 0709 12:18:24.518615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.784372 (+1265757us) service_pool.cc:224] Handling call
0709 12:18:25.814199 (+ 29827us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.814811 (+   612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.815002 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.816334 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:25.817469 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1301) took 1310 ms. Trace:
I20260709 12:18:25.813290 27659 rpcz_store.cc:276] 0709 12:18:24.566122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.783654 (+1217532us) service_pool.cc:224] Handling call
0709 12:18:25.784830 (+  1176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.810846 (+ 26016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.811690 (+   844us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.817898 27661 rpcz_store.cc:276] 0709 12:18:24.507293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.776172 (+1268879us) service_pool.cc:224] Handling call
0709 12:18:25.802726 (+ 26554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.817126 (+ 14400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.817164 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:25.817208 (+    44us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:25.817243 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.817302 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.817413 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:25.766278 29416 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.819806 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:25.821058 29416 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.821205 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:25.824831 29427 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:25.825935 29427 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:25.826566 29427 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:25.826913 29427 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:25.798820 28328 rpcz_store.cc:276] 0709 12:18:24.624382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.659827 (+ 35445us) service_pool.cc:224] Handling call
0709 12:18:24.660222 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.869489 (+209267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.871216 (+  1727us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:24.877150 (+  5934us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:24.881105 (+  3955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.790685 (+909580us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54002fab54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:25.790753 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.790910 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":389223680}
I20260709 12:18:25.827288 29427 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:25.817849 28952 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:25.839911 29458 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:25.840677 29458 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:25.843626 29462 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:25.844347 29462 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.846695 29427 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.848533 29419 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:25.849001 29440 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:18:25.849625 29444 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:25.851814 29450 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.865479 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 109) took 2131 ms. Trace:
I20260709 12:18:25.865716 29427 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.867893 29181 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.879144 28971 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:25.881138 29343 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:25.847002 29462 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:25.885494 27589 rpcz_store.cc:276] 0709 12:18:23.733777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:23.735123 (+  1346us) service_pool.cc:224] Handling call
0709 12:18:23.737124 (+  2001us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:23.739221 (+  2097us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:23.739555 (+   334us) log.cc:844] Serialized 14 byte log entry
0709 12:18:24.368342 (+628787us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b18000dfde0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:24.460219 (+ 91877us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b18000dfde0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:18:24.886951 (+426732us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.893432 (+  6481us) log.cc:844] Serialized 26 byte log entry
0709 12:18:24.893583 (+   151us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:24.893615 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.893676 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.753117 (+859441us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b18000dfde0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:25.865224 (+112107us) raft_consensus.cc:1685] finished
0709 12:18:25.865257 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.865409 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1702,"prepare.queue_time_us":1909,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":2402,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":16486,"spinlock_wait_cycles":10874496,"thread_start_us":411160,"threads_started":1,"wal-append.queue_time_us":1532186}
I20260709 12:18:25.889585 29458 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:25.893415 29343 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:25.893841 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 364) took 1155 ms. Trace:
I20260709 12:18:25.894104 28971 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:25.929894 29393 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:25.940975 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:25.950699 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.940455 29343 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.960966 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.961433 29343 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:25.961679 27662 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
W20260709 12:18:25.962150 27731 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.075s	user 0.014s	sys 0.037s
I20260709 12:18:25.962540 27731 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:25.963197 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.914781 28330 rpcz_store.cc:276] 0709 12:18:24.738138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.739687 (+  1549us) service_pool.cc:224] Handling call
0709 12:18:25.870471 (+1130784us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:25.870739 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.871000 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.871042 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:25.871116 (+    74us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:25.871169 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.871247 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.890087 (+ 18840us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8355200}
I20260709 12:18:25.964586 27737 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:25.962790 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.982607 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:25.984395 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:25.965791 29391 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.987665 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.983489 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:25.988941 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:25.989571 27655 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:25.990386 27733 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 12:18:25.986877 29391 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:25.994681 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:25.996711 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:25.963260 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1355) took 1403 ms. Trace:
I20260709 12:18:25.965844 29424 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:25.964479 29393 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:25.990386 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:26.014050 29432 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.079s	user 0.005s	sys 0.004s
I20260709 12:18:26.015951 29232 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:25.986212 27628 connection.cc:256] RPC call timeout handler was delayed by -1.2097218290000455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
I20260709 12:18:26.001410 29416 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:26.001173 29311 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:25.943993 28971 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.031229 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:26.035362 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 236222us
W20260709 12:18:26.064932 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 243781us
W20260709 12:18:26.066567 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:26.021013 29311 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:26.068723 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 113) took 1869 ms. Trace:
I20260709 12:18:26.001876 27731 rpcz_store.cc:276] 0709 12:18:24.560163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.564759 (+  4596us) service_pool.cc:224] Handling call
0709 12:18:25.963003 (+1398244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:26.024161 29462 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:26.065809 27627 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:26.087827 27627 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:18:26.020267 29393 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.069370 27581 rpcz_store.cc:276] 0709 12:18:24.199181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.199684 (+   503us) service_pool.cc:224] Handling call
0709 12:18:24.200105 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.252969 (+ 52864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.253014 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.267055 (+ 14041us) log.cc:844] Serialized 26 byte log entry
0709 12:18:24.722013 (+454958us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:24.773727 (+ 51714us) log.cc:844] Serialized 14 byte log entry
0709 12:18:24.818226 (+ 44499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.818300 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.013257 (+1194957us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:26.034593 (+ 21336us) raft_consensus.cc:1685] finished
0709 12:18:26.068329 (+ 33736us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.068529 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107067,"prepare.queue_time_us":25264,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":104928,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":10420,"spinlock_wait_cycles":11708160,"thread_start_us":418482,"threads_started":1,"wal-append.queue_time_us":540637}
I20260709 12:18:26.021842 29424 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:26.088842 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:26.098309 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:26.094894 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:26.100036 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:26.072397 29311 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:26.073443 29459 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:26.096253 29389 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:26.129443 29391 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:26.121176 29459 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.123809 29470 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:26.124809 29465 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:18:26.140143 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 116) took 1438 ms. Trace:
I20260709 12:18:26.127776 29475 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:18:26.129604 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1478) took 1064 ms. Trace:
I20260709 12:18:26.095726 28971 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:26.140591 27586 rpcz_store.cc:276] 0709 12:18:24.701755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.988994 (+287239us) service_pool.cc:224] Handling call
0709 12:18:24.989507 (+   513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.045682 (+ 56175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.047088 (+  1406us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:25.185554 (+138466us) log.cc:844] Serialized 26 byte log entry
0709 12:18:25.928910 (+743356us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:25.928986 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.929073 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.139901 (+210828us) raft_consensus.cc:1685] finished
0709 12:18:26.139929 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.140079 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34081,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":3617,"thread_start_us":399949,"threads_started":1,"wal-append.queue_time_us":741536}
I20260709 12:18:26.131160 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1297) took 1625 ms. Trace:
I20260709 12:18:26.138484 29459 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:26.138859 29424 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:26.147684 27660 rpcz_store.cc:276] 0709 12:18:24.506124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.065748 (+559624us) service_pool.cc:224] Handling call
0709 12:18:25.066408 (+   660us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.687872 (+621464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.687917 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:25.688492 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:18:26.020790 (+332298us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:26.021653 (+   863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.021736 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.130896 (+109160us) raft_consensus.cc:1685] finished
0709 12:18:26.130913 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.131075 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":636297,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":940695,"threads_started":2,"wal-append.queue_time_us":425097}
I20260709 12:18:26.141182 27658 rpcz_store.cc:276] 0709 12:18:25.065220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.806022 (+740802us) service_pool.cc:224] Handling call
0709 12:18:25.806428 (+   406us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:25.806853 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.806894 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:25.811622 (+  4728us) log.cc:844] Serialized 52 byte log entry
0709 12:18:26.096707 (+285085us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:26.096766 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.096863 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.129380 (+ 32517us) raft_consensus.cc:1685] finished
0709 12:18:26.129406 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.129549 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12510,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":5329,"thread_start_us":282190,"threads_started":1,"wal-append.queue_time_us":313547}
W20260709 12:18:26.150831 29433 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.962s	user 0.008s	sys 0.000s
I20260709 12:18:26.097947 29462 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:26.162636 29462 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:26.163350 29451 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:26.118074 29343 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:26.172797 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 118) took 1470 ms. Trace:
W20260709 12:18:26.243047 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 144670us
I20260709 12:18:26.245929 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:26.250579 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:26.252784 29471 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:26.253388 29471 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.253716 29471 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:26.258023 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 159795us
I20260709 12:18:26.258750 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:26.260036 29443 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:26.263705 29443 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.264083 29443 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:26.275951 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:26.279243 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1480) took 1213 ms. Trace:
I20260709 12:18:26.282802 27582 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:26.283038 27659 rpcz_store.cc:276] 0709 12:18:25.065748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.818340 (+752592us) service_pool.cc:224] Handling call
0709 12:18:25.818702 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:26.087409 (+268707us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:18:26.087444 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:26.087965 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:18:26.244789 (+156824us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:26.244832 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.244901 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.278965 (+ 34064us) raft_consensus.cc:1685] finished
0709 12:18:26.278993 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.279173 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6669,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":60,"thread_start_us":362984,"threads_started":2,"wal-append.queue_time_us":157832}
I20260709 12:18:26.168156 29433 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:26.278942 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 143441us
I20260709 12:18:26.287423 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:26.288019 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:26.288719 27732 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:26.289580 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:26.288079 27738 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:26.296226 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:26.297525 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1527) took 1776 ms (client timeout 1850 ms). Trace:
W20260709 12:18:26.297930 27582 rpcz_store.cc:269] 0709 12:18:24.509984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.510162 (+   178us) service_pool.cc:224] Handling call
0709 12:18:26.275769 (+1765607us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:26.286249 (+ 10480us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46140672}
I20260709 12:18:26.298671 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:26.301156 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:26.306679 29456 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:26.307243 29456 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:26.307752 29456 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:26.319818 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 61111.2us
I20260709 12:18:26.321933 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:26.249178 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1479) took 1183 ms. Trace:
I20260709 12:18:26.324095 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.334092 28337 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:26.335167 27658 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:26.336701 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.278419 29036 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:26.328125 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.343794 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:26.287457 28971 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.338984 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.277457 29065 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:26.268548 29393 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:26.327541 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:26.340036 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.378435 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:26.342428 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.329772 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:26.304504 29057 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:26.302361 29464 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:26.318506 29456 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:26.382017 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54564.5us
I20260709 12:18:26.341318 29319 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:26.332331 29459 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:26.375638 28971 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.376104 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:26.376461 29480 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:26.386941 29387 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:26.340457 29451 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:18:26.394804 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:26.393784 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:26.379249 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79500us
I20260709 12:18:26.403067 29480 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:26.463187 29311 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:26.534925 29480 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:26.537417 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:26.395979 29481 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:26.386075 29057 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:26.463977 29477 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:18:26.536046 29348 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:26.404745 29456 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:26.472219 29392 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:26.547907 29348 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:26.546741 29481 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.555336 29480 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:26.391147 29464 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.517576 29387 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:26.564960 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:26.555843 29443 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.558436 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:26.470932 29433 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:26.473676 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:26.391683 29319 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.403750 28971 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:26.560506 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 139829us
I20260709 12:18:26.380944 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:26.565662 27732 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:26.560233 29066 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:26.476349 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 3) took 1541 ms. Trace:
W20260709 12:18:26.568766 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:26.568791 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:26.560294 28329 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:26.572898 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:26.575620 27660 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:26.554889 29471 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:26.547256 29036 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:18:26.581485 29385 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:26.586882 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 192172us
I20260709 12:18:26.586833 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:26.590122 27581 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 1.
I20260709 12:18:26.568468 29387 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:26.582528 29319 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:26.591689 29385 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:26.583287 27730 rpcz_store.cc:276] 0709 12:18:24.935701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.937046 (+  1345us) service_pool.cc:224] Handling call
0709 12:18:26.476044 (+1538998us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:26.476231 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62617344}
W20260709 12:18:26.593065 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1526) took 2084 ms (client timeout 1589 ms). Trace:
I20260709 12:18:26.594519 29456 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:26.575753 29391 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:26.570344 29484 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:26.589226 28971 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:27.030450 27580 rpcz_store.cc:269] 0709 12:18:24.509082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.525494 (+ 16412us) service_pool.cc:224] Handling call
0709 12:18:26.473442 (+1947948us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:26.592996 (+119554us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51605376}
I20260709 12:18:27.032693 29456 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.032825 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:26.949009 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.035385 27730 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:26.576488 29311 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.035774 29456 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:26.609129 29057 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:26.608412 27628 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
W20260709 12:18:27.113477 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 505200us
I20260709 12:18:26.766882 29128 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:26.644395 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:26.895663 29457 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.217s	user 0.000s	sys 0.000s
I20260709 12:18:26.333426 27657 rpcz_store.cc:276] 0709 12:18:25.065495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.816390 (+750895us) service_pool.cc:224] Handling call
0709 12:18:25.816753 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.817126 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.817163 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:25.817654 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:18:26.120898 (+303244us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:26.135774 (+ 14876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.166495 (+ 30721us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.169195 (+  2700us) raft_consensus.cc:1685] finished
0709 12:18:26.171008 (+  1813us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.248672 (+ 77664us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26,"raft.queue_time_us":8428,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":292476,"threads_started":1,"wal-append.queue_time_us":338195}
W20260709 12:18:26.848924 29485 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.260s	user 0.000s	sys 0.000s
I20260709 12:18:27.104414 29480 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.094141 29391 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.595670 29490 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:27.095762 27732 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.506s	user 0.015s	sys 0.004s
I20260709 12:18:27.134933 27732 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:27.031173 28971 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.109444 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.103655 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:27.111649 27704 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 2.038s (SENT) [suppressed 2 similar messages]
I20260709 12:18:27.108680 29464 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.112505 29478 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.118620 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.118777 29078 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.118696 28336 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:26.595858 29348 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.121533 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:26.589156 29065 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:27.106557 29347 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.112258 29471 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:26.659605 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:26.611744 27662 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:26.849664 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.107515 29484 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.150982 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.179368 27732 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:27.146910 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.152549 29385 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.156896 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 125) took 2762 ms. Trace:
I20260709 12:18:27.188844 27663 rpcz_store.cc:276] 0709 12:18:24.394984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.404568 (+  9584us) service_pool.cc:224] Handling call
0709 12:18:24.404941 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.487509 (+ 82568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.487537 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.487967 (+   430us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.101124 (+2613157us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b1800427660. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:18:27.110413 (+  9289us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:27.110513 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.111055 (+   542us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:27.153175 (+ 42120us) raft_consensus.cc:1685] finished
0709 12:18:27.153199 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.156811 (+  3612us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82145,"raft.queue_time_us":86170,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":10481,"spinlock_wait_cycles":2395520,"thread_start_us":9033,"threads_started":1,"wal-append.queue_time_us":2639722}
I20260709 12:18:27.158313 28334 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:27.192804 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.193773 27735 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:27.194252 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1452) took 2696 ms (client timeout 1756 ms). Trace:
I20260709 12:18:27.134438 29391 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:27.197139 29387 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.543s	user 0.005s	sys 0.003s
I20260709 12:18:27.197551 29387 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.137180 29507 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:27.199672 29507 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.202205 29507 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.153005 28971 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.149725 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.156145 27655 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.203797 28335 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:27.204931 27581 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:27.176759 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.171109 29506 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.126595 29128 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.167516 29065 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:27.186471 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:27.163880 29078 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.177186 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.210212 29506 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.180668 29288 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:27.194686 27735 rpcz_store.cc:269] 0709 12:18:24.498211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.500171 (+  1960us) service_pool.cc:224] Handling call
0709 12:18:27.192642 (+2692471us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:27.194170 (+  1528us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9518464}
I20260709 12:18:27.138962 29490 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:27.216239 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.185639 29348 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:27.216709 29065 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.225311 29387 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.226085 29128 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.228292 29078 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.228735 29490 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:27.223049 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 373487us
I20260709 12:18:27.230113 29359 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.231372 29428 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.232694 28988 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:27.236294 29505 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:18:27.216666 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1525) took 2717 ms. Trace:
I20260709 12:18:27.151942 29478 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.238677 27584 rpcz_store.cc:276] 0709 12:18:24.498956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.499185 (+   229us) service_pool.cc:224] Handling call
0709 12:18:27.215924 (+2716739us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:27.216109 (+   185us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.216286 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.216317 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.216358 (+    41us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:27.216389 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.216441 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.216564 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34104704}
W20260709 12:18:27.216425 29499 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:18:27.244966 29484 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.214339 29450 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:27.238734 27657 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:27.249135 27581 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
W20260709 12:18:27.255927 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.256304 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.264956 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.249941 29450 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.250448 29428 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:27.259902 29369 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.259817 29490 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.259596 29348 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.258998 29484 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.261023 29078 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:27.262491 29504 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.262625 28971 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:27.275031 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93976.4us
I20260709 12:18:27.268153 29450 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:27.281028 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:27.281817 27735 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.283543 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:27.285032 27732 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:18:27.291107 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:27.294356 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 182717us
I20260709 12:18:27.292466 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:27.298393 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.299804 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.301956 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:27.308032 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.309028 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.310019 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:27.290360 29463 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.409s	user 0.014s	sys 0.028s
I20260709 12:18:27.311288 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.276862 29429 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:27.313227 29463 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.432s	user 0.014s	sys 0.029s
I20260709 12:18:27.271380 29428 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.293624 28971 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.291625 29504 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.312254 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:27.289623 29078 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.275554 29369 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.328832 29481 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.278561 29348 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.279439 29319 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.279151 29484 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.277402 29490 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.330230 29429 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.327406 29181 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:27.327665 29512 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.310535 29416 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:27.313556 29510 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:27.339314 29065 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.343758 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.345368 29506 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.309706 29452 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
W20260709 12:18:27.345616 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1289) took 2932 ms (client timeout 2077 ms). Trace:
I20260709 12:18:27.341320 29513 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.348249 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.348637 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.349009 29187 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.349650 27738 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:27.350252 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.339148 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.327561 29359 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:27.351457 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.350839 29454 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:27.361654 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44806.9us
I20260709 12:18:27.364276 27735 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:27.355986 29408 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:27.370167 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 130) took 2304 ms (client timeout 2115 ms). Trace:
I20260709 12:18:27.342103 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:27.380689 27584 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:27.359205 29480 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.343755 29391 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.385339 29506 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.383422 29513 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.366164 29207 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:27.368083 29416 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:27.398943 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
W20260709 12:18:27.396966 27736 rpcz_store.cc:269] 0709 12:18:25.065767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.067522 (+  1755us) service_pool.cc:224] Handling call
0709 12:18:27.343558 (+2276036us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:27.370070 (+ 26512us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6385408}
I20260709 12:18:27.401994 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:27.382329 27664 rpcz_store.cc:269] 0709 12:18:24.413010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.414166 (+  1156us) service_pool.cc:224] Handling call
0709 12:18:25.940728 (+1526562us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:27.345367 (+1404639us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:27.345513 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":327797248}
I20260709 12:18:27.408036 27735 raft_consensus.cc:606] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:27.408452 27588 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:27.386775 29509 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:18:27.414000 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:27.400184 28328 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:18:27.419458 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 68683.2us
I20260709 12:18:27.421789 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.401073 29359 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:27.440093 29387 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:27.448383 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.411711 29416 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:27.370154 29512 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.443764 29444 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:27.410828 28996 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:18:27.380336 29510 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.401539 29511 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:18:27.469697 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.407481 29480 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.449827 29299 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:27.446728 29520 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.397475 29454 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.462600 27730 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6.
I20260709 12:18:27.477526 27588 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:27.413666 29506 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:27.484064 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50441.6us
W20260709 12:18:27.485741 27630 connection.cc:256] RPC call timeout handler was delayed by -1.470103158000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
I20260709 12:18:27.486124 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:27.487509 28988 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:27.486147 27630 connection.cc:256] RPC call timeout handler was delayed by -1.2352031580001039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
I20260709 12:18:27.487951 28988 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.488502 28988 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.461251 29387 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:27.463276 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:27.492995 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 176322us
I20260709 12:18:27.468880 29181 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.472525 28996 consensus_queue.cc:1243] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:27.496268 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.497671 27588 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:27.497432 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.482151 29065 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.480669 29299 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:27.495376 29428 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.495005 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.534317 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.516923 29319 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.510469 28971 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.519443 29484 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.548058 29391 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.541929 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.522301 29429 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.522517 27729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c204963842941859d4313114fd92c93"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:27.561568 27729 raft_consensus.cc:606] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:27.561968 27729 raft_consensus.cc:614] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:18:27.546162 29519 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
W20260709 12:18:27.563714 29525 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:18:26.211931 27583 rpcz_store.cc:276] 0709 12:18:24.702381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.611910 (+909529us) service_pool.cc:224] Handling call
0709 12:18:25.612255 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.612440 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:26.018612 (+406172us) log.cc:844] Serialized 14 byte log entry
0709 12:18:26.172208 (+153596us) log.cc:844] Serialized 14 byte log entry
0709 12:18:26.172391 (+   183us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:26.172438 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:26.172484 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.172574 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.172715 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":141006,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":30,"raft.queue_time_us":537,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":69706,"spinlock_wait_cycles":3926272,"thread_start_us":539638,"threads_started":2}
I20260709 12:18:27.526798 29416 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.556128 29387 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.566174 29506 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.541785 29530 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.500033 29181 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.559988 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.509742 29359 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.542394 29528 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:18:27.565023 29510 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.491592 29520 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.565586 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.508193 29450 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.578974 29513 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.579732 29429 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:27.580089 29536 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:27.580782 29512 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.582363 29416 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.583123 29181 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:27.549446 29517 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.103s	user 0.001s	sys 0.001s
I20260709 12:18:27.607911 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.615648 27583 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:27.616631 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:27.617108 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 273232us
I20260709 12:18:27.617307 27583 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:27.571861 29408 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:27.543090 28331 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:27.637852 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 139598us
I20260709 12:18:27.638512 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.611111 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.618177 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.611090 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:27.640203 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.634683 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.641731 27583 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:27.642611 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.642978 27583 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:27.641743 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:27.644901 27736 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:27.647261 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.647897 27735 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:27.650328 27738 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:27.650610 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:27.651363 27664 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.651564 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.652220 27662 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:27.655598 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.656288 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.656790 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.616325 29345 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:27.567047 29530 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.649410 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.657027 27587 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:27.657414 27732 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:18:27.660529 27658 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 12:18:27.594769 29299 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.850272 27736 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:27.719151 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.851904 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.827911 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:27.852202 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.667464 27664 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:27.853235 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.828150 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.693702 27662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:47336
I20260709 12:18:27.831724 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1350) took 3431 ms. Trace:
I20260709 12:18:27.729785 27583 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:27.831295 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:27.811470 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.858026 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.692834 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.670161 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 361) took 2929 ms. Trace:
I20260709 12:18:27.859537 27583 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
W20260709 12:18:27.829756 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.858390 28335 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
W20260709 12:18:27.862934 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:27.862843 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.865445 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:27.861157 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.868412 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.868057 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.868798 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.859536 28332 rpcz_store.cc:276] 0709 12:18:24.728399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.728720 (+   321us) service_pool.cc:224] Handling call
0709 12:18:24.740044 (+ 11324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.939423 (+199379us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:18:24.940898 (+  1475us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.199577 (+2258679us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b2c000471f8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:27.200002 (+   425us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.365627 (+165625us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:27.365666 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.373232 (+  7566us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:27.636929 (+263697us) raft_consensus.cc:1685] finished
0709 12:18:27.636956 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.657468 (+ 20512us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3397,"raft.queue_time_us":86343,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":31852,"spinlock_wait_cycles":7958912,"thread_start_us":363753,"threads_started":2,"wal-append.queue_time_us":187572}
I20260709 12:18:27.870231 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.872611 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.867488 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.873098 27664 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:27.874029 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.874126 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.874739 27587 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:27.858675 27662 raft_consensus.cc:606] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:27.691934 29537 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:18:27.879709 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.874930 27664 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:27.679518 28996 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:18:27.883845 27552 connection.cc:256] RPC call timeout handler was delayed by -2.2289253419999113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
I20260709 12:18:27.715793 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.866515 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.875697 29081 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:27.878885 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.867563 29288 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.879403 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:27.878500 29460 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:27.900707 29460 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.882122 29534 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.904377 29534 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.908583 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.912127 29460 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.888110 28996 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.912860 28996 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:27.914052 29504 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.916405 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.907099 29534 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.917059 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:27.917898 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.918967 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.919914 29544 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:27.920300 29544 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.920634 29544 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:27.840392 28331 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:27.825416 29520 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:27.772249 29541 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:18:27.881990 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.658381 29503 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:27.770349 29204 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.857205 27734 rpcz_store.cc:276] 0709 12:18:24.400478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.401027 (+   549us) service_pool.cc:224] Handling call
0709 12:18:24.401320 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.403087 (+  1767us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:18:24.874542 (+471455us) log.cc:844] Serialized 14 byte log entry
0709 12:18:27.330120 (+2455578us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b18000368e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:18:27.413227 (+ 83107us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.432483 (+ 19256us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.432649 (+   166us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:27.432699 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.432805 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:27.830931 (+398126us) raft_consensus.cc:1685] finished
0709 12:18:27.831191 (+   260us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.831511 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6836,"prepare.queue_time_us":496047,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":9494,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":28711,"spinlock_wait_cycles":6570240,"thread_start_us":497857,"threads_started":2,"wal-append.queue_time_us":2575862}
I20260709 12:18:27.762027 29450 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.884812 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.885269 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.930320 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.888691 29531 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.679071 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:27.761173 28995 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.718758 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.774058 29530 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:27.886658 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0406036589999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1562s for a response
I20260709 12:18:27.876099 27662 raft_consensus.cc:614] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:27.692181 29480 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.718050 29539 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.867134 29065 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:27.934914 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6235725950000415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
W20260709 12:18:27.935688 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5563777740001115s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18230000000000002s for a response
I20260709 12:18:27.734197 29454 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:27.936039 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4171056600000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
W20260709 12:18:27.936419 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4166544629999862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
W20260709 12:18:27.936726 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2244194889999562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260709 12:18:27.937019 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1974093830001493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
W20260709 12:18:27.937197 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1591211470001781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260709 12:18:27.937361 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1591211470001781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
I20260709 12:18:27.657738 29383 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:27.828161 28988 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.870874 29387 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:27.939333 29065 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.941709 29387 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:27.941713 29539 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.719463 29512 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.168s
I20260709 12:18:27.878916 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.894773 27587 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:27.878320 29507 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:27.946817 29507 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:27.947798 29507 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.948222 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:27.949210 28330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 371) took 1650 ms (client timeout 1802 ms). Trace:
W20260709 12:18:27.949803 28330 rpcz_store.cc:269] 0709 12:18:26.298672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.301367 (+  2695us) service_pool.cc:224] Handling call
0709 12:18:27.948033 (+1646666us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:27.949119 (+  1086us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68893952}
I20260709 12:18:27.951066 29301 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.896504 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.671113 29345 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.886371 29510 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.660256 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.883247 29513 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.899364 29288 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.922148 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:27.938241 29534 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:27.881063 29528 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.883229 29429 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:27.886219 29299 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.960310 29513 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.959251 29510 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.927948 27588 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:27.931005 27734 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:27.968389 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.934793 29531 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.932637 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.938373 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:27.927948 29503 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:27.940284 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.939122 29520 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:27.943245 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 125) took 1288 ms. Trace:
I20260709 12:18:27.945020 28337 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:27.942750 29387 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.946138 29065 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.952153 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:27.953173 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:27.953785 27738 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:27.957167 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.923097 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.949290 27729 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:27.883756 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:27.988868 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:27.963088 29513 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.969148 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:27.971495 29506 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:27.972000 29531 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:27.972563 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:27.971038 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.991663 27588 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:27.974283 27736 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:27.972206 29443 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.976020 27585 rpcz_store.cc:276] 0709 12:18:26.654698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.654844 (+   146us) service_pool.cc:224] Handling call
0709 12:18:27.294118 (+639274us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:27.294315 (+   197us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.382890 (+ 88575us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:27.403958 (+ 21068us) log.cc:844] Serialized 14 byte log entry
0709 12:18:27.444020 (+ 40062us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.444075 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:27.444125 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.943038 (+498913us) spinlock_profiling.cc:243] Waited 88.4 ms on lock 0x7b54001892d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:27.943084 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.943193 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":75905,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":3035,"raft.queue_time_us":102353,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":13082,"spinlock_wait_cycles":232971264,"thread_start_us":60418,"threads_started":2,"wal-append.queue_time_us":69508}
I20260709 12:18:27.993283 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:27.968734 28996 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.995393 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:27.972440 29360 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:27.987246 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:27.975050 29503 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:27.996652 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:27.963639 29534 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.997154 27587 raft_consensus.cc:2376] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4. Re-sending same reply.
I20260709 12:18:27.985314 29510 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:27.997371 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.000840 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.002135 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:27.960922 29528 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:27.991055 29506 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:27.992920 29065 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:28.002496 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.000188 29513 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:28.004446 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:27.979270 29520 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:27.999886 29443 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:28.000917 29531 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.008837 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:28.009325 29392 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:28.012719 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.017309 29392 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.017558 27731 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:28.018262 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 131) took 1732 ms. Trace:
I20260709 12:18:28.018496 29392 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.009809 29443 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:28.020607 29443 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:28.020924 29345 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.017355 28994 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.009590 29504 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.005800 29478 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.005393 29506 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:28.006634 29510 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.036312 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.037087 27581 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.026420 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.039065 27587 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.026775 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 119) took 1441 ms. Trace:
I20260709 12:18:28.010334 29065 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.012372 29531 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:28.034322 28958 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:28.018644 27731 rpcz_store.cc:276] 0709 12:18:26.286357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.286979 (+   622us) service_pool.cc:224] Handling call
0709 12:18:28.017040 (+1730061us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:28.017358 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.017990 (+   632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.018205 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6798080}
I20260709 12:18:28.047039 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1490) took 1718 ms. Trace:
I20260709 12:18:28.022267 29534 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.046451 29539 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.047430 27660 rpcz_store.cc:276] 0709 12:18:26.329164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.393574 (+ 64410us) service_pool.cc:224] Handling call
0709 12:18:26.397594 (+  4020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.585783 (+188189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.046815 (+1461032us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b540005fd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:28.046986 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":409010944}
I20260709 12:18:28.039880 27582 rpcz_store.cc:276] 0709 12:18:26.585731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.585955 (+   224us) service_pool.cc:224] Handling call
0709 12:18:28.026109 (+1440154us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:28.026301 (+   192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.026470 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.026496 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.026539 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:28.026570 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.026626 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.026726 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33038208}
I20260709 12:18:28.048561 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.050693 27581 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:28.061524 27587 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:28.044221 29510 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:28.074491 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.076335 27587 raft_consensus.cc:2376] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3. Re-sending same reply.
I20260709 12:18:28.076894 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.071357 27588 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:28.044253 29520 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.048245 29531 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.074704 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:28.079419 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.076427 29510 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:28.083554 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60653.9us
I20260709 12:18:28.083978 27582 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:28.083856 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.086416 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.086691 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:28.089267 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.089408 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.090384 27581 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:28.090231 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.091317 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.092427 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.094163 27581 raft_consensus.cc:2376] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3. Re-sending same reply.
W20260709 12:18:28.095788 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78980.8us
I20260709 12:18:28.096331 27587 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:28.097021 28994 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:28.100346 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1536) took 1259 ms. Trace:
I20260709 12:18:28.102177 29520 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:28.102425 27588 rpcz_store.cc:276] 0709 12:18:26.840433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.099507 (+1259074us) service_pool.cc:224] Handling call
0709 12:18:28.099896 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.100064 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.100088 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.100129 (+    41us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:28.100161 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.100227 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.100303 (+    76us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2432}
I20260709 12:18:28.106910 29369 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.108958 29078 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.095003 29507 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.110121 29507 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:28.110623 29507 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.111680 28996 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:28.114147 29551 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:18:28.115918 29528 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:28.096495 29549 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:18:28.101071 27584 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:28.087157 29065 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.085063 28958 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:28.123764 28958 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.110149 29544 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.124689 28958 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.125224 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 339) took 1568 ms. Trace:
I20260709 12:18:28.125667 28328 rpcz_store.cc:276] 0709 12:18:26.556604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.564002 (+  7398us) service_pool.cc:224] Handling call
0709 12:18:26.583896 (+ 19894us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.440024 (+856128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.124940 (+684916us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:28.125165 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1833365888}
I20260709 12:18:28.126456 29544 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.096046 29478 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:28.127583 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.127815 29478 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.128432 29066 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.128763 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.129529 27657 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:28.129859 29478 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.130087 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.130396 29478 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.123265 27584 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:18:28.130993 27584 pending_rounds.cc:85] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
I20260709 12:18:28.131222 27584 pending_rounds.cc:107] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:18:28.131453 27584 raft_consensus.cc:2889] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:28.115842 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.133824 27584 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:28.134714 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.097187 27582 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.135491 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.136085 27662 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:28.136147 27588 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:28.141611 29520 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:28.142045 29520 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.142114 29369 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:28.142194 29365 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.142544 29520 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:28.142804 29369 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:28.143746 29369 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.142066 29066 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.148412 27729 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:28.149761 29520 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.153293 29520 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.153183 29443 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.153506 29348 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.156792 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.157938 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:28.147140 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.159885 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.161132 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.162302 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.164537 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 128) took 1324 ms. Trace:
I20260709 12:18:28.165911 27582 rpcz_store.cc:276] 0709 12:18:26.839794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.096727 (+1256933us) service_pool.cc:224] Handling call
0709 12:18:28.097042 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.164222 (+ 67180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.164467 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:28.168691 27582 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:18:28.171295 27582 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
I20260709 12:18:28.172215 27582 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:18:28.172276 27663 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.173277 27582 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:28.174506 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.176214 28334 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:28.176954 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.174499 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.186080 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.186854 27663 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:28.186081 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.188638 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.093061 29539 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.189949 29539 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.183080 27662 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:28.193558 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.160540 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.193219 29534 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.204366 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.205201 29539 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.205627 27583 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.195552 28996 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:28.184162 29520 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.184007 29348 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:28.213141 29348 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:28.213660 29365 raft_consensus.cc:2764] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:18:28.130857 29478 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.196033 29513 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:28.214334 29348 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.192764 29528 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:28.192713 29078 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:28.212378 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.210456 27586 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:28.206972 29531 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.240804 29365 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:28.241557 29365 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:18:28.208021 27663 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:28.196856 29443 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:28.199976 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.200349 29484 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.245646 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.236901 29078 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:28.222445 29513 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.248071 29503 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.210062 29520 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.235689 29458 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:28.238049 29528 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:28.251718 29478 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.251976 29057 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:28.293285 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.295522 28337 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:28.316401 27732 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:28.319415 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.320665 29207 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:28.321230 29207 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:28.322331 29207 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.278611 27663 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:28.326828 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.327088 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.327574 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:28.328049 29270 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.328449 29270 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.328537 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.328907 29270 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.329737 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.331041 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.331715 27663 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:18:28.295970 29078 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.301323 29066 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:28.301442 29360 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:28.300529 28994 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:18:28.256896 27485 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 5, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 5 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:28.303754 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.304531 29510 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.305716 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:28.314456 29528 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:28.255851 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.257135 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.244089 29500 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:18:28.347002 28328 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:28.255254 29507 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.257781 29484 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.355322 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1455) took 3306 ms. Trace:
I20260709 12:18:28.308233 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.309330 28331 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:28.310902 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.310951 29506 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.317181 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.355734 27737 rpcz_store.cc:276] 0709 12:18:25.048748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.048874 (+   126us) service_pool.cc:224] Handling call
0709 12:18:25.964400 (+915526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:25.965065 (+   665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.355112 (+2390047us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b540017e8d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:28.355262 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2078990720}
I20260709 12:18:28.358398 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.298897 27661 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:28.296293 29513 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:28.337163 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.359313 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.360554 29207 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:18:28.296701 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107350us
I20260709 12:18:28.360957 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:28.360989 29207 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.361677 29207 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:28.361846 29409 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.362339 29409 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.362695 29409 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.362874 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.363898 27580 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260709 12:18:28.364104 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.313916 29520 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.304600 29293 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:28.341324 28333 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:28.258309 29553 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
I20260709 12:18:28.344245 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.371728 29553 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:28.372124 29553 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:28.372466 29507 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:28.353850 29065 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:18:28.367646 29560 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:28.373996 29560 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:28.346174 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:28.314534 29057 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.348838 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.336567 29531 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:28.308036 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 120) took 1722 ms (client timeout 2224 ms). Trace:
I20260709 12:18:28.356025 27660 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.368556 29503 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:28.368362 29365 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:28.369231 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.370515 29510 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.371857 27659 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:28.345629 29270 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.364853 29429 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:28.384450 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:28.358659 29478 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.391557 27583 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6.
I20260709 12:18:28.367147 29544 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:28.394426 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:28.366791 29562 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:28.380729 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.381750 29444 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:28.411746 27660 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:28.374858 29560 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.416417 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.366914 29456 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:18:28.393000 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.379971 29347 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:18:28.395025 29429 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:28.394165 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:28.445766 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:28.408717 29389 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.445257 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:28.449894 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84542.9us
I20260709 12:18:28.417600 27655 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:28.458492 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:28.453872 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:28.419852 27630 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6.
I20260709 12:18:28.402253 29544 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:28.449864 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.459167 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.394047 29500 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:18:28.444900 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.463688 27630 leader_election.cc:403] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:18:28.390434 27586 rpcz_store.cc:269] 0709 12:18:26.586056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.588051 (+  1995us) service_pool.cc:224] Handling call
0709 12:18:28.160428 (+1572377us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:28.307813 (+147385us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b54001fefd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:28.307964 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179095680}
I20260709 12:18:28.415117 29444 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.397866 29458 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:18:28.453675 29429 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:28.407975 29066 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:18:28.471598 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:28.481691 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62071.8us
I20260709 12:18:28.409310 29503 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:18:28.481938 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:28.457667 29389 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:28.467458 29544 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:28.470520 29562 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:28.469555 29231 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:28.460140 29460 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:18:28.474082 29348 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 12:18:28.448799 29553 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.487306 27661 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:28.485678 29347 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6.
I20260709 12:18:28.506604 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.513434 27583 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:28.488654 27731 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.497666 29389 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.493917 29270 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.500313 29562 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:28.517231 29478 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.504570 29506 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.529033 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:28.533041 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 122) took 1944 ms (client timeout 1896 ms). Trace:
W20260709 12:18:28.533509 27589 rpcz_store.cc:269] 0709 12:18:26.588547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.588751 (+   204us) service_pool.cc:224] Handling call
0709 12:18:28.528826 (+1940075us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:28.532958 (+  4132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2672000}
W20260709 12:18:28.526309 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54802us
I20260709 12:18:28.510861 29430 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.517413 29348 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:28.513566 29360 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:28.512794 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 373) took 1410 ms. Trace:
I20260709 12:18:28.541419 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.544782 27660 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:28.526782 29562 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:28.552855 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.528066 29565 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:28.589641 29565 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:28.590149 29565 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:28.591890 29270 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:28.531459 29347 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:28.593060 29347 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.595209 29428 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:18:28.596879 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.598150 29409 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:28.559684 29564 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:18:28.568601 29288 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.570755 29207 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.574816 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.575665 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.577325 28333 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:28.577358 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.612402 28337 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:28.543604 29360 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.580746 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.546464 28329 rpcz_store.cc:276] 0709 12:18:27.102221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.104474 (+  2253us) service_pool.cc:224] Handling call
0709 12:18:27.119104 (+ 14630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.226735 (+107631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.226767 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.226818 (+    51us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:27.226856 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.501521 (+1274665us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:28.504038 (+  2517us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.512713 (+  8675us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":247015808}
I20260709 12:18:28.529908 29478 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.584379 29066 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:28.527911 29078 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.587986 28988 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.600373 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.601118 27663 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.622011 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.622861 27589 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
W20260709 12:18:28.594455 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1469) took 1554 ms (client timeout 1610 ms). Trace:
W20260709 12:18:28.601198 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48461.9us
I20260709 12:18:28.601490 27583 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:28.626632 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.608726 27662 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:28.612387 27737 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:28.577931 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.628899 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.630759 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.556684 29531 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.614841 27738 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:28.603574 29181 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:28.604928 29504 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:28.617338 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:28.616485 29503 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.617825 29288 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:28.651744 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.646194 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.698478 27583 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:28.645968 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:28.661901 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:28.647002 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.649820 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.707435 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.705363 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.735984 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.625149 29347 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.647691 29568 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:28.737749 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.738168 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:28.642351 29181 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:28.644008 29207 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.644351 29409 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.749044 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.641800 29569 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:28.696727 29231 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:28.742862 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.743396 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.743553 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.745433 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.644526 29478 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.620860 29458 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20260709 12:18:28.746447 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:28.742583 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 92873.4us
I20260709 12:18:28.649999 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:28.750708 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.707646 29531 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.751783 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:28.697561 29513 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.641659 29528 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.779846 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:28.759416 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 169551us
I20260709 12:18:28.769321 27733 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:28.742146 29568 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:28.745461 29503 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:18:28.843813 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64121.5us
I20260709 12:18:28.774205 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.775610 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.845498 27583 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:28.846733 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.848183 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.736857 29348 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:18:28.706370 29429 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.769853 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.762247 29231 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:28.635557 27730 rpcz_store.cc:269] 0709 12:18:27.040675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.131586 (+ 90911us) service_pool.cc:224] Handling call
0709 12:18:28.594252 (+1462666us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54000a5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:28.594393 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553318144}
I20260709 12:18:28.759238 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.773139 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.850906 27589 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:28.863876 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.864368 27734 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:28.854452 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:28.836603 29566 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:18:28.860239 29570 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:18:28.876745 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.881958 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.850014 29478 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:28.892863 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.879170 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.856734 29062 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:28.887902 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:28.883297 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.838167 29520 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.900775 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.903846 27660 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.862584 29231 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.923748 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.924870 29456 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:28.925132 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.925401 29456 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.926776 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.927984 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.928845 29348 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:28.930050 29456 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:28.932708 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.933784 27587 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:28.904759 29293 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.529s	user 0.012s	sys 0.006s
W20260709 12:18:28.905336 29571 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 12:18:28.905998 29429 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.896064 28994 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:18:28.908243 29568 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.909116 29057 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.892531 29478 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:28.910563 29078 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.862648 29181 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.905277 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.944676 28994 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:28.913056 29520 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.945083 28994 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.915343 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:28.945848 27589 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:28.946033 28994 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.947358 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1544) took 1505 ms. Trace:
I20260709 12:18:28.947767 27589 rpcz_store.cc:276] 0709 12:18:27.442477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.946438 (+1503961us) service_pool.cc:224] Handling call
0709 12:18:28.946741 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.947012 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.947041 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.947103 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:28.947131 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.947187 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.947286 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:28.915678 29062 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:28.916086 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:28.916667 29528 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.972188 27659 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:18:28.916975 29387 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:28.973405 29387 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:28.919148 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.920090 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:28.974277 29387 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:28.975279 29387 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.975822 27583 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:28.922933 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:28.940579 27587 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:28.980587 27587 raft_consensus.cc:606] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:28.909637 29544 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:28.960470 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.945118 27657 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:28.964440 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:28.966563 29484 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.966603 27660 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:28.989292 29512 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:28.989718 29512 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:28.989892 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:28.991792 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:28.967566 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 151785us
I20260709 12:18:28.993577 27660 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:28.993853 29544 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:28.968129 29513 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:28.972072 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1359) took 1334 ms (client timeout 1631 ms). Trace:
I20260709 12:18:28.919745 29458 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:28.948449 28994 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:28.949573 29478 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:28.983168 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64087.3us
I20260709 12:18:28.988296 28971 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:29.005614 27736 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:28.995916 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:28.997076 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:29.007025 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:29.007753 29579 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:29.001345 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.007900 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:29.008296 29579 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:29.001729 27733 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:29.009596 29513 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:29.009230 29579 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.010123 29513 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:29.011220 29456 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:29.011009 29513 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.014109 29456 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:29.014283 29528 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:29.014987 29528 consensus_queue.cc:1243] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:29.002555 27587 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:18:29.016067 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1542) took 1402 ms. Trace:
I20260709 12:18:29.016384 27587 rpcz_store.cc:276] 0709 12:18:27.613802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.001978 (+1388176us) service_pool.cc:224] Handling call
0709 12:18:29.002287 (+   309us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:29.015761 (+ 13474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.016009 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.023416 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:29.029166 29544 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.029819 29544 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.031147 29456 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:29.033654 29544 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:28.987869 29507 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:29.045861 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.990624 29512 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.053225 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:29.054193 28337 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:29.055987 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:29.056932 28337 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:28.985976 29389 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.003157 29458 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:28.971002 29444 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:29.081413 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:29.091140 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1517) took 1783 ms (client timeout 2249 ms). Trace:
W20260709 12:18:29.091920 27585 rpcz_store.cc:269] 0709 12:18:27.311023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.993064 (+682041us) service_pool.cc:224] Handling call
0709 12:18:29.090960 (+1097896us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:29.091083 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34417024}
I20260709 12:18:29.093319 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:29.094228 27585 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:29.096208 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:29.097059 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:29.089640 29512 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:29.101608 29490 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:29.102190 29444 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:29.102398 29500 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:29.102778 29500 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:29.062207 29462 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:18:29.103667 29500 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.104949 29500 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:29.091945 29584 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:18:29.106055 27588 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:28.998198 27729 rpcz_store.cc:269] 0709 12:18:27.637401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.638676 (+  1275us) service_pool.cc:224] Handling call
0709 12:18:28.971909 (+1333233us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:28.972011 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25026944}
I20260709 12:18:29.108397 29538 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:29.108825 29538 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:29.071369 28337 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:29.109652 29538 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.110895 29538 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:29.112659 29565 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.000s
I20260709 12:18:29.113049 29565 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.116717 28328 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:29.118244 28330 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:29.119190 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:18:29.124166 27582 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:29.003595 29231 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.126003 27588 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:29.127100 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:29.145951 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.172024 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.119190 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:29.175125 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48118.6us
I20260709 12:18:29.177150 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.179340 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:29.136979 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.189407 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.190783 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.192970 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:29.181265 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:29.197316 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.209982 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:29.210179 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:29.214660 29562 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.643s	user 0.007s	sys 0.003s
I20260709 12:18:29.222010 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.229481 27582 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
W20260709 12:18:29.233070 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.232710 27657 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:29.239226 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1505) took 1040 ms. Trace:
I20260709 12:18:29.213295 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:29.239634 27664 rpcz_store.cc:276] 0709 12:18:28.198565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.198745 (+   180us) service_pool.cc:224] Handling call
0709 12:18:29.210034 (+1011289us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:29.239164 (+ 29130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2382080}
I20260709 12:18:29.241019 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:29.244874 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63698.5us
I20260709 12:18:29.250651 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.252357 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.253732 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.077190 28996 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.132s
W20260709 12:18:29.257476 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 64583.8us
I20260709 12:18:29.267246 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:29.268827 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.270220 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:29.259289 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46226us
I20260709 12:18:29.254972 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.098995 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:29.198169 29387 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:29.211102 27661 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:29.212201 29585 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:29.273355 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:29.294860 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.279562 27737 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:29.296379 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.265241 29428 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:29.281672 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71779.3us
I20260709 12:18:29.300809 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:29.211810 29460 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:29.303853 27737 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:29.280362 29553 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:29.289512 27659 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:29.306399 27737 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:29.263309 29385 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:29.266103 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:29.281881 29579 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:29.278245 29544 raft_consensus.cc:2764] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:18:29.323022 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1538) took 1788 ms. Trace:
I20260709 12:18:29.326558 27589 rpcz_store.cc:276] 0709 12:18:27.534630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.957737 (+1423107us) service_pool.cc:224] Handling call
0709 12:18:28.962704 (+  4967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.962892 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:18:29.191548 (+228656us) log.cc:844] Serialized 14 byte log entry
0709 12:18:29.322577 (+131029us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.322649 (+    72us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:29.322709 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.322798 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.322949 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4961,"raft.queue_time_us":5117,"thread_start_us":313243,"threads_started":2}
I20260709 12:18:29.296221 29565 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:29.336709 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61467.9us
I20260709 12:18:29.356694 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.332840 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:29.316918 29314 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:29.300124 29428 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:29.321907 29460 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:29.314591 29299 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:29.256129 29562 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.388144 29428 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:29.395498 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.396289 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.397246 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:29.397969 29586 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:18:29.398582 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:29.380100 29314 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:29.399765 29314 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:29.399900 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:29.400643 28336 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:18:29.399919 29590 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:18:29.401286 28330 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:29.402300 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1541) took 1995 ms. Trace:
I20260709 12:18:29.402657 27584 rpcz_store.cc:276] 0709 12:18:27.406579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.941583 (+1535004us) service_pool.cc:224] Handling call
0709 12:18:28.941899 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:28.942421 (+   522us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.942448 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.942940 (+   492us) log.cc:844] Serialized 26 byte log entry
0709 12:18:29.071182 (+128242us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:29.071219 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.071297 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.402109 (+330812us) raft_consensus.cc:1685] finished
0709 12:18:29.402128 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.402244 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":208,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":6863,"thread_start_us":127251,"threads_started":1,"wal-append.queue_time_us":358437}
I20260709 12:18:29.403801 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:29.404631 27584 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:18:29.405241 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1551) took 1054 ms. Trace:
I20260709 12:18:29.405720 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.356165 29387 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:29.331473 29589 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:29.381304 29584 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:29.418633 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.405565 27584 rpcz_store.cc:276] 0709 12:18:28.351266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.404137 (+1052871us) service_pool.cc:224] Handling call
0709 12:18:29.404428 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.405015 (+   587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.405190 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.321735 29187 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
W20260709 12:18:29.422200 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 200234us
I20260709 12:18:29.478324 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.305712 29458 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.493657 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:29.496245 27735 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:18:29.497872 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.390275 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.416725 29580 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:29.419574 28996 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:18:29.356412 29585 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260709 12:18:29.416808 29387 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:29.518601 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:29.495191 29534 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:29.545019 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.551844 27735 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
W20260709 12:18:29.557727 29591 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:18:29.558053 29587 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
W20260709 12:18:29.559311 29592 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
W20260709 12:18:29.560235 29598 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 12:18:29.562268 29562 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:29.562731 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.563961 28996 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:29.564594 28996 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:29.565056 28996 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.515389 29389 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:29.542249 29584 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:29.549346 29387 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:29.516521 29187 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:29.562813 29314 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.602346 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:29.602789 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 224758us
I20260709 12:18:29.567943 29490 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:29.614604 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 302240us
W20260709 12:18:29.616971 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 298404us
I20260709 12:18:29.619161 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.600076 29231 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:29.625723 27659 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:29.626935 28332 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:29.634037 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.642133 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.643018 27735 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:29.606671 29187 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:29.648108 27733 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:29.649194 29612 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.649729 29612 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.652046 29612 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:29.601866 29597 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:18:29.618515 29603 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:29.638985 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:29.805054 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1336) took 1068 ms. Trace:
W20260709 12:18:29.806106 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 167443us
I20260709 12:18:29.770524 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1550) took 1426 ms. Trace:
I20260709 12:18:29.676426 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:29.812565 27664 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:29.816056 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:29.778988 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 234113us
I20260709 12:18:29.794588 27655 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:29.820175 29314 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:29.819902 29535 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:18:29.668769 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:29.675966 29187 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.825783 29187 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:29.807567 27589 rpcz_store.cc:276] 0709 12:18:28.343445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.380274 (+1036829us) service_pool.cc:224] Handling call
0709 12:18:29.380592 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.386215 (+  5623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.386246 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:29.406139 (+ 19893us) log.cc:844] Serialized 26 byte log entry
0709 12:18:29.513587 (+107448us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:29.513643 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.513706 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.769743 (+256037us) raft_consensus.cc:1685] finished
0709 12:18:29.769786 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.770202 (+   416us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5191,"raft.queue_time_us":6227,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":27502,"thread_start_us":4375,"threads_started":1,"wal-append.queue_time_us":186849}
I20260709 12:18:29.644348 29231 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:18:29.827314 29565 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:29.716116 29428 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.624842 29389 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:18:29.604501 29460 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:29.726789 29478 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.677s	user 0.006s	sys 0.001s
I20260709 12:18:29.616492 29490 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:29.730506 29500 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.000s
I20260709 12:18:29.678036 29510 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:29.647601 29512 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.544s	user 0.008s	sys 0.002s
I20260709 12:18:29.724687 29531 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:29.735937 29568 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:29.835286 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.850422 27735 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:29.857842 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1553) took 1480 ms (client timeout 1842 ms). Trace:
I20260709 12:18:29.858605 29389 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:29.859120 29389 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:29.860308 27587 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:29.860998 29389 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.862339 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1545) took 1615 ms. Trace:
W20260709 12:18:29.864233 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1518) took 2548 ms (client timeout 1736 ms). Trace:
I20260709 12:18:29.864166 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:29.864868 27580 rpcz_store.cc:269] 0709 12:18:27.315786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.549740 (+233954us) service_pool.cc:224] Handling call
0709 12:18:29.863631 (+2313891us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54001fead4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:29.863948 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1769457536}
I20260709 12:18:29.695443 29584 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.838500 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:29.869143 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:29.695400 29589 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.821591 29314 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:29.871757 29589 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.873741 27662 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:29.701275 29597 consensus_queue.cc:1243] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:29.874634 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 142) took 1271 ms. Trace:
I20260709 12:18:29.641206 29604 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:29.875715 29604 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:29.877365 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1347) took 1025 ms. Trace:
I20260709 12:18:29.877442 29604 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.878197 27659 rpcz_store.cc:276] 0709 12:18:28.851773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.317692 (+465919us) service_pool.cc:224] Handling call
0709 12:18:29.376177 (+ 58485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.627198 (+251021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.876631 (+249433us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:29.877078 (+   447us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":573421312}
W20260709 12:18:29.806559 28994 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.803s	user 0.007s	sys 0.002s
I20260709 12:18:29.672719 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:29.809434 29603 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:29.810652 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1519) took 1506 ms. Trace:
I20260709 12:18:29.823112 29535 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:29.866942 27656 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:29.870584 27657 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:29.802945 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.882395 28996 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:29.840215 29478 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.919967 27660 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:29.921770 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1342) took 1158 ms. Trace:
I20260709 12:18:29.922309 27660 rpcz_store.cc:276] 0709 12:18:28.763766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.040149 (+276383us) service_pool.cc:224] Handling call
0709 12:18:29.040418 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.920518 (+880100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.921439 (+   921us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b5400291654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:29.921705 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2021718272}
I20260709 12:18:29.922940 29603 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:29.845182 29531 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:29.844143 29510 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:29.843152 29490 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:29.880848 27585 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:18:29.858386 27582 rpcz_store.cc:269] 0709 12:18:28.377541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.743283 (+365742us) service_pool.cc:224] Handling call
0709 12:18:29.857297 (+1114014us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54000b1154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:29.857554 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":831904128}
I20260709 12:18:29.894980 27655 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:29.895766 27664 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:29.882611 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 143) took 1238 ms. Trace:
I20260709 12:18:29.880784 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:29.883478 29538 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.772s	user 0.015s	sys 0.001s
I20260709 12:18:29.892089 27662 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:18:29.875262 27589 rpcz_store.cc:276] 0709 12:18:28.602748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.872956 (+1270208us) service_pool.cc:224] Handling call
0709 12:18:29.873649 (+   693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.874136 (+   487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.874170 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.874248 (+    78us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:29.874294 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.874367 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.874526 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.844887 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1502) took 1238 ms. Trace:
I20260709 12:18:29.777156 27581 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:29.862836 27587 rpcz_store.cc:276] 0709 12:18:28.246557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.017133 (+770576us) service_pool.cc:224] Handling call
0709 12:18:29.859548 (+842415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.860932 (+  1384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.861983 (+  1051us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b54000b1150. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:29.862223 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1936286976}
I20260709 12:18:29.805780 27661 rpcz_store.cc:276] 0709 12:18:28.736831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.980020 (+243189us) service_pool.cc:224] Handling call
0709 12:18:28.984039 (+  4019us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.302977 (+318938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.804252 (+501275us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:29.804796 (+   544us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417153792}
W20260709 12:18:29.912417 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 64120.9us
I20260709 12:18:29.908632 29389 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:29.914865 27663 rpcz_store.cc:276] 0709 12:18:28.304041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.376920 (+ 72879us) service_pool.cc:224] Handling call
0709 12:18:28.377305 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.621176 (+243871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.809899 (+1188723us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400211e54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:29.810348 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":514275200}
I20260709 12:18:29.916265 29535 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:29.931348 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1499) took 1774 ms (client timeout 1570 ms). Trace:
I20260709 12:18:29.933085 29604 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:29.932793 29500 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:18:29.934264 28994 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:29.933916 29535 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.934772 28994 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:29.936158 29510 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.932282 27589 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:29.932262 27588 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:29.929356 27734 rpcz_store.cc:276] 0709 12:18:28.606584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.762478 (+155894us) service_pool.cc:224] Handling call
0709 12:18:28.785084 (+ 22606us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.864781 (+ 79697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.844277 (+979496us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:29.844662 (+   385us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176197888}
W20260709 12:18:29.941282 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:29.931905 27656 rpcz_store.cc:269] 0709 12:18:28.156844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.174828 (+ 17984us) service_pool.cc:224] Handling call
0709 12:18:29.236468 (+1061640us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:29.929888 (+693420us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2760576}
I20260709 12:18:29.932474 28329 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:29.849287 27658 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:29.947682 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1556) took 1188 ms. Trace:
I20260709 12:18:29.948379 27589 rpcz_store.cc:276] 0709 12:18:28.751989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.931390 (+1179401us) service_pool.cc:224] Handling call
0709 12:18:29.931926 (+   536us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.940364 (+  8438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.940577 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.938300 29510 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:29.960119 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1555) took 1208 ms. Trace:
I20260709 12:18:29.960634 27586 rpcz_store.cc:276] 0709 12:18:28.751625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.920823 (+1169198us) service_pool.cc:224] Handling call
0709 12:18:29.921333 (+   510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.939582 (+ 18249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.941666 (+  2084us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.959542 (+ 17876us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:29.959649 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.959745 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.959926 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.958659 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:29.976882 27735 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:29.970935 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:29.936295 28994 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:29.929140 27580 rpcz_store.cc:276] 0709 12:18:28.643892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.881086 (+1237194us) service_pool.cc:224] Handling call
0709 12:18:29.881632 (+   546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.882090 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.882149 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.882245 (+    96us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:29.882317 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.882395 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.882540 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.977506 29207 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:18:29.982352 27580 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:29.948091 29500 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:29.921572 29360 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:29.983063 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1557) took 1230 ms. Trace:
I20260709 12:18:29.985951 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.008157 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:30.007030 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44549.7us
I20260709 12:18:30.019548 29385 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:30.020174 29385 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:30.021374 29062 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:18:30.023180 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:30.024683 27657 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:30.026324 27657 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:29.978075 29478 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:30.026809 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.027271 27664 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:30.027971 29428 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:30.009784 29535 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:30.029789 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1547) took 1782 ms. Trace:
I20260709 12:18:30.031380 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1558) took 1278 ms. Trace:
W20260709 12:18:30.042745 27583 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.005s	sys 0.053s
I20260709 12:18:30.046139 27587 rpcz_store.cc:276] 0709 12:18:28.752585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.030231 (+1277646us) service_pool.cc:224] Handling call
0709 12:18:30.030647 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.030975 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:30.031007 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.031075 (+    68us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:30.031117 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.031193 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.031298 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.998066 28988 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:30.037250 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.000337 29538 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:30.040364 27581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 134) took 1717 ms (client timeout 1695 ms). Trace:
I20260709 12:18:30.041590 29360 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:30.027839 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:30.051753 27581 rpcz_store.cc:269] 0709 12:18:28.322597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.600160 (+277563us) service_pool.cc:224] Handling call
0709 12:18:30.040075 (+1439915us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:30.040253 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624328576}
I20260709 12:18:30.045774 27585 rpcz_store.cc:276] 0709 12:18:28.247157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.097904 (+850747us) service_pool.cc:224] Handling call
0709 12:18:29.879991 (+782087us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.927828 (+ 47837us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.029513 (+101685us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b54001fead0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:30.029714 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1797492992}
W20260709 12:18:30.060477 27583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.005s	sys 0.053s
I20260709 12:18:30.061789 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:29.994539 29500 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.051011 28988 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.008453 27580 rpcz_store.cc:276] 0709 12:18:28.752302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.981432 (+1229130us) service_pool.cc:224] Handling call
0709 12:18:29.981869 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.982759 (+   890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.982984 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.082975 27589 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 12:18:30.100073 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 219616us
W20260709 12:18:30.102455 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.097092 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1567) took 1260 ms. Trace:
I20260709 12:18:30.104771 27589 rpcz_store.cc:276] 0709 12:18:28.836834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.055235 (+1218401us) service_pool.cc:224] Handling call
0709 12:18:30.082499 (+ 27264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:30.096802 (+ 14303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.097034 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.085119 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.106714 27660 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:30.126062 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.126236 29299 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260709 12:18:30.127925 29299 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:30.128067 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.128486 29057 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:30.128831 29299 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:30.118399 29428 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.134328 29458 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:30.138437 28988 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.144886 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:30.160737 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:30.161669 29603 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.163806 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:30.167946 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.165246 27580 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:30.135139 29299 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.171415 27589 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:30.173750 27586 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:30.174703 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1565) took 1154 ms. Trace:
I20260709 12:18:30.178087 29531 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.165695 29428 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:30.186764 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.188186 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1539) took 2653 ms. Trace:
I20260709 12:18:30.139503 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.188882 27585 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:18:30.196573 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1562) took 1171 ms. Trace:
I20260709 12:18:30.197163 27589 rpcz_store.cc:276] 0709 12:18:29.018299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.108667 (+1090368us) service_pool.cc:224] Handling call
0709 12:18:30.129751 (+ 21084us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.171930 (+ 42179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.189164 (+ 17234us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:30.198266 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70333.3us
W20260709 12:18:30.199759 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74423.5us
I20260709 12:18:30.200505 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1560) took 1426 ms. Trace:
I20260709 12:18:30.201045 27587 rpcz_store.cc:276] 0709 12:18:28.765772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.053188 (+1287416us) service_pool.cc:224] Handling call
0709 12:18:30.120347 (+ 67159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.170894 (+ 50547us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:30.170925 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.170971 (+    46us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:30.192020 (+ 21049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.192123 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.192252 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.166407 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:30.203459 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1561) took 1199 ms. Trace:
I20260709 12:18:30.207091 27585 rpcz_store.cc:276] 0709 12:18:29.003728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.073405 (+1069677us) service_pool.cc:224] Handling call
0709 12:18:30.122442 (+ 49037us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:30.200294 (+ 77852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.203398 (+  3104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.179302 29531 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:30.212636 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:30.175194 27586 rpcz_store.cc:276] 0709 12:18:29.020383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.146081 (+1125698us) service_pool.cc:224] Handling call
0709 12:18:30.173482 (+ 27401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.174301 (+   819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.174579 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.217046 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1564) took 1197 ms. Trace:
I20260709 12:18:30.187467 28988 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:30.182034 29428 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:30.205809 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:30.223887 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164549us
W20260709 12:18:30.224421 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46895.5us
I20260709 12:18:30.226961 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:30.227681 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:30.201135 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1538) took 1038 ms. Trace:
I20260709 12:18:30.230684 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:30.192305 27583 rpcz_store.cc:276] 0709 12:18:27.534973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.975237 (+1440264us) service_pool.cc:224] Handling call
0709 12:18:28.975540 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.975711 (+   171us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:29.015734 (+ 40023us) log.cc:844] Serialized 14 byte log entry
0709 12:18:30.078161 (+1062427us) log.cc:844] Serialized 14 byte log entry
0709 12:18:30.132498 (+ 54337us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.146450 (+ 13952us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:30.187665 (+ 41215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.187820 (+   155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.188075 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":286483,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":3238,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":9229,"thread_start_us":1004596,"threads_started":2,"wal-append.queue_time_us":495886}
I20260709 12:18:30.238548 29562 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:30.245561 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.247170 27735 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:30.229981 29458 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:30.223927 27580 rpcz_store.cc:276] 0709 12:18:29.019809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.121238 (+1101429us) service_pool.cc:224] Handling call
0709 12:18:30.164996 (+ 43758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.191449 (+ 26453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.216974 (+ 25525us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.257409 27589 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:18:30.248523 27655 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:18:30.259971 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 145) took 1189 ms. Trace:
I20260709 12:18:30.248190 29603 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:30.261142 27589 rpcz_store.cc:276] 0709 12:18:29.070264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.230831 (+1160567us) service_pool.cc:224] Handling call
0709 12:18:30.250660 (+ 19829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.259765 (+  9105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.259923 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.228900 29602 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:18:30.231796 27661 rpcz_store.cc:276] 0709 12:18:29.153735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.951462 (+797727us) service_pool.cc:224] Handling call
0709 12:18:29.957439 (+  5977us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.958258 (+   819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.958302 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:29.959078 (+   776us) log.cc:844] Serialized 26 byte log entry
0709 12:18:30.016351 (+ 57273us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:30.044235 (+ 27884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.051436 (+  7201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:30.163899 (+112463us) raft_consensus.cc:1685] finished
0709 12:18:30.166190 (+  2291us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.189137 (+ 22947us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":153,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":218,"thread_start_us":33246,"threads_started":1,"wal-append.queue_time_us":117175}
I20260709 12:18:30.269459 27586 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:30.265267 29538 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:30.279103 29562 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.285202 29365 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:30.288297 29365 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.283967 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:30.292099 27583 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:18:30.292614 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 148) took 1165 ms. Trace:
I20260709 12:18:30.292814 27583 rpcz_store.cc:276] 0709 12:18:29.127518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.243562 (+1116044us) service_pool.cc:224] Handling call
0709 12:18:30.270486 (+ 26924us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.292431 (+ 21945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.292573 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.294600 29066 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
W20260709 12:18:30.295522 29637 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.079s	user 0.002s	sys 0.000s
I20260709 12:18:30.295615 29066 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:30.296150 29066 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.298422 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:30.285109 29490 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.299839 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:30.300256 29490 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.278340 29458 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.282151 27580 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:30.305716 27485 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:30.309020 29589 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.310552 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1552) took 1957 ms. Trace:
W20260709 12:18:30.311120 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 108005us
I20260709 12:18:30.279170 29456 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
W20260709 12:18:30.315109 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93051.4us
I20260709 12:18:30.314133 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 146) took 1236 ms. Trace:
I20260709 12:18:30.318497 27586 rpcz_store.cc:276] 0709 12:18:29.076642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.231182 (+1154540us) service_pool.cc:224] Handling call
0709 12:18:30.257751 (+ 26569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.282985 (+ 25234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.312691 (+ 29706us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.321014 29589 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:30.321471 29589 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.321847 29589 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:30.313848 27584 rpcz_store.cc:276] 0709 12:18:28.352727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.441898 (+1089171us) service_pool.cc:224] Handling call
0709 12:18:29.442366 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:29.443106 (+   740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.443150 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:29.576647 (+133497us) log.cc:844] Serialized 52 byte log entry
0709 12:18:30.043195 (+466548us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:30.045388 (+  2193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.059510 (+ 14122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:30.304747 (+245237us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b18002f2f00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:30.310075 (+  5328us) raft_consensus.cc:1685] finished
0709 12:18:30.310099 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.310377 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":236,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":50476,"spinlock_wait_cycles":27858688,"thread_start_us":334326,"threads_started":1,"wal-append.queue_time_us":354645}
I20260709 12:18:30.325045 29507 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:30.325454 27580 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:30.286288 29500 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:30.330892 27583 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:30.333513 29584 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:18:30.335580 27660 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:18:30.340077 29387 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.723s	user 0.003s	sys 0.013s
I20260709 12:18:30.340590 29387 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.341831 28328 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:30.342686 27586 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:30.297165 29066 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.349821 29458 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:30.352953 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.359460 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.360327 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1567) took 1139 ms. Trace:
I20260709 12:18:30.360700 27589 rpcz_store.cc:276] 0709 12:18:29.221198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.276351 (+1055153us) service_pool.cc:224] Handling call
0709 12:18:30.314519 (+ 38168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.354277 (+ 39758us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:30.354296 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.354357 (+    61us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:30.360148 (+  5791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.360207 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.360285 (+    78us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:30.330319 29602 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 12:18:30.351217 29365 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.369236 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.370041 29613 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:30.371857 29181 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:30.372328 29299 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.373405 29299 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:30.375593 29490 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.377810 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.315228 29562 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.392755 29365 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:30.393985 29628 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.235s	user 0.003s	sys 0.003s
I20260709 12:18:30.380862 29500 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:30.379233 29066 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:30.402521 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 140) took 2078 ms. Trace:
I20260709 12:18:30.411146 28331 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:30.413045 29365 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.413902 29365 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.414011 27485 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:30.414816 29575 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:30.416391 29365 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.416276 29385 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.417912 29385 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:30.414161 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 240277us
W20260709 12:18:30.418360 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 187749us
I20260709 12:18:30.418874 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:30.419543 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.420635 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.420743 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1559) took 1667 ms. Trace:
I20260709 12:18:30.421516 29478 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:30.421949 29478 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.422353 29478 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:30.423272 29512 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:30.423837 29512 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:30.424829 29512 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.402973 27588 rpcz_store.cc:276] 0709 12:18:28.324403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.175952 (+851549us) service_pool.cc:224] Handling call
0709 12:18:29.176316 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.943594 (+767278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.402204 (+458610us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b5400186fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:30.402443 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1733813504}
I20260709 12:18:30.428911 29207 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.429541 29207 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.416318 27584 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:30.421262 27582 rpcz_store.cc:276] 0709 12:18:28.752909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.033795 (+1280886us) service_pool.cc:224] Handling call
0709 12:18:30.034249 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.118099 (+ 83850us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:30.118120 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.118145 (+    25us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:30.118167 (+    22us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.420458 (+302291us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:30.420498 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.420684 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100203392}
W20260709 12:18:30.437145 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.386569 29360 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:18:30.440647 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.441587 29500 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:30.442982 29500 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:30.443670 29500 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:30.444032 29500 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.444319 29535 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:30.444536 29065 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:30.445044 29065 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.445221 29535 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:30.444924 29500 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.446084 29065 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.446844 29500 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:30.429754 27485 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:30.447187 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1530) took 1296 ms. Trace:
W20260709 12:18:30.447942 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 308591us
W20260709 12:18:30.448588 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.448808 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.449543 29314 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.449013 27485 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:30.449791 29539 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.450095 29314 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.450318 29539 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:30.451401 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82336.1us
I20260709 12:18:30.452044 29301 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:30.452497 29535 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:30.452693 29301 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.453068 29535 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:30.453538 29535 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:30.453665 29301 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.448197 27658 rpcz_store.cc:276] 0709 12:18:29.150930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.806815 (+655885us) service_pool.cc:224] Handling call
0709 12:18:29.807466 (+   651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.848859 (+ 41393us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:30.237705 (+388846us) log.cc:844] Serialized 14 byte log entry
0709 12:18:30.446656 (+208951us) log.cc:844] Serialized 14 byte log entry
0709 12:18:30.446827 (+   171us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.446870 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:30.446922 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.446993 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.447122 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117231,"prepare.queue_time_us":947,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":119436,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"thread_start_us":59505,"threads_started":1,"wal-append.queue_time_us":123734}
I20260709 12:18:30.452232 29314 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.452208 29539 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.455152 29301 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:30.456429 29568 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.611s	user 0.006s	sys 0.009s
I20260709 12:18:30.456791 29539 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:30.457312 28994 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:30.458029 29539 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:30.458385 29539 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:30.452548 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.450688 27485 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:18:30.522929 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 131193us
I20260709 12:18:30.523641 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:30.508731 27485 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:18:30.507654 29604 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.002s
I20260709 12:18:30.512606 29365 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:30.544710 29387 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:30.541030 29065 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:30.547492 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:30.548403 29588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.010s	sys 0.043s
W20260709 12:18:30.548776 29588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.010s	sys 0.043s
I20260709 12:18:30.552922 27738 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:18:30.551282 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:30.566116 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 125580us
W20260709 12:18:30.565156 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 128633us
I20260709 12:18:30.563098 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:30.567611 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115215us
I20260709 12:18:30.556191 29057 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:18:30.573988 29585 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:18:30.574275 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.574934 27586 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:30.575201 29585 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:30.576040 27583 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:30.576519 27580 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:30.576687 29585 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:18:30.562129 29562 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.578028 29387 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:18:30.578131 29585 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:30.578603 29387 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:30.459291 29539 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.578994 29585 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.579843 29387 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.580647 29539 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:30.581418 29387 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:30.560832 28996 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 12:18:30.584277 29585 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.559429 29360 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 12:18:30.586406 29605 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.587009 29605 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:30.589808 29638 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:18:30.591090 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:30.598177 29589 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.599869 29535 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.602979 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:30.607965 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.570040 27485 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 2, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:30.609320 29057 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.609916 29507 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.610194 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 161) took 1604 ms. Trace:
I20260709 12:18:30.610980 29360 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:30.613519 29301 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:18:30.615463 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.616878 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:30.630115 27738 rpcz_store.cc:276] 0709 12:18:29.005483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.005621 (+   138us) service_pool.cc:224] Handling call
0709 12:18:29.005928 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.563881 (+1557953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.609856 (+ 45975us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:30.610130 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3556382464}
W20260709 12:18:30.634135 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:30.619453 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170722us
W20260709 12:18:30.620138 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.623571 29604 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.626139 29605 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:30.626667 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:30.628557 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:30.630056 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.633176 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:30.644584 27735 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:30.646706 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:30.635061 29301 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.633759 29065 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.633654 29360 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.655485 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:30.658313 27485 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:30.655189 29389 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:30.663401 29588 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:30.652522 29490 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:30.665556 29604 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.667459 29270 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
W20260709 12:18:30.669687 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62009.4us
I20260709 12:18:30.665676 29360 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.658551 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:18:30.652729 29510 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.677s	user 0.012s	sys 0.007s
I20260709 12:18:30.676757 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:30.677502 28328 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:30.678941 29207 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.646590 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:30.680002 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.636574 29507 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.679416 29314 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:30.643877 29057 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.681627 27586 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260709 12:18:30.691825 27586 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 1
I20260709 12:18:30.694346 27485 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:30.662986 29389 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.698508 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:30.659965 29584 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:30.696671 27659 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:30.692163 27586 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:18:30.701320 27586 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:30.681562 29365 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:30.671211 29562 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:30.713979 27485 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:30.716439 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:30.715802 27628 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7274fd8a2115405c9b78c61214bff736. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:18:30.729338 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56870.7us
I20260709 12:18:30.733774 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:30.733966 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:30.734635 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48013.8us
I20260709 12:18:30.735430 29456 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:30.736515 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:30.675737 29065 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:30.740899 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89106.8us
I20260709 12:18:30.731149 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:30.739534 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:30.745945 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:30.717561 29360 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:30.742830 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.753995 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:30.756085 27485 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:30.759166 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:30.779122 27485 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:30.753276 29605 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:30.747921 29365 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:30.758659 29589 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:30.812197 29639 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:18:30.813608 29065 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:30.814568 29389 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:30.778728 29531 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.547s	user 0.006s	sys 0.015s
I20260709 12:18:30.816102 29057 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:30.818606 27581 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:30.818787 29458 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:30.821053 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1570) took 1983 ms. Trace:
I20260709 12:18:30.821743 29365 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:30.822424 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:30.822820 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:30.824156 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.824023 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:30.824685 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.824769 27733 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:30.826287 28332 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:18:30.785420 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.806445 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:30.792484 29409 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:30.792917 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 120) took 14710 ms (client timeout 15000 ms). Trace:
W20260709 12:18:30.795373 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 89826.6us
I20260709 12:18:30.754299 29584 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:18:30.803828 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:30.804997 27737 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:30.807785 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:30.811591 29535 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.748160 29066 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20260709 12:18:30.854928 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82953.4us
I20260709 12:18:30.811617 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:30.791441 27485 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:30.821417 27581 rpcz_store.cc:276] 0709 12:18:28.837632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.063794 (+1226162us) service_pool.cc:224] Handling call
0709 12:18:30.077602 (+ 13808us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.819042 (+741440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.820809 (+  1767us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:30.820967 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1703109888}
W20260709 12:18:30.845392 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82356.3us
W20260709 12:18:30.858551 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 52194.5us
I20260709 12:18:30.822317 29583 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:30.865774 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.862291 27732 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:30.781774 29490 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:30.894893 29604 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:30.827755 29605 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.763710 29562 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:30.906105 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:30.916038 27733 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:30.892633 29603 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:30.919354 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.906810 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:30.922513 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.923238 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:30.907171 29428 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260709 12:18:30.928072 27585 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:30.928364 29428 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:30.928858 29428 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:30.904266 29583 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:30.935181 27582 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:30.914158 29605 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:30.914484 29535 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.908866 29409 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:30.940105 29645 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:18:30.899279 29181 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:18:30.916208 29460 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:30.918834 29462 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.169s
W20260709 12:18:30.947739 28988 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.721s	user 0.016s	sys 0.008s
I20260709 12:18:30.919637 29603 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.920179 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:30.960945 29301 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:30.962293 29301 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:30.964097 29507 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:30.828680 27483 rpcz_store.cc:269] 0709 12:18:15.987146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.987290 (+   144us) service_pool.cc:224] Handling call
0709 12:18:30.693970 (+14706680us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:24.169677 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:24.298848 (+129171us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:24.300331 (+  1483us) write_op.cc:435] Acquired schema lock
0709 12:18:24.302360 (+  2029us) tablet.cc:662] Decoding operations
0709 12:18:24.373829 (+ 71469us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:24.373935 (+   106us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:24.373976 (+    41us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:18:24.374405 (+   429us) tablet.cc:701] Row locks acquired
0709 12:18:24.374429 (+    24us) write_op.cc:260] PREPARE: finished
0709 12:18:24.374607 (+   178us) write_op.cc:270] Start()
0709 12:18:24.374750 (+   143us) write_op.cc:276] Timestamp: P: 1783599504374554 usec, L: 0
0709 12:18:24.374778 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:18:24.377538 (+  2760us) log.cc:844] Serialized 1289 byte log entry
0709 12:18:25.880473 (+1502935us) op_driver.cc:464] REPLICATION: finished
0709 12:18:27.194850 (+1314377us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e9723e0ac 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e9dd7eef0
0709 12:18:27.339608 (+144758us) write_op.cc:301] APPLY: starting
0709 12:18:27.339697 (+    89us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:27.339841 (+   144us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:27.339859 (+    18us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:27.340291 (+   432us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:27.340461 (+   170us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:18:27.340507 (+    46us) write_op.cc:312] APPLY: finished
0709 12:18:27.340757 (+   250us) log.cc:844] Serialized 35 byte log entry
0709 12:18:27.446289 (+105532us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:27.450880 (+  4591us) write_op.cc:454] Released schema lock
0709 12:18:27.456297 (+  5417us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:27.493034 (+ 36737us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:27.494370 (+  1336us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1996130,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":1458847,"apply.run_cpu_time_us":20928,"apply.run_wall_time_us":136243,"num_ops":3,"prepare.queue_time_us":327974,"prepare.run_cpu_time_us":27573,"prepare.run_wall_time_us":754491,"raft.queue_time_us":689823,"raft.run_cpu_time_us":43952,"raft.run_wall_time_us":1455389,"replication_time_us":1505498,"spinlock_wait_cycles":9609088,"thread_start_us":2679224,"threads_started":5,"wal-append.queue_time_us":718311,"wal-append.run_cpu_time_us":54397,"wal-append.run_wall_time_us":2873898}]]}
W20260709 12:18:30.950340 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47724.8us
I20260709 12:18:30.948196 29535 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:30.952172 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144553us
I20260709 12:18:30.956331 29409 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:30.956725 29603 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:30.973803 29643 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:18:30.974121 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:30.979466 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:30.980819 29562 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:30.983060 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:30.978799 27582 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
W20260709 12:18:30.990545 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:30.979132 29299 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.605s	user 0.007s	sys 0.007s
I20260709 12:18:30.929821 29428 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:31.013172 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:31.014392 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.950032 29460 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:31.016674 27662 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:31.012840 27660 raft_consensus.cc:2376] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5. Re-sending same reply.
I20260709 12:18:30.980837 27661 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260709 12:18:31.018913 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:31.030047 28328 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 3: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:31.032927 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.034336 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:31.035539 29389 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:31.035892 29583 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.037027 29583 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.037820 29605 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.040664 29535 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.040751 29605 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.041338 29535 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:31.041795 29535 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:31.042201 29535 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.046926 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:31.048743 29604 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:31.049024 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:31.049445 29604 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.050294 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:31.050776 29651 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:31.051213 29651 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.051438 29650 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:31.051913 29651 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:31.052134 29650 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:18:31.052843 29478 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.630s	user 0.019s	sys 0.000s
I20260709 12:18:31.053140 29650 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.021791 29460 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.054445 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:31.055387 29462 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:31.058358 27657 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
W20260709 12:18:31.058782 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 352301us
I20260709 12:18:31.051786 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:31.053277 29478 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:31.062995 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.068006 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:31.070951 29589 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:31.071564 29589 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:31.072454 29589 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:31.074406 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.074841 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.078595 27655 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:31.084288 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.026970 29360 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:18:31.094110 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 43894.7us
I20260709 12:18:31.105852 27658 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:31.114575 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:31.115857 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.117007 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.085278 29387 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:31.098033 29428 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:31.155248 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:31.155936 27732 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:31.097764 29360 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:31.157255 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45188.3us
I20260709 12:18:31.114657 29181 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:31.159374 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.162586 27735 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:31.143751 29539 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:31.138489 29512 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:31.106314 29584 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
W20260709 12:18:31.173872 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97087.8us
I20260709 12:18:31.149483 29387 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.185911 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:31.190892 27736 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:18:31.154081 28996 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:31.194434 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.150653 29507 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.159669 29360 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:31.146965 28988 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:18:31.175869 29512 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.173291 29539 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:31.191056 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.192371 29387 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:31.232779 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.213929 29507 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.234879 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:31.235620 28332 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:31.235991 29507 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.198954 28996 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:31.249084 28332 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:31.255003 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:31.258755 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:31.264287 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106414us
I20260709 12:18:31.264508 29270 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.219985 29604 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.227567 29512 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.228055 29539 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:31.186978 29531 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 12:18:31.253870 29490 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:31.258734 28996 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:31.287690 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65549.5us
W20260709 12:18:31.292837 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 137978us
I20260709 12:18:31.295495 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:31.305168 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.314906 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.330776 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 75825us
I20260709 12:18:31.315546 29314 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:18:31.336522 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.321553 29583 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.285669 29270 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:31.379683 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84337.8us
W20260709 12:18:31.353034 29656 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:18:31.356113 29409 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:31.357952 29478 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.398006 29478 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.398551 29478 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.374881 29672 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.399781 29672 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.402124 29672 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.402669 29299 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:31.341706 29490 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:31.403218 29299 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.322888 29535 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.406332 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.420831 29562 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:31.421440 29562 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:31.429014 29409 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:31.429819 29409 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.435688 29207 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.436494 29207 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.441473 29389 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:31.348163 29583 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.445979 29583 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.446130 29065 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:31.446780 29065 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:31.447898 29065 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.450013 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.452049 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:31.454466 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:18:31.461162 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86859.4us
I20260709 12:18:31.384217 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:31.463598 27660 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:31.422533 29562 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:31.465663 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 180286us
I20260709 12:18:31.468369 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:31.458227 28994 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.473675 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.459523 29589 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:31.480860 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:31.481652 27661 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:31.484717 29299 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.467518 29207 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.492668 27656 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 2 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:31.477249 28994 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.476189 29589 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.491300 28328 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:31.498430 27729 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:31.499565 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:31.502923 28994 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.503823 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.496245 29314 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:31.506371 29535 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.506122 29314 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.508149 29478 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.490548 29360 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.500869 29589 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.592437 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.569905 29562 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:31.598377 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.599058 27736 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 2.
I20260709 12:18:31.601897 29531 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:31.603292 29360 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.503819 29387 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.617981 28333 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 12:18:31.618067 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.622488 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.623148 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.623451 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.625025 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:31.626196 29603 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.001s
I20260709 12:18:31.626684 29603 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.628939 29603 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:31.595239 29647 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:18:31.632179 29409 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:31.632262 29651 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.580s	user 0.012s	sys 0.000s
I20260709 12:18:31.501513 29365 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.097s
I20260709 12:18:31.632886 29531 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:31.633251 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:31.635912 29512 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.594475 29270 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.510655 29539 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.594321 29669 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.646963 29387 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.663538 29669 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.664912 29512 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.667816 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.668802 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:31.670965 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:31.644434 29651 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.672850 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.672654 28996 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:31.651093 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152428us
I20260709 12:18:31.674901 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.653326 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.680116 27584 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:31.681654 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:31.654232 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 436635us
W20260709 12:18:31.675797 29659 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.332s	user 0.004s	sys 0.003s
I20260709 12:18:31.651603 29531 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.676013 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:31.685940 27734 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:31.686420 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.679530 29678 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:31.677796 29057 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:31.660817 29460 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.606s	user 0.011s	sys 0.001s
I20260709 12:18:31.687343 27583 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:31.676483 29365 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:18:31.696658 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.700067 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.704090 29057 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.706627 29365 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:31.694012 29672 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:18:31.710987 29065 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.713382 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.714030 29531 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.714330 29605 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:31.714880 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.711011 29512 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:31.700052 29602 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:31.716187 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.707096 29365 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:31.715444 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.718647 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 43657.2us
I20260709 12:18:31.704761 29387 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:31.704814 29669 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.719511 29605 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.731197 29683 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.736625 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.724339 29602 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:31.740031 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.725636 29387 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.747778 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.747231 29387 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.749240 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:31.709255 29460 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.751281 27657 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:31.741731 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.744010 29057 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.744772 29539 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.754043 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:31.755476 29512 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:31.756681 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:31.757187 29669 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:31.757268 27588 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:31.758351 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:31.759024 28328 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:31.740520 29683 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.761188 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:31.761824 28332 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:18:31.762333 29683 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.740612 29602 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:31.764171 27587 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:31.765885 29651 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.766906 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:31.767699 28329 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:31.769256 29670 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:31.771463 27588 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:31.772799 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:31.757798 29669 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:31.730378 29531 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:31.769039 29365 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:31.773877 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.773746 29314 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:18:31.774863 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:31.783553 28329 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.784142 29589 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.786401 29589 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:31.787458 29605 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:31.771592 29676 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:31.778801 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:31.788115 29676 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:31.788595 28332 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:31.778085 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.789757 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.789204 29676 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.790485 29462 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:31.790882 29462 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:31.791036 27628 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:31.791409 27628 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:31.780824 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:31.791347 29462 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:31.792708 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:31.782776 29365 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.795414 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:31.796070 27587 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.795507 29588 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:31.797868 29588 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.795997 29385 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:31.798285 29588 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:31.798668 29385 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.799086 29385 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.799674 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:31.777791 27734 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:31.802480 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:31.791504 29651 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:31.779163 29562 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:18:31.810123 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.810925 27733 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:31.810649 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.809927 29531 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:18:31.811932 27655 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:31.813815 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:31.807631 29539 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.814831 27733 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:31.815865 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:31.816159 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:31.816577 27729 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:31.817829 27656 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:18:31.817899 29360 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:31.820124 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.822438 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:31.822815 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.823091 27736 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.802282 29460 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:31.827188 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.827296 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.822351 29673 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:18:31.828704 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.807818 29651 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.833362 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:31.833945 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:31.834182 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.810503 29589 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:31.838766 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.842458 28994 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:31.842945 28994 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:31.847450 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:31.848081 28994 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.830102 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:31.832971 29512 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:31.837553 29684 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:31.854135 27731 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:31.856298 27586 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:31.856370 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:31.857313 27655 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:31.857523 28329 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:31.857460 29387 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.858525 29669 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.837180 29683 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:31.859990 29683 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:31.860356 29683 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.839982 29678 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:31.851496 29589 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:31.861475 29678 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.837281 29651 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:31.864315 29678 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:31.839869 29409 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:18:31.867929 27585 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:31.870734 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:31.872613 28994 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:31.835915 29531 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:31.851547 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:31.871847 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61287.4us
I20260709 12:18:31.901259 28994 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:31.902384 28994 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.905619 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:31.906581 27585 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:31.839932 29490 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:31.907649 29490 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.909722 29490 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:31.920631 27736 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:31.924399 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:31.924584 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:31.916146 29507 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.930090 27663 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
W20260709 12:18:31.938606 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54704.9us
W20260709 12:18:31.939028 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 101499us
I20260709 12:18:31.939273 27585 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:31.862715 29589 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:31.947688 27660 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:31.950136 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:31.949290 27663 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:31.954144 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:31.955118 27588 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:31.949906 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.940404 29490 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:31.967521 29584 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:18:31.968067 29688 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:18:31.968308 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:31.970252 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:31.976349 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:31.979142 27658 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:31.979961 27586 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:31.934214 29507 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:31.989668 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:32.006188 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:31.967159 29539 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:32.008957 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:31.997931 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:31.948664 29065 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.062s
I20260709 12:18:31.977723 29584 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:32.018353 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:32.006430 28996 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:32.006186 29387 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.025561 29389 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:18:32.026930 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102639us
I20260709 12:18:32.035959 27663 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:18:32.019076 29409 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:32.042011 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:32.024847 29584 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:32.024384 29065 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260709 12:18:32.050989 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.047091 27663 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:32.054919 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.045974 27660 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:32.056207 27662 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:32.026396 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:32.056228 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:32.056679 27659 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:18:32.067859 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.044181 29409 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:32.045318 29299 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:32.062031 29683 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.073742 27658 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:32.074038 29490 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:32.098162 29602 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.099073 29683 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.077961 29409 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:32.143992 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 173816us
W20260709 12:18:32.145328 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.146056 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1562) took 1100 ms. Trace:
I20260709 12:18:32.146468 27657 rpcz_store.cc:276] 0709 12:18:31.045426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.059162 (+ 13736us) service_pool.cc:224] Handling call
0709 12:18:31.059485 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:31.753172 (+693687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.145773 (+392601us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:32.145970 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1590501248}
I20260709 12:18:32.149183 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:32.150028 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:32.150054 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:32.107406 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3610437560000719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19920000000000002s for a response
I20260709 12:18:32.151773 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:32.152328 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:32.153316 29301 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:32.154296 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:32.156189 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:32.156371 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:32.156931 27659 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:32.153175 29683 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:18:32.161516 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 169616us
I20260709 12:18:32.164796 27587 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
I20260709 12:18:32.165141 27587 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:18:32.167346 27587 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:32.169394 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:32.170121 29699 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:18:32.171082 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.171553 29507 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:32.172551 29507 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:32.144377 29692 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:32.155982 29301 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:32.170142 27658 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:32.173518 29692 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.165892 29666 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:32.163069 29531 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:32.163496 29387 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:32.163300 29670 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:32.172987 29507 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:32.174531 29385 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.181483 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.187613 29602 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:32.158370 28996 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 12:18:32.191179 28333 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:32.182636 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:32.106209 29490 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.145083 29389 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:18:32.189796 29670 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:32.180827 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 207523us
I20260709 12:18:32.200554 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.202251 27585 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:18:32.180560 29301 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:32.189103 29683 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.201421 29389 consensus_queue.cc:1243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:32.166862 29065 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:18:32.207840 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.178844 29462 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.223966 27585 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:32.225580 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:32.208634 29490 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.199481 29387 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:32.230365 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47797.7us
I20260709 12:18:32.198423 29531 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:32.300835 27655 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:18:32.338829 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:32.346088 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 157635us
I20260709 12:18:32.352982 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.232798 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.278918 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.282580 27733 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:32.230377 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:32.211082 29697 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:18:32.354182 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.362802 27663 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:32.365142 27553 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:32.365648 27553 leader_election.cc:403] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:32.369197 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:32.213860 29385 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.226639 29389 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.226244 29314 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:32.206799 29701 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:32.298691 29703 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.204s	user 0.002s	sys 0.000s
W20260709 12:18:32.318256 29698 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.222s	user 0.003s	sys 0.001s
I20260709 12:18:32.384654 29360 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.297235 29605 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:32.323134 29676 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:32.231990 29702 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.354785 29568 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:32.223021 29683 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.413617 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:32.403982 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:32.382575 29490 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:32.413254 28994 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20260709 12:18:32.416332 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.420132 28332 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:18:32.400810 29539 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:32.400568 29605 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:32.398193 29704 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.428870 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:32.430159 27733 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:32.432346 28994 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:32.433167 28994 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:32.415052 29705 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.419353 29568 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:32.429525 29588 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.631s	user 0.012s	sys 0.000s
I20260709 12:18:32.405332 29360 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.399226 29389 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:32.425128 29692 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.451004 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:32.452867 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48925.6us
I20260709 12:18:32.438385 29673 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:32.444875 29512 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:32.400665 29670 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.418331 29603 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:18:32.410593 29701 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.455122 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.473076 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.475675 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:32.475850 27589 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:32.470937 27734 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:32.473845 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.474576 29539 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:18:32.479305 29701 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:32.418447 29507 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.487226 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.488042 27588 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:32.488256 29683 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:32.492913 27588 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:32.493332 27588 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:32.494235 27588 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.495908 27588 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:32.496490 29678 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:32.434479 29695 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:18:32.415525 29702 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.500123 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.424480 29531 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.535144 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.529948 27588 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:18:32.537796 27588 pending_rounds.cc:85] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:18:32.538162 27588 pending_rounds.cc:107] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:18:32.538483 27588 raft_consensus.cc:2889] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:32.532369 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:32.508070 29462 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:32.510741 29299 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:32.439692 29704 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.506943 27661 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:32.561372 27738 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 4. Ops: []
I20260709 12:18:32.470487 29490 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260709 12:18:32.567603 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.589499 27586 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:32.459561 29588 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.582084 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.591943 27732 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:32.545217 29651 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.678s	user 0.011s	sys 0.000s
I20260709 12:18:32.450090 29705 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.591806 27655 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:32.465528 29387 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.597867 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:32.525578 29562 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260709 12:18:32.552313 29701 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.604668 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.551890 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.469561 29605 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.611985 27736 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:18:32.529389 29365 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20260709 12:18:32.477324 27585 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:32.476035 29673 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.542106 29692 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.069s
I20260709 12:18:32.613994 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:32.617115 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.675194 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.624202 27589 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:32.687952 27736 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:32.611795 27738 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:32.674973 29588 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.675097 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:32.615670 29301 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.615738 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:32.698571 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:32.677556 29584 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.612s	user 0.009s	sys 0.001s
I20260709 12:18:32.677757 29716 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:32.678515 29715 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:32.597136 29651 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.679752 29702 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:32.578610 29710 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:32.686364 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.611289 27731 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:18:32.693982 29181 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:32.712455 28333 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:32.714769 27732 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:32.762343 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.606375 29713 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:32.747776 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.749810 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.772640 27589 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:32.750852 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.699657 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.763310 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.695570 29365 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:32.764668 29705 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:32.694445 29057 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.774405 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.822544 27586 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:32.822610 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.698575 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.601841 29568 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:32.618075 29701 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.787237 27729 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:32.787298 27736 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:32.765712 29710 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.818328 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.821758 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.781733 27735 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:32.763263 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:32.706760 29683 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.094s
I20260709 12:18:32.844714 27589 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:32.834033 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:32.845556 27732 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:32.845726 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.846429 27586 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:32.850741 27731 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:18:32.853029 29301 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:32.847582 27580 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:32.824761 29365 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.859884 29365 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.862680 29365 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:32.865899 29651 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:32.867233 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:32.868952 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.869434 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.870205 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:32.870913 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:32.870443 27584 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:32.711654 29584 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.878259 27733 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
W20260709 12:18:32.883505 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 351209us
W20260709 12:18:32.887673 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194276us
I20260709 12:18:32.892760 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:32.900487 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.872875 29562 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:32.915674 27586 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 1
I20260709 12:18:32.916024 27586 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260709 12:18:32.916270 27586 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:32.911494 27589 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:18:32.899545 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:32.837944 29057 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.750381 29716 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.908108 29728 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:32.781001 29715 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:32.903854 29676 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:32.787371 29713 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:32.945057 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52359.9us
I20260709 12:18:32.917383 29562 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:33.095252 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.098839 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.099781 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:32.941571 29673 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.099011 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.113489 27663 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:32.966025 29365 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:32.965376 29728 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:33.129101 29409 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.974s	user 0.011s	sys 0.000s
I20260709 12:18:32.949184 29181 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:33.096802 29562 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:33.186408 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.095485 29676 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.196199 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:33.198304 27589 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:33.201218 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.201874 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:33.175248 29692 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.218520 29181 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.218969 29057 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.190177 29365 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:18:33.220574 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:33.229588 29726 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:18:33.248104 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 190) took 1439 ms. Trace:
I20260709 12:18:33.249634 29716 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:33.251780 27485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 234) took 11511 ms (client timeout 15000 ms). Trace:
I20260709 12:18:33.255383 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:33.256238 27733 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:33.257442 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.258174 27730 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:33.258610 28328 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:18:33.259752 27731 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:33.260125 29701 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.262209 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:33.262521 29713 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.263048 27588 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:33.264330 29715 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.265228 29710 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.268512 28994 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:33.268934 28994 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:33.269801 28994 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.272734 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:33.273733 28331 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:33.274499 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:33.276124 29710 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:18:33.276674 29710 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:33.277693 29710 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.247611 29728 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.159341 29409 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:33.308126 27737 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.617s	user 0.006s	sys 0.005s
I20260709 12:18:33.271201 28994 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:33.309165 27737 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:33.315346 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:33.248677 27661 rpcz_store.cc:276] 0709 12:18:31.809198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.810006 (+   808us) service_pool.cc:224] Handling call
0709 12:18:33.247890 (+1437884us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:33.248034 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":487812864}
I20260709 12:18:33.326836 27660 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
W20260709 12:18:33.252231 27485 rpcz_store.cc:269] 0709 12:18:21.740263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.744900 (+  4637us) service_pool.cc:224] Handling call
0709 12:18:33.251708 (+11506808us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:31.399614 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:31.415204 (+ 15590us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:31.415278 (+    74us) write_op.cc:435] Acquired schema lock
0709 12:18:31.415306 (+    28us) tablet.cc:662] Decoding operations
0709 12:18:31.417160 (+  1854us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:31.417297 (+   137us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:31.417341 (+    44us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:18:31.611384 (+194043us) tablet.cc:701] Row locks acquired
0709 12:18:31.611457 (+    73us) write_op.cc:260] PREPARE: finished
0709 12:18:31.611646 (+   189us) write_op.cc:270] Start()
0709 12:18:31.611855 (+   209us) write_op.cc:276] Timestamp: P: 1783599511611579 usec, L: 0
0709 12:18:31.611886 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:18:31.612708 (+   822us) log.cc:844] Serialized 5529 byte log entry
0709 12:18:31.857862 (+245154us) op_driver.cc:464] REPLICATION: finished
0709 12:18:32.011746 (+153884us) write_op.cc:301] APPLY: starting
0709 12:18:32.014625 (+  2879us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:32.026158 (+ 11533us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:32.055901 (+ 29743us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:32.169019 (+113118us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:32.169395 (+   376us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 12:18:32.169451 (+    56us) write_op.cc:312] APPLY: finished
0709 12:18:32.169919 (+   468us) log.cc:844] Serialized 101 byte log entry
0709 12:18:32.170138 (+   219us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:32.170307 (+   169us) write_op.cc:454] Released schema lock
0709 12:18:32.170386 (+    79us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:32.170818 (+   432us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:32.170855 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2341241,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":93805,"apply.run_cpu_time_us":6675,"apply.run_wall_time_us":199823,"num_ops":14,"prepare.queue_time_us":304560,"prepare.run_cpu_time_us":45389,"prepare.run_wall_time_us":377901,"raft.queue_time_us":73028,"raft.run_cpu_time_us":5883,"raft.run_wall_time_us":102952,"replication_time_us":245792,"thread_start_us":348353,"threads_started":4,"wal-append.queue_time_us":139464}]]}
I20260709 12:18:33.334049 27658 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
W20260709 12:18:33.315620 29726 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:18:33.308893 29207 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.217386 29678 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:33.354825 29678 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.357460 29678 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:33.351846 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.349368 29713 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:33.334475 29588 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:18:33.352895 29716 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:33.364872 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:33.217752 29692 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.385521 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.388532 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.389389 27660 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:33.351096 29718 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:33.393395 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:33.393909 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.421886 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:33.423622 27660 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:33.388563 27661 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 1.
I20260709 12:18:33.424657 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.426050 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:33.432878 27659 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:33.434285 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1367) took 1213 ms. Trace:
I20260709 12:18:33.434772 27659 rpcz_store.cc:276] 0709 12:18:32.220841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.278524 (+ 57683us) service_pool.cc:224] Handling call
0709 12:18:33.434040 (+1155516us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:33.434214 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2479626112}
I20260709 12:18:33.441215 29409 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.442456 29704 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.443719 29676 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.443969 29733 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:33.394259 29716 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:33.445719 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:33.446302 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.447506 29716 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.447531 29744 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:33.449023 27588 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:33.449867 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:33.450269 29736 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:33.450780 28328 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:33.450829 29736 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.453147 29736 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.455430 29562 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.456029 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 192) took 1019 ms. Trace:
I20260709 12:18:33.456393 27729 rpcz_store.cc:276] 0709 12:18:32.436792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.437049 (+   257us) service_pool.cc:224] Handling call
0709 12:18:33.455828 (+1018779us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:33.455960 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":728044416}
I20260709 12:18:33.459657 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:33.358404 27483 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 2 to 4, term changed from 2 to 4, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1), VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3), VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.386931 29729 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:33.449887 29692 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.515082 28994 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.515818 28994 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.465921 29314 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:33.501744 27483 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 3 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.520478 27483 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 3, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.522610 27483 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 4 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.524749 27483 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 4, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.468981 29736 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:33.468454 29744 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:33.387261 29713 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.599408 29734 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:33.600088 29734 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.531240 27483 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.602316 29734 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.601688 27483 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.596328 29389 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.603497 27483 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.466717 29718 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:33.605566 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1403) took 1027 ms. Trace:
I20260709 12:18:33.605721 29718 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.381791 29207 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.603284 27581 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:18:33.609179 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1582) took 1085 ms. Trace:
I20260709 12:18:33.609690 27581 rpcz_store.cc:276] 0709 12:18:32.523602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.524578 (+   976us) service_pool.cc:224] Handling call
0709 12:18:33.602738 (+1078160us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:33.602955 (+   217us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:33.608806 (+  5851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.609104 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38904576}
I20260709 12:18:33.606150 27734 rpcz_store.cc:276] 0709 12:18:32.578479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.592016 (+ 13537us) service_pool.cc:224] Handling call
0709 12:18:32.606851 (+ 14835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:33.113998 (+507147us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:33.114030 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:33.114075 (+    45us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:33.114112 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.605286 (+491174us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54001d47d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:33.605332 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:33.605473 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1165765888}
W20260709 12:18:33.608693 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184125us
I20260709 12:18:33.530396 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:33.615018 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84736.9us
W20260709 12:18:33.615815 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.612943 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:33.617197 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.618350 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.675101 27737 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:33.605249 27483 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 4, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:18:33.520283 29683 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.000s	sys 0.015s
I20260709 12:18:33.599438 29736 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:33.602958 29729 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.607231 29718 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:33.503885 29588 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.681017 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:33.681566 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68702.5us
I20260709 12:18:33.694139 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:33.694407 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:33.680044 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61230.9us
I20260709 12:18:33.680584 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.697320 27729 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:33.697649 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.691082 28994 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:33.698527 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 374245us
I20260709 12:18:33.595921 29562 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:33.619963 29670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.006s	sys 0.012s
I20260709 12:18:33.507863 29740 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.608153 29744 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:33.599452 29314 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:33.621485 29723 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.701359 27630 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:33.721624 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.723627 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.724313 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:33.724740 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:33.725656 29666 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:33.726208 29666 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:33.726707 29666 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:33.727487 29713 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.729750 29207 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.732616 29729 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.696566 29715 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:33.689903 27483 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 2 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.674376 29678 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:33.707247 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:33.735633 29678 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:33.736090 29678 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:33.738708 29365 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:33.739639 29365 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:33.741402 29365 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:33.742240 29365 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:33.745894 29562 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:33.715900 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1579) took 1307 ms. Trace:
I20260709 12:18:33.753470 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1362) took 1572 ms. Trace:
I20260709 12:18:33.753882 27658 rpcz_store.cc:276] 0709 12:18:32.181022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.181217 (+   195us) service_pool.cc:224] Handling call
0709 12:18:33.737673 (+1556456us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:33.753410 (+ 15737us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2631383808}
I20260709 12:18:33.705371 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:33.756520 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52344.4us
I20260709 12:18:33.758239 27483 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 5, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.760403 27483 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:33.716470 29314 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.747818 29360 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.695441 29736 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:33.705281 29460 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.700865 29588 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:33.683831 29683 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.001s	sys 0.018s
I20260709 12:18:33.731949 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:33.823534 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91707.7us
I20260709 12:18:33.757522 29365 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:33.795043 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.021s	sys 0.026s
I20260709 12:18:33.758500 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:33.700698 29181 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:33.773625 27731 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
W20260709 12:18:33.773356 29568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.018s	sys 0.038s
I20260709 12:18:33.753580 27664 rpcz_store.cc:276] 0709 12:18:32.408460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.476406 (+ 67946us) service_pool.cc:224] Handling call
0709 12:18:32.479737 (+  3331us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:32.539042 (+ 59305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:32.539074 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:32.591862 (+ 52788us) log.cc:844] Serialized 78 byte log entry
0709 12:18:32.884688 (+292826us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:33.132882 (+248194us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.277622 (+144740us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b18003ac9c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:18:33.712744 (+435122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.712824 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:33.712869 (+    45us) raft_consensus.cc:1685] finished
0709 12:18:33.715670 (+  2801us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:33.715842 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16209,"prepare.queue_time_us":259401,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":7143,"raft.queue_time_us":230049,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":141,"spinlock_wait_cycles":16052608,"thread_start_us":977012,"threads_started":4,"wal-append.queue_time_us":417819}
I20260709 12:18:33.763549 29744 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:33.756970 29715 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.764894 29584 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:33.762337 27483 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 5, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 5 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.756357 29740 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.821297 29460 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:33.827450 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69005.1us
W20260709 12:18:33.830485 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.021s	sys 0.028s
W20260709 12:18:33.834887 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.770488 29723 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.845077 28333 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:33.828114 29365 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:33.831576 29568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.019s	sys 0.040s
I20260709 12:18:33.827440 29742 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.774870 29562 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.829823 29683 consensus_queue.cc:1243] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:33.831564 29677 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:33.857697 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.004s	sys 0.028s
W20260709 12:18:33.870770 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.005s	sys 0.029s
I20260709 12:18:33.841065 27483 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:33.872843 27483 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 3, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.838433 29729 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:33.837548 29750 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:33.876247 27483 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.882360 27483 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 4, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.886296 27483 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 3, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:33.930752 27737 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:33.843173 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:33.901363 29460 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.901796 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.938088 29666 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:33.911351 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.939067 29676 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:33.939818 27734 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:33.940274 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:33.912364 29745 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:18:33.940923 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
W20260709 12:18:33.914212 29706 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.379s	user 0.008s	sys 0.017s
I20260709 12:18:33.921320 27583 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:33.843016 29678 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.846096 29715 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.933481 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.875207 29677 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:33.936671 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:33.951668 27732 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:33.941757 27583 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:33.953293 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:33.954082 28994 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:33.936884 27483 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:33.879406 29389 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 12:18:33.974274 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.974615 27483 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.976739 27483 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:18:33.978176 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.891228 29729 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.981377 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:33.978515 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 271324us
I20260709 12:18:33.882545 29754 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260709 12:18:33.971367 27583 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:33.982354 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:33.875569 29742 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:33.982303 27483 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:33.940900 29666 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:33.967742 29677 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:33.989694 27734 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:33.990784 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:33.991304 27583 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:33.991042 27483 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 3 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:33.993180 27483 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 5, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 5 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:33.994149 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:33.982086 27732 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:34.004652 27483 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 4, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:34.015693 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.016517 27732 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:34.018280 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:34.019054 27734 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:33.988819 29360 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:33.988711 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:34.030486 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.015535 29743 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:34.040314 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:33.996471 29758 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.001696 29584 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:33.989039 29750 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.049630 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.052110 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1598) took 1183 ms. Trace:
I20260709 12:18:34.030915 29684 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.052523 27586 rpcz_store.cc:276] 0709 12:18:32.869170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.900427 (+ 31257us) service_pool.cc:224] Handling call
0709 12:18:32.915424 (+ 14997us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:32.946014 (+ 30590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:32.946075 (+    61us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:32.966497 (+ 20422us) log.cc:844] Serialized 52 byte log entry
0709 12:18:33.467150 (+500653us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:33.507003 (+ 39853us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.924076 (+417073us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.924302 (+   226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.924404 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.051874 (+127470us) raft_consensus.cc:1685] finished
0709 12:18:34.051895 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.052046 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41213,"prepare.queue_time_us":37672,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":89,"raft.queue_time_us":255875,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":86,"spinlock_wait_cycles":17237760,"thread_start_us":741177,"threads_started":2,"wal-append.queue_time_us":720730}
I20260709 12:18:34.046022 29666 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.050935 29584 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.058535 29734 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:34.075567 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1370) took 1101 ms. Trace:
I20260709 12:18:34.075948 27655 rpcz_store.cc:276] 0709 12:18:32.973969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.975018 (+  1049us) service_pool.cc:224] Handling call
0709 12:18:33.102598 (+127580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:33.461891 (+359293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:33.461938 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:33.609218 (+147280us) log.cc:844] Serialized 26 byte log entry
0709 12:18:33.979045 (+369827us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:33.979105 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.979194 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.075380 (+ 96186us) raft_consensus.cc:1685] finished
0709 12:18:34.075397 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.075518 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40055,"raft.queue_time_us":251326,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":599234,"threads_started":2,"wal-append.queue_time_us":390615}
I20260709 12:18:34.080447 29729 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.081873 29678 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.062110 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1578) took 1652 ms. Trace:
I20260709 12:18:34.082845 27656 rpcz_store.cc:276] 0709 12:18:32.408164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.474945 (+ 66781us) service_pool.cc:224] Handling call
0709 12:18:32.492229 (+ 17284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:32.492616 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:32.492650 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:33.279070 (+786420us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:33.279602 (+   532us) log.cc:844] Serialized 26 byte log entry
0709 12:18:33.855475 (+575873us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:33.950755 (+ 95280us) log.cc:844] Serialized 14 byte log entry
0709 12:18:34.003699 (+ 52944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.013829 (+ 10130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.017078 (+  3249us) raft_consensus.cc:1685] finished
0709 12:18:34.047506 (+ 30428us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.060180 (+ 12674us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57257,"prepare.queue_time_us":46449,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":22327,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":19163,"spinlock_wait_cycles":18973056,"thread_start_us":566967,"threads_started":1,"wal-append.queue_time_us":614940}
I20260709 12:18:34.062630 29710 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:34.086341 29710 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.086907 29710 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:34.088917 29748 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.248s	user 0.000s	sys 0.002s
I20260709 12:18:34.091710 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1415) took 1473 ms. Trace:
I20260709 12:18:34.070128 29666 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.092129 27738 rpcz_store.cc:276] 0709 12:18:32.617842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.832520 (+214678us) service_pool.cc:224] Handling call
0709 12:18:32.843284 (+ 10764us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:32.846196 (+  2912us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:18:33.132758 (+286562us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.520682 (+387924us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:33.701720 (+181038us) log.cc:844] Serialized 26 byte log entry
0709 12:18:33.701948 (+   228us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:33.701997 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.702114 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.088328 (+386214us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:34.091397 (+  3069us) raft_consensus.cc:1685] finished
0709 12:18:34.091493 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.091646 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":486691,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":12806,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":83391,"spinlock_wait_cycles":8540416,"thread_start_us":572856,"threads_started":2,"wal-append.queue_time_us":636890}
I20260709 12:18:34.094236 29666 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.071750 29588 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.095351 29715 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.096164 29742 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.095666 29744 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:34.096845 29747 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
W20260709 12:18:34.097040 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.097482 29740 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.098650 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.099556 28331 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:34.100981 29732 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:34.101078 29744 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.101457 29732 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:34.073139 29704 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.102183 29736 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.102411 29732 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.102566 29704 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.104797 29763 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.105964 29605 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:34.105372 29763 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.059391 29460 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:18:34.052961 29758 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.108201 29512 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:34.108568 29512 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:34.108940 29763 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.109683 29763 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.074918 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1371) took 1100 ms. Trace:
I20260709 12:18:34.068022 29181 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:34.111517 29181 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.110433 27657 rpcz_store.cc:276] 0709 12:18:32.974275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.975101 (+   826us) service_pool.cc:224] Handling call
0709 12:18:33.037865 (+ 62764us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:33.204114 (+166249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:33.204255 (+   141us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:33.317629 (+113374us) log.cc:844] Serialized 52 byte log entry
0709 12:18:33.871506 (+553877us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:33.871565 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.871667 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.074610 (+202943us) raft_consensus.cc:1685] finished
0709 12:18:34.074641 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.074835 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":255663,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":119,"thread_start_us":630811,"threads_started":2,"wal-append.queue_time_us":643700}
I20260709 12:18:34.111928 29181 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:34.070878 29584 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:34.069506 29746 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.115228 29365 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.115748 29746 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.117494 29365 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.094612 29562 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.118078 29365 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.118544 29365 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:34.119259 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81239.3us
I20260709 12:18:34.120299 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:34.109951 29758 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:34.121750 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67703.4us
I20260709 12:18:34.106119 29723 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:34.145562 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.149026 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:34.070281 29695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.517s	user 0.044s	sys 0.034s
W20260709 12:18:34.155051 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.109553 29512 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.161887 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:34.162814 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:34.156899 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.146s (SENT) [suppressed 16 similar messages]
I20260709 12:18:34.129179 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:34.119026 29746 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.151738 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.176108 27657 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:34.176462 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:34.161847 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.162741 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.163419 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.193451 27664 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:34.165275 29728 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:34.165302 28331 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:18:34.147825 29460 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:34.154371 29695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.611s	user 0.044s	sys 0.034s
I20260709 12:18:34.133819 29678 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.168861 29677 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.105378 29704 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.151458 29409 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.173523 29666 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.187397 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.172248 29729 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.144845 29763 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.191684 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.216423 28333 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:34.191732 27663 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:34.151609 29712 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.057s
I20260709 12:18:34.151245 29760 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:34.059409 29750 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:34.202343 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 205600us
I20260709 12:18:34.153374 29732 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:34.185169 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.066144 29684 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.189538 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.213132 28994 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.235635 28994 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.172045 29743 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.214872 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.237097 27657 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:34.192729 27662 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:34.192592 29389 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.220126 29710 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.192198 29734 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.222209 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:34.226621 27552 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:34.244949 27552 leader_election.cc:403] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:34.205713 29753 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:18:34.252511 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 261768us
I20260709 12:18:34.233783 29512 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:34.210829 29704 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.245097 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:34.256191 27663 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:34.256552 27655 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:18:34.256911 27655 pending_rounds.cc:85] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:18:34.254073 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.257143 27655 pending_rounds.cc:107] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:18:34.253774 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.257522 27655 raft_consensus.cc:2889] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:34.257757 27664 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:34.225693 29760 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.258673 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:34.256071 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.259680 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.261682 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.210970 29678 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.207549 29460 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.274215 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.265035 27662 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:34.264827 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.246343 29314 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.225590 29718 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:18:34.213884 29409 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.279201 28994 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.277029 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.223304 29729 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.283058 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.281024 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:34.284622 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.222868 29666 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.290728 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.294973 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.298062 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.298801 27657 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:34.266494 29692 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:18:34.307760 28994 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:34.302858 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.304381 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.304996 27588 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:18:34.301630 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:34.308665 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 151794us
I20260709 12:18:34.309608 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:34.310292 27588 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:18:34.268855 29360 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:34.267648 29760 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:34.260478 29734 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.325091 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.308270 28994 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.306804 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:34.306386 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.250401 29743 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.262368 29764 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:34.280947 29753 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:34.250664 29389 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.302363 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:34.321645 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.349344 27582 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:34.349401 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47145.7us
I20260709 12:18:34.283375 29490 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:34.326390 28994 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:34.283949 29744 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:34.272871 29762 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:34.278200 29704 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.368448 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.308665 29562 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:34.323740 29742 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.308243 29773 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.377923 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.287943 29314 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.347950 29753 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.320266 29736 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.330020 29764 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.348621 29695 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.378074 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:34.330579 29673 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:18:34.359535 29588 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:18:34.388067 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.384848 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:34.410259 27735 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:34.387147 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:34.370342 29365 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.358455 29490 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:34.433869 27586 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:34.443432 28994 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.379033 29562 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.353996 29605 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:18:34.460008 28994 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:34.385656 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 131951us
I20260709 12:18:34.389397 29753 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:34.330240 29758 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
W20260709 12:18:34.442041 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54980.4us
W20260709 12:18:34.436415 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85980.5us
I20260709 12:18:34.437870 29736 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.438905 29715 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.449090 29409 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:34.451915 29765 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:18:34.452548 29584 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:34.521389 29772 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:18:34.454100 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.523227 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.523661 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.524063 27732 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:34.524346 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.525372 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.525465 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.525978 27732 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:34.523660 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:34.454442 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:34.454721 29728 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.456419 27583 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:18:34.528412 27583 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
I20260709 12:18:34.528723 27583 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:18:34.528997 27583 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:34.457317 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:34.313210 29740 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:34.382654 29744 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:34.437215 27588 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:34.469360 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:34.396790 29762 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:34.471390 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:34.384012 29531 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.118s	user 0.009s	sys 0.050s
I20260709 12:18:34.476217 27734 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:34.479316 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.480224 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:34.485386 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:34.448313 29365 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:34.398078 29764 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:34.489813 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:34.456897 29490 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:34.492713 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.493854 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:34.333961 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.461936 29562 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:34.515862 29710 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:34.516197 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:34.516930 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.468514 29713 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:34.519682 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:34.520309 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:34.530113 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72863.9us
I20260709 12:18:34.530457 27582 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:34.541509 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.541672 29579 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:34.541843 29727 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:34.542217 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.451239 29181 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.385097 29775 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.151s
I20260709 12:18:34.390471 29742 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.390798 29773 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.460316 29360 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.447439 29695 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.792794 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:34.651464 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 368725us
I20260709 12:18:34.825460 27735 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:34.825734 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:34.614333 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 291863us
I20260709 12:18:34.826280 27588 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:34.546361 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1586) took 1966 ms. Trace:
I20260709 12:18:34.582499 27586 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
W20260709 12:18:34.604482 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 124360us
I20260709 12:18:34.842147 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:34.842617 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 348839us
I20260709 12:18:34.843042 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:34.545671 27733 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
W20260709 12:18:34.766590 27663 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.502s	user 0.005s	sys 0.001s
I20260709 12:18:34.824596 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.830029 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:34.923398 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:34.922960 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:34.827201 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 337439us
I20260709 12:18:34.925700 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:34.861903 27662 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:34.923472 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59604.1us
I20260709 12:18:34.844102 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:34.667479 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:34.928635 27552 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:34.928980 27664 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:34.861074 27657 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:18:34.546669 29720 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 12:18:34.930380 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:34.537237 29531 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.276s	user 0.016s	sys 0.080s
W20260709 12:18:34.734654 29766 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.360s	user 0.000s	sys 0.001s
I20260709 12:18:34.930315 27734 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:18:34.898641 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.807181 29584 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:34.933988 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:34.823058 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.650513 29460 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.802347 29719 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:34.627333 29677 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.651083 29539 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.821308 29743 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.545920 29728 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:34.790717 29732 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.550s	user 0.010s	sys 0.001s
I20260709 12:18:34.526829 29763 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.790556 29389 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.627202 29409 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:34.953926 27729 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:34.954339 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:35.269677 27734 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:34.955508 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.270776 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:34.839205 29704 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.934252 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:34.963996 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.272769 27586 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:34.859295 29718 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:34.631510 29684 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.656613 29314 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:34.537074 29762 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:34.964645 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 462818us
I20260709 12:18:34.964862 27735 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:34.652923 29734 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.653493 29678 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.823127 29729 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:34.894759 29512 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.637s	user 0.012s	sys 0.000s
I20260709 12:18:34.888000 29588 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:34.653285 29750 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:34.536736 29740 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:35.377933 29740 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:34.704072 29774 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:34.784826 29788 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.379761 29788 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.840286 27583 rpcz_store.cc:276] 0709 12:18:32.580293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.580473 (+   180us) service_pool.cc:224] Handling call
0709 12:18:32.580905 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:33.921212 (+1340307us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54000d3ca0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:18:34.546106 (+624894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.546305 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1046708096}
I20260709 12:18:34.840977 29579 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.923070 29746 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:35.381450 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:34.972600 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.859375 29723 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:34.500032 29736 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.974144 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:34.973707 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67893.1us
I20260709 12:18:34.505445 29715 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.771907 29673 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:34.542795 29767 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:18:34.987481 27585 raft_consensus.cc:606] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:34.988189 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:34.990312 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 164673us
I20260709 12:18:34.541673 27661 raft_consensus.cc:1217] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 12:18:34.993268 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:34.997457 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 154475us
W20260709 12:18:34.996800 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67351.3us
I20260709 12:18:34.900358 29713 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.022792 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:35.046623 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:35.085042 29695 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.018579 29539 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.044461 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:35.082487 27580 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:35.059549 29769 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:34.657110 29744 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:35.058874 29786 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.083734 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:35.217206 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:34.644615 29760 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:34.932682 29365 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.987082 29719 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:35.084734 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:35.046741 29770 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.278060 27582 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:18:35.082573 29753 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.587s	user 0.007s	sys 0.002s
I20260709 12:18:35.286249 29790 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:35.085501 29743 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:35.272172 29692 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.950s	user 0.023s	sys 0.032s
I20260709 12:18:35.290680 27588 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:35.322214 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:35.268352 29460 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:35.341799 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1376) took 1155 ms. Trace:
I20260709 12:18:35.344128 27656 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:18:35.376915 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.379722 29774 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:34.971943 27658 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:34.828162 29666 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:35.320986 29742 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:35.551856 27586 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:35.552259 27737 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:35.552934 27660 rpcz_store.cc:276] 0709 12:18:34.186814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.187001 (+   187us) service_pool.cc:224] Handling call
0709 12:18:35.341570 (+1154569us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b5400211454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:35.341726 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1664804096}
I20260709 12:18:35.554198 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1610) took 1256 ms. Trace:
I20260709 12:18:35.554622 27582 rpcz_store.cc:276] 0709 12:18:34.298150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.277339 (+979189us) service_pool.cc:224] Handling call
0709 12:18:35.277696 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.553943 (+276247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:35.554144 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:35.555095 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1384) took 1069 ms. Trace:
I20260709 12:18:35.555518 27658 rpcz_store.cc:276] 0709 12:18:34.485996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.825615 (+339619us) service_pool.cc:224] Handling call
0709 12:18:35.555037 (+729422us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:35.556968 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:35.558640 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:35.555616 27583 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:35.570124 27659 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:35.575902 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.578074 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:35.550977 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:35.566540 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:35.345001 29704 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.326869 29762 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:35.463505 29794 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
W20260709 12:18:35.580149 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 258098us
I20260709 12:18:35.589129 27589 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:35.373461 29775 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.590840 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:35.566474 29579 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:35.576217 29743 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.566349 29692 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:35.578512 29734 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.567144 29460 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:35.462399 29782 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:35.592509 29683 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:35.576422 29769 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:35.566912 29753 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.576772 29673 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:35.551772 29746 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:35.600644 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1597) took 2731 ms. Trace:
I20260709 12:18:35.602334 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1615) took 1060 ms. Trace:
I20260709 12:18:35.685523 27588 rpcz_store.cc:276] 0709 12:18:34.542152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.923049 (+380897us) service_pool.cc:224] Handling call
0709 12:18:35.602176 (+679127us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:35.602270 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":839619840}
I20260709 12:18:35.598488 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.612865 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.381628 29710 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:35.613178 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52483.8us
I20260709 12:18:35.534301 29729 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.534240 29778 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.589414 27661 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:18:35.623639 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:35.667997 29666 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:35.598664 29793 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.222s	user 0.003s	sys 0.006s
I20260709 12:18:35.659897 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:35.663082 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44295.4us
I20260709 12:18:35.589154 29750 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:35.670899 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:35.789336 27734 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:35.599143 29588 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:18:35.643376 29360 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.010s	sys 0.059s
I20260709 12:18:35.595494 29797 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:35.610893 29704 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.599170 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1605) took 1302 ms. Trace:
I20260709 12:18:35.612455 29314 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.595409 29365 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.598565 29715 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.723984 29773 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:35.596696 29760 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:35.694267 27729 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:18:35.614790 29584 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:35.684285 27584 rpcz_store.cc:276] 0709 12:18:32.868816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.900316 (+ 31500us) service_pool.cc:224] Handling call
0709 12:18:32.915976 (+ 15660us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:32.952997 (+ 37021us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:32.953061 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:33.921961 (+968900us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b2c000014c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:33.927788 (+  5827us) log.cc:844] Serialized 26 byte log entry
0709 12:18:34.069424 (+141636us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:34.172992 (+103568us) log.cc:844] Serialized 14 byte log entry
0709 12:18:34.173224 (+   232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.173318 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:35.600311 (+1426993us) raft_consensus.cc:1685] finished
0709 12:18:35.600415 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:35.600559 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32354,"prepare.queue_time_us":106607,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":61,"raft.queue_time_us":39451,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":154175,"spinlock_wait_cycles":37991296,"thread_start_us":132491,"threads_started":1,"wal-append.queue_time_us":326514}
I20260709 12:18:35.597208 29788 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.701622 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:35.688894 29712 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.027s	sys 0.047s
I20260709 12:18:35.659818 29743 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.588676 29512 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
I20260709 12:18:35.659246 29460 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:35.667730 29684 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:35.579376 29409 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:35.717931 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:35.857017 27588 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:35.718928 28332 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:18:35.720046 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:35.833618 29712 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.027s	sys 0.048s
I20260709 12:18:35.664338 29795 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:35.831542 29801 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.726001 27657 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:35.678004 29683 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:35.577509 29723 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.735070 27655 raft_consensus.cc:2368] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:35.689924 29746 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:35.788084 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:35.588594 29786 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:35.800932 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:35.575721 29770 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.600389 29539 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.665205 29734 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.813262 28994 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:35.588966 29790 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:35.795172 29360 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.010s	sys 0.060s
W20260709 12:18:35.804589 29764 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.855s	user 0.011s	sys 0.000s
W20260709 12:18:35.826979 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 240918us
I20260709 12:18:35.791007 29750 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:35.830185 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 128648us
I20260709 12:18:35.703588 29710 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:35.716529 29729 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.800489 29797 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:35.656260 29775 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:35.828356 29562 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.905s	user 0.009s	sys 0.003s
I20260709 12:18:35.840826 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:35.786530 29666 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.819366 29728 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.808477 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1379) took 1334 ms. Trace:
I20260709 12:18:35.818794 29181 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:35.662494 29727 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:35.662919 29782 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.593114 29719 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.821710 29740 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.722496 29778 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:35.826607 29314 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:35.833155 29588 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.133s
I20260709 12:18:35.964645 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:35.836441 29365 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:35.831935 29773 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:36.199391 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 234833us
I20260709 12:18:35.836915 29584 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.200075 29562 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.200203 29788 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:36.110396 29744 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:35.913743 27581 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:36.200918 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:36.233775 27733 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:36.201254 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:36.124763 29713 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:36.199954 29683 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.202050 29764 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.249522 29314 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.250957 29764 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:36.287350 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:36.288442 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.292333 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1596) took 2145 ms. Trace:
I20260709 12:18:36.292943 27656 rpcz_store.cc:276] 0709 12:18:34.146954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.151039 (+  4085us) service_pool.cc:224] Handling call
0709 12:18:34.162309 (+ 11270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.554584 (+1392275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.291996 (+737412us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:36.292248 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2028647296}
W20260709 12:18:36.295481 29579 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.648s	user 0.008s	sys 0.003s
I20260709 12:18:36.295966 29579 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.296676 29790 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.298252 29579 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:36.301013 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:36.302608 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.202837 29782 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:35.845789 29684 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.203157 29801 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.203823 27584 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:36.204205 29797 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:36.122189 29802 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.205703 29778 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:36.182150 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 488867us
W20260709 12:18:36.141140 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 467399us
I20260709 12:18:35.856492 29409 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:35.833411 29785 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.980284 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:35.839082 28329 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:36.180176 29804 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
W20260709 12:18:36.226426 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 269024us
W20260709 12:18:36.322789 29732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.947s	user 0.010s	sys 0.010s
I20260709 12:18:36.230408 27735 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:35.858773 29798 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.860236 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.199282 29736 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.207700 29786 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.243289 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:36.214516 29666 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.211642 29775 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:36.220237 29800 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:36.248925 29809 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:35.901450 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.348799 27588 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:36.251528 27587 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:36.252682 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:36.257158 29584 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.257876 29769 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.215771 29750 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:35.816112 27580 rpcz_store.cc:276] 0709 12:18:34.296787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.046631 (+749844us) service_pool.cc:224] Handling call
0709 12:18:35.047040 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.579069 (+532029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:35.599095 (+ 20026us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.227828 29773 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.228109 29795 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.405035 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1579) took 1510 ms. Trace:
I20260709 12:18:36.405898 29795 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.406462 29795 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.259686 29704 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:36.407059 27581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1603) took 2110 ms (client timeout 2215 ms). Trace:
I20260709 12:18:36.407852 27664 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:36.408142 27587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 189) took 2117 ms (client timeout 1773 ms). Trace:
W20260709 12:18:36.408486 27587 rpcz_store.cc:269] 0709 12:18:34.291129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.292364 (+  1235us) service_pool.cc:224] Handling call
0709 12:18:36.407982 (+2115618us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.408075 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4428246400}
I20260709 12:18:36.408885 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.409493 27664 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:36.409596 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.410305 27587 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:36.410465 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.410737 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1639) took 1128 ms. Trace:
I20260709 12:18:36.327221 29785 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.224136 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.412297 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.212328 29670 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:36.242960 29744 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.232121 29719 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:36.240425 29753 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.262569 29562 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.263518 27582 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
W20260709 12:18:36.264935 29490 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.314s	user 0.011s	sys 0.000s
I20260709 12:18:36.445391 29490 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.446882 29746 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.447899 29490 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.266265 29734 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.267572 29673 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:36.267856 29770 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:36.207262 29729 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.268289 29723 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.221882 29727 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.207788 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1428) took 1721 ms. Trace:
I20260709 12:18:36.242422 29692 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:36.269518 29740 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.269774 29710 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.214612 27661 rpcz_store.cc:276] 0709 12:18:34.474349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.481483 (+  7134us) service_pool.cc:224] Handling call
0709 12:18:34.541564 (+ 60081us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:35.722779 (+1181215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:35.805522 (+ 82743us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.269662 29460 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.318605 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1640) took 1035 ms. Trace:
I20260709 12:18:36.468319 27584 rpcz_store.cc:276] 0709 12:18:35.283093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.828175 (+545082us) service_pool.cc:224] Handling call
0709 12:18:36.318512 (+490337us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.296720 29788 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.315294 29728 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:36.322312 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.321964 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.313421 29790 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.315304 29684 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.312804 29764 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:36.458395 29723 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:36.330878 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:36.492512 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 161738us
I20260709 12:18:36.493147 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:36.314141 29760 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:36.340694 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.006s	sys 0.011s
W20260709 12:18:36.341967 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98752.7us
W20260709 12:18:36.493829 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.006s	sys 0.011s
I20260709 12:18:36.494320 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:36.326287 29409 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:36.474484 29762 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.818s	user 0.011s	sys 0.000s
I20260709 12:18:36.343259 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 379) took 1365 ms. Trace:
I20260709 12:18:36.321810 29802 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.332402 29389 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:36.327387 29181 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.348482 29809 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:36.353016 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:36.401572 28335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.000s	sys 0.020s
I20260709 12:18:36.401453 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.227602 29365 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:36.407624 27581 rpcz_store.cc:269] 0709 12:18:34.296197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.443263 (+147066us) service_pool.cc:224] Handling call
0709 12:18:36.406877 (+1963614us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54000bd6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.406993 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2642463488}
I20260709 12:18:36.411054 27587 rpcz_store.cc:276] 0709 12:18:35.282798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.409307 (+1126509us) service_pool.cc:224] Handling call
0709 12:18:36.410690 (+  1383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.417865 27588 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:36.501312 29753 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:36.421175 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.343827 29736 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.351241 29778 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:36.432730 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.423921 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:18:36.438452 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 191) took 2144 ms (client timeout 1954 ms). Trace:
I20260709 12:18:36.406913 29795 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.342294 29798 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.533335 29795 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.440418 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 376) took 1490 ms. Trace:
I20260709 12:18:36.535043 29798 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.533996 28333 rpcz_store.cc:276] 0709 12:18:34.950158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.950308 (+   150us) service_pool.cc:224] Handling call
0709 12:18:34.974261 (+ 23953us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:34.974474 (+   213us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:34.993134 (+ 18660us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:34.993162 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:35.288599 (+295437us) log.cc:844] Serialized 52 byte log entry
0709 12:18:35.577569 (+288970us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:36.120806 (+543237us) log.cc:844] Serialized 14 byte log entry
0709 12:18:36.212132 (+ 91326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.212242 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.440089 (+227847us) raft_consensus.cc:1685] finished
0709 12:18:36.440191 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.440333 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106439,"prepare.queue_time_us":448022,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":60,"raft.queue_time_us":112678,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":27505,"spinlock_wait_cycles":4315392,"thread_start_us":697313,"threads_started":2,"wal-append.queue_time_us":509174}
I20260709 12:18:36.537982 28328 raft_consensus.cc:2368] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:36.538818 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:36.539489 27730 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:36.440774 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.442255 29773 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.420924 29785 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:36.443559 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:36.444566 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.449858 27586 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:36.450320 27657 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:36.462908 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.347052 29786 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:36.470737 29801 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.471923 29797 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.474043 27736 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:36.443594 29744 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.215085 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:36.526324 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:36.548966 29820 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:18:36.549539 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.550387 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
W20260709 12:18:36.332607 29732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.011s	sys 0.011s
I20260709 12:18:36.500411 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:36.551764 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51446.1us
I20260709 12:18:36.545701 29828 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:36.552965 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:36.553572 27655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1383) took 2068 ms (client timeout 1917 ms). Trace:
I20260709 12:18:36.554347 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:36.519727 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.555212 27584 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:36.556505 29797 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.557137 29771 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:36.557595 29771 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.547277 29786 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.557971 29771 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.516448 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 423) took 1237 ms. Trace:
I20260709 12:18:36.559631 29365 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.559476 28335 rpcz_store.cc:276] 0709 12:18:35.279215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.364164 (+ 84949us) service_pool.cc:224] Handling call
0709 12:18:35.585126 (+220962us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:35.616801 (+ 31675us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:35.616848 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:35.677071 (+ 60223us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.516120 (+839049us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:36.516173 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.516220 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.516250 (+    30us) raft_consensus.cc:1685] finished
0709 12:18:36.516256 (+     6us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.516382 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51021,"raft.queue_time_us":26763,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11601,"thread_start_us":713267,"threads_started":1,"wal-append.queue_time_us":624080}
I20260709 12:18:36.505329 28336 rpcz_store.cc:276] 0709 12:18:34.974587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.084353 (+109766us) service_pool.cc:224] Handling call
0709 12:18:35.964659 (+880306us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:36.339939 (+375280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432753920}
W20260709 12:18:36.561878 29803 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.577s	user 0.003s	sys 0.000s
I20260709 12:18:36.405532 27733 rpcz_store.cc:276] 0709 12:18:34.894835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.949200 (+ 54365us) service_pool.cc:224] Handling call
0709 12:18:36.404815 (+1455615us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54001d24d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.404939 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1567831936}
I20260709 12:18:36.563678 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1641) took 1280 ms. Trace:
I20260709 12:18:36.563762 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.564386 27733 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:36.505854 29734 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:36.502987 29710 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:36.569092 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 371) took 1682 ms. Trace:
W20260709 12:18:36.529001 27553 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39872 dropped due to backpressure: service queue is full with 50 items
I20260709 12:18:36.500159 29746 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.505333 29762 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.527884 27656 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:36.515880 29712 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:18:36.571745 29710 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.547889 27580 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:36.573701 29712 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:36.573892 29710 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:36.507747 29770 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:36.529809 29736 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.576673 29770 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.578079 29736 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.533172 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.578787 29770 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:36.532191 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.579267 29460 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.579504 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.579876 27587 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:36.580884 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.579530 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.491705 29790 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.581514 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
W20260709 12:18:36.543773 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 80925.8us
I20260709 12:18:36.582672 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.475188 29788 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.495178 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:36.547580 28994 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:36.585879 28994 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.506122 29729 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:18:36.501511 29760 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.500885 29764 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.500427 29723 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:36.553978 27655 rpcz_store.cc:269] 0709 12:18:34.484994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.485137 (+   143us) service_pool.cc:224] Handling call
0709 12:18:36.553383 (+2068246us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400211e54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.553506 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2714430336}
I20260709 12:18:36.555111 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.528057 29778 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.527568 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:36.564085 27582 rpcz_store.cc:276] 0709 12:18:35.283414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.577064 (+293650us) service_pool.cc:224] Handling call
0709 12:18:36.563520 (+986456us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b5400087ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.563616 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326833920}
I20260709 12:18:36.476490 29827 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.476289 29728 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:36.530432 27589 rpcz_store.cc:269] 0709 12:18:34.293897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.294694 (+   797us) service_pool.cc:224] Handling call
0709 12:18:36.438158 (+2143464us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540011ce54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.438352 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2974057344}
I20260709 12:18:36.520337 29683 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:18:36.516641 29750 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.625545 27658 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:36.626610 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 220) took 1004 ms. Trace:
I20260709 12:18:36.626668 29750 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.540472 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.627038 27658 rpcz_store.cc:276] 0709 12:18:35.622674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.211500 (+588826us) service_pool.cc:224] Handling call
0709 12:18:36.211835 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.625911 (+414076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.626413 (+   502us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400229f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:36.626560 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":950977408}
I20260709 12:18:36.628299 29790 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.629992 29760 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.630371 27730 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:36.631563 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.632463 28336 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:36.632578 29788 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.606714 29770 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:36.633747 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.569489 28332 rpcz_store.cc:276] 0709 12:18:34.887448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.930108 (+ 42660us) service_pool.cc:224] Handling call
0709 12:18:36.568428 (+1638320us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b540024a254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.569035 (+   607us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1800852736}
I20260709 12:18:36.635188 29723 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.571719 28337 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:18:36.636740 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.637431 27656 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 1.
I20260709 12:18:36.569092 29734 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:36.638228 29734 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:36.519390 29809 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.648458 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:36.459738 27731 rpcz_store.cc:276] 0709 12:18:34.486668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.659706 (+173038us) service_pool.cc:224] Handling call
0709 12:18:34.660055 (+   349us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:34.924760 (+264705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:34.930220 (+  5460us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:35.372977 (+442757us) log.cc:844] Serialized 52 byte log entry
0709 12:18:35.794865 (+421888us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:35.805321 (+ 10456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:35.816256 (+ 10935us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.207416 (+391160us) raft_consensus.cc:1685] finished
0709 12:18:36.207456 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.207677 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79724,"raft.queue_time_us":182484,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":70630,"thread_start_us":320114,"threads_started":1,"wal-append.queue_time_us":585651}
I20260709 12:18:36.581465 27587 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:36.650182 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.650751 27587 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:36.590528 29802 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.578101 29727 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.592849 29719 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:36.651350 29673 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:36.594691 27584 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:36.652402 29673 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:36.652770 29673 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:36.652983 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.653095 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.595795 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.653614 27587 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:36.654038 27581 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.654474 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.521130 29181 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.598791 27552 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:36.655175 27552 leader_election.cc:403] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 12:18:36.600549 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 105431us
I20260709 12:18:36.653635 29673 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.656602 29673 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:36.657196 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:36.600647 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1577) took 1706 ms (client timeout 1978 ms). Trace:
W20260709 12:18:36.659443 27732 rpcz_store.cc:269] 0709 12:18:34.894107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.931654 (+ 37547us) service_pool.cc:224] Handling call
0709 12:18:36.600476 (+1668822us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.600586 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525500160}
W20260709 12:18:36.603884 27659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1602) took 2447 ms (client timeout 2205 ms). Trace:
W20260709 12:18:36.660429 27659 rpcz_store.cc:269] 0709 12:18:34.156036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.156153 (+   117us) service_pool.cc:224] Handling call
0709 12:18:36.603713 (+2447560us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.603822 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2823033600}
I20260709 12:18:36.571511 29785 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:36.661450 29785 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:36.661875 29785 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:36.662876 29808 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.669s	user 0.001s	sys 0.002s
I20260709 12:18:36.605551 29786 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.569837 27553 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39872 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1636)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1626)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1621)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1624)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1627)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1664)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1628)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1623)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 193)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 195)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 196)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1649)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1652)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1658)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1659)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1663)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 197)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 199)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1630)
I20260709 12:18:36.617966 29778 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:36.655064 27587 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:36.618415 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:36.667269 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:36.580888 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:36.620399 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:56174 (request call id 7) took 4143 ms. Trace:
I20260709 12:18:36.622000 29728 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.687748 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 217) took 1066 ms. Trace:
W20260709 12:18:36.687443 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44484.4us
I20260709 12:18:36.622346 29684 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:36.623521 29683 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:18:36.624759 29764 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:36.636930 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 424) took 1065 ms. Trace:
I20260709 12:18:36.515631 29389 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.644078 27583 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:36.622606 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.720181 27589 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:36.688174 27585 rpcz_store.cc:276] 0709 12:18:32.469956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.471761 (+  1805us) service_pool.cc:224] Handling call
0709 12:18:36.605163 (+4133402us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.605264 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5332738944}
W20260709 12:18:36.688830 27628 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7274fd8a2115405c9b78c61214bff736. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:36.688292 27660 rpcz_store.cc:276] 0709 12:18:35.621785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.622894 (+  1109us) service_pool.cc:224] Handling call
0709 12:18:35.623241 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:35.657005 (+ 33764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:35.657065 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:35.657660 (+   595us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.494845 (+837185us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b180043c720. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:18:36.506539 (+ 11694us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:36.506576 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.506646 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.639828 (+133182us) raft_consensus.cc:1685] finished
0709 12:18:36.639848 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.687673 (+ 47825us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33299,"raft.queue_time_us":98494,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":64483,"spinlock_wait_cycles":13718784,"thread_start_us":11510,"threads_started":1,"wal-append.queue_time_us":833986}
I20260709 12:18:36.619621 29773 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:36.722070 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:36.718669 28337 rpcz_store.cc:276] 0709 12:18:35.572000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.614578 (+ 42578us) service_pool.cc:224] Handling call
0709 12:18:36.569036 (+954458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.636075 (+ 67039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.636738 (+   663us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b540024a250. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.636870 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2075226880}
I20260709 12:18:36.724388 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.609066 29670 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:36.760948 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 221) took 1138 ms. Trace:
I20260709 12:18:36.762619 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.757273 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:36.779760 27588 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:36.728741 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.763773 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:36.728125 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:36.703517 29826 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:18:36.780736 27585 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:36.787537 27660 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:36.783074 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.869165 27553 leader_election.cc:400] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:36.869571 28332 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:36.869647 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:36.870100 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:36.778795 27657 rpcz_store.cc:276] 0709 12:18:35.622954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.216264 (+593310us) service_pool.cc:224] Handling call
0709 12:18:36.216597 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.544338 (+327741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.760617 (+216279us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400291b54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:36.760877 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":537008384}
I20260709 12:18:36.803891 28334 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:36.790079 27658 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:36.812670 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.780772 29773 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.823483 29774 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:36.856922 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 258819us
I20260709 12:18:36.656628 29181 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:36.857707 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 190518us
I20260709 12:18:36.667501 27555 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:39856 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1621)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1624)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1627)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1664)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1628)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1623)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 193)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 195)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 196)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1649)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1652)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1658)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1659)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1663)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 197)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 199)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1671)
W20260709 12:18:36.867502 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1647) took 1547 ms (client timeout 1930 ms). Trace:
I20260709 12:18:36.867502 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1642) took 1583 ms. Trace:
I20260709 12:18:36.728084 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.869545 27553 leader_election.cc:403] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:18:36.871086 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.756762 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.911888 29365 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:36.880585 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:36.914847 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45836.2us
I20260709 12:18:36.916399 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:36.713656 27661 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:36.879585 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:36.924005 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.923933 27589 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:36.925473 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.925788 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.926476 27582 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:18:36.926571 27587 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:36.926800 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:36.927439 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:36.927529 27586 rpcz_store.cc:269] 0709 12:18:35.320173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.578324 (+258151us) service_pool.cc:224] Handling call
0709 12:18:36.815413 (+1237089us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b540011bf54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.867435 (+ 52022us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1754868608}
I20260709 12:18:36.928351 27582 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:36.929325 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.728468 29784 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:36.718060 29683 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:36.927486 27660 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.929989 27582 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:36.931382 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.932276 27582 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:36.933429 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.929992 27583 rpcz_store.cc:276] 0709 12:18:35.283691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.575628 (+291937us) service_pool.cc:224] Handling call
0709 12:18:36.825263 (+1249635us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000bdbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:36.867441 (+ 42178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2453753600}
I20260709 12:18:36.938623 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.941452 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.943066 29750 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:36.942175 29727 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:36.943470 29750 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:36.943470 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:36.943984 29750 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:36.944864 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.945804 29746 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:36.946195 29746 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.946565 29746 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:36.946273 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:36.947546 29460 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:36.947667 29814 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:36.948006 29460 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:36.948115 29814 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:36.948187 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:36.949312 29712 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:36.949700 29712 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:36.949807 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:36.948596 29814 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:36.951121 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:36.948505 29460 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:36.955550 29666 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:36.955972 29666 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.956341 29666 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:36.959535 29797 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:36.960182 29797 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:36.961119 29797 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.961628 29746 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:36.950135 29712 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:36.964178 29746 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:36.723686 29786 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:36.968029 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:36.782177 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:36.971879 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.984728 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:36.972651 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.985872 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1644) took 1666 ms. Trace:
I20260709 12:18:36.973361 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.702034 29584 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.218s
W20260709 12:18:36.980934 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.986281 27589 rpcz_store.cc:276] 0709 12:18:35.319305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.972407 (+1653102us) service_pool.cc:224] Handling call
0709 12:18:36.985820 (+ 13413us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.967988 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:36.983143 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.914502 29579 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:36.914149 29773 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:18:36.981331 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101837us
I20260709 12:18:36.986610 27583 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:37.031524 28994 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.053715 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:37.054713 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.057170 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 193) took 2518 ms. Trace:
I20260709 12:18:36.845921 29670 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:37.110777 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 186498us
I20260709 12:18:37.109088 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:37.113220 27581 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:37.107574 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:36.968279 29774 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:37.059186 27588 raft_consensus.cc:2368] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:36.702248 29824 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.282s
I20260709 12:18:36.671339 29827 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:37.181684 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 301150us
I20260709 12:18:37.211414 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.060609 27664 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:37.230376 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 215) took 1436 ms. Trace:
I20260709 12:18:37.322748 27589 rpcz_store.cc:276] 0709 12:18:34.538835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.056155 (+2517320us) service_pool.cc:224] Handling call
0709 12:18:37.056530 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.056751 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.056787 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.056841 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:37.056885 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.056954 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.057099 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.324254 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1628) took 1021 ms. Trace:
I20260709 12:18:37.333516 27588 rpcz_store.cc:276] 0709 12:18:36.302564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.915348 (+612784us) service_pool.cc:224] Handling call
0709 12:18:37.324190 (+408842us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.198096 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 194) took 2269 ms. Trace:
I20260709 12:18:37.127503 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:37.060467 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.324031 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1656) took 1520 ms. Trace:
I20260709 12:18:37.327246 27738 rpcz_store.cc:276] 0709 12:18:35.793824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.625990 (+832166us) service_pool.cc:224] Handling call
0709 12:18:36.626302 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.626507 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:36.626834 (+   327us) log.cc:844] Serialized 14 byte log entry
0709 12:18:37.229916 (+603082us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.229976 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:37.230039 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.230141 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.230304 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6581,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":142166,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":32671,"thread_start_us":602413,"threads_started":2,"wal-append.queue_time_us":625972}
I20260709 12:18:37.380724 27586 rpcz_store.cc:276] 0709 12:18:35.803845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.929066 (+1125221us) service_pool.cc:224] Handling call
0709 12:18:37.323956 (+394890us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.149608 29786 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:18:37.380195 27581 rpcz_store.cc:276] 0709 12:18:34.929159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.137537 (+2208378us) service_pool.cc:224] Handling call
0709 12:18:37.197431 (+ 59894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.197663 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.197701 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.197756 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:37.197801 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.197870 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.198005 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.968215 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 382) took 1623 ms. Trace:
I20260709 12:18:37.159548 27585 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:36.966859 29727 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:37.286530 29832 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
W20260709 12:18:37.229727 29692 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.762s	user 0.008s	sys 0.008s
I20260709 12:18:37.364733 27584 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:37.366782 28994 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:37.351979 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1637) took 2070 ms. Trace:
W20260709 12:18:37.381923 27580 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.808s	user 0.010s	sys 0.006s
W20260709 12:18:37.382464 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.491063 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 167416us
I20260709 12:18:37.325098 27659 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:37.317229 29389 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.428427 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1614) took 2889 ms. Trace:
I20260709 12:18:37.240835 29775 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:37.401072 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 196) took 2471 ms. Trace:
I20260709 12:18:37.512329 27588 rpcz_store.cc:276] 0709 12:18:34.929702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.334317 (+2404615us) service_pool.cc:224] Handling call
0709 12:18:37.400374 (+ 66057us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.400616 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.400653 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.400709 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:37.400778 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.400847 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.400992 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.293004 29823 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:37.126940 29784 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:37.446512 27581 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:18:37.517411 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1645) took 2197 ms. Trace:
I20260709 12:18:37.517907 27581 rpcz_store.cc:276] 0709 12:18:35.319641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.427190 (+2107549us) service_pool.cc:224] Handling call
0709 12:18:37.445766 (+ 18576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.517048 (+ 71282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.517339 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:37.434688 29831 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.505s	user 0.000s	sys 0.001s
W20260709 12:18:37.538873 27661 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.554s	user 0.004s	sys 0.000s
I20260709 12:18:37.485435 27589 rpcz_store.cc:276] 0709 12:18:35.282148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.350828 (+2068680us) service_pool.cc:224] Handling call
0709 12:18:37.351223 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.351503 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.351546 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.351617 (+    71us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:37.351669 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.351746 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.351902 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.054365 29723 raft_consensus.cc:2764] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:18:37.541759 27586 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:37.542546 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1646) took 2222 ms. Trace:
I20260709 12:18:37.542893 27586 rpcz_store.cc:276] 0709 12:18:35.319928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.489333 (+2169405us) service_pool.cc:224] Handling call
0709 12:18:37.541554 (+ 52221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.542294 (+   740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.542489 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.489396 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.544579 27657 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:37.347163 29774 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.491963 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:37.515466 29809 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.867s	user 0.016s	sys 0.011s
I20260709 12:18:37.116587 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:37.474131 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.061940 29734 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.420512 29729 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:37.512981 29409 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.005s	user 0.021s	sys 0.006s
I20260709 12:18:37.162312 29460 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.376302 29762 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.539253 27661 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:37.539947 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:37.541221 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:37.492283 29673 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.835s	user 0.010s	sys 0.005s
I20260709 12:18:37.433915 29786 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.508621 27585 rpcz_store.cc:276] 0709 12:18:34.539265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.060327 (+2521062us) service_pool.cc:224] Handling call
0709 12:18:37.159286 (+ 98959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.428073 (+268787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.428355 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:37.547446 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55570.9us
I20260709 12:18:37.547544 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.506229 29692 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:37.565389 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 216) took 1944 ms. Trace:
I20260709 12:18:37.506843 27580 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:37.607731 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:37.213785 29683 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.623363 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:37.764210 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:37.767073 27581 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:37.681068 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.681495 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.661327 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.766140 27589 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:37.768571 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:37.768842 27655 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:37.768165 27664 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:37.769737 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.553818 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:37.770243 27655 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
W20260709 12:18:37.770731 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 217013us
I20260709 12:18:37.765431 27586 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:37.646164 27660 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.716s	user 0.010s	sys 0.001s
I20260709 12:18:37.772111 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:37.772341 27655 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:37.773226 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1605) took 1103 ms. Trace:
I20260709 12:18:37.766446 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 223) took 1196 ms. Trace:
W20260709 12:18:37.637885 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.006s	sys 0.060s
I20260709 12:18:37.773878 27660 rpcz_store.cc:276] 0709 12:18:36.669647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.926562 (+256915us) service_pool.cc:224] Handling call
0709 12:18:37.773167 (+846605us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.767419 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:37.774384 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.019s	sys 0.084s
I20260709 12:18:37.774006 27661 rpcz_store.cc:276] 0709 12:18:36.569724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.580613 (+ 10889us) service_pool.cc:224] Handling call
0709 12:18:37.766395 (+1185782us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:37.775017 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 838587us
I20260709 12:18:37.765427 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.775410 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:37.777863 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.701809 29750 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.757s	user 0.006s	sys 0.003s
I20260709 12:18:37.780223 29750 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.780565 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:37.780691 29676 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:18:37.781046 29785 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.119s	user 0.014s	sys 0.004s
I20260709 12:18:37.781191 29676 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.781594 29676 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:18:37.781472 29785 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.782280 29750 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.782876 29764 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:37.667474 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 338668us
I20260709 12:18:37.554767 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:37.764494 27662 rpcz_store.cc:276] 0709 12:18:35.621458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.622769 (+  1311us) service_pool.cc:224] Handling call
0709 12:18:35.668193 (+ 45424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:36.535873 (+867680us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:36.535903 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:36.536372 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.612471 (+ 76099us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:36.612544 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.826009 (+213465us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:36.826061 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:37.565008 (+738947us) raft_consensus.cc:1685] finished
0709 12:18:37.565037 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.565313 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":973,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":1994593664,"thread_start_us":75797,"threads_started":1,"wal-append.queue_time_us":382339}
I20260709 12:18:37.517342 29727 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.783600 27659 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
W20260709 12:18:37.785065 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 230371us
I20260709 12:18:37.790817 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
W20260709 12:18:37.791760 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.792204 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:37.564109 29728 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:37.707674 29833 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
W20260709 12:18:37.701475 29837 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:18:37.793381 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:37.653697 29579 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.015s	sys 0.047s
W20260709 12:18:37.704245 29778 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.040s	user 0.008s	sys 0.004s
I20260709 12:18:37.789324 27655 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:37.785183 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:37.817663 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:37.818766 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:37.704698 29814 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.754s	user 0.007s	sys 0.003s
W20260709 12:18:37.706225 29710 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.132s	user 0.010s	sys 0.000s
I20260709 12:18:37.551258 29774 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:37.657450 29712 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.694s	user 0.003s	sys 0.008s
I20260709 12:18:37.796990 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:37.564726 29784 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:37.547847 29823 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:37.798053 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:37.783114 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:37.793366 29824 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:18:37.744319 29762 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:37.807883 29838 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.553s	user 0.003s	sys 0.000s
I20260709 12:18:37.811602 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:37.701241 29827 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.640488 29670 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.708842 29460 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:37.787568 29846 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:37.628314 29409 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:37.809439 29771 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.251s	user 0.010s	sys 0.010s
I20260709 12:18:37.789969 29750 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:37.820775 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48832.3us
I20260709 12:18:37.821278 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:37.872655 27585 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:37.822350 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:37.874049 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51791.1us
I20260709 12:18:37.874996 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:37.783360 29844 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:37.793414 29723 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:18:37.790580 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1648) took 2212 ms (client timeout 2205 ms). Trace:
I20260709 12:18:37.819751 29778 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.823050 29845 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:37.823185 29667 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:18:37.564338 29809 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.442616 28329 rpcz_store.cc:276] 0709 12:18:35.344665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.375758 (+ 31093us) service_pool.cc:224] Handling call
0709 12:18:35.376244 (+   486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.329285 (+953041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.825640 (+496355us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54002fab54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:36.925845 (+100205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1037124352}
I20260709 12:18:37.832743 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:37.894400 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61747.7us
I20260709 12:18:37.882222 29824 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:37.813271 29579 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:37.838241 29785 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:37.918555 29579 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.920368 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:37.921080 27586 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:37.922058 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:37.922715 27584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1626) took 1620 ms (client timeout 1885 ms). Trace:
W20260709 12:18:37.923419 27584 rpcz_store.cc:269] 0709 12:18:36.301937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.652873 (+350936us) service_pool.cc:224] Handling call
0709 12:18:37.922566 (+1269693us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54000be0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:37.922664 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1634884480}
I20260709 12:18:37.924513 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:37.925596 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:37.926257 27584 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:37.830315 29814 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.831842 29712 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.892028 29845 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.839058 29842 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:37.793793 29727 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.824688 29704 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:37.923148 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:37.829515 29584 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:37.863212 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.877s	user 0.019s	sys 0.016s
W20260709 12:18:37.940255 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.019s	sys 0.016s
I20260709 12:18:37.942708 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:37.836611 29719 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:37.843118 29734 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:37.864637 29847 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:37.876188 29750 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.881217 27660 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:37.832863 29710 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.952652 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:37.952925 29750 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:37.947901 29719 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.954561 29710 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:37.878206 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:37.885053 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 214) took 2093 ms. Trace:
I20260709 12:18:37.936156 29842 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.877987 29771 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.888720 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:37.865312 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:37.850507 29762 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.955729 29842 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:37.957799 29762 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:37.837049 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53996.5us
I20260709 12:18:37.958282 29845 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.783722 29786 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:18:37.878893 29844 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:37.960249 29844 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:37.960726 29844 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:37.891444 29784 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:37.901109 29365 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.988s	user 0.009s	sys 0.001s
I20260709 12:18:37.962244 29365 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:37.904516 29684 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:37.903606 29823 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:37.881459 27580 rpcz_store.cc:269] 0709 12:18:35.576851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.444105 (+867254us) service_pool.cc:224] Handling call
0709 12:18:37.788742 (+1344637us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.877120 29849 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:18:37.931135 29676 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.968125 29849 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:37.968608 29849 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:37.968818 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1652) took 2178 ms. Trace:
I20260709 12:18:37.969414 29676 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:37.969727 29849 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:37.916138 29809 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:37.917241 29666 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.960s	user 0.004s	sys 0.006s
I20260709 12:18:37.971313 29771 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:37.971695 27657 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:37.971760 29666 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.973176 29409 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.974411 29666 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:37.926160 29778 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:37.927846 27658 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:37.928151 29773 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.563s	user 0.001s	sys 0.009s
I20260709 12:18:37.955435 27735 rpcz_store.cc:276] 0709 12:18:35.791375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.618974 (+827599us) service_pool.cc:224] Handling call
0709 12:18:36.619375 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:36.635740 (+ 16365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:36.635768 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:36.636247 (+   479us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.728491 (+ 92244us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:36.728544 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.828446 (+ 99902us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:36.828504 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:37.790294 (+961790us) spinlock_profiling.cc:243] Waited 7.94 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:37.884838 (+ 94544us) raft_consensus.cc:1685] finished
0709 12:18:37.884857 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.884993 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55293952,"thread_start_us":65227,"threads_started":1,"wal-append.queue_time_us":487743}
I20260709 12:18:37.975926 29773 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.976744 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1653) took 2185 ms. Trace:
I20260709 12:18:37.977169 27588 rpcz_store.cc:276] 0709 12:18:35.791163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.975897 (+2184734us) service_pool.cc:224] Handling call
0709 12:18:37.976252 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.976440 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.976467 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.976508 (+    41us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:37.976541 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.976589 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.976687 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.978698 29773 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:37.978925 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1654) took 2187 ms. Trace:
I20260709 12:18:37.929136 29814 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:37.931869 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.990844 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.992094 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 113997us
I20260709 12:18:37.992553 28994 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:37.993021 28994 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:37.931663 29775 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:37.998281 29775 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:37.935952 27661 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:37.829805 29782 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:18:37.947235 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.003s	sys 0.019s
W20260709 12:18:37.999799 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.003s	sys 0.019s
I20260709 12:18:37.947623 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:37.954963 29719 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:37.955812 27738 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
W20260709 12:18:38.003888 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211750us
I20260709 12:18:37.915814 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.005688 27581 raft_consensus.cc:2376] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6. Re-sending same reply.
I20260709 12:18:37.969237 27585 rpcz_store.cc:276] 0709 12:18:35.790770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.968034 (+2177264us) service_pool.cc:224] Handling call
0709 12:18:37.968333 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.968495 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.968528 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.968569 (+    41us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:37.968598 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.968646 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.968732 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.947388 29704 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:37.952697 29847 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:37.981081 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1623) took 1680 ms. Trace:
I20260709 12:18:38.009348 27582 rpcz_store.cc:276] 0709 12:18:36.301024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.933125 (+632101us) service_pool.cc:224] Handling call
0709 12:18:37.980911 (+1047786us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:37.981020 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1969554816}
I20260709 12:18:37.951306 29387 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.982015 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:37.982950 27584 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:18:38.010681 27735 raft_consensus.cc:2368] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:37.992857 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:37.788286 29729 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:37.994402 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:37.995231 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1610) took 1012 ms. Trace:
I20260709 12:18:38.010926 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1649) took 2387 ms. Trace:
W20260709 12:18:38.018787 27583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.002s	sys 0.030s
W20260709 12:18:38.022989 27583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.002s	sys 0.030s
I20260709 12:18:38.021847 27658 rpcz_store.cc:276] 0709 12:18:36.982430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.982698 (+   268us) service_pool.cc:224] Handling call
0709 12:18:37.115733 (+133035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.975494 (+859761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.995002 (+ 19508us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:37.995172 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867095296}
I20260709 12:18:37.995285 27587 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:38.022018 27584 rpcz_store.cc:276] 0709 12:18:35.623564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.937461 (+2313897us) service_pool.cc:224] Handling call
0709 12:18:37.982651 (+ 45190us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.010603 (+ 27952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.010861 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.993619 29584 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:37.928342 29712 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:38.048884 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.005839 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.049938 27738 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:37.971240 29727 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:38.010516 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:38.051786 27581 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:38.053771 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1658) took 2241 ms. Trace:
I20260709 12:18:38.054215 27581 rpcz_store.cc:276] 0709 12:18:35.812454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.052809 (+2240355us) service_pool.cc:224] Handling call
0709 12:18:38.053181 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.053389 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.053425 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.053482 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:38.053524 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.053593 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.053709 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.024598 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:37.950783 27656 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:38.059450 27736 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:38.029692 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.022531 27630 leader_election.cc:400] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:38.060572 27630 leader_election.cc:403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:18:37.995023 29849 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:38.037642 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44863.5us
I20260709 12:18:38.000114 29713 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:38.000767 29770 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:38.049211 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54868.4us
I20260709 12:18:38.066484 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:38.067482 27735 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:38.057548 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1659) took 2235 ms. Trace:
W20260709 12:18:38.069583 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47294.4us
I20260709 12:18:38.061416 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:38.003726 29771 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:38.062191 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.007768 29782 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:38.073659 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1655) took 2281 ms. Trace:
I20260709 12:18:38.009573 29704 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:38.061149 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:38.079044 28332 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:18:38.070006 27581 rpcz_store.cc:276] 0709 12:18:35.825707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.056592 (+2230885us) service_pool.cc:224] Handling call
0709 12:18:38.056956 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.057169 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.057206 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.057260 (+    54us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:38.057302 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.057370 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.057490 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.069882 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:38.073590 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:37.988469 27588 rpcz_store.cc:276] 0709 12:18:35.791490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.978141 (+2186651us) service_pool.cc:224] Handling call
0709 12:18:37.978445 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.978629 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.978655 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.978699 (+    44us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:37.978732 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.978783 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.978880 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.074339 27585 rpcz_store.cc:276] 0709 12:18:35.791791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.035917 (+2244126us) service_pool.cc:224] Handling call
0709 12:18:38.073013 (+ 37096us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.073258 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.073297 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.073352 (+    55us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:38.073404 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.073472 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.073595 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.075313 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.076186 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:38.084941 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:38.085676 27581 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:38.085964 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:38.086613 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1662) took 2229 ms. Trace:
I20260709 12:18:38.082974 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:38.087639 27581 rpcz_store.cc:276] 0709 12:18:35.856956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.085111 (+2228155us) service_pool.cc:224] Handling call
0709 12:18:38.085461 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.086327 (+   866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.086551 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:38.103295 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.087311 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:38.048857 29692 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:38.049249 29389 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:38.130139 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.003968 29774 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.147889 27588 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:38.148574 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1660) took 2318 ms. Trace:
I20260709 12:18:38.148926 27588 rpcz_store.cc:276] 0709 12:18:35.830362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.083748 (+2253386us) service_pool.cc:224] Handling call
0709 12:18:38.147571 (+ 63823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.148316 (+   745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.148518 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.150441 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.151268 27735 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
W20260709 12:18:38.151463 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 299404us
I20260709 12:18:38.072502 29771 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:38.155040 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.078486 29712 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:38.036413 29387 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.130281 29785 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:38.127164 29773 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:38.166954 28332 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:38.176569 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.177349 27585 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.129464 29666 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:38.133473 29734 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:38.133358 29727 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:38.144047 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:38.143517 29409 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:38.185326 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.181017 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.186335 27664 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6.
I20260709 12:18:38.077888 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:38.126258 29814 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260709 12:18:38.076169 29770 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:38.183763 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 287010us
I20260709 12:18:38.148731 29684 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:38.183081 29673 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:38.158562 29771 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:38.201048 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 118180us
I20260709 12:18:38.172866 29710 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:18:38.175753 29785 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:38.129472 29713 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:38.221565 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1604) took 1552 ms (client timeout 1999 ms). Trace:
I20260709 12:18:38.224715 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:38.175032 29773 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:38.181591 29734 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:38.181910 29727 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:38.230551 27585 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:38.246265 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.191540 29676 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:38.229027 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.181216 29666 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:38.245474 27656 rpcz_store.cc:269] 0709 12:18:36.669348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.688422 (+ 19074us) service_pool.cc:224] Handling call
0709 12:18:38.221320 (+1532898us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:38.221480 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":995815296}
I20260709 12:18:38.192023 29855 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:38.260056 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.189328 29732 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:38.157305 29847 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.227259 29778 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.066s
W20260709 12:18:38.309197 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 136187us
W20260709 12:18:38.292459 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 138556us
I20260709 12:18:38.241889 29785 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.270321 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.335891 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:38.334558 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 108277us
W20260709 12:18:38.346208 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 279780us
W20260709 12:18:38.343030 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132695us
I20260709 12:18:38.216804 29782 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.324712 27731 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:38.363885 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.246438 29784 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20260709 12:18:38.366444 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.344033 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.345418 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:38.249462 29727 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:38.351037 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:38.351688 29713 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:38.258383 29666 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:38.249747 29734 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:38.259439 29773 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.258867 29676 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:38.243400 29852 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:38.369781 29861 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:38.369805 29859 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:38.348824 29732 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.367429 29855 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:38.383903 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.364980 29853 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.386615 29387 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:38.373858 29854 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:18:38.378737 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.391145 29692 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:38.398567 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.399619 29764 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:38.400082 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.405391 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:38.405781 27587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 201) took 1973 ms (client timeout 2166 ms). Trace:
W20260709 12:18:38.403051 29846 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.526s	user 0.004s	sys 0.002s
I20260709 12:18:38.457937 29846 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:38.410426 28292 leader_election.cc:400] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6.
W20260709 12:18:38.414911 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69573.9us
I20260709 12:18:38.357141 29729 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:38.366204 29857 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:18:38.458567 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:38.459442 27585 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:38.460500 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:38.460983 29824 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.524s	user 0.004s	sys 0.005s
I20260709 12:18:38.461369 29824 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:38.461656 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.462384 27662 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:38.371206 29873 ts_tablet_manager.cc:933] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:38.463496 27738 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:38.458743 28292 leader_election.cc:403] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:18:38.464833 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54627.7us
I20260709 12:18:38.465750 29875 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6.
I20260709 12:18:38.424157 29778 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:18:38.424883 29856 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:38.468282 29856 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.460521 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:38.470893 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.471449 29774 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:38.432992 29842 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.472723 29719 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.474054 27586 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:38.474450 29842 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:38.427583 29879 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:38.475579 29879 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:38.475998 29879 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:38.477098 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.478824 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:38.407239 29684 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:38.479907 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.479972 29684 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.481005 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.481523 27581 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:38.483292 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 195) took 3553 ms. Trace:
I20260709 12:18:38.483873 27737 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:38.483873 27583 rpcz_store.cc:276] 0709 12:18:34.929449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.321975 (+2392526us) service_pool.cc:224] Handling call
0709 12:18:37.322342 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:37.322740 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.322778 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:37.351278 (+ 28500us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.024676 (+673398us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:38.024719 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.024792 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:38.483083 (+458291us) raft_consensus.cc:1685] finished
0709 12:18:38.483103 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.483237 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":21100,"thread_start_us":532259,"threads_started":1,"wal-append.queue_time_us":761041}
I20260709 12:18:38.418383 29676 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:38.386745 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.444674 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.373229 29784 consensus_queue.cc:1243] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:18:38.452623 27655 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:38.402966 29797 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:18:38.456087 27587 rpcz_store.cc:269] 0709 12:18:36.431902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.972304 (+540402us) service_pool.cc:224] Handling call
0709 12:18:38.405614 (+1433310us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:38.405727 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1952626560}
I20260709 12:18:38.456094 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.457289 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:38.295804 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.486568 27664 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 5
W20260709 12:18:38.488437 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88477.6us
I20260709 12:18:38.491622 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.471614 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.463055 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.458726 29844 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.455051 29764 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:38.494716 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.454380 29692 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:38.470458 29856 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:38.488565 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:38.495563 29723 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:38.429711 29859 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.497323 29704 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.499270 27581 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:38.501638 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1620) took 2201 ms. Trace:
I20260709 12:18:38.501986 28334 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.503300 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:38.486615 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1663) took 2522 ms. Trace:
I20260709 12:18:38.504658 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.507548 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.482709 29584 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:38.453083 29855 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:38.445189 29750 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.453696 29786 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.444902 29853 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.458606 29846 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.111s
W20260709 12:18:38.571820 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.693387 27658 raft_consensus.cc:2368] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:38.652750 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1648) took 1156 ms. Trace:
I20260709 12:18:38.694270 27586 rpcz_store.cc:276] 0709 12:18:37.496201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.922894 (+426693us) service_pool.cc:224] Handling call
0709 12:18:38.637956 (+715062us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b540011b554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:38.652670 (+ 14714us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1227047680}
I20260709 12:18:38.517071 28994 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:38.696545 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 208046us
I20260709 12:18:38.696979 28994 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:38.611073 27733 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:38.592422 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 161326us
W20260709 12:18:38.700793 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 197620us
I20260709 12:18:38.659802 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.701591 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.700382 27655 raft_consensus.cc:2368] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:38.702051 27729 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:38.702728 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.703011 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:38.702862 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.704456 27583 rpcz_store.cc:276] 0709 12:18:35.964264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.485913 (+2521649us) service_pool.cc:224] Handling call
0709 12:18:38.486184 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.486347 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.486368 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.486404 (+    36us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:38.486434 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.486480 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.486568 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.693964 27587 rpcz_store.cc:276] 0709 12:18:36.300113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.500609 (+2200496us) service_pool.cc:224] Handling call
0709 12:18:38.501012 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.501228 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.501264 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.501318 (+    54us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:38.501367 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.501436 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.501572 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.708017 27587 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:18:38.708377 27583 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:38.708705 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1630) took 2364 ms. Trace:
I20260709 12:18:38.708958 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1671) took 2044 ms. Trace:
I20260709 12:18:38.709100 27587 rpcz_store.cc:276] 0709 12:18:36.344333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.707443 (+2363110us) service_pool.cc:224] Handling call
0709 12:18:38.707735 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.708419 (+   684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.708589 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.709337 27583 rpcz_store.cc:276] 0709 12:18:36.664922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.707950 (+2043028us) service_pool.cc:224] Handling call
0709 12:18:38.708214 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.708752 (+   538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.708912 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.710724 27587 raft_consensus.cc:1217] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-6.4]   Dedup: 3.2->[4.3-6.4]
I20260709 12:18:38.621845 27730 raft_consensus.cc:2368] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:38.721338 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:38.718354 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.712095 27587 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
W20260709 12:18:38.635162 29863 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
I20260709 12:18:38.723922 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.511684 29588 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:38.724130 27586 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:38.722476 27587 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:18:38.725513 27587 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:38.726011 27582 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:38.725539 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 198) took 2422 ms. Trace:
I20260709 12:18:38.726681 27586 rpcz_store.cc:276] 0709 12:18:36.303171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.695284 (+2392113us) service_pool.cc:224] Handling call
0709 12:18:38.723959 (+ 28675us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.725230 (+  1271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.725470 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.500311 29797 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:38.654284 29871 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:38.608240 29774 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:38.733914 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.727468 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:38.729210 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.743685 27661 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:38.644490 29770 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:38.731536 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.499585 29734 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.704314 29764 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.732900 27732 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:38.731463 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.692368 29873 tablet_copy_client.cc:323] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
W20260709 12:18:38.742528 29775 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.744s	user 0.012s	sys 0.000s
I20260709 12:18:38.705633 29728 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:38.728569 29588 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:38.705950 29827 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:38.713308 29762 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:38.731597 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:38.700914 29723 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:38.683455 29719 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:38.507078 29666 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.757968 27586 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:38.706388 29823 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:38.759042 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.760241 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:38.750361 29712 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.000s
W20260709 12:18:38.764453 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58311.5us
I20260709 12:18:38.745664 29871 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:38.765105 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.720247 29713 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:18:38.748189 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1650) took 3075 ms. Trace:
I20260709 12:18:38.747157 29872 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.771045 27659 raft_consensus.cc:2368] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:38.750140 29778 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:38.728418 29842 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:18:38.782228 29762 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:38.761550 29728 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:38.844754 27661 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:38.844647 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.844650 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:38.744074 29860 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20260709 12:18:38.750823 29774 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.777758 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:38.841759 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:38.848286 29712 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:38.757704 29878 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.115s
I20260709 12:18:38.858777 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.860059 27552 leader_election.cc:400] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:38.864684 27655 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:38.865170 27552 leader_election.cc:400] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:38.865509 27552 leader_election.cc:403] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:18:38.865785 27586 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:38.861090 27661 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:38.830463 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:38.763659 29775 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:38.868183 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90517.7us
I20260709 12:18:38.868322 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.770159 29773 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.867233 27661 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:38.841547 29823 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:38.871541 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:38.841526 29888 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:38.868578 27655 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:18:38.869212 27586 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:38.840150 29723 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.838104 29784 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
W20260709 12:18:38.862703 29785 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.501s	user 0.005s	sys 0.006s
I20260709 12:18:38.858273 29762 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:38.878757 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:38.861608 29728 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:38.883256 28994 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:38.863963 29719 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:38.880568 29771 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.641s	user 0.008s	sys 0.003s
W20260709 12:18:38.891789 29886 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:18:38.882798 29844 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:38.897624 29881 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:18:38.899713 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 155860us
I20260709 12:18:38.981210 27662 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:38.901170 29784 consensus_queue.cc:1243] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:38.885845 29588 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.921342 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.885085 29827 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:38.924804 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1625) took 1213 ms. Trace:
I20260709 12:18:39.004674 27657 rpcz_store.cc:276] 0709 12:18:37.711231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.776023 (+ 64792us) service_pool.cc:224] Handling call
0709 12:18:37.776227 (+   204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.972190 (+195963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.924470 (+952280us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b5400229054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:38.924727 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":449117568}
I20260709 12:18:38.878373 29713 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 12:18:38.923411 28994 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:39.006322 28994 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:39.007474 28994 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.008723 28994 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.875072 29823 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:38.874578 29710 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:38.937366 29849 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.865s	user 0.004s	sys 0.007s
I20260709 12:18:38.923960 29890 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:38.938846 29365 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:38.966374 29771 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.019778 29365 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.021749 27735 raft_consensus.cc:2368] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:39.021811 29365 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:38.941751 29727 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.561s	user 0.007s	sys 0.000s
I20260709 12:18:38.996768 29704 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:39.023080 29883 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.192s	user 0.003s	sys 0.000s
W20260709 12:18:38.944082 29889 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:18:38.844681 29797 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.025424 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 197) took 2722 ms. Trace:
I20260709 12:18:39.026000 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 221) took 1083 ms. Trace:
I20260709 12:18:39.025806 27585 rpcz_store.cc:276] 0709 12:18:36.302877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.504080 (+2201203us) service_pool.cc:224] Handling call
0709 12:18:38.567165 (+ 63085us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.567443 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.567476 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:38.568063 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.723595 (+155532us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:38.726351 (+  2756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.726422 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.025182 (+298760us) raft_consensus.cc:1685] finished
0709 12:18:39.025201 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.025349 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":78673,"threads_started":1,"wal-append.queue_time_us":219475}
W20260709 12:18:38.944474 29835 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.445s	user 0.000s	sys 0.006s
I20260709 12:18:38.945560 29734 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:38.947384 29774 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:38.948719 29884 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:18:38.952059 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:38.989532 29786 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:38.952127 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:39.033915 27664 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:38.895783 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:38.952970 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 70968.2us
I20260709 12:18:38.956935 29683 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:38.885389 29785 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.853639 29872 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:38.943073 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:38.870256 29778 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:38.878443 29584 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.072s
I20260709 12:18:38.988272 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 7274fd8a2115405c9b78c61214bff736 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:18:38.988034 29846 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:38.830442 29855 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:39.011879 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138975us
I20260709 12:18:39.027839 29853 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.018150 29719 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:39.024107 29843 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:18:39.024464 29775 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.026476 27734 rpcz_store.cc:276] 0709 12:18:37.942122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.942223 (+   101us) service_pool.cc:224] Handling call
0709 12:18:37.942527 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:37.949197 (+  6670us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.949234 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:37.949725 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.385483 (+435758us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b18001633c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:18:38.483631 (+ 98148us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:38.483667 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.483768 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.025738 (+541970us) raft_consensus.cc:1685] finished
0709 12:18:39.025756 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.025949 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8813,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":47019264,"thread_start_us":97957,"threads_started":1,"wal-append.queue_time_us":668787}
I20260709 12:18:39.028177 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:39.028906 27586 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 3: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:39.036063 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.061020 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:39.093214 27738 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 1.
I20260709 12:18:39.022818 29727 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.047863 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:38.905369 29860 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20260709 12:18:39.031773 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.093740 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.131001 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.094363 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:38.930068 29871 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:38.882921 29888 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.097908 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:39.101322 29723 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.094511 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 7274fd8a2115405c9b78c61214bff736 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-7274fd8a2115405c9b78c61214bff736
I20260709 12:18:39.103402 29713 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:39.103798 29704 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:39.104830 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:39.105747 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 206) took 1111 ms. Trace:
I20260709 12:18:39.107151 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.079805 27729 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:39.121212 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.147084 27735 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:39.108107 29389 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:39.126538 29827 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:39.131817 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 251608us
W20260709 12:18:39.122596 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74840.5us
I20260709 12:18:39.050379 29666 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.135041 27664 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
W20260709 12:18:39.138629 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.056710 29750 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.143255 27656 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:18:39.144863 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.146854 27582 rpcz_store.cc:276] 0709 12:18:37.993914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.024346 (+ 30432us) service_pool.cc:224] Handling call
0709 12:18:39.105556 (+1081210us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:39.105693 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540697344}
I20260709 12:18:39.148242 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.045605 28331 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:39.157742 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:39.158329 27735 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:39.107928 29670 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:18:39.158931 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:39.159055 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.159685 27582 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:39.150315 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:39.133488 27585 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:39.160915 27588 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:39.164024 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:39.154580 27730 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:39.156702 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.156904 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.149083 27589 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:39.142295 29713 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:39.146138 29871 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:39.153787 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:39.151412 29389 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:39.170100 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.165738 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1647) took 1669 ms. Trace:
I20260709 12:18:39.177706 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.146227 29704 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:39.342387 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1641) took 1848 ms. Trace:
I20260709 12:18:39.576342 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:39.479209 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:39.537035 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.583577 27589 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:18:39.633165 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 479931us
I20260709 12:18:39.348299 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:39.352646 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1653) took 1212 ms. Trace:
I20260709 12:18:39.633805 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1652) took 2083 ms. Trace:
I20260709 12:18:39.634068 27552 leader_election.cc:400] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 2. Current term is 3.
I20260709 12:18:39.363940 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:39.179809 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.338874 27660 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:39.635038 27552 leader_election.cc:400] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
W20260709 12:18:39.224927 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.641350 27552 leader_election.cc:403] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:18:39.634186 27584 rpcz_store.cc:276] 0709 12:18:38.139799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.351399 (+1211600us) service_pool.cc:224] Handling call
0709 12:18:39.351761 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.352159 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:39.352208 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:39.352270 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:39.352320 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.352404 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.352556 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15460,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":16643}
I20260709 12:18:39.178985 27582 rpcz_store.cc:276] 0709 12:18:37.495909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.164393 (+1668484us) service_pool.cc:224] Handling call
0709 12:18:39.164830 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.165009 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:39.165040 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:39.165440 (+   400us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:39.165492 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.165559 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.165682 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.645009 27582 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:18:39.500509 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1631) took 1135 ms. Trace:
I20260709 12:18:38.848503 27580 rpcz_store.cc:276] 0709 12:18:35.672773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.966533 (+2293760us) service_pool.cc:224] Handling call
0709 12:18:37.966902 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.967109 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:18:38.230836 (+263727us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.615161 (+384325us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.700028 (+ 84867us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.705408 (+  5380us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:38.720227 (+ 14819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.727567 (+  7340us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.745644 (+ 18077us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":359183,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":3548,"raft.queue_time_us":78354,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":69287,"spinlock_wait_cycles":51590912,"thread_start_us":468746,"threads_started":2,"wal-append.queue_time_us":373770}
I20260709 12:18:39.181545 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:39.287031 29879 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.811s	user 0.011s	sys 0.000s
I20260709 12:18:39.347007 27674 tablet_copy_source_session.cc:215] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:39.657883 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 208) took 1295 ms. Trace:
I20260709 12:18:39.638950 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.179682 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:39.634343 27589 rpcz_store.cc:276] 0709 12:18:37.550789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.178697 (+1627908us) service_pool.cc:224] Handling call
0709 12:18:39.583108 (+404411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.633517 (+ 50409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.633751 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.493094 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.658483 27584 rpcz_store.cc:276] 0709 12:18:38.362555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.643054 (+1280499us) service_pool.cc:224] Handling call
0709 12:18:39.657402 (+ 14348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.657587 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:39.657618 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:39.657661 (+    43us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:39.657698 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.657750 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.657841 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.285889 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:39.336083 27628 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.4:39321 timed out after 1.626s (SENT) [suppressed 29 similar messages]
I20260709 12:18:39.339143 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 219) took 1397 ms. Trace:
I20260709 12:18:39.488819 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1636) took 1083 ms. Trace:
I20260709 12:18:39.502054 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1458) took 1173 ms. Trace:
I20260709 12:18:39.662973 27737 rpcz_store.cc:276] 0709 12:18:37.941596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.941704 (+   108us) service_pool.cc:224] Handling call
0709 12:18:37.952713 (+ 11009us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.487658 (+534945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.338873 (+851215us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:39.339069 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1221155200}
I20260709 12:18:39.663465 27732 rpcz_store.cc:276] 0709 12:18:38.330374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.494452 (+164078us) service_pool.cc:224] Handling call
0709 12:18:39.501838 (+1007386us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:39.501989 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79472896}
I20260709 12:18:39.664566 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.665038 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.665956 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1463) took 1309 ms. Trace:
I20260709 12:18:39.571165 27586 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:39.646072 29460 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:39.227008 27588 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:39.631477 27585 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:39.667919 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1649) took 2159 ms. Trace:
I20260709 12:18:39.668541 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:39.668292 27588 rpcz_store.cc:276] 0709 12:18:37.509242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.164191 (+1654949us) service_pool.cc:224] Handling call
0709 12:18:39.226409 (+ 62218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.667669 (+441260us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.667867 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.670461 29859 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.663110 27662 rpcz_store.cc:276] 0709 12:18:38.405308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.462862 (+ 57554us) service_pool.cc:224] Handling call
0709 12:18:39.488516 (+1025654us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400211454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:39.488732 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":846217472}
W20260709 12:18:39.672016 29728 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.748s	user 0.011s	sys 0.000s
I20260709 12:18:39.672614 29728 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.672919 27662 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:18:39.673029 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1598) took 1611 ms. Trace:
I20260709 12:18:39.674496 27662 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:39.673532 27736 rpcz_store.cc:276] 0709 12:18:38.061922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.062152 (+   230us) service_pool.cc:224] Handling call
0709 12:18:38.068681 (+  6529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.271825 (+203144us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:18:38.706261 (+434436us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.987245 (+280984us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.987345 (+   100us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:38.987419 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.672825 (+685406us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400316d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:39.672871 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.672977 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50974,"prepare.queue_time_us":84748,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":52743,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":466464000,"thread_start_us":627102,"threads_started":2,"wal-append.queue_time_us":316393}
I20260709 12:18:39.674782 29728 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.646387 28994 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:39.657860 29727 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.666328 27732 rpcz_store.cc:276] 0709 12:18:38.356352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.664764 (+1308412us) service_pool.cc:224] Handling call
0709 12:18:39.665888 (+  1124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.675541 28994 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.676538 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.676690 29389 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.677299 27732 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:39.677630 28994 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.677770 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1606) took 1354 ms. Trace:
I20260709 12:18:39.646843 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.678160 27732 rpcz_store.cc:276] 0709 12:18:38.323717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.676273 (+1352556us) service_pool.cc:224] Handling call
0709 12:18:39.677711 (+  1438us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.678475 27738 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:39.678689 29389 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.679391 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.392966 29713 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.680078 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:39.680600 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1464) took 1321 ms. Trace:
I20260709 12:18:39.680889 29899 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:18:39.297971 27731 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.566s	user 0.009s	sys 0.000s
I20260709 12:18:39.681525 29899 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.681975 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:39.680970 27738 rpcz_store.cc:276] 0709 12:18:38.358850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.679141 (+1320291us) service_pool.cc:224] Handling call
0709 12:18:39.680545 (+  1404us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.683599 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.252465 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:39.646610 29892 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.648497 27656 rpcz_store.cc:276] 0709 12:18:38.366228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.366400 (+   172us) service_pool.cc:224] Handling call
0709 12:18:38.366975 (+   575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:39.156163 (+789188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.500162 (+343999us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:39.500427 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1784615808}
W20260709 12:18:39.638665 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 274844us
I20260709 12:18:39.162969 29584 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.495s
W20260709 12:18:39.498742 29673 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.283s	user 0.004s	sys 0.007s
I20260709 12:18:39.693326 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:39.694372 29892 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.648936 29786 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:39.655893 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:39.568683 29764 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.798s	user 0.008s	sys 0.002s
W20260709 12:18:39.501502 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 382) took 2630 ms (client timeout 2011 ms). Trace:
I20260709 12:18:39.495784 29710 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:39.586921 29588 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:39.476555 29845 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:39.658468 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:39.662873 29872 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.252772 29844 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:39.569528 29852 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:39.334704 29802 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:39.699514 29673 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:39.709777 29728 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.713231 29802 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.713744 29728 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.701735 29764 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.717080 29728 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.717622 29764 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.658557 29879 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.702046 29710 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:39.719053 29710 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.719806 29879 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:39.720197 29710 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:39.661000 27735 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:18:39.483877 29824 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:39.662007 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:39.726135 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1607) took 1365 ms. Trace:
I20260709 12:18:39.640213 29797 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:39.180625 29773 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.667269 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 207) took 1319 ms. Trace:
I20260709 12:18:39.726562 27735 rpcz_store.cc:276] 0709 12:18:38.360539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.638596 (+1278057us) service_pool.cc:224] Handling call
0709 12:18:39.726070 (+ 87474us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.727382 27586 rpcz_store.cc:276] 0709 12:18:38.347651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.570590 (+1222939us) service_pool.cc:224] Handling call
0709 12:18:39.570972 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.666987 (+ 96015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.667211 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.728574 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.730757 27737 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:39.684932 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.685144 27664 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:39.735459 27657 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:39.734968 27738 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:39.728977 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:39.631821 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.631705 27587 rpcz_store.cc:276] 0709 12:18:37.494178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.710219 (+1216041us) service_pool.cc:224] Handling call
0709 12:18:38.710526 (+   307us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:38.726191 (+ 15665us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:18:38.744287 (+ 18096us) log.cc:844] Serialized 14 byte log entry
0709 12:18:39.032692 (+288405us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:39.033314 (+   622us) log.cc:844] Serialized 52 byte log entry
0709 12:18:39.033476 (+   162us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:39.033720 (+   244us) log.cc:844] Serialized 14 byte log entry
0709 12:18:39.044426 (+ 10706us) log.cc:844] Serialized 14 byte log entry
0709 12:18:39.044732 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.045057 (+   325us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.342149 (+297092us) raft_consensus.cc:1685] finished
0709 12:18:39.342169 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.342322 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10206,"prepare.queue_time_us":119273,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":6261,"raft.queue_time_us":55408,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":72519,"spinlock_wait_cycles":14160640,"thread_start_us":275414,"threads_started":1,"wal-append.queue_time_us":411948}
I20260709 12:18:39.685751 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1596) took 1648 ms. Trace:
I20260709 12:18:39.737987 27733 rpcz_store.cc:276] 0709 12:18:38.037450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.037608 (+   158us) service_pool.cc:224] Handling call
0709 12:18:38.037914 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.701000 (+663086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.683292 (+982292us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54002e90d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:39.685678 (+  2386us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1317443456}
I20260709 12:18:39.229246 29900 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:39.679101 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.658245 29719 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:39.743678 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1622) took 1065 ms. Trace:
I20260709 12:18:39.744067 27734 rpcz_store.cc:276] 0709 12:18:38.678322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.631552 (+953230us) service_pool.cc:224] Handling call
0709 12:18:39.743597 (+112045us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:39.661553 29676 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.154s	user 0.011s	sys 0.002s
I20260709 12:18:39.651506 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:39.698902 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:39.749331 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50466.3us
I20260709 12:18:39.672976 29732 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.299296 29823 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.583880 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:39.706889 29892 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:39.700549 29389 raft_consensus.cc:2764] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:18:39.694389 29723 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:39.639968 29684 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.165454 29778 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.671969 29893 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:39.700412 28331 rpcz_store.cc:269] 0709 12:18:36.870848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.878025 (+  7177us) service_pool.cc:224] Handling call
0709 12:18:37.850312 (+972287us) spinlock_profiling.cc:243] Waited 5.73 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:39.501242 (+1650930us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400520dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:39.501386 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2456127104}
W20260709 12:18:39.732256 29692 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.013s	user 0.007s	sys 0.006s
W20260709 12:18:39.732697 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.733150 29802 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.922151 29684 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.916735 29893 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:39.924698 29893 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.916728 29732 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:39.736814 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1620) took 1059 ms. Trace:
I20260709 12:18:39.927227 27738 rpcz_store.cc:276] 0709 12:18:38.677780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.684714 (+1006934us) service_pool.cc:224] Handling call
0709 12:18:39.736733 (+ 52019us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.927893 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 390) took 1900 ms. Trace:
I20260709 12:18:39.868520 29842 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:39.928617 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.929170 29842 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.929694 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:39.930248 27582 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:39.930374 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:39.286427 29891 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
I20260709 12:18:39.737608 27586 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:39.702881 29899 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:39.686952 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:39.739598 27732 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:39.712598 29852 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:39.704476 29845 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:39.729224 27730 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:18:39.740896 27587 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:39.741380 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:39.733229 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1612) took 1278 ms (client timeout 1512 ms). Trace:
I20260709 12:18:39.751159 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:39.752573 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101132us
W20260709 12:18:39.962461 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211359us
I20260709 12:18:39.702138 29786 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.795722 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 235) took 1375 ms. Trace:
I20260709 12:18:39.971844 27660 rpcz_store.cc:276] 0709 12:18:38.420557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.476851 (+ 56294us) service_pool.cc:224] Handling call
0709 12:18:39.795495 (+1318644us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54000e56d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:39.795650 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1980234112}
I20260709 12:18:39.859104 27664 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
W20260709 12:18:39.909078 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 325273us
I20260709 12:18:39.727126 29844 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:39.725804 29824 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:39.696692 29704 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:39.981619 28334 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:39.918856 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:39.749477 29676 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:39.931301 29842 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:39.939720 27738 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:39.940784 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1633) took 1206 ms. Trace:
I20260709 12:18:39.940834 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1605) took 1670 ms. Trace:
I20260709 12:18:39.986091 27736 rpcz_store.cc:276] 0709 12:18:38.734011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.675288 (+941277us) service_pool.cc:224] Handling call
0709 12:18:39.940684 (+265396us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.986217 27732 rpcz_store.cc:276] 0709 12:18:38.270743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.678848 (+1408105us) service_pool.cc:224] Handling call
0709 12:18:39.940773 (+261925us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.870208 29728 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:39.869683 29727 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:39.961715 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1682) took 1243 ms. Trace:
I20260709 12:18:40.014385 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:39.961719 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1459) took 1629 ms. Trace:
I20260709 12:18:39.962744 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.014983 27733 rpcz_store.cc:276] 0709 12:18:38.333752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.741036 (+1407284us) service_pool.cc:224] Handling call
0709 12:18:39.961665 (+220629us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.015640 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53018.6us
I20260709 12:18:39.965164 27586 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
W20260709 12:18:39.952173 29762 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.041s	user 0.004s	sys 0.007s
I20260709 12:18:39.980629 27582 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:40.021445 27582 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:40.023000 27582 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:39.984689 27664 raft_consensus.cc:1217] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3]   Dedup: 4.3->[]
I20260709 12:18:40.026373 27664 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:18:39.996848 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 199) took 3693 ms. Trace:
I20260709 12:18:40.014403 27587 rpcz_store.cc:276] 0709 12:18:38.718020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.738032 (+1020012us) service_pool.cc:224] Handling call
0709 12:18:39.738398 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.961382 (+222984us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.961649 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.027417 27581 rpcz_store.cc:276] 0709 12:18:36.303530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.704159 (+2400629us) service_pool.cc:224] Handling call
0709 12:18:38.705533 (+  1374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.706276 (+   743us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.706301 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:38.706766 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.857683 (+150917us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:38.857716 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.857782 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.996516 (+1138734us) raft_consensus.cc:1685] finished
0709 12:18:39.996548 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.996763 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":110674,"threads_started":1,"wal-append.queue_time_us":204873}
I20260709 12:18:39.998703 28994 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:40.030952 28994 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:40.031147 27587 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:40.008250 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.032295 28334 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:40.006304 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:39.982115 29460 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:40.035964 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.037231 27734 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:18:40.037492 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 701384us
I20260709 12:18:40.016609 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:39.918762 29692 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:39.987454 29906 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.397s	user 0.001s	sys 0.000s
I20260709 12:18:40.029539 27582 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
W20260709 12:18:40.455107 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 438306us
I20260709 12:18:40.497166 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.035990 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:40.500291 27588 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:40.455993 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:40.473071 27655 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:40.473071 27657 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:40.084218 27664 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:40.509102 29764 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:40.511420 27664 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:40.511704 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.473994 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:40.512935 29764 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.513358 27660 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:40.475772 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:40.515677 27663 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:40.476358 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:40.516757 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.517007 27662 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:40.518787 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.477470 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:40.473089 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 497661us
W20260709 12:18:40.077123 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160118us
I20260709 12:18:40.521967 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:40.522451 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.033731 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:40.524188 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.525390 27738 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:39.964116 27731 rpcz_store.cc:269] 0709 12:18:38.455186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.470621 (+ 15435us) service_pool.cc:224] Handling call
0709 12:18:39.733029 (+1262408us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:39.733161 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589572352}
W20260709 12:18:40.525617 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1610) took 2117 ms (client timeout 2167 ms). Trace:
I20260709 12:18:40.525792 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1468) took 1466 ms. Trace:
I20260709 12:18:40.526207 27738 rpcz_store.cc:276] 0709 12:18:39.059567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.523550 (+1463983us) service_pool.cc:224] Handling call
0709 12:18:40.525714 (+  2164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.526852 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.527287 27736 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:40.527989 27731 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:39.993458 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1456) took 1672 ms (client timeout 2200 ms). Trace:
I20260709 12:18:40.527825 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1461) took 2187 ms. Trace:
I20260709 12:18:40.080499 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.084662 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:40.158754 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.529476 29784 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:40.529866 27733 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:40.480634 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:40.530444 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1623) took 1851 ms (client timeout 2448 ms). Trace:
W20260709 12:18:40.366503 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:40.092916 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91250.6us
I20260709 12:18:40.135931 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.270601 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.114343 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.155862 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.117431 28336 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:40.536912 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:40.537986 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 453252us
I20260709 12:18:40.538570 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.490470 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:40.136211 28337 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260709 12:18:40.528712 27730 rpcz_store.cc:269] 0709 12:18:38.319889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.273125 (+953236us) service_pool.cc:224] Handling call
0709 12:18:39.991890 (+718765us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.541038 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.542454 29873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7274fd8a2115405c9b78c61214bff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:40.542580 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.126627 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.554822 28334 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:39.996595 29389 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.081s
I20260709 12:18:40.557592 29897 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:40.558393 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.558768 29897 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.088403 29460 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:40.562012 29848 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:18:40.562438 29897 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:40.369640 29588 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.664s	user 0.008s	sys 0.004s
I20260709 12:18:40.282969 29676 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.357160 29684 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:40.282025 29692 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:40.015800 29704 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.565712 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 6, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 6 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:40.382005 29710 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.661s	user 0.008s	sys 0.004s
I20260709 12:18:40.338229 29713 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:40.569662 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 3 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:40.395141 29719 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.647s	user 0.011s	sys 0.000s
I20260709 12:18:40.572006 29719 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:40.019397 29723 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:40.573494 29723 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.074888 29727 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.085s
I20260709 12:18:40.030948 29728 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.031489 29762 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.288983 29771 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:40.093509 29773 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:40.336438 29785 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:40.256565 29786 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:40.361467 29797 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.623s	user 0.010s	sys 0.000s
I20260709 12:18:40.198007 29802 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:40.602425 29802 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.388232 29823 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:40.081573 29824 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.074090 29827 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.606597 29824 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:40.097448 29844 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.078626 29845 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.392522 29852 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:40.327365 29871 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.108s	user 0.014s	sys 0.000s
W20260709 12:18:40.136211 29882 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:18:40.330304 29888 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:40.079279 29890 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:40.271832 29893 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:40.289420 29909 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:18:40.680413 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1683) took 1957 ms. Trace:
W20260709 12:18:40.362807 29914 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.277s	user 0.003s	sys 0.000s
W20260709 12:18:40.497309 29924 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:18:40.680894 27586 rpcz_store.cc:276] 0709 12:18:38.722973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.016879 (+1293906us) service_pool.cc:224] Handling call
0709 12:18:40.017327 (+   448us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:40.026135 (+  8808us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:40.026166 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:40.030098 (+  3932us) log.cc:844] Serialized 52 byte log entry
0709 12:18:40.462091 (+431993us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b18002b7d40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:18:40.598841 (+136750us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:40.598910 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.599018 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:40.679864 (+ 80846us) raft_consensus.cc:1685] finished
0709 12:18:40.679881 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.680198 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8288,"raft.queue_time_us":12550,"raft.run_cpu_time_us":6521,"raft.run_wall_time_us":58917,"spinlock_wait_cycles":3696000,"thread_start_us":57217,"threads_started":1,"wal-append.queue_time_us":597812}
W20260709 12:18:40.503163 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47654.5us
I20260709 12:18:40.474355 27659 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:40.507910 29365 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:40.508471 29778 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.005s	sys 0.010s
I20260709 12:18:40.508736 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:40.527825 29900 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:40.528606 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1641) took 1376 ms. Trace:
I20260709 12:18:40.536470 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.538645 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.689670 28328 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:40.685506 29900 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.079402 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.691892 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:40.664228 27487 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:40.693154 29890 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:39.928326 28330 rpcz_store.cc:276] 0709 12:18:38.027644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.036391 (+  8747us) service_pool.cc:224] Handling call
0709 12:18:38.473127 (+436736us) spinlock_profiling.cc:243] Waited 55.4 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:38.473339 (+   212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.750361 (+277022us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.758413 (+  8052us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.761826 (+  3413us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:38.763987 (+  2161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.927604 (+1163617us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:39.927670 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.927829 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":743300096}
I20260709 12:18:40.341341 28994 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.693616 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:40.693861 29890 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:40.694839 28994 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:40.695919 29890 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:40.525641 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 491834us
I20260709 12:18:40.548830 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.528635 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:40.530880 27733 rpcz_store.cc:269] 0709 12:18:38.678587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.035503 (+1356916us) service_pool.cc:224] Handling call
0709 12:18:40.530360 (+494857us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.535228 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.549810 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.551496 28337 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:40.536912 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.553496 29856 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:40.526150 27737 rpcz_store.cc:269] 0709 12:18:38.407958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.664333 (+1256375us) service_pool.cc:224] Handling call
0709 12:18:40.525141 (+860808us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:40.525383 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70802816}
I20260709 12:18:40.562582 29460 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.583841 29723 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.551208 29670 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:40.693573 27487 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 4, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 4 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:40.592510 29684 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:40.592592 29727 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:40.595362 29762 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.598210 29764 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:40.605000 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.606513 29827 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:40.633806 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112167us
W20260709 12:18:40.636749 29922 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:18:40.637423 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:40.637557 29773 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:40.616598 29809 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:40.637132 29676 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.640053 29389 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.640364 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:40.640727 29797 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.641479 29732 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:40.642521 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:40.708253 29670 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:40.644070 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.644690 29870 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:18:40.651275 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:40.649165 29802 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:40.639005 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.725972 27589 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:40.730923 27487 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 5, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:40.733177 29389 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:40.663535 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.668310 27580 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:40.667609 29588 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.735931 29389 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.736176 29727 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.736291 29588 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.736827 29797 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.666256 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.691172 29900 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:40.530824 27734 raft_consensus.cc:2368] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
W20260709 12:18:40.739614 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1637) took 1682 ms (client timeout 2074 ms). Trace:
I20260709 12:18:40.739894 27730 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:18:40.739995 27734 rpcz_store.cc:269] 0709 12:18:39.059522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.046310 (+986788us) service_pool.cc:224] Handling call
0709 12:18:40.739522 (+693212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.740586 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1640) took 1592 ms. Trace:
W20260709 12:18:40.696226 29778 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.006s	sys 0.010s
I20260709 12:18:40.741040 27730 rpcz_store.cc:276] 0709 12:18:39.148163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.604485 (+1456322us) service_pool.cc:224] Handling call
0709 12:18:40.740497 (+136012us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.696007 27655 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:40.528774 27736 rpcz_store.cc:276] 0709 12:18:38.340565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.521888 (+2181323us) service_pool.cc:224] Handling call
0709 12:18:40.527767 (+  5879us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.695039 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:40.742415 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.743211 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.744093 28333 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:40.744211 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1608) took 2380 ms. Trace:
I20260709 12:18:40.744249 29827 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:40.744593 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:40.725749 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1671) took 1030 ms. Trace:
I20260709 12:18:40.744782 27736 rpcz_store.cc:276] 0709 12:18:38.364196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.742851 (+2378655us) service_pool.cc:224] Handling call
0709 12:18:40.744148 (+  1297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.745229 27584 rpcz_store.cc:276] 0709 12:18:39.691771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.695984 (+  4213us) service_pool.cc:224] Handling call
0709 12:18:40.528254 (+832270us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:40.722236 (+193982us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41827200}
I20260709 12:18:40.745914 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.746332 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:40.693961 27729 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:40.590441 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.726220 29670 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:40.746652 27736 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.749085 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:40.749245 29588 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.749925 27738 raft_consensus.cc:2376] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6. Re-sending same reply.
I20260709 12:18:40.750504 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1642) took 1595 ms. Trace:
I20260709 12:18:40.717168 29809 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:40.746857 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:40.751688 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.752085 27584 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:40.752307 29809 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.747707 29775 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:40.747709 29778 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:40.753420 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.753623 29809 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.753795 29778 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.754364 29778 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:40.718942 27732 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:40.755580 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.756062 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 228) took 1112 ms. Trace:
I20260709 12:18:40.755716 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 2, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:18:40.728811 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86467.1us
I20260709 12:18:40.725540 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.723095 29856 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.757488 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:40.757773 29786 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:18:40.757483 27737 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:40.758648 29786 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:40.758873 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1613) took 2269 ms (client timeout 2926 ms). Trace:
I20260709 12:18:40.615867 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:40.731812 29676 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:40.759346 27737 rpcz_store.cc:269] 0709 12:18:38.489511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.709805 (+2220294us) service_pool.cc:224] Handling call
0709 12:18:40.758793 (+ 48988us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.759758 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 170292us
I20260709 12:18:40.760483 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.761430 29676 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.761873 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.762110 29676 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:40.762578 29676 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.762981 29676 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:40.694723 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1645) took 3199 ms. Trace:
I20260709 12:18:40.785403 27583 rpcz_store.cc:276] 0709 12:18:37.495333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.710616 (+1215283us) service_pool.cc:224] Handling call
0709 12:18:38.710872 (+   256us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:39.042609 (+331737us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:18:39.162315 (+119706us) log.cc:844] Serialized 14 byte log entry
0709 12:18:39.752167 (+589852us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:39.752699 (+   532us) log.cc:844] Serialized 26 byte log entry
0709 12:18:39.752861 (+   162us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:39.753123 (+   262us) log.cc:844] Serialized 14 byte log entry
0709 12:18:39.753307 (+   184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.753385 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:40.214984 (+461599us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b1800348400. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:40.270016 (+ 55032us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b1800348400. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:18:40.685379 (+415363us) raft_consensus.cc:1685] finished
0709 12:18:40.694258 (+  8879us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.694505 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1978,"prepare.queue_time_us":56389,"prepare.run_cpu_time_us":204,"prepare.run_wall_time_us":4278,"raft.queue_time_us":62069,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":60083,"spinlock_wait_cycles":361151104,"thread_start_us":880741,"threads_started":3,"wal-append.queue_time_us":794251}
I20260709 12:18:40.787528 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1668) took 1142 ms. Trace:
I20260709 12:18:40.788003 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:40.789446 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:40.790191 27583 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:40.790377 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:40.743299 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48372.1us
I20260709 12:18:40.684654 27731 rpcz_store.cc:276] 0709 12:18:39.152107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.526408 (+1374301us) service_pool.cc:224] Handling call
0709 12:18:40.528533 (+  2125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.782861 27584 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:40.793756 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.794528 27731 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:40.795610 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:40.796285 27655 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: [5.6-6.7]
I20260709 12:18:40.696043 29917 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:40.797217 29917 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.797497 27736 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:40.797636 29917 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:40.798004 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1617) took 2178 ms. Trace:
I20260709 12:18:40.798388 27736 rpcz_store.cc:276] 0709 12:18:38.619596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.745544 (+2125948us) service_pool.cc:224] Handling call
0709 12:18:40.797942 (+ 52398us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.713018 27585 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:40.799808 27736 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:18:40.800432 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1460) took 2462 ms. Trace:
I20260709 12:18:40.746973 29802 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:40.756552 27733 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:40.747570 29876 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:40.732756 29732 consensus_queue.cc:1243] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:40.730854 29684 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:40.744249 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 104138us
I20260709 12:18:40.731614 29773 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.763613 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.763083 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:40.765102 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.830982 28334 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:40.765235 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:40.767625 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.767868 29797 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.759207 29856 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:40.768026 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.817338 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:40.846951 27731 raft_consensus.cc:606] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:40.847432 27731 raft_consensus.cc:614] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:40.847584 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:40.847942 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 6) took 1976 ms. Trace:
I20260709 12:18:40.848340 27656 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:40.848311 27731 rpcz_store.cc:276] 0709 12:18:38.878780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.795286 (+1916506us) service_pool.cc:224] Handling call
0709 12:18:40.847844 (+ 52558us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.752646 29900 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.770005 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:40.770700 29871 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:40.771605 29389 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:40.861498 29389 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:40.862097 29389 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:18:40.863884 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 238523us
I20260709 12:18:40.771621 28328 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:40.773535 29778 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.870440 29778 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:40.726719 27586 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:40.872516 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:40.873849 27731 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:18:40.874032 27586 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:40.874855 29728 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:40.739228 29727 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:40.780082 29927 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:18:40.756450 27732 rpcz_store.cc:276] 0709 12:18:39.643862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.549327 (+905465us) service_pool.cc:224] Handling call
0709 12:18:40.756008 (+206681us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.783595 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1614) took 2281 ms (client timeout 3016 ms). Trace:
W20260709 12:18:40.876040 27730 rpcz_store.cc:269] 0709 12:18:38.501885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.741969 (+2240084us) service_pool.cc:224] Handling call
0709 12:18:40.783529 (+ 41560us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.876698 27732 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:40.877321 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1615) took 2366 ms. Trace:
I20260709 12:18:40.877552 27730 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:40.877611 27732 rpcz_store.cc:276] 0709 12:18:38.510561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.876032 (+2365471us) service_pool.cc:224] Handling call
0709 12:18:40.876401 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.877069 (+   668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.877256 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.878244 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1626) took 2199 ms. Trace:
I20260709 12:18:40.784678 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.794350 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:40.750844 27738 rpcz_store.cc:276] 0709 12:18:39.155560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.527271 (+1371711us) service_pool.cc:224] Handling call
0709 12:18:40.750447 (+223176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.790973 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:40.788033 27580 rpcz_store.cc:276] 0709 12:18:39.645373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.662637 (+ 17264us) service_pool.cc:224] Handling call
0709 12:18:40.490217 (+827580us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223f8c 00007f0e97223ef6 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:40.787265 (+297048us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b54000bd1d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:40.787441 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110363392}
I20260709 12:18:40.807281 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1643) took 1645 ms. Trace:
I20260709 12:18:40.880224 27733 rpcz_store.cc:276] 0709 12:18:39.160899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.724959 (+1564060us) service_pool.cc:224] Handling call
0709 12:18:40.806103 (+ 81144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.800935 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:40.828334 29460 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:18:40.829272 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71907.7us
I20260709 12:18:40.843706 27734 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:40.845086 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:40.892210 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46410.4us
I20260709 12:18:40.847132 28330 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:40.853816 27735 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 2.
I20260709 12:18:40.894816 27657 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:40.854954 29365 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:40.855597 29900 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:40.855959 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.856267 29824 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:18:40.858475 29684 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.859223 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.859941 27588 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.860252 27737 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:40.902009 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 153079us
I20260709 12:18:40.902540 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1646) took 1715 ms. Trace:
I20260709 12:18:40.807428 29852 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:40.867159 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1621) took 2189 ms (client timeout 1846 ms). Trace:
I20260709 12:18:40.903038 27737 rpcz_store.cc:276] 0709 12:18:39.187293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.762790 (+1575497us) service_pool.cc:224] Handling call
0709 12:18:40.902463 (+139673us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.867913 29876 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:40.868039 29845 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:40.875259 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1462) took 2526 ms. Trace:
I20260709 12:18:40.800768 27736 rpcz_store.cc:276] 0709 12:18:38.337979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.799101 (+2461122us) service_pool.cc:224] Handling call
0709 12:18:40.799469 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:40.800165 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.800356 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.892786 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1644) took 1730 ms. Trace:
I20260709 12:18:40.894039 27732 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:40.894232 27586 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:40.910569 27734 rpcz_store.cc:276] 0709 12:18:39.162804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.740973 (+1578169us) service_pool.cc:224] Handling call
0709 12:18:40.892678 (+151705us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.891484 27483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 229) took 9072 ms. Trace:
I20260709 12:18:40.878685 27730 rpcz_store.cc:276] 0709 12:18:38.679317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.876787 (+2197470us) service_pool.cc:224] Handling call
0709 12:18:40.877178 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.878016 (+   838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.878198 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.903606 27729 rpcz_store.cc:269] 0709 12:18:38.678055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.492755 (+814700us) service_pool.cc:224] Handling call
0709 12:18:40.866951 (+1374196us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001d15d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:40.867080 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2353083136}
I20260709 12:18:40.909596 27731 rpcz_store.cc:276] 0709 12:18:38.348739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.849317 (+2500578us) service_pool.cc:224] Handling call
0709 12:18:40.873503 (+ 24186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.874208 (+   705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.875054 (+   846us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b54001d15d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:40.875176 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45543680}
I20260709 12:18:40.916028 27580 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:18:40.918859 27584 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:40.912776 27583 raft_consensus.cc:2376] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7. Re-sending same reply.
I20260709 12:18:40.923949 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1467) took 1873 ms. Trace:
I20260709 12:18:40.924602 27734 rpcz_store.cc:276] 0709 12:18:39.050884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.914616 (+1863732us) service_pool.cc:224] Handling call
0709 12:18:40.922943 (+  8327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.923403 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:40.923447 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:40.923515 (+    68us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:40.923585 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.923693 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.923848 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.926431 29389 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.927625 27736 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [1.2-3.3]
I20260709 12:18:40.929086 27589 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:40.930847 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 225) took 2206 ms. Trace:
I20260709 12:18:40.931182 27732 rpcz_store.cc:276] 0709 12:18:38.710605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.878849 (+2168244us) service_pool.cc:224] Handling call
0709 12:18:40.879161 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.915159 (+ 35998us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.917147 (+  1988us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.934865 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:40.936074 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1639) took 1803 ms. Trace:
I20260709 12:18:40.936645 27736 rpcz_store.cc:276] 0709 12:18:39.125555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.926026 (+1800471us) service_pool.cc:224] Handling call
0709 12:18:40.927041 (+  1015us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:40.928190 (+  1149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.928546 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.900965 29809 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.907112 29460 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:40.944391 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:40.941483 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:40.932485 29827 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:40.940322 29684 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:18:40.956082 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 171550us
I20260709 12:18:40.964607 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:40.959901 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 100868us
I20260709 12:18:40.967626 29900 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:40.968945 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:40.969551 29876 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:40.970702 27627 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:40.971643 27583 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:40.973140 29684 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:18:40.963593 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51794.5us
I20260709 12:18:40.974304 27583 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:40.975651 29365 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:40.975579 29824 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:18:40.948001 29871 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:40.977706 29775 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:40.971112 27627 leader_election.cc:403] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:40.974092 29704 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:40.984527 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:40.987695 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:40.990171 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48961.6us
I20260709 12:18:40.984560 27733 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:40.940426 29870 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:18:40.995484 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1635) took 1958 ms. Trace:
I20260709 12:18:40.996038 27733 rpcz_store.cc:276] 0709 12:18:39.037188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.895781 (+1858593us) service_pool.cc:224] Handling call
0709 12:18:40.942812 (+ 47031us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.995085 (+ 52273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.995406 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.998878 27737 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:40.994540 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 226) took 2277 ms. Trace:
I20260709 12:18:41.002070 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:41.000648 27738 rpcz_store.cc:276] 0709 12:18:38.710921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.895417 (+2184496us) service_pool.cc:224] Handling call
0709 12:18:40.940468 (+ 45051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.987585 (+ 47117us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:40.987635 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:40.987698 (+    63us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:40.987733 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.987790 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.987934 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.953277 29389 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:40.999882 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1466) took 1950 ms. Trace:
I20260709 12:18:41.005255 27737 rpcz_store.cc:276] 0709 12:18:39.049576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.917568 (+1867992us) service_pool.cc:224] Handling call
0709 12:18:40.954779 (+ 37211us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.999515 (+ 44736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.999808 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.998101 29870 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:41.007645 27729 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:41.009753 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.009769 27732 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:41.011149 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1647) took 1791 ms. Trace:
I20260709 12:18:41.011422 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:41.011621 27732 rpcz_store.cc:276] 0709 12:18:39.219730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.939358 (+1719628us) service_pool.cc:224] Handling call
0709 12:18:40.994076 (+ 54718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.010803 (+ 16727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.011080 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.021322 27738 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:41.024823 27588 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:41.025465 29917 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.026528 27589 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:18:41.029505 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1616) took 2420 ms (client timeout 2053 ms). Trace:
I20260709 12:18:41.031544 27581 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:18:40.975132 29852 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260709 12:18:41.032966 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50945us
I20260709 12:18:41.032608 29460 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.031570 27583 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:41.033543 29778 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:18:41.043588 29778 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.044703 29778 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.046476 29778 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:18:41.033720 29684 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:41.047657 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:40.990942 29704 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:41.024897 27580 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:18:41.036266 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1638) took 1926 ms. Trace:
I20260709 12:18:40.981437 29809 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.968941 29900 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20260709 12:18:41.038614 27735 rpcz_store.cc:269] 0709 12:18:38.609005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.728192 (+1119187us) service_pool.cc:224] Handling call
0709 12:18:41.029251 (+1301059us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:41.029413 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2155030272}
I20260709 12:18:41.051138 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:41.052867 27729 rpcz_store.cc:276] 0709 12:18:39.109920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.932728 (+1822808us) service_pool.cc:224] Handling call
0709 12:18:40.980831 (+ 48103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.008272 (+ 27441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.036182 (+ 27910us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.009830 29824 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:41.007721 29870 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:41.062654 29848 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:41.062656 27731 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:41.063476 29870 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:41.064646 29932 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.236s	user 0.004s	sys 0.000s
I20260709 12:18:41.065079 29870 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.063155 29848 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.065982 29848 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:41.055809 29727 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260709 12:18:41.057742 29890 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.067656 29890 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.065629 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:41.058182 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:41.061491 29732 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5.
I20260709 12:18:41.071171 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:41.054155 29704 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:40.998613 29670 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:40.911362 27483 rpcz_store.cc:276] 0709 12:18:31.595829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.603423 (+  7594us) service_pool.cc:224] Handling call
0709 12:18:31.854463 (+251040us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a350f3a 00007f0ea0106e9c 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc 00007f0e9e6c5392 00007f0e9868348c 00007f0e986828c6
0709 12:18:40.661472 (+8807009us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:34.771786 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:34.772468 (+   682us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:34.772518 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:18:34.772555 (+    37us) tablet.cc:662] Decoding operations
0709 12:18:34.785915 (+ 13360us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:34.786003 (+    88us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:34.786048 (+    45us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:18:35.287971 (+501923us) tablet.cc:701] Row locks acquired
0709 12:18:35.287997 (+    26us) write_op.cc:260] PREPARE: finished
0709 12:18:35.288190 (+   193us) write_op.cc:270] Start()
0709 12:18:35.288306 (+   116us) write_op.cc:276] Timestamp: P: 1783599515288137 usec, L: 0
0709 12:18:35.288328 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:18:35.364305 (+ 75977us) log.cc:844] Serialized 9787 byte log entry
0709 12:18:35.601047 (+236742us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e9723e0ac 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b
0709 12:18:36.661146 (+1060099us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:37.328051 (+666905us) op_driver.cc:464] REPLICATION: finished
0709 12:18:37.884102 (+556051us) write_op.cc:301] APPLY: starting
0709 12:18:37.884204 (+   102us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:37.886609 (+  2405us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:37.886635 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:37.934384 (+ 47749us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:37.934729 (+   345us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=25
0709 12:18:37.934782 (+    53us) write_op.cc:312] APPLY: finished
0709 12:18:37.935228 (+   446us) log.cc:844] Serialized 170 byte log entry
0709 12:18:37.948327 (+ 13099us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:37.996094 (+ 47767us) write_op.cc:454] Released schema lock
0709 12:18:37.996170 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:37.996589 (+   419us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:37.996619 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10228,"reported_tablets":39,"spinlock_wait_cycles":70554368,"child_traces":[["op",{"apply.queue_time_us":534063,"apply.run_cpu_time_us":5871,"apply.run_wall_time_us":112477,"num_ops":25,"prepare.queue_time_us":583142,"prepare.run_cpu_time_us":39415,"prepare.run_wall_time_us":1061253,"raft.queue_time_us":497537,"raft.run_cpu_time_us":42323,"raft.run_wall_time_us":607440,"replication_time_us":1990209,"spinlock_wait_cycles":174096896,"thread_start_us":1331884,"threads_started":4,"wal-append.queue_time_us":620903,"wal-append.run_cpu_time_us":16730,"wal-append.run_wall_time_us":2996066}]]}
I20260709 12:18:41.075258 29676 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.076154 29673 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:41.076380 29893 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 12:18:41.051120 28329 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:41.070195 29890 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:41.084921 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 144847us
I20260709 12:18:41.079437 29900 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:41.078625 29809 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:41.092886 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.085739 29389 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:41.086481 29673 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.098562 29890 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.100862 29917 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:41.102488 29934 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:18:41.102986 29676 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:41.103888 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.126945 27589 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.110971 29786 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:41.098496 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:41.112030 29875 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:41.081287 29588 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:18:41.112391 28994 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:18:41.113195 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4073864590000085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
I20260709 12:18:41.113822 29900 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:41.128036 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46362.6us
I20260709 12:18:41.128183 29852 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:41.112089 29827 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.129942 28994 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.131626 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.130439 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:41.134313 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:41.135583 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 165085us
I20260709 12:18:41.134325 29875 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:41.099361 29389 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:41.147720 29917 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:41.148128 29389 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:41.148473 29917 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:41.149804 29917 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.151472 29365 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.152093 29365 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.154376 29365 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:41.144685 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50809.9us
I20260709 12:18:41.146489 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:41.147490 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.124570 29673 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.156174 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:41.129834 29786 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:41.165370 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.154922 29365 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.140304 29676 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:41.178175 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 90180.1us
I20260709 12:18:41.183120 29809 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:41.183657 29824 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:18:41.183956 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48472.1us
I20260709 12:18:41.185210 29824 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:41.186211 29824 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:41.186621 29824 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.187681 29775 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:41.187633 29824 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.189388 29824 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.190126 29888 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:41.192445 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:41.193835 29809 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:41.195560 29809 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:41.196491 29809 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:41.199514 29809 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:41.201335 29809 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:41.205744 27583 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:41.216970 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.218199 27583 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:41.221314 27734 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:41.220451 28994 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.158674 29852 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.227355 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.230190 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.224939 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.224175 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:41.241276 27729 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:41.249878 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.141218 29827 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.160279 29890 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.255244 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.238641 29809 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:41.179646 29365 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.173446 29900 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:41.260895 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87848.6us
I20260709 12:18:41.262072 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.264134 27731 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:41.266382 27732 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:41.266834 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.267791 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:41.269253 27735 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.263790 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.276271 27737 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:41.276491 27658 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:41.226575 29676 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.253607 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:41.281046 28994 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:41.284621 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:41.284750 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.287911 27732 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:41.288108 27656 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 1. Current term is 2. Ops: []
I20260709 12:18:41.290383 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.296860 29670 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:41.298633 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 229) took 1654 ms. Trace:
I20260709 12:18:41.301486 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.304930 29936 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.123s
I20260709 12:18:41.307551 29365 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:41.306551 29676 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:41.309278 27732 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:41.312129 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.311363 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.314580 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.315017 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.315891 27664 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:41.315779 29852 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:41.319427 29939 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 12:18:41.319712 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.319931 29845 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:41.320367 29845 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:41.320693 27658 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:41.260113 29786 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.321576 27655 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:18:41.321738 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.322460 29786 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:41.323203 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:41.324473 29900 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:41.322600 27660 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:41.327769 28332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 432) took 1270 ms (client timeout 1634 ms). Trace:
I20260709 12:18:41.299100 27736 rpcz_store.cc:276] 0709 12:18:39.644154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.942030 (+1297876us) service_pool.cc:224] Handling call
0709 12:18:41.001387 (+ 59357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.071052 (+ 69665us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:18:41.129398 (+ 58346us) log.cc:844] Serialized 14 byte log entry
0709 12:18:41.297940 (+168542us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b180040b1c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a
0709 12:18:41.298200 (+   260us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:41.298257 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:41.298306 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.298390 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:41.298548 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":72177,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":3456,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"spinlock_wait_cycles":7957248,"thread_start_us":81829,"threads_started":2,"wal-append.queue_time_us":137787}
W20260709 12:18:41.328262 28332 rpcz_store.cc:269] 0709 12:18:40.014006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.014142 (+   136us) service_pool.cc:224] Handling call
0709 12:18:41.279273 (+1265131us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:41.284650 (+  5377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2338286464}
I20260709 12:18:41.323966 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.259503 29809 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.331491 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.333887 27733 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
W20260709 12:18:41.344328 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:41.362198 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138153us
I20260709 12:18:41.364784 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.365810 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:41.332368 27588 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:41.368000 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.368945 27729 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:41.374378 27730 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:41.375720 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.377066 27736 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:41.384366 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.391892 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.392773 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.239423 29875 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.397171 27580 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:41.372010 29728 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:18:41.404482 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 311892us
I20260709 12:18:41.384651 28330 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:41.412081 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.412953 27731 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.403905 27588 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 1
I20260709 12:18:41.420768 27589 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:41.424033 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 230) took 1779 ms. Trace:
I20260709 12:18:41.423616 27735 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
W20260709 12:18:41.423468 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 231107us
I20260709 12:18:41.426811 27738 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:41.427276 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:41.429013 27585 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:18:41.429344 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.425312 27730 rpcz_store.cc:276] 0709 12:18:39.644429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.943099 (+1298670us) service_pool.cc:224] Handling call
0709 12:18:41.109084 (+165985us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:41.109451 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.378531 (+269080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.423756 (+ 45225us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:41.423949 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":615962368}
I20260709 12:18:41.413821 27588 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:18:41.430459 27588 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:41.431070 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.431694 27580 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:41.432767 29713 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:41.433236 29713 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.428741 29773 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.434165 29713 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.434240 29773 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.330719 29673 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.422252 29845 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:41.456113 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 170180us
I20260709 12:18:41.456312 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.458926 27589 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:41.460184 28332 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:18:41.461258 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.461938 29871 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:41.462550 29871 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.462406 29809 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.463666 29775 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.463784 29871 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.464371 29775 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.461436 27584 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
W20260709 12:18:41.436074 29944 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:18:41.425381 29778 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.440923 29845 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.435307 29713 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:41.482331 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94765.9us
W20260709 12:18:41.498838 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58027us
I20260709 12:18:41.438289 29773 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:41.502933 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.507722 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:41.479430 29948 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:18:41.508805 27580 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:41.509569 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:41.511260 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84108.6us
I20260709 12:18:41.431668 29389 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.473234 29673 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:41.440534 29945 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260709 12:18:41.521975 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.439283 29676 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 12:18:41.522468 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:41.524485 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.479131 29827 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:41.439564 29588 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 12:18:41.527132 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:41.500046 29778 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.480728 29762 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:18:41.530309 29844 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:18:41.532064 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.534142 28994 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:41.499944 29824 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.534437 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:41.536849 28994 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:41.537122 27580 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:41.537463 28994 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.548660 27552 leader_election.cc:400] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:41.534420 29890 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:41.536285 29844 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:41.544478 29809 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.561968 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:41.546780 29950 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:41.548439 29797 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.562994 29873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58246 (request call id 393) took 3514 ms. Trace:
I20260709 12:18:41.563482 29827 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:41.587231 28994 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.400825 29933 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:41.598063 28994 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:41.562731 29824 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.583940 29732 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:41.600773 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82918.7us
I20260709 12:18:41.592338 29960 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:18:41.603492 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77942.9us
I20260709 12:18:41.615862 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:41.602090 29732 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:41.621318 29952 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:18:41.622900 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:41.624639 29762 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:41.625976 29890 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:18:41.587925 29873 rpcz_store.cc:276] 0709 12:18:38.049658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.065409 (+ 15751us) service_pool.cc:224] Handling call
0709 12:18:38.463566 (+398157us) ts_tablet_manager.cc:934] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
0709 12:18:40.540314 (+2076748us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:41.068352 (+528038us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:41.068483 (+   131us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:41.439021 (+370538us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:41.467329 (+ 28308us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:41.523213 (+ 55884us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:41.543431 (+ 20218us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":147947,"tablet-copy.queue_time_us":298084,"thread_start_us":238921,"threads_started":2}
I20260709 12:18:41.601857 29950 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:41.585953 29797 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.641834 29945 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.641961 29873 tablet_copy_client.cc:806] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
W20260709 12:18:41.645526 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.657229 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.658102 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.621289 29732 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:41.663972 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.664971 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:41.666253 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.667246 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.608021 29960 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:41.627413 29778 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:41.673491 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.636344 29950 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.676856 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:41.673496 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 149722us
I20260709 12:18:41.676155 27729 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:18:41.676396 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.689075 27733 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:41.676596 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:41.676625 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.676643 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.626880 29365 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:18:41.646601 29775 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:41.660778 29673 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:41.695760 27656 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:41.655598 29917 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:18:41.698529 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:41.686722 27729 pending_rounds.cc:85] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 4
I20260709 12:18:41.706732 27729 pending_rounds.cc:107] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:18:41.707099 27729 raft_consensus.cc:2889] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:41.702633 27655 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:18:41.632638 29844 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.677417 29460 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.631901 29827 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.693213 29389 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:41.660926 29945 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:41.712652 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.687763 29950 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.739410 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:41.712468 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:41.742837 28332 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:41.718196 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:41.756883 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.757584 27733 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:41.709776 29873 tablet_copy_client.cc:670] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
W20260709 12:18:41.767388 29971 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:18:41.770678 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.774977 29960 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.775877 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.776223 29960 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.777685 27582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:41.778321 27582 raft_consensus.cc:606] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:41.778707 27582 raft_consensus.cc:614] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:41.778484 29950 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.779448 29954 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:18:41.809504 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:41.810776 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 248904us
I20260709 12:18:41.816897 27735 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.780930 29890 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:18:41.818334 29890 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:41.809772 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.780263 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:41.779610 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:41.823141 27737 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.822959 29871 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:41.827625 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:41.828466 29871 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:41.828627 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 150135us
I20260709 12:18:41.821826 27732 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:41.810235 27731 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:41.820218 28328 raft_consensus.cc:2376] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6. Re-sending same reply.
I20260709 12:18:41.786909 27587 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:41.723996 29900 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:41.724628 29673 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:41.824708 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107989us
I20260709 12:18:41.828893 29871 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.781816 27582 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:41.827067 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:41.784917 29870 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.719s	user 0.014s	sys 0.000s
W20260709 12:18:41.840611 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:41.787773 29961 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
W20260709 12:18:41.786121 29848 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.720s	user 0.013s	sys 0.000s
I20260709 12:18:41.836556 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:41.846163 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.846321 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:41.836733 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:41.776502 27584 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.793066 29824 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:41.744192 29460 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:41.838778 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.840273 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:41.784322 29797 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:41.833470 29684 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:41.833535 29389 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:41.838888 29953 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:41.851046 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:41.858997 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.859615 27580 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:41.860071 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:41.854065 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:41.846812 27582 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:41.857522 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:41.841150 29871 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:41.863127 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:41.863940 29389 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:41.865458 29727 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:41.847728 29917 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:41.866462 27583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:56174
I20260709 12:18:41.886184 27583 raft_consensus.cc:606] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:41.886663 27583 raft_consensus.cc:614] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:41.866843 29945 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:41.867770 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206566us
I20260709 12:18:41.867946 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:41.868557 29969 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.096s	user 0.000s	sys 0.002s
I20260709 12:18:41.869108 29673 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.858379 29954 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.855038 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.877897 27584 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:41.899173 27581 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:41.880587 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:41.881494 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:41.855270 29684 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:41.890156 27589 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:18:41.895457 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:41.896049 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:41.902963 27735 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:41.872185 29844 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.839664 29900 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.907213 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:41.908161 28334 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:41.913357 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.894431 29727 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.824590 29890 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:41.894735 29870 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:41.951946 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.853024 29848 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:41.902447 29893 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:41.959113 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:41.961966 27659 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:41.961943 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:41.964246 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.960539 27662 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:41.966548 27660 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:41.967485 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51489.3us
I20260709 12:18:41.901221 29988 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:18:41.959564 29929 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:18:41.979970 27735 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
W20260709 12:18:41.980161 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 139931us
I20260709 12:18:41.980520 27655 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.952991 29809 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:41.957757 29727 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.982887 28336 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:41.986176 27581 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:41.953887 29827 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:41.955364 29870 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.975802 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:41.971627 29969 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:18:41.965476 29950 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:42.003587 29704 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.913s	user 0.012s	sys 0.005s
I20260709 12:18:41.986575 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:42.006747 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 146725us
I20260709 12:18:42.010149 29684 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.011062 27581 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:42.011564 29684 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:41.983459 29988 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.988381 29917 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:41.992125 29809 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.990162 29893 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:42.017700 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:42.023155 27583 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:18:42.018378 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.016126 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:42.021605 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45877.4us
I20260709 12:18:41.972364 29824 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 12:18:42.007833 29870 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:42.025676 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71349.6us
I20260709 12:18:42.028012 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:42.024704 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:42.016784 29460 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:42.016582 29827 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:42.024905 29809 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:42.100301 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72392us
I20260709 12:18:42.191449 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:42.019358 29917 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:42.036562 29977 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:18:42.326962 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:42.028645 29775 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:42.017414 29988 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:42.044128 29797 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.241s
W20260709 12:18:42.292214 29984 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:18:42.036989 29732 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.028973 29890 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:42.031509 29389 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:42.311741 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 187187us
I20260709 12:18:42.313462 27659 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
W20260709 12:18:42.324985 29933 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.576s	user 0.007s	sys 0.007s
I20260709 12:18:42.048005 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:42.029561 29856 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:42.318622 27655 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:42.319185 27731 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:42.327454 29917 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:42.014882 29704 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.022370 29982 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:42.332309 29775 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.356663 29775 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:42.357494 29965 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:42.357926 29965 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:42.358839 29965 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.359957 29965 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:42.361176 29727 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.363813 29848 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:42.366190 29960 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.011s
W20260709 12:18:42.341041 27701 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.682s (SENT)
I20260709 12:18:42.349117 27731 raft_consensus.cc:614] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:18:42.347714 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 299746us
I20260709 12:18:42.350626 29983 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:42.035904 29893 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.354763 27658 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
W20260709 12:18:42.355628 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44297.4us
I20260709 12:18:42.372712 27734 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:42.373497 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:42.322271 27737 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:42.384095 29873 tablet_copy_client.cc:538] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:42.383025 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:42.353789 29982 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.386114 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.376657 29888 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.394758 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:42.395687 27657 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:42.399585 29967 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:18:42.400956 29845 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:42.401055 29978 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:18:42.401428 29845 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:42.402403 29973 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.634s	user 0.000s	sys 0.016s
I20260709 12:18:42.402571 29845 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.404253 29845 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:42.408162 27657 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:42.392302 29941 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:18:42.410400 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 433643us
I20260709 12:18:42.389478 29982 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:42.434307 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53334.2us
I20260709 12:18:42.412241 29983 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:42.435742 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.453100 27581 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:42.436664 29727 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:42.454896 29987 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:18:42.452453 29809 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.460804 29809 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:42.465745 29676 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:42.466027 29732 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:42.466249 29676 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:42.467285 29954 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.561s	user 0.004s	sys 0.011s
I20260709 12:18:42.467823 29954 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:42.468539 29954 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.469060 29954 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.471055 27580 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:42.470304 29870 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.441313 29990 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:42.475467 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 445) took 1164 ms. Trace:
I20260709 12:18:42.475878 28336 rpcz_store.cc:276] 0709 12:18:41.311531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.322928 (+ 11397us) service_pool.cc:224] Handling call
0709 12:18:42.475234 (+1152306us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:42.475397 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1513391488}
I20260709 12:18:42.480211 29704 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:42.407239 29888 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:42.529736 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94120.4us
W20260709 12:18:42.481684 29844 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.574s	user 0.009s	sys 0.001s
I20260709 12:18:42.426967 29941 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:42.487215 29973 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:18:42.543565 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 525201us
I20260709 12:18:42.544571 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:42.533037 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1712) took 1088 ms. Trace:
W20260709 12:18:42.542493 29684 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.530s	user 0.012s	sys 0.004s
W20260709 12:18:42.555724 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 206772us
I20260709 12:18:42.485801 29954 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:42.556829 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.540457 29732 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:42.567312 29870 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:42.569415 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:42.558928 29871 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.662s	user 0.009s	sys 0.001s
I20260709 12:18:42.574323 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:42.582023 27737 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:42.584550 29365 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:42.580531 29389 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:18:42.585927 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:42.583515 29684 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.589372 29684 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.592366 29873 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:42.592684 28329 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:18:42.593164 28329 pending_rounds.cc:85] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 6
I20260709 12:18:42.593608 28329 pending_rounds.cc:107] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:18:42.591724 29684 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:42.593894 28329 raft_consensus.cc:2889] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:42.595974 29844 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:42.597137 29995 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:42.604372 29365 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:42.604624 29888 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:42.605746 29365 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.609678 29732 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.610477 29732 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.611488 29941 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.614202 29732 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:42.611585 29888 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:42.615767 29888 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.617163 29900 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:42.618677 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:42.622226 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.623311 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:18:42.626966 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.712318 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:42.716010 27582 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:42.636973 29871 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.467303 29676 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.713161 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 232) took 1489 ms. Trace:
W20260709 12:18:42.713845 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.548388 27585 rpcz_store.cc:276] 0709 12:18:41.444654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.526371 (+ 81717us) service_pool.cc:224] Handling call
0709 12:18:41.543440 (+ 17069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.561290 (+ 17850us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:18:41.847223 (+285933us) log.cc:844] Serialized 14 byte log entry
0709 12:18:42.231213 (+383990us) log.cc:844] Serialized 14 byte log entry
0709 12:18:42.532507 (+301294us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:42.532605 (+    98us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:42.532682 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.532789 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:42.532957 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36171,"prepare.queue_time_us":75346,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":3734,"raft.queue_time_us":361295,"thread_start_us":815970,"threads_started":4,"wal-append.queue_time_us":168611}
I20260709 12:18:42.714323 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1685) took 1351 ms. Trace:
W20260709 12:18:42.736371 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 181011us
I20260709 12:18:42.714354 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
W20260709 12:18:42.717679 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.725394 29670 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.727941 29875 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.727703 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:42.728502 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:42.738356 27735 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:42.738356 27583 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:42.729563 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1483) took 1093 ms. Trace:
I20260709 12:18:42.734720 27734 rpcz_store.cc:276] 0709 12:18:41.223595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.229907 (+  6312us) service_pool.cc:224] Handling call
0709 12:18:42.648714 (+1418807us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400316d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:42.713068 (+ 64354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2625025792}
I20260709 12:18:42.737455 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.743616 27738 rpcz_store.cc:276] 0709 12:18:41.635895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.636052 (+   157us) service_pool.cc:224] Handling call
0709 12:18:41.642248 (+  6196us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:41.642698 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:41.642744 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:41.656362 (+ 13618us) log.cc:844] Serialized 26 byte log entry
0709 12:18:41.764384 (+108022us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18002c7ac0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1
0709 12:18:42.729135 (+964751us) spinlock_profiling.cc:243] Waited 8.26 ms on lock 0x7b54000472a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a2bf6b8 00007f0e9a34892b 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c
0709 12:18:42.729204 (+    69us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:42.729250 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.729317 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:42.729361 (+    44us) raft_consensus.cc:1685] finished
0709 12:18:42.729373 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:42.729484 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6302,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":9222,"spinlock_wait_cycles":23619712,"thread_start_us":15585,"threads_started":1,"wal-append.queue_time_us":35289}
I20260709 12:18:42.736009 27587 rpcz_store.cc:276] 0709 12:18:41.362539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.374029 (+ 11490us) service_pool.cc:224] Handling call
0709 12:18:42.630590 (+1256561us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:42.714254 (+ 83664us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":906817920}
I20260709 12:18:42.713689 29762 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:42.764182 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1686) took 1383 ms. Trace:
I20260709 12:18:42.764652 27588 rpcz_store.cc:276] 0709 12:18:41.380686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.382216 (+  1530us) service_pool.cc:224] Handling call
0709 12:18:41.403669 (+ 21453us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:41.478549 (+ 74880us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:41.478602 (+    53us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:41.479487 (+   885us) log.cc:844] Serialized 78 byte log entry
0709 12:18:41.641590 (+162103us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:41.674488 (+ 32898us) log.cc:844] Serialized 14 byte log entry
0709 12:18:41.683374 (+  8886us) log.cc:844] Serialized 14 byte log entry
0709 12:18:41.683834 (+   460us) log.cc:844] Serialized 14 byte log entry
0709 12:18:41.684046 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.684156 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:41.789031 (+104875us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b1800348460. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f
0709 12:18:41.790958 (+  1927us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:18:42.401925 (+610967us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b240013aaf8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:18:42.763915 (+361990us) raft_consensus.cc:1685] finished
0709 12:18:42.763940 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:42.764114 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5910,"prepare.queue_time_us":228971,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":5933,"raft.run_cpu_time_us":9267,"raft.run_wall_time_us":163042,"spinlock_wait_cycles":53835520,"thread_start_us":161717,"threads_started":1,"wal-append.queue_time_us":288089}
I20260709 12:18:42.767263 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:42.769843 29989 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:42.770255 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57484.5us
I20260709 12:18:42.771386 27627 leader_election.cc:400] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3.
I20260709 12:18:42.762490 29875 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.769505 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.762601 29670 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.773337 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.773749 29875 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:42.774184 29670 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:42.769493 27585 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:42.775411 29786 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:42.774623 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:42.775156 29990 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:42.778178 29995 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:42.717000 29844 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:42.778602 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:42.779222 29844 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.780575 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:42.781273 27587 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:42.732878 29704 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.781980 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:42.782779 29704 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.784368 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:42.782274 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:42.758693 27735 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
W20260709 12:18:42.798712 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.738859 29933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:36344 (request call id 9) took 13874 ms. Trace:
I20260709 12:18:42.808673 27589 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:18:42.782337 29995 raft_consensus.cc:2764] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:18:42.817107 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:42.818224 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:42.822566 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:42.746747 29809 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:18:42.829173 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:42.856171 27582 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:42.830690 27589 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:42.803175 29786 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:42.824746 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:42.786366 29762 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:42.802726 29888 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:42.812445 29890 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.856055 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:42.879808 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.881248 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.882577 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.882604 29827 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:42.750629 29978 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:18:42.862375 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:42.890280 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119119us
I20260709 12:18:42.835120 29704 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:42.843685 29871 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:42.889127 29988 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.540s	user 0.010s	sys 0.000s
I20260709 12:18:42.907945 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:42.912886 27580 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:42.885740 29827 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.892251 29888 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.897922 29954 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:42.910270 29989 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:42.934206 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 197415us
W20260709 12:18:42.934441 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 197464us
I20260709 12:18:42.943092 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:42.912537 29704 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.947549 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:42.948251 27580 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:18:42.931740 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:42.929054 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:42.951264 27588 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:42.898213 29998 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:42.898124 29786 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:42.942878 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.892488 29890 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.930680 29827 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:42.956758 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:42.933779 29990 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:42.930271 29988 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:42.932889 29871 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.961979 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:42.963981 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:42.964587 29936 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:42.965057 29997 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:42.961683 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:42.941617 29888 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:42.970538 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:42.954015 29995 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:42.987816 27582 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:42.916059 29996 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:18:42.952487 29989 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:42.914172 29986 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:18:42.954538 29998 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:43.005013 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:43.006999 29999 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:18:43.007547 30005 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:18:43.010666 27586 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:42.962918 29876 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:43.017460 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:43.026000 27588 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
W20260709 12:18:42.986040 29775 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.629s	user 0.013s	sys 0.000s
I20260709 12:18:42.987392 29802 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:18:42.998346 29917 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.624s	user 0.010s	sys 0.002s
I20260709 12:18:42.987725 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:43.054765 29917 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.011444 29998 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:42.995944 29670 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:42.813376 29933 rpcz_store.cc:276] 0709 12:18:28.842705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.322175 (+1479470us) service_pool.cc:224] Handling call
0709 12:18:30.339169 (+ 16994us) log.cc:844] Serialized 499 byte log entry
0709 12:18:30.639507 (+300338us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b180034f6c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a2a1b0b 00007f0e9a326b63 00007f0e9a34e752 00007f0e9f30ed29
0709 12:18:42.673967 (+12034460us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":127078,"raft.queue_time_us":189626,"raft.run_cpu_time_us":2627,"raft.run_wall_time_us":176000,"spinlock_wait_cycles":4358528,"thread_start_us":140419,"threads_started":1,"wal-append.queue_time_us":235811,"wal-append.run_cpu_time_us":5762,"wal-append.run_wall_time_us":1011399}
I20260709 12:18:43.009910 29786 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.032169 29809 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.048501 29990 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:43.011581 29797 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
W20260709 12:18:43.049268 29845 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.641s	user 0.010s	sys 0.001s
I20260709 12:18:43.032126 29876 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:43.050864 30014 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:43.053836 30015 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.057657 27660 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:43.057577 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:43.080344 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 123408us
I20260709 12:18:42.968338 29704 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.011183 29989 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:42.987928 29732 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:18:42.985401 29871 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:43.109486 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 130715us
I20260709 12:18:43.035964 29775 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.112068 29995 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:43.112506 29460 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:43.114105 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96761.4us
I20260709 12:18:43.035413 29893 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:43.058301 29802 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.130515 29876 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:42.963126 29762 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:43.201783 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 144297us
I20260709 12:18:43.123884 29670 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:43.090606 29809 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.091866 29990 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:43.308069 27701 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.510s (SENT)
W20260709 12:18:43.271991 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 283553us
I20260709 12:18:43.351408 27701 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:43.316167 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 262) took 1287 ms. Trace:
I20260709 12:18:43.348162 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 408) took 1273 ms. Trace:
W20260709 12:18:43.260375 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 348773us
I20260709 12:18:43.156427 29917 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:43.312700 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1723) took 1407 ms. Trace:
I20260709 12:18:43.360212 27586 rpcz_store.cc:276] 0709 12:18:41.905265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.026773 (+121508us) service_pool.cc:224] Handling call
0709 12:18:42.027197 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:43.028119 (+1000922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.312435 (+284316us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b5400186fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:43.312633 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2227886336}
I20260709 12:18:43.356105 27660 rpcz_store.cc:276] 0709 12:18:42.029258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.029733 (+   475us) service_pool.cc:224] Handling call
0709 12:18:42.035681 (+  5948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:43.141372 (+1105691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.315851 (+174479us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:43.316061 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2349851008}
I20260709 12:18:43.369242 29996 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:43.369704 29996 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:43.370607 29996 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.372035 29996 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:43.372548 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.259212 30003 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.268s	user 0.000s	sys 0.000s
I20260709 12:18:43.289386 30026 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:43.309170 29982 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.868s	user 0.006s	sys 0.006s
I20260709 12:18:43.379581 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:43.135421 30010 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:43.152556 29704 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:43.384716 30007 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.371s	user 0.000s	sys 0.001s
I20260709 12:18:43.385118 29972 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:43.352840 29802 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:43.186609 30014 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:43.389853 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:43.354187 28334 rpcz_store.cc:276] 0709 12:18:42.047696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.047874 (+   178us) service_pool.cc:224] Handling call
0709 12:18:42.104102 (+ 56228us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:42.312345 (+208243us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:42.716900 (+404555us) log.cc:844] Serialized 14 byte log entry
0709 12:18:43.320424 (+603524us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:43.320503 (+    79us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:43.320566 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.320761 (+   195us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:43.320927 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4262,"prepare.queue_time_us":578774,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":61044,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":966844,"threads_started":3}
I20260709 12:18:43.361565 29844 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.413353 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
W20260709 12:18:43.415360 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.374908 29988 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:43.376924 29762 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:43.418939 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 110925us
W20260709 12:18:43.418371 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.421355 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.425094 29670 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.425629 29941 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:18:43.426910 29953 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:43.385927 29890 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:43.447635 30039 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.448244 30039 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.448909 29809 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.450402 30039 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:43.451589 30013 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:18:43.436082 29844 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:43.459167 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81517.8us
I20260709 12:18:43.390743 29460 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.419620 30010 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:43.187120 30015 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.430778 29713 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:43.464723 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:43.453354 29844 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:43.489306 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.459746 29933 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:43.392109 29982 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.530102 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:43.531005 29982 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:43.531803 30010 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.391779 30026 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.464084 29809 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:43.534507 29809 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.511207 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:43.535853 27735 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:43.516185 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:18:43.517066 30008 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:18:43.517433 28336 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:18:43.517966 30031 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:18:43.516232 30032 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.523138 30015 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:43.519889 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140705us
I20260709 12:18:43.520212 30038 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.522362 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:43.505829 29713 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.523600 30014 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.526587 29704 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:43.528827 29844 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:43.532279 30036 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.515684 30035 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:43.655254 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.538028 27735 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:43.655700 30035 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.555460 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:43.655298 30036 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.555934 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:43.657572 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101718us
I20260709 12:18:43.657960 30035 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:43.658565 30043 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:43.606598 27738 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:18:43.659072 30043 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.659227 30036 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.519469 30029 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.662038 30029 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:43.607897 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 89885.9us
I20260709 12:18:43.641994 27732 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:18:43.664834 27660 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:43.666493 27656 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:18:43.670300 30001 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.794s	user 0.000s	sys 0.002s
W20260709 12:18:43.673136 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.673702 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.481921 29888 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.679800 30038 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.682221 30038 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:43.542490 29972 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.683153 29972 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.511849 29945 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:18:43.687525 29945 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.687937 29945 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:18:43.690164 29870 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:43.690582 29870 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.690902 29870 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:43.691852 29775 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.611739 30026 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.693498 29965 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.694007 29965 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.535028 29809 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.470695 29827 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.705219 29827 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:43.789666 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:43.674969 29985 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:43.806551 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:43.750959 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:43.772346 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 423322us
W20260709 12:18:43.779554 27701 leader_election.cc:336] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.670s (SENT)
I20260709 12:18:43.804886 27586 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:43.809967 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:43.713629 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 452) took 1114 ms. Trace:
W20260709 12:18:43.779062 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61164.1us
I20260709 12:18:43.527156 29998 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.811731 27582 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:43.817646 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:43.473174 29995 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.820753 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:43.816994 28332 rpcz_store.cc:276] 0709 12:18:42.599059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.599199 (+   140us) service_pool.cc:224] Handling call
0709 12:18:42.599659 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:42.600095 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:42.600142 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:42.600701 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:18:42.886222 (+285521us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:42.886312 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.886417 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:43.713334 (+826917us) raft_consensus.cc:1685] finished
0709 12:18:43.713368 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:43.713553 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1450,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"thread_start_us":284955,"threads_started":1,"wal-append.queue_time_us":437729}
I20260709 12:18:43.826642 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:43.811895 29933 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.757537 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:43.715185 29890 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.711138 29893 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:43.853277 29996 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:43.817757 30001 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.147s	user 0.000s	sys 0.004s
I20260709 12:18:43.761649 29954 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:43.812727 29676 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:43.857450 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77938.1us
I20260709 12:18:43.739831 29365 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:43.706759 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:43.812445 29985 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:43.813707 29871 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.819123 29727 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:43.847661 29684 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.776445 29941 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
W20260709 12:18:43.925777 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.641057 30032 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.834386 29960 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
W20260709 12:18:43.898695 27701 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.908s (SENT)
I20260709 12:18:43.827702 29876 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.847860 29978 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:43.911767 27732 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:43.750007 29967 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
W20260709 12:18:43.911934 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94370.5us
W20260709 12:18:43.913635 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.919281 27582 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:43.920784 28334 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:43.826534 29673 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
W20260709 12:18:43.901809 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81168.4us
I20260709 12:18:43.951315 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:43.889869 29933 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.901968 27656 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:18:43.899294 29983 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:43.932559 27735 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:43.935276 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:43.929462 29809 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:43.902385 29954 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.919991 29727 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.912897 30043 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:43.926916 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 282331us
I20260709 12:18:43.973428 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:43.925149 30038 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:43.939011 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:43.906406 29996 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:43.950651 30016 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:18:43.955827 30047 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.890292 29802 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.894749 29890 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.926976 29888 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.926496 29965 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.906664 29732 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 12:18:43.923305 28330 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:43.960121 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 116659us
I20260709 12:18:43.933313 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:43.914496 29365 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:43.940250 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:43.938844 30053 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:43.932444 29684 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:43.929121 30026 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.077260 29978 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.979095 27734 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:43.957470 29972 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:43.983422 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:43.964124 30020 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:43.982378 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.088778 27582 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:44.089725 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.090376 27582 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:18:43.977573 29460 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:43.973192 29727 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:44.092053 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:44.045419 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63688.1us
I20260709 12:18:43.938607 30021 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.166s
I20260709 12:18:44.051170 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.097920 27580 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:44.058904 27701 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.609s (SENT)
I20260709 12:18:44.067802 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:44.067847 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 127662us
I20260709 12:18:44.043912 29936 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:44.056999 30029 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:44.071509 29670 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260709 12:18:44.081926 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.115398 28332 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:44.066936 29713 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:44.118580 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59747us
I20260709 12:18:44.082662 30055 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.083178 29684 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.084694 27586 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:44.045221 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.122306 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.073128 30050 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.091434 30026 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.107522 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.130945 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.066576 30019 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.047439 29809 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:18:44.078231 30053 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.093729 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:43.980536 29985 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.123234 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:44.123203 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.127091 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.129268 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.043731 30047 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.130211 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.156261 27735 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:44.156668 27584 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:44.131590 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:44.088115 30010 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.061131 29888 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.135313 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.110106 30020 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:44.152797 27738 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 12:18:44.108377 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:44.153271 27737 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:44.153216 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:44.130390 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.157725 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.165410 27584 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:44.125216 30043 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.167485 30043 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.169636 30043 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.158535 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:44.170132 30062 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:18:44.134750 30050 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.171371 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.172868 30050 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:44.130868 30029 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.164630 27582 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:44.164726 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:44.176456 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.179677 27735 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:44.138291 29998 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.175901 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:44.181998 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73713.3us
I20260709 12:18:44.182196 27734 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:44.176065 27580 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:18:44.188633 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:44.192626 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.154855 30053 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:44.088411 29978 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.200515 27582 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:44.163352 29986 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:18:44.183282 29989 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.911s	user 0.011s	sys 0.000s
I20260709 12:18:44.181439 29827 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:44.169638 29996 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:44.213963 27701 leader_election.cc:336] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.672s (SENT)
I20260709 12:18:44.154819 30028 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.259s
I20260709 12:18:44.220086 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:44.220089 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:44.149989 30019 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.185949 29936 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:44.224143 30063 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:18:44.224236 30030 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:44.164534 30010 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.190655 29876 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.131186 30055 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.214722 29998 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:44.218075 29827 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.186230 30043 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:18:44.185935 30029 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.231667 29460 raft_consensus.cc:2764] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260709 12:18:44.215489 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.224890 30030 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:44.245117 29873 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:18:44.213915 30045 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:18:44.214418 29989 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.238269 30021 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.241665 29998 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.240242 29809 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:44.247901 29827 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:44.246920 29996 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:44.251737 30058 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:18:44.248605 30064 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:44.251902 30020 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.304574 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:44.577410 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:44.554030 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.252744 30030 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:44.438773 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.308503 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:44.617249 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:44.300997 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.254832 30027 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:44.622404 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.313329 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:44.259333 30054 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:18:44.633705 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:44.345929 29990 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.980s	user 0.011s	sys 0.000s
W20260709 12:18:44.598928 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.629168 27656 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:44.630302 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:44.639454 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.639789 27660 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:44.593724 29993 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.631930 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:44.304811 30056 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
W20260709 12:18:44.352716 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.650s (SENT) [suppressed 20 similar messages]
I20260709 12:18:44.609222 29936 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:44.274508 29890 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.635432 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:44.627211 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.635915 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:44.274540 27535 auto_rebalancer.cc:629] Failed to schedule move for tablet b8ac8041c87e49308784e1cad6aeec03: Illegal state: Replica 3f90188458da46f6ba138d5f82c55b69 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:44.629091 27658 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:44.575433 29365 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.553854 29991 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:44.407449 30065 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
W20260709 12:18:44.305166 30060 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
W20260709 12:18:44.586086 30071 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
W20260709 12:18:44.365988 30070 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260709 12:18:44.640893 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 357012us
I20260709 12:18:44.573868 30045 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.645519 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:44.351821 30059 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:44.646813 29993 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.647142 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:44.649223 30061 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.760s	user 0.006s	sys 0.008s
I20260709 12:18:44.688316 27657 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:44.623422 29996 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.688973 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.689182 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.259450 30044 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.690764 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:44.426954 29888 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.577481 30033 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:44.352759 30038 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:44.667547 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 314833us
I20260709 12:18:44.685745 30029 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.667096 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 410) took 1153 ms. Trace:
I20260709 12:18:44.703498 27582 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:44.686609 30059 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:44.281842 29989 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.281066 30055 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.353178 29933 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.615753 29998 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:44.285737 30012 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:44.547408 29995 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:44.675690 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.675913 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.395954 30019 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.625360 30064 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:44.310926 29954 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.574471 30032 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.279747 30026 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:44.675132 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.679872 29991 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:44.671957 29945 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.984s	user 0.009s	sys 0.003s
W20260709 12:18:44.677390 29870 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.986s	user 0.010s	sys 0.002s
I20260709 12:18:44.572625 29876 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:44.581143 29727 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.689745 27658 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
W20260709 12:18:44.685398 29676 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.778s	user 0.008s	sys 0.004s
I20260709 12:18:44.280192 30021 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.680811 29917 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:44.694460 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.690505 29936 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:18:44.695827 29684 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:44.696453 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.738842 27662 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:44.741452 27585 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:44.697365 29985 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:44.746281 29991 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.697836 29993 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.642939 29990 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.575736 29762 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.737421 29876 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.702593 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.734697 29954 raft_consensus.cc:2764] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:18:44.702714 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:44.693480 28330 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.609s	user 0.011s	sys 0.000s
I20260709 12:18:44.703707 28334 rpcz_store.cc:276] 0709 12:18:43.513452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.515365 (+  1913us) service_pool.cc:224] Handling call
0709 12:18:43.515713 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:43.950176 (+434463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:44.666737 (+716561us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:44.667014 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":931096960}
I20260709 12:18:44.608822 29978 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:44.700320 30047 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.741353 29848 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.735780 29870 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.694727 30044 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.707628 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:44.707991 30053 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.767932 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:44.713515 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.718165 27657 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:44.719412 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:44.779495 27655 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:44.679983 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.722239 27660 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:44.722994 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:44.723969 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.724691 30059 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.727581 30064 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.729907 29996 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.730448 30030 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:44.731761 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56678.1us
I20260709 12:18:44.721251 29875 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:44.691540 30045 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.719764 30012 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.742506 27581 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:44.743880 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:43.901436 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:18:44.744473 30077 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:18:44.744822 29873 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260709 12:18:44.807391 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.808157 27661 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:44.808557 29873 tablet_bootstrap.cc:492] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:44.808880 30027 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:44.809118 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94853.9us
I20260709 12:18:44.809331 29873 ts_tablet_manager.cc:1403] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 2.217s	user 0.043s	sys 0.016s
I20260709 12:18:44.734913 30055 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.696959 30033 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:44.735903 29945 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.747152 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.752686 30021 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.753279 29684 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:44.755515 27582 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:44.758809 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.767160 27586 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:44.701934 30038 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:44.760048 27656 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:44.780319 27588 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:44.805058 30012 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:44.763785 28328 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:44.768345 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.786420 30059 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:44.733762 29995 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.781723 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:44.817852 30053 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:44.789222 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.755836 29762 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.794416 27660 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:44.755290 29990 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.848608 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.849275 28331 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:44.852134 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:44.758213 29876 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.802561 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.760392 28330 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:44.857226 28335 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:18:44.806476 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:44.761893 29848 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.811762 27735 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:44.757130 29802 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.784757 29954 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:44.785212 29870 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.792233 29983 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:44.791950 29875 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:44.796691 29460 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:44.871009 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:44.830536 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:44.817370 27482 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 2 to 3, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:44.872387 27733 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:44.830724 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.872692 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.809348 30027 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:44.838197 27582 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:44.879280 27482 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 2 to 3, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:44.880276 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.880990 27582 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:44.881733 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:44.881919 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.843379 28336 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:44.883136 27582 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 12:18:44.884949 29875 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.885761 29727 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:44.842796 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74519.6us
I20260709 12:18:44.888101 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.888460 27586 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:18:44.888846 30030 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.801673 27655 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:44.845646 30026 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:18:44.867370 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.889520 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:44.889866 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.890085 27586 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:18:44.889559 30026 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:44.890650 30026 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:18:44.890676 30037 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:44.891253 30037 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:44.814710 29993 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:18:44.891649 30037 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:44.891974 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:44.873241 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:44.871536 27587 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:44.762080 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 121) took 12175 ms (client timeout 15000 ms). Trace:
I20260709 12:18:44.874272 29990 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:44.893841 29990 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.893635 29684 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:44.895867 27583 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:44.874358 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.896351 30038 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.897712 30038 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:44.827464 29676 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:18:44.899358 28332 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:44.899958 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 454) took 1073 ms. Trace:
I20260709 12:18:44.900312 28336 rpcz_store.cc:276] 0709 12:18:43.826197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.826357 (+   160us) service_pool.cc:224] Handling call
0709 12:18:44.899798 (+1073441us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x7b540024a254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:44.899912 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2282913024}
I20260709 12:18:44.875695 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.901638 27729 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:44.874936 29960 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:44.876993 30033 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.888916 30053 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:18:44.818921 30064 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:44.873565 27734 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:44.894232 29990 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:44.820834 30055 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:44.899142 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 455) took 1071 ms. Trace:
I20260709 12:18:44.874846 29954 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:44.949954 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63364.3us
I20260709 12:18:44.951277 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:44.902802 27701 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.026s (SENT)
I20260709 12:18:44.876570 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.918154 29875 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:44.909497 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:44.965260 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55819.6us
I20260709 12:18:44.965752 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.966262 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:44.910176 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103737us
I20260709 12:18:44.917222 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:44.968920 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51834.3us
I20260709 12:18:44.969696 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:44.887166 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 407) took 1165 ms. Trace:
I20260709 12:18:44.903780 27482 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 3, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 3 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:44.924237 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.934703 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:44.976670 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:44.977355 27583 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:44.939556 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:44.977444 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:44.977883 27729 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:44.865522 29762 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:44.865571 30045 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:44.945792 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:44.953234 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:44.961344 27587 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:44.962319 27586 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:44.996328 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.952131 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:44.907797 27582 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:18:44.918780 29991 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:44.898752 30010 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:44.972702 27662 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:44.972606 27733 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:44.925760 30053 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.014710 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:44.948273 29983 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:44.967862 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:45.022217 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54448.2us
I20260709 12:18:45.023926 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:44.984758 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:44.991688 27738 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:18:44.992787 27736 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:45.025064 27656 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 3.
I20260709 12:18:44.935732 29945 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:18:44.999006 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:45.026008 27582 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:18:45.018363 27482 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:44.973119 29978 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:44.999788 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:44.983965 30029 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:45.006136 27735 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
W20260709 12:18:45.006445 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152521us
I20260709 12:18:45.042932 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.043727 27729 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:44.971275 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:45.009122 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.010010 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:45.011534 27586 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:18:45.012619 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.045205 27657 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:45.003074 29965 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:45.052704 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:45.012508 29848 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:45.001662 29991 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:45.017072 29802 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:45.013324 29673 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:45.028201 27589 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:45.039901 29888 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:44.945577 28328 rpcz_store.cc:276] 0709 12:18:43.827626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.890153 (+ 62527us) service_pool.cc:224] Handling call
0709 12:18:44.898919 (+1008766us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54003705d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:44.899081 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1616531200}
W20260709 12:18:45.040941 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138162us
I20260709 12:18:45.065243 29991 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:45.046295 27482 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 3, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:45.048388 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:44.893679 27487 rpcz_store.cc:269] 0709 12:18:32.586892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.587145 (+   253us) service_pool.cc:224] Handling call
0709 12:18:44.761999 (+12174854us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:41.058986 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:41.075815 (+ 16829us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:41.075877 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:18:41.075899 (+    22us) tablet.cc:662] Decoding operations
0709 12:18:41.076908 (+  1009us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:41.076990 (+    82us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:41.077022 (+    32us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:18:41.077804 (+   782us) tablet.cc:701] Row locks acquired
0709 12:18:41.077862 (+    58us) write_op.cc:260] PREPARE: finished
0709 12:18:41.078057 (+   195us) write_op.cc:270] Start()
0709 12:18:41.078227 (+   170us) write_op.cc:276] Timestamp: P: 1783599521077969 usec, L: 0
0709 12:18:41.078249 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:18:41.099837 (+ 21588us) log.cc:844] Serialized 2434 byte log entry
0709 12:18:41.330179 (+230342us) op_driver.cc:464] REPLICATION: finished
0709 12:18:41.585543 (+255364us) write_op.cc:301] APPLY: starting
0709 12:18:41.596248 (+ 10705us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:41.643041 (+ 46793us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:41.696341 (+ 53300us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:41.763665 (+ 67324us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:41.763972 (+   307us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0709 12:18:41.764020 (+    48us) write_op.cc:312] APPLY: finished
0709 12:18:41.764358 (+   338us) log.cc:844] Serialized 53 byte log entry
0709 12:18:41.764534 (+   176us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:41.764658 (+   124us) write_op.cc:454] Released schema lock
0709 12:18:41.764737 (+    79us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:41.765198 (+   461us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:41.765231 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":677948,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":221565,"apply.run_cpu_time_us":3906,"apply.run_wall_time_us":195853,"num_ops":6,"prepare.queue_time_us":217907,"prepare.run_cpu_time_us":11965,"prepare.run_wall_time_us":87212,"raft.queue_time_us":180555,"raft.run_cpu_time_us":43875,"raft.run_wall_time_us":146273,"replication_time_us":251727,"spinlock_wait_cycles":1762176,"thread_start_us":407933,"threads_started":4,"wal-append.queue_time_us":28338,"wal-append.run_cpu_time_us":50297,"wal-append.run_wall_time_us":1638842}]]}
W20260709 12:18:45.049923 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 78716.6us
I20260709 12:18:45.001204 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:45.043504 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:45.058142 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48224.8us
W20260709 12:18:45.094015 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50670.5us
W20260709 12:18:45.062947 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.019759 30053 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:18:44.973945 29917 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260709 12:18:45.002179 27581 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:44.966701 29873 raft_consensus.cc:359] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 NON_PARTICIPANT]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:45.075125 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.085639 29673 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.077631 27582 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:18:44.976348 28330 rpcz_store.cc:276] 0709 12:18:43.706151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.706364 (+   213us) service_pool.cc:224] Handling call
0709 12:18:44.871085 (+1164721us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:44.991432 29827 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.028380 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:45.012818 30044 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:45.027452 29995 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:45.043174 30045 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:45.063441 30010 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:45.012938 30052 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:18:45.045848 29762 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:45.050367 30029 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.087584 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:45.142979 27731 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:45.089139 27701 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.472s (SENT)
I20260709 12:18:45.143821 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:45.140342 30045 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.987154 30028 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:45.001816 30034 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:18:45.092895 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:45.071758 29965 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:45.094633 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:45.102082 27729 raft_consensus.cc:2368] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:45.105779 29917 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:18:45.105778 30082 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:18:45.129807 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.144953 30029 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:45.133036 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.072680 29727 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
W20260709 12:18:45.140039 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 138961us
I20260709 12:18:45.102093 30033 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:45.147679 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:45.189457 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100317us
I20260709 12:18:45.148530 30028 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.142352 30010 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:45.221350 27701 leader_election.cc:336] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.638s (SENT)
I20260709 12:18:45.088363 29876 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:45.202312 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:45.207212 27731 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:45.144531 30045 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:45.148689 29978 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:45.130311 29673 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:45.240594 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 254971us
I20260709 12:18:45.150336 29460 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
W20260709 12:18:45.243660 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 149086us
I20260709 12:18:45.250773 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:45.221261 30028 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:45.218051 30033 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:45.224766 30010 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:45.220546 29965 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:45.256237 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.249425 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:45.270759 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:45.141435 29983 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.234897 29762 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:45.227285 29876 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:45.236188 29978 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:45.236312 30095 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:45.260355 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.222886 29676 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:18:45.252978 29995 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:45.312922 27535 auto_rebalancer.cc:629] Failed to schedule move for tablet 70329e70bb724d43a3185508e1186f4b: Illegal state: Replica 3f90188458da46f6ba138d5f82c55b69 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:45.308218 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:45.318060 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:45.324991 30090 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:18:45.326370 29993 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:45.327099 29993 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.330474 30012 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.331089 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:45.332660 30085 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
W20260709 12:18:45.338102 29998 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.616s	user 0.006s	sys 0.003s
I20260709 12:18:45.338578 29998 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.340307 27729 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:45.263554 29954 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:45.271735 29873 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:45.268965 30030 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:45.346590 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:45.411967 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.287619 29978 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.283834 30055 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:45.414978 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68508.9us
W20260709 12:18:45.413302 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 269038us
I20260709 12:18:45.280642 29996 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:45.264081 30033 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:45.282932 29888 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:45.280654 29876 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:45.433816 27733 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:45.343519 29995 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:45.419539 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.244234 30094 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.125s
I20260709 12:18:45.287258 30032 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:45.452280 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:45.421367 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 273784us
I20260709 12:18:45.422773 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:45.423065 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 204190us
I20260709 12:18:45.453572 27736 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:45.453645 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:45.290107 30095 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:45.421682 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:45.460953 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.461056 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:45.462049 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.462390 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.463304 27661 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:45.436305 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.380226 29993 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:45.268461 30044 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.154s
W20260709 12:18:45.415632 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 267944us
I20260709 12:18:45.440896 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.412016 29827 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:45.443869 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 232585us
I20260709 12:18:45.418715 29978 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:45.454695 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.453568 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:45.424006 29888 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.422320 29954 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:45.408957 30019 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:45.465628 30034 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:45.442833 29995 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:45.454241 29965 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.472774 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:45.477118 30026 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.586s	user 0.012s	sys 0.003s
W20260709 12:18:45.471084 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 378231us
I20260709 12:18:45.466594 27663 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:45.467355 30047 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:18:45.481922 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.489256 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:45.490360 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:45.491274 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:45.496217 30047 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:45.499097 30047 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.483048 30034 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.474838 30032 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:18:45.490073 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:45.479763 30019 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:45.484824 30089 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 12:18:45.475981 29888 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:45.552781 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.514150 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:45.455714 29873 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:45.465034 29762 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:45.555069 27701 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.766s (SENT)
I20260709 12:18:45.561159 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:45.556712 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74784.1us
I20260709 12:18:45.504415 30101 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:45.560067 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 139170us
W20260709 12:18:45.562134 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72147.3us
W20260709 12:18:45.563519 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73245.9us
I20260709 12:18:45.549068 27735 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:45.516897 30034 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:45.577967 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:45.521425 30055 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:45.547297 30019 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:45.555788 29996 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:45.596936 27582 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:45.557156 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:45.584837 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93575.6us
W20260709 12:18:45.585310 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69098.8us
I20260709 12:18:45.592443 29684 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:18:45.593240 30100 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:45.492194 30026 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.641805 27552 leader_election.cc:400] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:45.642424 27552 leader_election.cc:403] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:45.517766 30094 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 12:18:45.637593 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.637861 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.560235 30104 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:45.565542 30101 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:45.647140 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.647375 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:45.648681 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.651763 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:45.667745 30088 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:18:45.668608 30026 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:45.656860 29998 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:45.671829 27664 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:45.641908 29954 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:18:45.673025 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.673868 30100 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:45.674171 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:45.674674 27664 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:45.674716 30100 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:45.677167 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1751) took 1426 ms. Trace:
I20260709 12:18:45.677670 27584 rpcz_store.cc:276] 0709 12:18:44.251238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.821663 (+570425us) service_pool.cc:224] Handling call
0709 12:18:44.829640 (+  7977us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:44.836715 (+  7075us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:44.836745 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:44.843799 (+  7054us) log.cc:844] Serialized 26 byte log entry
0709 12:18:45.085216 (+241417us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:45.086283 (+  1067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.086388 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:45.676898 (+590510us) raft_consensus.cc:1685] finished
0709 12:18:45.676927 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:45.677093 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1768,"raft.queue_time_us":97,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":9482,"thread_start_us":165548,"threads_started":1,"wal-append.queue_time_us":430246}
I20260709 12:18:45.683655 27738 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 4: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:45.515940 30095 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.691525 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:45.692467 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:45.675123 30100 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:45.708141 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.714071 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:45.685269 30044 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:45.647568 30055 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:45.640255 30112 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:45.667126 30110 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:45.657462 30104 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.675644 29827 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5.
I20260709 12:18:45.726996 30094 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.727625 30108 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:45.730145 30120 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:45.688989 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:45.744553 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:45.806587 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117793us
I20260709 12:18:45.806936 27738 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:45.690948 30012 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:45.749963 30100 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.719533 29954 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:45.759038 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.788650 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:45.745910 29684 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:45.746495 30095 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:45.767197 30112 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.767376 30104 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:45.737932 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:45.834424 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.809769 30045 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.567s	user 0.009s	sys 0.000s
I20260709 12:18:45.824335 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:45.798243 30108 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:45.827198 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.845172 27662 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:45.847163 27738 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 2: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:45.823957 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:45.771070 30044 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.814100 30120 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.826512 30101 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.803791 29996 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:45.805841 30094 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:46.065670 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.299230 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.232580 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 420791us
I20260709 12:18:45.881266 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:45.894659 27664 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:45.910204 27581 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260709 12:18:46.109031 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.024s	sys 0.008s
W20260709 12:18:46.322258 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.025s	sys 0.013s
W20260709 12:18:46.322561 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 756523us
I20260709 12:18:46.003386 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:45.849418 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:46.323625 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:46.301138 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:46.302466 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.325901 27738 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:45.881186 30114 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
W20260709 12:18:46.327163 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.945422 27662 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
W20260709 12:18:46.318513 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.003s	sys 0.007s
I20260709 12:18:46.330474 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:45.833863 29954 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:46.329371 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:46.338205 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.004s	sys 0.007s
W20260709 12:18:46.343691 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 788587us
W20260709 12:18:46.231215 29673 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.978s	user 0.009s	sys 0.000s
W20260709 12:18:46.283028 29978 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.802s	user 0.013s	sys 0.009s
W20260709 12:18:46.282735 30029 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.050s	user 0.008s	sys 0.000s
W20260709 12:18:46.286576 30037 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.394s	user 0.007s	sys 0.006s
W20260709 12:18:46.297263 29990 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.335s	user 0.010s	sys 0.002s
W20260709 12:18:46.331212 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 327901us
I20260709 12:18:46.366559 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:46.332831 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:46.329831 30111 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:46.295154 30053 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.168s	user 0.008s	sys 0.003s
I20260709 12:18:46.347870 28335 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 12:18:45.891774 30065 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:18:45.835311 30112 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:46.232263 30121 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:18:45.965013 29873 ts_tablet_manager.cc:1434] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 1.150s	user 0.010s	sys 0.002s
I20260709 12:18:45.892822 30012 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:46.353770 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.840716 29802 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.368605 29993 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:18:45.894879 30027 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.015s	user 0.006s	sys 0.020s
W20260709 12:18:46.356065 27701 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.641s (SENT)
W20260709 12:18:46.230768 30115 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:18:46.029040 29876 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.586s	user 0.010s	sys 0.010s
I20260709 12:18:46.278987 29827 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.360075 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:46.392326 29991 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.291s	user 0.009s	sys 0.003s
I20260709 12:18:46.362922 27732 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:46.330515 30064 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.384s	user 0.012s	sys 0.000s
I20260709 12:18:46.119453 29365 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:46.125355 29985 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.380s	user 0.014s	sys 0.009s
W20260709 12:18:46.356875 30038 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.450s	user 0.013s	sys 0.001s
I20260709 12:18:46.329744 28328 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:46.298322 30100 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:46.375888 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1514) took 1367 ms. Trace:
I20260709 12:18:46.384917 28333 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 12:18:46.385531 27735 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.006s	sys 0.029s
I20260709 12:18:46.387008 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.352980 29954 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.884903 29995 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:45.891485 30033 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.397585 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:46.368746 30055 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.004s	sys 0.016s
I20260709 12:18:46.484375 27733 rpcz_store.cc:276] 0709 12:18:45.008371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.008506 (+   135us) service_pool.cc:224] Handling call
0709 12:18:46.375610 (+1367104us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:46.375811 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":892230016}
I20260709 12:18:46.286950 30104 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.362779 27664 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:18:46.400590 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.066877 30110 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.313s
W20260709 12:18:46.376502 30106 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.014s	sys 0.015s
I20260709 12:18:46.282948 29684 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.403645 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 238) took 1942 ms. Trace:
I20260709 12:18:46.397711 30012 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.364758 29978 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.228150 30108 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.362594 29673 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.411816 29876 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.476557 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:46.401758 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45758.4us
I20260709 12:18:46.477344 27655 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:46.369011 30029 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:46.484046 27735 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.008s	sys 0.030s
I20260709 12:18:46.552217 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:46.394001 29873 ts_tablet_manager.cc:1467] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Tablet startup took 1584ms
I20260709 12:18:46.497490 30038 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:46.495198 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 165914us
I20260709 12:18:46.495355 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:46.370822 30037 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.457703 29762 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:18:46.475797 29888 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.910s	user 0.009s	sys 0.001s
I20260709 12:18:46.400411 30126 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:46.486923 30059 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.635s	user 0.013s	sys 0.015s
I20260709 12:18:46.388520 30053 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:46.512652 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.411015 30027 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.506865 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1470) took 1063 ms. Trace:
I20260709 12:18:46.488131 29991 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.388345 30111 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:46.493309 30055 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.004s	sys 0.016s
I20260709 12:18:46.486402 30129 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:46.403404 29993 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:46.457916 30101 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:46.493115 29460 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.601866 29673 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:46.555137 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.601982 29460 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.603119 27735 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:18:46.603489 27660 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 12:18:46.603886 27660 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 3
I20260709 12:18:46.604120 27660 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:18:46.604349 27660 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:46.604291 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:46.605151 27735 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:45.848156 30045 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:46.534469 30106 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.015s	sys 0.016s
W20260709 12:18:46.556142 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 232912us
I20260709 12:18:46.556782 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.607717 27729 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:46.608570 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:46.608860 27587 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:18:46.609015 30104 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:46.486239 29998 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.609870 29998 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.376319 29990 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.476311 29365 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.572471 30108 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:46.612386 29978 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:46.614284 27737 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:46.563864 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.565171 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:46.568166 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 201686us
I20260709 12:18:46.532385 29985 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:46.549188 30012 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:46.601787 30131 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:18:46.575874 27736 raft_consensus.cc:2368] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7.
I20260709 12:18:46.543253 27580 rpcz_store.cc:276] 0709 12:18:44.461191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.838887 (+377696us) service_pool.cc:224] Handling call
0709 12:18:44.946845 (+107958us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:44.947298 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:44.947397 (+    99us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:44.957263 (+  9866us) log.cc:844] Serialized 26 byte log entry
0709 12:18:45.076889 (+119626us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:45.076947 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.077054 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:45.675710 (+598656us) raft_consensus.cc:1685] finished
0709 12:18:45.675729 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:46.384252 (+708523us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000f00d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:46.403572 (+ 19320us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":619,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"spinlock_wait_cycles":247331712,"thread_start_us":119123,"threads_started":1,"wal-append.queue_time_us":185341}
I20260709 12:18:46.403973 29802 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:46.451026 29827 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.584955 27731 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:46.577762 27656 rpcz_store.cc:276] 0709 12:18:45.443012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.443121 (+   109us) service_pool.cc:224] Handling call
0709 12:18:45.459094 (+ 15973us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:45.459530 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:45.459572 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:45.460161 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:18:45.602327 (+142166us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:45.602372 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.602444 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:46.506527 (+904083us) raft_consensus.cc:1685] finished
0709 12:18:46.506554 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:46.506777 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35719,"raft.queue_time_us":45300,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":41450,"thread_start_us":118881,"threads_started":1,"wal-append.queue_time_us":367478}
W20260709 12:18:46.574483 30096 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.341s	user 0.011s	sys 0.010s
I20260709 12:18:46.572240 29762 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.600603 30055 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:46.650628 29949 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.651156 29762 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:46.651705 29993 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.652508 29673 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:46.652916 29993 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:46.653204 29673 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.654835 29960 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:46.654960 30120 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:46.655534 29960 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.577252 29888 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:46.619673 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54629.4us
W20260709 12:18:46.684825 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.459980 30127 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
I20260709 12:18:46.613639 28330 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:46.580416 30038 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:46.582448 30026 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.588608 30138 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:46.704035 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 152219us
I20260709 12:18:46.558596 30128 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260709 12:18:46.640990 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:46.710162 27656 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:46.648274 30037 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:46.551273 30132 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 12:18:46.649863 30044 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.610965 29991 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:46.575387 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.693332 27662 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:18:46.728999 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1671) took 1046 ms. Trace:
I20260709 12:18:46.631788 29985 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.696265 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.615314 30139 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:18:46.706991 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.707005 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:46.734727 27738 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:46.599972 30129 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.629750 30146 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:46.658499 30094 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.748687 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 282) took 1792 ms. Trace:
W20260709 12:18:46.749886 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.716892 30132 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:46.752445 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.722713 29888 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:46.719870 30026 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.720103 30138 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.729461 27662 rpcz_store.cc:276] 0709 12:18:45.682298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.692747 (+1010449us) service_pool.cc:224] Handling call
0709 12:18:46.693096 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:46.728677 (+ 35581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.728943 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:46.695416 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:46.711691 30141 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:46.581141 30126 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.731920 27656 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:46.735570 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1525) took 1018 ms. Trace:
I20260709 12:18:46.763886 27738 rpcz_store.cc:276] 0709 12:18:45.717247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.323359 (+606112us) service_pool.cc:224] Handling call
0709 12:18:46.735488 (+412129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:46.741103 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1660) took 1085 ms. Trace:
I20260709 12:18:46.632256 30143 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:46.762763 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:46.720098 30029 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:46.739003 29960 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.738065 30139 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:46.712808 30038 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:46.757287 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62014us
W20260709 12:18:46.740918 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.777019 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.749101 27658 rpcz_store.cc:276] 0709 12:18:44.956512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.956615 (+   103us) service_pool.cc:224] Handling call
0709 12:18:44.972827 (+ 16212us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:44.991132 (+ 18305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:44.991168 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:45.054988 (+ 63820us) log.cc:844] Serialized 26 byte log entry
0709 12:18:45.135905 (+ 80917us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:45.135964 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.136061 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:45.850405 (+714344us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:18:46.387358 (+536953us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:46.748456 (+361098us) raft_consensus.cc:1685] finished
0709 12:18:46.748480 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:46.748624 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":482,"raft.queue_time_us":18055,"raft.run_cpu_time_us":2828,"raft.run_wall_time_us":901562,"spinlock_wait_cycles":138959616,"thread_start_us":41585,"threads_started":1,"wal-append.queue_time_us":119895}
I20260709 12:18:46.766073 27662 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:46.742839 30037 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:46.764740 27729 rpcz_store.cc:276] 0709 12:18:45.656055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.609435 (+953380us) service_pool.cc:224] Handling call
0709 12:18:46.741030 (+131595us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:46.764106 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:46.772730 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:46.747653 30120 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:46.691438 30055 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.776072 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:46.784432 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.778294 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:46.742957 30129 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:46.787425 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:46.788100 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:46.791667 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:46.752727 30132 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:46.659592 29949 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:46.763139 29996 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:18:46.777371 30019 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.137s	user 0.016s	sys 0.000s
I20260709 12:18:46.794443 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:46.763155 29995 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.777616 29998 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:46.800462 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:46.781343 30037 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.776860 29876 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.780956 29365 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.804337 27658 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:46.802271 27656 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:46.802685 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:46.780189 30139 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:46.780156 30146 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:18:46.811841 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.792554 29960 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:46.786743 30105 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:46.812865 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1671) took 1143 ms. Trace:
I20260709 12:18:46.818497 27735 rpcz_store.cc:276] 0709 12:18:45.669372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.603959 (+934587us) service_pool.cc:224] Handling call
0709 12:18:46.812808 (+208849us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:46.820662 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.825989 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:46.826423 27729 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:46.827582 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:46.792337 29727 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
W20260709 12:18:46.558524 29873 ts_tablet_manager.cc:1469] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Trace:
0709 12:18:38.049658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.065409 (+ 15751us) service_pool.cc:224] Handling call
0709 12:18:38.463566 (+398157us) ts_tablet_manager.cc:934] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
0709 12:18:40.540314 (+2076748us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:41.068352 (+528038us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:41.068483 (+   131us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:41.439021 (+370538us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:41.467329 (+ 28308us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:41.523213 (+ 55884us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:41.543431 (+ 20218us) inbound_call.cc:177] Queueing success response
0709 12:18:42.591904 (+1048473us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:18:44.809702 (+2217798us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:18:44.881144 (+ 71442us) tablet_replica.cc:261] Starting instrumentation
0709 12:18:44.881745 (+   601us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":147947,"raft.queue_time_us":33527,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":39364,"tablet-copy.queue_time_us":298084,"tablet-download-pool.queue_time_us":208,"tablet-download-pool.run_cpu_time_us":1238,"tablet-download-pool.run_wall_time_us":623533,"thread_start_us":238921,"threads_started":2}
I20260709 12:18:46.828675 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1653) took 1217 ms. Trace:
I20260709 12:18:46.829139 27736 rpcz_store.cc:276] 0709 12:18:45.611652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.637284 (+ 25632us) service_pool.cc:224] Handling call
0709 12:18:46.828508 (+1191224us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54002a0654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:46.828621 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2133430016}
I20260709 12:18:46.830084 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:46.813170 29995 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:46.805788 30111 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.805971 30037 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:46.765658 30141 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.792356 30120 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.812906 30053 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:46.817900 30044 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:46.833459 29673 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.848294 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:46.832417 29365 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:46.801855 30019 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.780246 30143 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.827316 29960 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:46.859084 30095 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.026s	user 0.011s	sys 0.004s
I20260709 12:18:46.828904 30027 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:46.862231 29802 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.849565 27583 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:46.836447 29996 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:46.868301 28337 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
W20260709 12:18:46.853030 30028 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.577s	user 0.016s	sys 0.012s
I20260709 12:18:46.869372 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:46.997845 27729 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:18:46.860507 27735 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:18:46.868587 30010 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.578s	user 0.014s	sys 0.024s
I20260709 12:18:46.933385 30038 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:46.896330 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91408.8us
I20260709 12:18:46.990671 27588 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:18:46.991045 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:46.865409 30111 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:46.995627 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:46.871915 30053 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 12:18:46.856837 30146 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:46.999033 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1661) took 1332 ms. Trace:
I20260709 12:18:47.001117 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 417) took 1582 ms. Trace:
I20260709 12:18:47.003691 27658 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:18:47.011406 27735 rpcz_store.cc:276] 0709 12:18:45.666514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.835892 (+1169378us) service_pool.cc:224] Handling call
0709 12:18:46.858829 (+ 22937us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.998797 (+139968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.998984 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:47.008210 27657 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:47.013339 30152 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:18:47.014014 30120 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:47.018388 28328 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.512s	user 0.013s	sys 0.001s
I20260709 12:18:47.011531 28337 rpcz_store.cc:276] 0709 12:18:45.419188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.419380 (+   192us) service_pool.cc:224] Handling call
0709 12:18:46.867905 (+1448525us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:46.868118 (+   213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:47.000858 (+132740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.001069 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6653312}
I20260709 12:18:47.021261 27657 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
W20260709 12:18:47.025403 27701 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.650s (SENT)
W20260709 12:18:47.026326 27701 leader_election.cc:336] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.693s (SENT)
I20260709 12:18:46.999598 29365 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:47.034271 27701 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.103s (SENT)
I20260709 12:18:47.034807 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:46.885430 30132 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.007048 30010 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.006456 29996 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:47.044575 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:47.045359 27729 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:47.045576 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:47.031574 27656 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
W20260709 12:18:47.037458 27701 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.135s (SENT)
I20260709 12:18:46.823932 30105 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.008236 29876 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.022336 30038 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:47.003404 30028 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.018776 28328 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:47.040860 30120 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:47.054311 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:47.033514 29985 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:47.054059 27734 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:47.012360 30111 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.056192 27656 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
W20260709 12:18:47.048061 27701 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.186s (SENT)
I20260709 12:18:47.059480 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:47.047077 30132 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:47.092769 27655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1463) took 2109 ms (client timeout 2660 ms). Trace:
W20260709 12:18:47.095067 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47028.8us
W20260709 12:18:47.056108 30030 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.621s	user 0.026s	sys 0.047s
W20260709 12:18:47.093464 27655 rpcz_store.cc:269] 0709 12:18:44.950918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.951052 (+   134us) service_pool.cc:224] Handling call
0709 12:18:47.059740 (+2108688us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.059819 (+    79us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3507942784}
I20260709 12:18:47.137815 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:47.138219 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:47.138631 30155 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:18:47.139012 27655 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:47.139163 29365 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.140092 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:47.141701 27701 leader_election.cc:336] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 3.133s (SENT)
I20260709 12:18:47.142316 27655 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:47.142287 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:47.143394 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.143940 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:47.145110 27656 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:47.138554 27729 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:47.147046 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:47.060081 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 465) took 1314 ms. Trace:
I20260709 12:18:47.151906 28328 rpcz_store.cc:276] 0709 12:18:45.746046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.747691 (+  1645us) service_pool.cc:224] Handling call
0709 12:18:47.057340 (+1309649us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:47.155170 30141 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:47.157095 27689 heartbeater.cc:646] Failed to heartbeat to 127.25.202.62:43715 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.25.202.62:43715 timed out after 15.000s (SENT)
I20260709 12:18:47.148622 30120 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:47.158991 29802 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.190935 29995 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:47.162181 30010 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:47.163546 30019 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.140822 29954 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:47.165685 27733 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:47.435756 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1659) took 1788 ms. Trace:
I20260709 12:18:47.436199 27733 rpcz_store.cc:276] 0709 12:18:45.647776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.616197 (+968421us) service_pool.cc:224] Handling call
0709 12:18:47.165243 (+549046us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:47.165442 (+   199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:47.435445 (+270003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.435687 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134479232}
I20260709 12:18:47.166323 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:47.437849 27586 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:47.058754 29876 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.158799 30129 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.055119 30038 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.046844 29960 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.172576 29996 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.174993 30111 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:47.176337 29993 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.523s	user 0.012s	sys 0.000s
I20260709 12:18:47.135757 30149 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:47.179679 30143 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.181345 30105 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:47.182732 27658 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:47.183079 30028 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.158717 30120 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.146673 29949 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:47.202987 30100 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.714s	user 0.012s	sys 0.005s
W20260709 12:18:47.288820 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:47.485258 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:47.338613 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:47.404322 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 262617us
I20260709 12:18:47.405647 30141 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:47.491472 30141 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.411978 29978 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:47.493216 30010 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.493482 30141 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.493816 30129 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:47.494004 30149 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:47.461752 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:47.413324 29762 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.762s	user 0.015s	sys 0.000s
I20260709 12:18:47.413863 30158 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:47.414705 30112 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.017s	user 0.016s	sys 0.002s
W20260709 12:18:47.413846 30012 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.776s	user 0.005s	sys 0.009s
I20260709 12:18:47.416885 29985 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:47.136382 30030 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.703s	user 0.026s	sys 0.048s
W20260709 12:18:47.422087 30139 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.003s
I20260709 12:18:47.442981 27589 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:47.445144 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:47.442991 29954 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:47.445645 30133 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.886s	user 0.009s	sys 0.001s
I20260709 12:18:47.445781 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:47.447296 29876 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:47.447613 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:47.423226 30037 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.000s
W20260709 12:18:47.434556 30108 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.822s	user 0.008s	sys 0.010s
I20260709 12:18:47.502568 29978 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.498780 29762 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.497810 30095 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.541054 29978 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.541568 30095 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:47.490437 27701 leader_election.cc:336] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.578s (SENT)
I20260709 12:18:47.493736 30010 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:47.412797 30034 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.841s	user 0.013s	sys 0.022s
I20260709 12:18:47.497756 27655 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:47.533301 27586 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:47.536696 27656 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:47.459596 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:47.494563 30149 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:47.967242 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.972065 27658 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:47.972751 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1538) took 1107 ms. Trace:
I20260709 12:18:47.501608 30158 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.887167 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:48.064019 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:48.079375 27586 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:18:47.580758 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 248) took 2123 ms. Trace:
I20260709 12:18:48.080147 27587 rpcz_store.cc:276] 0709 12:18:45.457751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.458081 (+   330us) service_pool.cc:224] Handling call
0709 12:18:45.458391 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:46.609380 (+1150989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.580383 (+971003us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:47.580635 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2645237632}
I20260709 12:18:48.081995 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.082871 27587 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:48.128930 30052 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:48.129570 30052 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.131964 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1738) took 2319 ms. Trace:
I20260709 12:18:48.132339 27583 rpcz_store.cc:276] 0709 12:18:45.812127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.812309 (+   182us) service_pool.cc:224] Handling call
0709 12:18:45.812635 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.997265 (+1184630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.131753 (+1134488us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400087ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:48.131910 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2355515648}
I20260709 12:18:47.971963 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1484) took 1109 ms. Trace:
I20260709 12:18:47.972043 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1483) took 1114 ms. Trace:
I20260709 12:18:48.143814 27655 rpcz_store.cc:276] 0709 12:18:46.862594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.139818 (+277224us) service_pool.cc:224] Handling call
0709 12:18:47.971895 (+832077us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:47.868144 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1666) took 2200 ms (client timeout 2204 ms). Trace:
I20260709 12:18:47.542677 29762 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.964354 30026 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:47.860363 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 419) took 2217 ms. Trace:
I20260709 12:18:47.539740 30139 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.858373 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1670) took 2176 ms. Trace:
W20260709 12:18:47.859125 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 272) took 2950 ms (client timeout 1900 ms). Trace:
I20260709 12:18:48.086493 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-7274fd8a2115405c9b78c61214bff736 received from {username='slave'} at 127.0.0.1:36354
W20260709 12:18:48.014763 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.011s	sys 0.049s
W20260709 12:18:47.611267 28329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 415) took 2340 ms (client timeout 1602 ms). Trace:
I20260709 12:18:48.167816 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:47.502902 30012 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:47.614075 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:48.153096 29960 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.010s	sys 0.008s
I20260709 12:18:47.577430 30120 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:47.582170 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1524) took 1865 ms. Trace:
I20260709 12:18:48.172505 27734 rpcz_store.cc:276] 0709 12:18:45.716507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.495016 (+778509us) service_pool.cc:224] Handling call
0709 12:18:47.581918 (+1086902us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.582084 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1124481920}
I20260709 12:18:47.534032 30030 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:47.585089 29933 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:47.560247 30037 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.090756 27586 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
W20260709 12:18:47.592881 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1664) took 1925 ms (client timeout 2344 ms). Trace:
W20260709 12:18:48.175887 27730 rpcz_store.cc:269] 0709 12:18:45.667347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.375261 (+707914us) service_pool.cc:224] Handling call
0709 12:18:47.592698 (+1217437us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.592823 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431863040}
I20260709 12:18:47.848665 27663 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:48.177587 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1487) took 1313 ms. Trace:
I20260709 12:18:47.531745 29985 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:48.218932 27627 service_pool.cc:136] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:50014 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20260709 12:18:47.978207 30034 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:47.592623 30033 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.168509 29762 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.824326 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:48.141789 30047 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.003s	sys 0.014s
I20260709 12:18:48.235735 27733 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:48.236044 27735 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:47.891757 30059 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:48.259120 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1533) took 1411 ms. Trace:
I20260709 12:18:48.259619 27735 rpcz_store.cc:276] 0709 12:18:46.847698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.823943 (+976245us) service_pool.cc:224] Handling call
0709 12:18:48.259045 (+435102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:47.569406 29978 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:47.611541 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.262006 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:47.852820 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1670) took 2183 ms (client timeout 1665 ms). Trace:
W20260709 12:18:48.264743 27732 rpcz_store.cc:269] 0709 12:18:45.669096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.737689 (+ 68593us) service_pool.cc:224] Handling call
0709 12:18:47.852521 (+2114832us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.852697 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1216544128}
I20260709 12:18:47.973241 27729 rpcz_store.cc:276] 0709 12:18:46.865075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.444015 (+578940us) service_pool.cc:224] Handling call
0709 12:18:47.972695 (+528680us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.056574 29954 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.631635 30141 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.864557 29460 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:47.875258 29827 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:47.856087 27657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1474) took 2140 ms (client timeout 2815 ms). Trace:
I20260709 12:18:47.586702 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.276748 27581 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:47.502132 30112 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.095502 29991 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:47.957497 30097 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.490s	user 0.000s	sys 0.032s
W20260709 12:18:48.074080 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.286019 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.973606 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:48.102550 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 139191us
I20260709 12:18:48.097939 29941 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:48.106719 29990 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.107039 30010 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.108243 30021 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.110016 30149 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:48.115667 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.077384 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:48.138850 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1748) took 1086 ms. Trace:
W20260709 12:18:48.308037 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 230757us
I20260709 12:18:48.300765 29941 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.307869 27585 rpcz_store.cc:276] 0709 12:18:47.052147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.052340 (+   193us) service_pool.cc:224] Handling call
0709 12:18:47.052697 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:47.056654 (+  3957us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:47.056690 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:47.061815 (+  5125us) log.cc:844] Serialized 26 byte log entry
0709 12:18:47.531151 (+469336us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:47.531215 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.531285 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:48.138565 (+607280us) raft_consensus.cc:1685] finished
0709 12:18:48.138593 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:48.138776 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":641,"raft.queue_time_us":160,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":6207,"thread_start_us":380752,"threads_started":1,"wal-append.queue_time_us":977370}
I20260709 12:18:48.309217 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:48.145195 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:48.145429 27656 rpcz_store.cc:276] 0709 12:18:46.857725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.146331 (+288606us) service_pool.cc:224] Handling call
0709 12:18:47.971984 (+825653us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:48.145270 27731 rpcz_store.cc:269] 0709 12:18:45.667885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.372582 (+704697us) service_pool.cc:224] Handling call
0709 12:18:47.867896 (+1495314us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.868067 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213778304}
I20260709 12:18:48.108381 30158 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.148150 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1743) took 1373 ms. Trace:
I20260709 12:18:48.316509 27584 rpcz_store.cc:276] 0709 12:18:46.775150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.775275 (+   125us) service_pool.cc:224] Handling call
0709 12:18:46.775545 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.867505 (+ 91960us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:46.867542 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:46.867592 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:46.867635 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.147845 (+1280210us) spinlock_profiling.cc:243] Waited 91.7 ms on lock 0x7b54000b1154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:48.147909 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:48.148077 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210963200}
I20260709 12:18:48.145916 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.149876 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:48.313114 27702 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c596ddd50b894549b567a5b66ac1a31e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:48.151304 27583 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:18:48.260649 29762 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.319474 27583 pending_rounds.cc:85] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 7
I20260709 12:18:48.319813 27583 pending_rounds.cc:107] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:18:48.320216 27583 raft_consensus.cc:2889] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:48.152465 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.156981 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:48.165447 27737 rpcz_store.cc:269] 0709 12:18:44.908894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.909274 (+   380us) service_pool.cc:224] Handling call
0709 12:18:47.858867 (+2949593us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54001d10d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.859032 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3724706560}
I20260709 12:18:48.154114 28333 rpcz_store.cc:276] 0709 12:18:45.643373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.643512 (+   139us) service_pool.cc:224] Handling call
0709 12:18:45.666148 (+ 22636us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:46.484157 (+818009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.860018 (+1375861us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54003705d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:47.860259 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1652194176}
W20260709 12:18:48.167950 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.014s	sys 0.053s
I20260709 12:18:48.318954 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:48.323226 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 850754us
I20260709 12:18:48.164508 27664 rpcz_store.cc:276] 0709 12:18:45.681962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.336868 (+654906us) service_pool.cc:224] Handling call
0709 12:18:46.337100 (+   232us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:46.545739 (+208639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.858045 (+1312306us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:47.858300 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58517760}
I20260709 12:18:47.992170 30136 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:48.153247 30028 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.010s	sys 0.009s
I20260709 12:18:48.330579 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:47.608130 30038 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.178041 27663 rpcz_store.cc:276] 0709 12:18:46.863762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.845260 (+981498us) service_pool.cc:224] Handling call
0709 12:18:47.848411 (+  3151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.177260 (+328849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.177521 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:48.064368 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.331655 27737 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:48.332688 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1747) took 1474 ms. Trace:
I20260709 12:18:48.333047 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:47.863543 30093 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:48.333693 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1535) took 1469 ms. Trace:
I20260709 12:18:48.334621 27733 rpcz_store.cc:276] 0709 12:18:46.864057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.538808 (+674751us) service_pool.cc:224] Handling call
0709 12:18:48.333503 (+794695us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:48.333639 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149438848}
I20260709 12:18:48.304246 30110 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:48.232743 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.171192 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-7274fd8a2115405c9b78c61214bff736 on tablet 7274fd8a2115405c9b78c61214bff736 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:48.169479 28330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 416) took 2897 ms (client timeout 2145 ms). Trace:
I20260709 12:18:48.282183 30141 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.333817 30093 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:48.218556 29933 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:48.268551 27662 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:18:48.331259 30028 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.010s	sys 0.009s
I20260709 12:18:48.372378 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:48.373102 27733 raft_consensus.cc:2368] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:48.273340 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.373965 27732 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
W20260709 12:18:48.276105 27657 rpcz_store.cc:269] 0709 12:18:45.715848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.332490 (+616642us) service_pool.cc:224] Handling call
0709 12:18:47.855827 (+1523337us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:47.855976 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1472381824}
I20260709 12:18:48.232795 30052 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:48.259137 30047 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.004s	sys 0.014s
I20260709 12:18:48.283356 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:48.335094 30149 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:48.291095 27729 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:48.259227 30059 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:48.264285 30033 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.392953 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1662) took 2726 ms. Trace:
W20260709 12:18:48.298281 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 807800us
I20260709 12:18:48.312330 28332 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:18:48.393765 27701 leader_election.cc:336] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.899s (SENT)
I20260709 12:18:48.394120 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.394332 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:48.313540 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:48.282611 29954 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.405539 29985 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.404901 29954 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.330729 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1468) took 2889 ms. Trace:
I20260709 12:18:48.333182 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.407719 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.408541 27731 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:48.408304 30033 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.409029 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.409437 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.410069 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.410341 30033 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.410539 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.223640 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.408131 30059 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:48.171552 29960 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.010s	sys 0.008s
I20260709 12:18:48.411798 29673 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:48.363961 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:48.412293 29978 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:18:48.412421 29673 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:48.311167 29941 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:48.412896 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49030.4us
I20260709 12:18:48.322781 30030 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.413642 27734 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:48.415036 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.412894 29673 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:48.388677 29827 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:48.415956 30037 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:48.416291 29827 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.413247 29933 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:48.417163 30047 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:48.418187 29933 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:48.418322 29827 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.419427 30139 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.419476 30163 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:18:48.420535 30163 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:48.420907 30163 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:48.333061 27582 rpcz_store.cc:276] 0709 12:18:46.858671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.866475 (+  7804us) service_pool.cc:224] Handling call
0709 12:18:48.332538 (+1466063us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:48.332634 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321762688}
I20260709 12:18:48.334018 30158 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.320472 30102 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:48.320758 29972 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:48.437453 29972 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:48.438105 29972 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:48.438309 29954 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.294763 29991 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.440627 30034 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.377632 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:48.441491 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.375440 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:48.442708 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:48.443243 27732 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:48.378952 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.444967 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:48.379611 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:48.366765 29888 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.054s	sys 0.069s
I20260709 12:18:48.219257 27627 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:50014 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
W20260709 12:18:48.366847 28330 rpcz_store.cc:269] 0709 12:18:45.272007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.276782 (+  4775us) service_pool.cc:224] Handling call
0709 12:18:48.148863 (+2872081us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:48.168163 (+ 19300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2545816960}
I20260709 12:18:48.382611 29762 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.383479 29990 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.367269 30026 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.385052 27735 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:48.169025 28329 rpcz_store.cc:269] 0709 12:18:45.270891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.276718 (+  5827us) service_pool.cc:224] Handling call
0709 12:18:47.456428 (+2179710us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:47.575487 (+119059us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37825792}
I20260709 12:18:48.386482 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.392621 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:48.396456 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 83000.9us
I20260709 12:18:48.393344 27729 rpcz_store.cc:276] 0709 12:18:45.666789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.266186 (+2599397us) service_pool.cc:224] Handling call
0709 12:18:48.290847 (+ 24661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.392707 (+101860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.392899 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.411839 30059 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:48.413236 29941 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:48.258327 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.415664 27734 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:48.520870 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:48.355285 27737 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:48.223592 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:48.411088 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.411130 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:48.407629 27660 rpcz_store.cc:276] 0709 12:18:45.441743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.441852 (+   109us) service_pool.cc:224] Handling call
0709 12:18:45.442124 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.605188 (+1163064us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.330463 (+1725275us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:48.330646 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2669951872}
I20260709 12:18:48.530160 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:48.429368 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1741) took 1654 ms. Trace:
I20260709 12:18:48.312582 30021 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.333771 30136 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.434340 29673 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.550113 27729 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:48.550945 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1668) took 2882 ms. Trace:
I20260709 12:18:48.551368 27729 rpcz_store.cc:276] 0709 12:18:45.668459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.520086 (+2851627us) service_pool.cc:224] Handling call
0709 12:18:48.549852 (+ 29766us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.550648 (+   796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.550878 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.434922 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 280) took 3479 ms. Trace:
I20260709 12:18:48.421705 30163 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.436915 30102 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:48.448064 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:48.582132 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:48.582557 27584 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:48.447999 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.583757 27582 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:18:48.449925 27581 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:48.471114 27585 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:48.585179 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.586669 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.472529 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:48.473637 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.428427 30141 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.430437 30095 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.430343 30149 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:48.471410 29954 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.497759 30012 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:48.448848 29888 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.054s	sys 0.069s
I20260709 12:18:48.486791 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:48.607599 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:48.607861 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:48.496117 30112 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.549059 30094 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.487515 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:48.449110 30038 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.489821 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1749) took 1435 ms. Trace:
I20260709 12:18:48.497524 30045 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.491120 27738 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:48.493793 30126 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.500633 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1477) took 1803 ms. Trace:
I20260709 12:18:48.483858 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.491190 29972 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:48.532256 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1744) took 1756 ms (client timeout 2216 ms). Trace:
I20260709 12:18:48.532598 27732 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:48.531817 27580 rpcz_store.cc:276] 0709 12:18:46.774570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.774702 (+   132us) service_pool.cc:224] Handling call
0709 12:18:46.774988 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.793547 (+ 18559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:46.793577 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:46.793615 (+    38us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:46.793646 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.429035 (+1635389us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b54001fead4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:48.429131 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:48.429298 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42185984}
I20260709 12:18:48.516135 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:48.542896 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1536) took 1678 ms. Trace:
W20260709 12:18:48.544691 27661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.015s	sys 0.033s
I20260709 12:18:48.530901 27660 raft_consensus.cc:2376] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4. Re-sending same reply.
I20260709 12:18:48.439759 30093 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.581220 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:48.553282 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:48.556422 27659 rpcz_store.cc:276] 0709 12:18:44.955862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.955974 (+   112us) service_pool.cc:224] Handling call
0709 12:18:44.982319 (+ 26345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:45.105020 (+122701us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:45.105061 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:45.105641 (+   580us) log.cc:844] Serialized 26 byte log entry
0709 12:18:45.275850 (+170209us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:45.275897 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.275990 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:45.332679 (+ 56689us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:18:47.956757 (+2624078us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:48.434685 (+477928us) raft_consensus.cc:1685] finished
0709 12:18:48.434712 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:48.434858 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9436,"raft.queue_time_us":31471,"raft.run_cpu_time_us":3306,"raft.run_wall_time_us":313774,"spinlock_wait_cycles":89630208,"thread_start_us":282592,"threads_started":2,"wal-append.queue_time_us":315538}
I20260709 12:18:48.591209 27585 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
W20260709 12:18:48.594627 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.595166 27583 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:48.593055 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:48.520480 29941 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:48.507651 30026 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.602871 28335 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:18:48.603382 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:48.605158 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:48.606678 27735 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:18:48.448252 30158 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.608348 27656 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:48.604348 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:48.483516 29990 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:48.609174 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:48.581548 29985 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.616855 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1672) took 2934 ms. Trace:
I20260709 12:18:48.589113 30102 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:48.617769 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:48.619619 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 276) took 1927 ms. Trace:
I20260709 12:18:48.620829 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:48.619681 27733 raft_consensus.cc:606] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:48.628531 27584 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:48.624627 27628 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49998 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1726)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
W20260709 12:18:48.685626 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.738905 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.622337 27589 rpcz_store.cc:276] 0709 12:18:47.054806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.054981 (+   175us) service_pool.cc:224] Handling call
0709 12:18:47.056235 (+  1254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:47.532761 (+476526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.489579 (+956818us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:48.489761 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":888862592}
I20260709 12:18:48.978271 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.984419 27736 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
W20260709 12:18:48.986513 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 377426us
I20260709 12:18:48.820881 29991 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.987751 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:48.988780 30137 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:48.989574 30137 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:48.990545 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:48.992373 27730 rpcz_store.cc:276] 0709 12:18:46.864494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.176998 (+1312504us) service_pool.cc:224] Handling call
0709 12:18:48.542755 (+365757us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.996189 27663 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260709 12:18:48.995908 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1688) took 1113 ms. Trace:
I20260709 12:18:48.997162 27660 rpcz_store.cc:276] 0709 12:18:47.882431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.529839 (+647408us) service_pool.cc:224] Handling call
0709 12:18:48.995847 (+466008us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:48.965032 27661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.997s	user 0.035s	sys 0.057s
W20260709 12:18:48.966006 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.975584 27588 rpcz_store.cc:269] 0709 12:18:46.775451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.775883 (+   432us) service_pool.cc:224] Handling call
0709 12:18:48.532076 (+1756193us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:48.532199 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":322085632}
I20260709 12:18:49.000033 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.001243 27588 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:48.969951 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1675) took 1091 ms. Trace:
I20260709 12:18:49.007438 27656 rpcz_store.cc:276] 0709 12:18:47.878374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.581841 (+703467us) service_pool.cc:224] Handling call
0709 12:18:48.969861 (+388020us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.972467 27581 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:48.984674 27627 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:50014 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1726)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
W20260709 12:18:48.916450 30159 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.014s	sys 0.035s
I20260709 12:18:48.610687 29954 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:48.974628 28337 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7.
I20260709 12:18:48.615733 28331 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:48.975735 30026 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:48.976336 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.027s	sys 0.049s
W20260709 12:18:49.018352 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.027s	sys 0.049s
W20260709 12:18:49.018826 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 603811us
I20260709 12:18:49.019665 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:49.020291 27730 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:49.022089 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.976625 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:48.915732 27701 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.015s	sys 0.035s
W20260709 12:18:49.024000 27701 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.016s	sys 0.036s
W20260709 12:18:49.024362 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 628611us
I20260709 12:18:49.024694 27659 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:18:49.025024 27701 leader_election.cc:336] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.523s (SENT)
I20260709 12:18:49.025458 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:48.804770 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.687739 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.977494 27655 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:49.026531 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.026731 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1691) took 1143 ms. Trace:
W20260709 12:18:49.027132 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.027100 27655 rpcz_store.cc:276] 0709 12:18:47.883292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.583047 (+699755us) service_pool.cc:224] Handling call
0709 12:18:49.026671 (+443624us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:49.027638 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:49.028088 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.975251 27662 rpcz_store.cc:276] 0709 12:18:46.696905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.589732 (+892827us) service_pool.cc:224] Handling call
0709 12:18:48.077926 (+488194us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.371065 (+293139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.500347 (+129282us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:48.500561 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":426917632}
I20260709 12:18:48.963207 29673 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:48.786489 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.965814 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1663) took 3298 ms. Trace:
W20260709 12:18:49.012638 30159 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.015s	sys 0.037s
I20260709 12:18:48.962801 30136 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:49.035832 27738 rpcz_store.cc:276] 0709 12:18:45.667050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.490624 (+2823574us) service_pool.cc:224] Handling call
0709 12:18:48.490933 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.965508 (+474575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.965755 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.982154 29972 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:48.796533 30138 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:48.983305 30174 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:49.000466 27628 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49998 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1726)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
W20260709 12:18:48.983780 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 430543us
I20260709 12:18:48.965927 27733 raft_consensus.cc:614] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:48.997599 27731 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:48.966477 29941 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:48.967168 27658 rpcz_store.cc:276] 0709 12:18:45.682631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.482398 (+1799767us) service_pool.cc:224] Handling call
0709 12:18:47.971880 (+489482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:47.972850 (+   970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.615301 (+642451us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400292550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:48.616778 (+  1477us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1124943616}
I20260709 12:18:48.964354 27737 rpcz_store.cc:276] 0709 12:18:46.692677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.609805 (+1917128us) service_pool.cc:224] Handling call
0709 12:18:48.618998 (+  9193us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.619204 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.619242 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:48.619298 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:48.619359 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.619426 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:48.619551 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.968988 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.058615 27584 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.058830 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 251) took 1585 ms. Trace:
I20260709 12:18:49.059605 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.003705 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1685) took 1122 ms. Trace:
I20260709 12:18:49.063498 27664 rpcz_store.cc:276] 0709 12:18:47.881533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.580887 (+699354us) service_pool.cc:224] Handling call
0709 12:18:49.003644 (+422757us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.971158 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.063695 27586 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 2.
I20260709 12:18:49.064805 27583 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:49.009864 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.918282 30149 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.021066 30093 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:49.010499 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.011405 27736 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:49.012656 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1682) took 1132 ms. Trace:
I20260709 12:18:49.075732 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.076128 27663 rpcz_store.cc:276] 0709 12:18:47.880561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.581782 (+701221us) service_pool.cc:224] Handling call
0709 12:18:49.012596 (+430814us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:48.935043 29933 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.516s	user 0.012s	sys 0.000s
I20260709 12:18:49.009543 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.077310 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.076260 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1677) took 3405 ms. Trace:
I20260709 12:18:48.995440 30163 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:49.029162 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.087808 27662 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:48.632606 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.088172 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:49.088725 27733 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:49.088878 27657 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:48.962406 30021 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:49.030560 27589 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:48.634953 30094 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:49.039602 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:49.042866 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:49.095645 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.012514 29985 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.095645 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1690) took 1212 ms. Trace:
I20260709 12:18:49.098003 27657 rpcz_store.cc:276] 0709 12:18:47.883010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.632258 (+749248us) service_pool.cc:224] Handling call
0709 12:18:49.095579 (+463321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:48.998457 30113 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:49.038702 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.052332 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:48.990741 30137 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.065373 27580 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.068544 27584 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:49.107409 27701 connection.cc:256] RPC call timeout handler was delayed by -1.352540265000016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24340000000000003s for a response
I20260709 12:18:49.069629 27588 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:49.071137 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.010452 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1532) took 2163 ms. Trace:
W20260709 12:18:49.108409 27701 leader_election.cc:336] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.934s (SENT)
I20260709 12:18:49.073400 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.108500 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:48.678411 30038 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.076946 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.075045 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.078423 27581 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:48.958956 30172 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:18:49.118779 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.119799 27585 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:49.120671 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.121232 27585 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:49.122866 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 285) took 3676 ms. Trace:
I20260709 12:18:49.080708 30158 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.080909 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.135762 27663 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.136021 30038 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.136332 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1715) took 1038 ms. Trace:
I20260709 12:18:49.084087 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1526) took 2357 ms. Trace:
I20260709 12:18:49.136722 27582 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:49.136802 29985 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:49.137097 27738 rpcz_store.cc:276] 0709 12:18:46.726959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.070516 (+2343557us) service_pool.cc:224] Handling call
0709 12:18:49.083621 (+ 13105us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.083782 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.083808 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.083849 (+    41us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:49.083878 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.083929 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.084031 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.138430 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:49.094933 30126 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.080708 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.090934 27730 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:18:49.140386 27658 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.140879 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1720) took 1021 ms. Trace:
I20260709 12:18:49.048054 30095 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.111219 30113 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.139086 27738 raft_consensus.cc:606] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:49.142603 27738 raft_consensus.cc:614] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:49.141506 30126 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.111053 30030 raft_consensus.cc:2764] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:18:49.059437 27587 rpcz_store.cc:276] 0709 12:18:47.473745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.127969 (+654224us) service_pool.cc:224] Handling call
0709 12:18:48.128239 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.446740 (+318501us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:48.837291 (+390551us) log.cc:844] Serialized 14 byte log entry
0709 12:18:49.044925 (+207634us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.045002 (+    77us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:49.045073 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.058536 (+ 13463us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54001fefd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:49.058602 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.058759 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64153,"prepare.queue_time_us":50457,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":77526,"spinlock_wait_cycles":731857792,"thread_start_us":505240,"threads_started":2}
W20260709 12:18:49.044183 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 528072us
I20260709 12:18:49.159060 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:49.159751 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.160535 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:49.048446 30174 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:49.098882 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.100790 27656 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.101323 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1706) took 1007 ms. Trace:
I20260709 12:18:49.108927 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:49.078963 27659 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260709 12:18:49.108425 27583 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:49.167155 27657 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:49.116035 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.125658 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.168577 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1716) took 1066 ms. Trace:
I20260709 12:18:49.168680 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.169051 27657 rpcz_store.cc:276] 0709 12:18:48.102332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.098653 (+996321us) service_pool.cc:224] Handling call
0709 12:18:49.168503 (+ 69850us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:49.127102 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77187.5us
I20260709 12:18:49.123311 27661 rpcz_store.cc:276] 0709 12:18:45.446416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.468457 (+ 22041us) service_pool.cc:224] Handling call
0709 12:18:45.477566 (+  9109us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:47.967246 (+2489680us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b18004165c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:18:47.967461 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:47.967505 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:47.968162 (+   657us) log.cc:844] Serialized 26 byte log entry
0709 12:18:48.998237 (+1030075us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:48.998309 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.998414 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.122549 (+124135us) raft_consensus.cc:1685] finished
0709 12:18:49.122572 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.122804 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46338,"raft.queue_time_us":105316,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":46279,"spinlock_wait_cycles":13801728,"thread_start_us":1129351,"threads_started":2,"wal-append.queue_time_us":775063}
I20260709 12:18:49.169741 27583 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:18:49.108613 27735 rpcz_store.cc:276] 0709 12:18:46.847098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.260455 (+1413357us) service_pool.cc:224] Handling call
0709 12:18:49.010384 (+749929us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.132193 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.141135 27730 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:18:49.141677 27658 rpcz_store.cc:276] 0709 12:18:48.119620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.058007 (+938387us) service_pool.cc:224] Handling call
0709 12:18:49.140821 (+ 82814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.170782 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1527) took 2443 ms. Trace:
I20260709 12:18:49.146677 30094 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:49.136723 27663 rpcz_store.cc:276] 0709 12:18:48.097580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.077045 (+979465us) service_pool.cc:224] Handling call
0709 12:18:49.136268 (+ 59223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.152275 29954 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.154356 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.155241 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.126683 27660 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:49.155496 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.157193 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:49.160775 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:49.082775 27627 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49998 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1726)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
I20260709 12:18:49.167728 27655 rpcz_store.cc:276] 0709 12:18:48.094336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.027984 (+933648us) service_pool.cc:224] Handling call
0709 12:18:49.101279 (+ 73295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.168848 27584 raft_consensus.cc:2376] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6. Re-sending same reply.
I20260709 12:18:49.168591 27581 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:49.126320 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:49.078886 27734 rpcz_store.cc:276] 0709 12:18:45.670981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.521292 (+2850311us) service_pool.cc:224] Handling call
0709 12:18:48.531312 (+ 10020us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:48.546515 (+ 15203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.546545 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:48.547007 (+   462us) log.cc:844] Serialized 26 byte log entry
0709 12:18:48.996216 (+449209us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:49.021401 (+ 25185us) log.cc:844] Serialized 14 byte log entry
0709 12:18:49.040274 (+ 18873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.040381 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.040443 (+    62us) raft_consensus.cc:1685] finished
0709 12:18:49.040463 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.076061 (+ 35598us) spinlock_profiling.cc:243] Waited 9.69 ms on lock 0x7b54002a1a50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:49.076199 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23839,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":34663,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":43569,"spinlock_wait_cycles":22286080,"thread_start_us":383477,"threads_started":1,"wal-append.queue_time_us":396201}
I20260709 12:18:49.171180 27730 rpcz_store.cc:276] 0709 12:18:46.727431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.140432 (+2413001us) service_pool.cc:224] Handling call
0709 12:18:49.140864 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:49.170525 (+ 29661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.170727 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.171914 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1680) took 1292 ms. Trace:
I20260709 12:18:49.172338 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.172362 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1696) took 1087 ms. Trace:
I20260709 12:18:49.170894 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.160909 27664 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:49.167152 27585 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.100144 30026 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:49.204052 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46939us
I20260709 12:18:49.196333 27656 rpcz_store.cc:276] 0709 12:18:48.085069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.009068 (+923999us) service_pool.cc:224] Handling call
0709 12:18:49.172303 (+163235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.196789 27580 raft_consensus.cc:2376] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8. Re-sending same reply.
I20260709 12:18:49.197232 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:18:49.205238 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96849.3us
I20260709 12:18:49.196789 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1687) took 1314 ms. Trace:
W20260709 12:18:49.171283 30059 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.644s	user 0.005s	sys 0.005s
I20260709 12:18:49.197567 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1697) took 1112 ms. Trace:
I20260709 12:18:49.199767 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:49.218591 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47781.3us
I20260709 12:18:49.219154 27660 rpcz_store.cc:276] 0709 12:18:48.085372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.076647 (+991275us) service_pool.cc:224] Handling call
0709 12:18:49.197506 (+120859us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.146163 30126 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.197402 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:49.230495 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.230822 27584 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:18:49.202512 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.231514 27655 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:18:49.231834 27658 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.204140 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.188300 27582 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:18:49.207376 27664 rpcz_store.cc:276] 0709 12:18:47.882145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.074762 (+1192617us) service_pool.cc:224] Handling call
0709 12:18:49.196718 (+121956us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.207208 27585 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:49.207917 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.206159 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:49.236169 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:49.166379 30130 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:49.239394 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 278) took 2546 ms. Trace:
I20260709 12:18:49.215617 27735 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:49.246018 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.263028 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.264195 27655 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.265476 27736 rpcz_store.cc:276] 0709 12:18:46.693220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.985685 (+2292465us) service_pool.cc:224] Handling call
0709 12:18:49.011182 (+ 25497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.075759 (+ 64577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.239112 (+163353us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b54002a2450. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:49.239297 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57688960}
W20260709 12:18:49.208492 27701 connection.cc:256] RPC call timeout handler was delayed by -1.1832125900000392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
I20260709 12:18:49.220227 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.199343 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 280) took 2459 ms. Trace:
I20260709 12:18:49.202570 30110 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:49.270460 27582 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:49.271631 27583 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:49.268891 27627 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:50014 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1745)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1746)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1747)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1740)
W20260709 12:18:49.258307 27628 service_pool.cc:136] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49998 dropped due to backpressure: service queue is full with 50 items [suppressed 4 similar messages]
I20260709 12:18:49.273754 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 279) took 2534 ms. Trace:
I20260709 12:18:49.274121 27735 rpcz_store.cc:276] 0709 12:18:46.739150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.170398 (+2431248us) service_pool.cc:224] Handling call
0709 12:18:49.215427 (+ 45029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.273478 (+ 58051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.273694 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.275573 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:49.276687 27701 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.569s (SENT)
I20260709 12:18:49.275583 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:49.214988 27587 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:49.222493 27586 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:49.200552 27659 rpcz_store.cc:276] 0709 12:18:47.879945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.076769 (+1196824us) service_pool.cc:224] Handling call
0709 12:18:49.171859 (+ 95090us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.233865 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.222095 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.243798 27584 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:49.273653 27628 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49998 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1745)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1746)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1747)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1740)
I20260709 12:18:49.280040 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.280773 27660 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:49.281248 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1705) took 1187 ms. Trace:
I20260709 12:18:49.281350 27584 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:18:49.281668 27660 rpcz_store.cc:276] 0709 12:18:48.094099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.279765 (+1185666us) service_pool.cc:224] Handling call
0709 12:18:49.281195 (+  1430us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.172905 30095 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.248152 27663 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.291401 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.291709 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1704) took 1197 ms. Trace:
I20260709 12:18:49.292208 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1707) took 1197 ms. Trace:
I20260709 12:18:49.292124 27663 rpcz_store.cc:276] 0709 12:18:48.093822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.203888 (+1110066us) service_pool.cc:224] Handling call
0709 12:18:49.291643 (+ 87755us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.285090 27584 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:18:49.292706 27655 rpcz_store.cc:276] 0709 12:18:48.094571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.264989 (+1170418us) service_pool.cc:224] Handling call
0709 12:18:49.292152 (+ 27163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.172670 30174 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:49.178061 30030 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:49.268198 29954 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:49.255954 30059 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.250744 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.251523 27661 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.252420 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1702) took 1159 ms. Trace:
I20260709 12:18:49.253608 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:49.215932 29972 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:49.254488 30187 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:18:49.254379 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.173913 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:49.208777 30120 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:18:49.236488 30038 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:18:49.267295 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47140.7us
I20260709 12:18:49.317354 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.331111 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.267107 27734 rpcz_store.cc:276] 0709 12:18:46.739829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.198193 (+2458364us) service_pool.cc:224] Handling call
0709 12:18:49.198640 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.198882 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.198925 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.198985 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:49.199035 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.199117 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.199253 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.324896 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.172286 30138 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260709 12:18:49.204180 30112 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:18:49.229612 30188 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.148s	user 0.003s	sys 0.000s
I20260709 12:18:49.278954 27664 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:49.337507 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1680) took 3593 ms. Trace:
I20260709 12:18:49.241788 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:49.287873 27701 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.655s (SENT)
I20260709 12:18:49.339890 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:49.339907 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98268us
I20260709 12:18:49.247519 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1694) took 1273 ms. Trace:
I20260709 12:18:49.293216 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.343959 27658 rpcz_store.cc:276] 0709 12:18:47.974553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.170729 (+1196176us) service_pool.cc:224] Handling call
0709 12:18:49.247431 (+ 76702us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.344588 27663 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.249069 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.294852 30047 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:18:49.305018 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69020.5us
I20260709 12:18:49.347941 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.348126 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.250406 29888 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:49.309835 27628 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36354 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1745)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1746)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1747)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 297)
I20260709 12:18:49.349777 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1686) took 1468 ms. Trace:
I20260709 12:18:49.322805 29972 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.330873 30021 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:49.308677 29673 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:18:49.316717 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1678) took 1437 ms. Trace:
I20260709 12:18:49.330506 29954 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:49.357722 27661 rpcz_store.cc:276] 0709 12:18:47.879343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.170117 (+1290774us) service_pool.cc:224] Handling call
0709 12:18:49.316634 (+146517us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.316875 27657 rpcz_store.cc:276] 0709 12:18:48.093255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.199432 (+1106177us) service_pool.cc:224] Handling call
0709 12:18:49.252339 (+ 52907us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.318781 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.350458 27628 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36354 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1745)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1746)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1747)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1749)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 297)
I20260709 12:18:49.359753 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1708) took 1265 ms. Trace:
I20260709 12:18:49.291133 30149 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:49.334832 27736 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:49.360190 27660 rpcz_store.cc:276] 0709 12:18:48.094800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.282402 (+1187602us) service_pool.cc:224] Handling call
0709 12:18:49.359679 (+ 77277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.360773 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1544) took 1068 ms. Trace:
I20260709 12:18:49.339032 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.345288 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1699) took 1259 ms. Trace:
I20260709 12:18:49.361195 27736 rpcz_store.cc:276] 0709 12:18:48.292650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.318414 (+1025764us) service_pool.cc:224] Handling call
0709 12:18:49.334480 (+ 16066us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.360479 (+ 25999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.360699 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.312186 27584 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:18:49.345935 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58094.6us
I20260709 12:18:49.217304 30026 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.350234 27663 rpcz_store.cc:276] 0709 12:18:47.881831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.292963 (+1411132us) service_pool.cc:224] Handling call
0709 12:18:49.349713 (+ 56750us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.325150 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.342074 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.337939 27729 rpcz_store.cc:276] 0709 12:18:45.744565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.552320 (+2807755us) service_pool.cc:224] Handling call
0709 12:18:48.602942 (+ 50622us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:48.611656 (+  8714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.611709 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:48.612225 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:18:49.041437 (+429212us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:49.077729 (+ 36292us) log.cc:844] Serialized 14 byte log entry
0709 12:18:49.077955 (+   226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.078020 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.337270 (+259250us) raft_consensus.cc:1685] finished
0709 12:18:49.337297 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.337445 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":36124,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":3784,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":62382,"thread_start_us":252996,"threads_started":1,"wal-append.queue_time_us":463683}
I20260709 12:18:49.330247 30038 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.316275 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:49.355408 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.356578 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.360383 27628 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36354 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 296)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1745)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1746)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1747)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1749)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 297)
I20260709 12:18:49.361445 27656 rpcz_store.cc:276] 0709 12:18:48.085948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.204972 (+1119024us) service_pool.cc:224] Handling call
0709 12:18:49.345226 (+140254us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:49.362608 27701 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.660s (SENT)
I20260709 12:18:49.363085 27655 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.276925 30094 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:18:49.345535 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 142364us
I20260709 12:18:49.362097 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.365784 27664 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:49.363988 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.349021 29990 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:49.349224 30120 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:49.354923 29972 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:49.432356 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70335.1us
I20260709 12:18:49.439641 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1684) took 1558 ms. Trace:
W20260709 12:18:49.433279 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.432392 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 293) took 1346 ms. Trace:
I20260709 12:18:49.442167 27664 rpcz_store.cc:276] 0709 12:18:48.085698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.233535 (+1147837us) service_pool.cc:224] Handling call
0709 12:18:49.432311 (+198776us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:49.413796 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.324272 30059 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.350003 29998 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:18:49.432364 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69790.4us
I20260709 12:18:49.357725 30126 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:18:49.436461 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72553.8us
W20260709 12:18:49.438679 28292 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36354 dropped due to backpressure: service queue is full with 50 items [suppressed 34 similar messages]
I20260709 12:18:49.472368 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.360113 30112 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:49.441392 27655 rpcz_store.cc:276] 0709 12:18:47.881209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.293421 (+1412212us) service_pool.cc:224] Handling call
0709 12:18:49.439557 (+146136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.459573 28335 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:49.359640 30021 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.460959 27736 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:18:49.433795 27662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1689) took 1551 ms (client timeout 2026 ms). Trace:
I20260709 12:18:49.411443 30138 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:49.492607 27662 rpcz_store.cc:269] 0709 12:18:47.882721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.250353 (+1367632us) service_pool.cc:224] Handling call
0709 12:18:49.433713 (+183360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.354861 30137 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:18:49.469867 27701 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.914s (SENT)
I20260709 12:18:49.343855 30102 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.473600 27664 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.366724 30173 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:49.486007 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:49.432904 27658 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.435415 30038 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:49.494839 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.367143 30045 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.130s
W20260709 12:18:49.496695 27701 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.051s (SENT)
I20260709 12:18:49.528051 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.437172 30030 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:49.497469 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:49.491922 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 216464us
I20260709 12:18:49.487703 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.494489 30021 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:49.433444 28329 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:49.500665 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1539) took 1237 ms. Trace:
I20260709 12:18:49.432684 30094 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:49.461778 30195 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:49.516201 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.517899 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:49.523878 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.528965 27658 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:49.531157 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43537.3us
I20260709 12:18:49.359409 30181 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.127s
W20260709 12:18:49.544993 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47650.4us
I20260709 12:18:49.551414 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.552109 27656 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
W20260709 12:18:49.552310 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.532896 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.534180 30110 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:49.496243 30138 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:49.536605 27731 rpcz_store.cc:276] 0709 12:18:48.262919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.216083 (+953164us) service_pool.cc:224] Handling call
0709 12:18:49.256223 (+ 40140us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.256443 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:18:49.287573 (+ 31130us) log.cc:844] Serialized 14 byte log entry
0709 12:18:49.500198 (+212625us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.500260 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:49.500319 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.500423 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.500587 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25072,"prepare.queue_time_us":34356,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":3898,"raft.queue_time_us":27031,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":187128,"threads_started":1}
I20260709 12:18:49.505910 30173 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.474603 29991 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:18:49.499807 30190 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:49.576072 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.553488 27660 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:49.506217 30137 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:49.534926 28336 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:18:49.546094 30094 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:49.531987 27701 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.071s (SENT)
I20260709 12:18:49.592160 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:49.567447 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.545162 29873 ts_tablet_manager.cc:933] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.594308 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.525482 29990 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 12:18:49.574327 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:49.553004 27663 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:49.576778 27656 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:49.577082 28330 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:18:49.607360 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1718) took 1490 ms. Trace:
I20260709 12:18:49.607949 27663 rpcz_store.cc:276] 0709 12:18:48.116545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.517425 (+1400880us) service_pool.cc:224] Handling call
0709 12:18:49.607257 (+ 89832us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.609138 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.571666 30170 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:49.604048 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.566025 30120 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.571954 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.575399 29985 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:49.574244 29945 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:49.604982 30095 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.610913 27663 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.574774 30045 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:49.573540 30173 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:49.572351 30138 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.621390 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.628762 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.591676 30110 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:49.583984 30195 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:49.611886 30170 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:49.566448 29954 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.591877 30112 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.619225 30197 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:18:49.651400 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.651291 30026 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:49.652155 30177 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.546s	user 0.003s	sys 0.001s
I20260709 12:18:49.635911 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.652647 30102 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.653867 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.654111 30059 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:49.653498 30170 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.654532 27663 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:49.592708 30190 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.636246 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.656575 27655 raft_consensus.cc:2410] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:18:49.656797 27656 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:49.635412 30192 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
W20260709 12:18:49.657290 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.657677 30190 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:49.658131 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1717) took 1545 ms. Trace:
I20260709 12:18:49.658555 27655 rpcz_store.cc:276] 0709 12:18:48.112969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.502845 (+1389876us) service_pool.cc:224] Handling call
0709 12:18:49.657932 (+155087us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400229f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:49.658063 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232032128}
I20260709 12:18:49.659610 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1528) took 2931 ms. Trace:
I20260709 12:18:49.660408 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 282) took 2874 ms. Trace:
I20260709 12:18:49.661167 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.661972 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:49.662120 27655 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.660871 27737 rpcz_store.cc:276] 0709 12:18:46.785962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.215074 (+2429112us) service_pool.cc:224] Handling call
0709 12:18:49.255752 (+ 40678us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:49.366313 (+110561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.366350 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:49.366849 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:18:49.552225 (+185376us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:49.552264 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.552331 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.660197 (+107866us) raft_consensus.cc:1685] finished
0709 12:18:49.660219 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.660349 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":40535,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":153,"thread_start_us":176381,"threads_started":2,"wal-append.queue_time_us":224763}
I20260709 12:18:49.663110 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:49.655411 30191 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.296s	user 0.000s	sys 0.001s
I20260709 12:18:49.664040 27655 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:49.665027 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.665843 27655 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.666718 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.660002 27738 rpcz_store.cc:276] 0709 12:18:46.727703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.143798 (+2416095us) service_pool.cc:224] Handling call
0709 12:18:49.144169 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:49.144848 (+   679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.144884 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:49.145411 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:18:49.354190 (+208779us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:49.354256 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.354362 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.659388 (+305026us) raft_consensus.cc:1685] finished
0709 12:18:49.659411 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.659547 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":208212,"threads_started":1,"wal-append.queue_time_us":215231}
I20260709 12:18:49.668085 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.668206 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 421) took 1541 ms. Trace:
W20260709 12:18:49.668779 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.668610 28336 rpcz_store.cc:276] 0709 12:18:48.126858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.145689 (+ 18831us) service_pool.cc:224] Handling call
0709 12:18:49.668000 (+1522311us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:49.668151 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2791558016}
I20260709 12:18:49.655949 30195 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.619625 30120 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.640512 30045 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:49.672645 27731 raft_consensus.cc:606] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:49.673164 27731 raft_consensus.cc:614] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:49.672979 30045 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.673956 30120 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:49.674525 30137 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:49.644095 30110 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.655278 30170 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:49.675137 30045 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.620235 29945 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:49.675483 30189 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.323s	user 0.001s	sys 0.000s
I20260709 12:18:49.657300 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1700) took 1571 ms. Trace:
I20260709 12:18:49.692088 27656 rpcz_store.cc:276] 0709 12:18:48.092597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.625370 (+1532773us) service_pool.cc:224] Handling call
0709 12:18:49.657243 (+ 31873us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.661091 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 289) took 1109 ms. Trace:
I20260709 12:18:49.693135 27734 rpcz_store.cc:276] 0709 12:18:48.551815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.332967 (+781152us) service_pool.cc:224] Handling call
0709 12:18:49.333406 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.365538 (+ 32132us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:18:49.365570 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.365630 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:18:49.365669 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.660876 (+295207us) spinlock_profiling.cc:243] Waited 7.39 ms on lock 0x7b54001d10d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:49.660923 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.661037 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45418,"raft.queue_time_us":19867,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":26446,"spinlock_wait_cycles":16997248}
W20260709 12:18:49.685530 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 246847us
I20260709 12:18:49.680543 27660 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:49.610168 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:49.701299 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 158973us
I20260709 12:18:49.701781 27664 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:49.699988 27581 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.502s	user 0.011s	sys 0.000s
I20260709 12:18:49.702648 27581 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:49.698041 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:49.703464 27587 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:49.668653 27655 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:49.711493 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.712766 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1695) took 1628 ms. Trace:
I20260709 12:18:49.713106 27664 rpcz_store.cc:276] 0709 12:18:48.084746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.609535 (+1524789us) service_pool.cc:224] Handling call
0709 12:18:49.712681 (+103146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.713922 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.725970 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.736959 27659 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 4: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:49.739037 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 253101us
I20260709 12:18:49.742012 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.746156 30045 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:49.746660 30045 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:49.747115 30045 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:49.752851 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1758) took 1204 ms. Trace:
I20260709 12:18:49.753298 27583 rpcz_store.cc:276] 0709 12:18:48.548852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.212318 (+663466us) service_pool.cc:224] Handling call
0709 12:18:49.271447 (+ 59129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.272047 (+   600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.752608 (+480561us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b54001879d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:49.752781 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135203200}
I20260709 12:18:49.756261 30045 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.758404 30045 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.760790 27552 leader_election.cc:400] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:49.761114 27552 leader_election.cc:403] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:18:49.761759 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:49.764183 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.765216 27661 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.765693 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1724) took 1266 ms. Trace:
I20260709 12:18:49.766053 27661 rpcz_store.cc:276] 0709 12:18:48.499479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.763828 (+1264349us) service_pool.cc:224] Handling call
0709 12:18:49.765623 (+  1795us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.766776 29365 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6.
I20260709 12:18:49.767478 29365 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:49.767854 29365 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:49.768196 29365 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.775655 29365 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.778770 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.781998 29365 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:49.785274 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.770936 27627 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:50014 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1490)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1747)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1749)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 304)
I20260709 12:18:49.788321 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:49.785907 30045 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:49.786185 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:49.789448 30045 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:49.791764 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1774) took 1207 ms. Trace:
I20260709 12:18:49.792626 29990 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:49.793088 29990 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:49.793457 29990 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.792210 27589 rpcz_store.cc:276] 0709 12:18:48.584683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.213913 (+629230us) service_pool.cc:224] Handling call
0709 12:18:49.214278 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.342474 (+128196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.342508 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.342559 (+    51us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:49.342604 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.791453 (+448849us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54001248d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:49.791513 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.791689 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":294335360}
I20260709 12:18:49.795955 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 287) took 1468 ms. Trace:
I20260709 12:18:49.799741 27733 rpcz_store.cc:276] 0709 12:18:48.327096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.322922 (+995826us) service_pool.cc:224] Handling call
0709 12:18:49.323352 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.777097 (+453745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.777126 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.777167 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:49.777197 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.795717 (+ 18520us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:49.795759 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.795897 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1042978560}
I20260709 12:18:49.800992 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.802093 27662 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:49.803197 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.803552 29990 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.804082 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.804211 27662 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:49.828249 27659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1712) took 1731 ms (client timeout 2135 ms). Trace:
I20260709 12:18:49.850838 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.767037 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.864370 27661 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.693343 30030 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:49.828728 27659 rpcz_store.cc:269] 0709 12:18:48.096817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.711161 (+1614344us) service_pool.cc:224] Handling call
0709 12:18:49.828161 (+117000us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:49.901969 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72062us
I20260709 12:18:49.902746 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.903484 29365 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:49.904147 29365 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:49.906322 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.907127 27659 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.911864 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.789927 30045 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:49.914150 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.914767 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:49.881700 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1693) took 1912 ms. Trace:
I20260709 12:18:49.914973 27657 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:49.916245 27656 rpcz_store.cc:276] 0709 12:18:47.968942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.692915 (+1723973us) service_pool.cc:224] Handling call
0709 12:18:49.881603 (+188688us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.918211 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:49.916992 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1713) took 1819 ms. Trace:
I20260709 12:18:49.922663 27657 rpcz_store.cc:276] 0709 12:18:48.097090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.913911 (+1816821us) service_pool.cc:224] Handling call
0709 12:18:49.916888 (+  2977us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.737686 29998 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:49.931674 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:49.908006 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.933892 27662 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.934412 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1692) took 1978 ms. Trace:
I20260709 12:18:49.934743 27662 rpcz_store.cc:276] 0709 12:18:47.956319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.826480 (+1870161us) service_pool.cc:224] Handling call
0709 12:18:49.934358 (+107878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.935936 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:18:49.923653 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1703) took 1830 ms. Trace:
I20260709 12:18:49.937058 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.903373 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:49.938489 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.927783 29365 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.940129 27663 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:49.942342 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.943225 27663 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:49.943964 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.944635 27663 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:49.946985 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.947664 27662 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:49.929663 27733 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:49.948596 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.936785 27658 rpcz_store.cc:276] 0709 12:18:48.093520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.805017 (+1711497us) service_pool.cc:224] Handling call
0709 12:18:49.923588 (+118571us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.959409 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.949524 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.710148 30038 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:49.961506 27657 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:49.926266 30093 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.847s	user 0.002s	sys 0.016s
I20260709 12:18:49.967645 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:49.968225 27655 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:18:49.968266 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 277) took 3275 ms. Trace:
I20260709 12:18:49.968533 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.968848 27732 rpcz_store.cc:276] 0709 12:18:46.692960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.976189 (+2283229us) service_pool.cc:224] Handling call
0709 12:18:48.976573 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:48.985015 (+  8442us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.985047 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:48.985597 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 12:18:49.163475 (+177878us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:49.163519 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.164782 (+  1263us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.967977 (+803195us) raft_consensus.cc:1685] finished
0709 12:18:49.968013 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.968189 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5776,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":177428,"threads_started":1,"wal-append.queue_time_us":328517}
W20260709 12:18:49.977898 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.930317 30163 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:49.979861 27660 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:49.979183 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:49.981256 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.007s	sys 0.006s
W20260709 12:18:49.984550 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.007s	sys 0.006s
I20260709 12:18:49.984720 27657 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:49.984905 27553 connection.cc:256] RPC call timeout handler was delayed by -1.9514280109999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
I20260709 12:18:49.979220 27588 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:49.941634 29933 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.949534 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:49.988826 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.990360 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 292) took 2933 ms. Trace:
I20260709 12:18:49.990670 27662 rpcz_store.cc:276] 0709 12:18:47.056899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.989699 (+2932800us) service_pool.cc:224] Handling call
0709 12:18:49.989938 (+   239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.990074 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.990097 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.990137 (+    40us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:49.990164 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.990212 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.990313 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:49.992127 27659 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:49.994976 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:49.996129 27657 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
W20260709 12:18:49.997704 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 465686us
I20260709 12:18:50.000082 27663 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:50.000928 27659 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:50.001519 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1681) took 2121 ms. Trace:
I20260709 12:18:50.001794 27659 rpcz_store.cc:276] 0709 12:18:47.880261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.000306 (+2120045us) service_pool.cc:224] Handling call
0709 12:18:50.000602 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.001294 (+   692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.001468 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:49.999457 27701 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.434s (SENT)
I20260709 12:18:49.964114 29673 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:50.010965 27657 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:49.967136 29991 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:18:50.012717 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64340.5us
I20260709 12:18:50.017438 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.027280 27583 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:50.018052 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:50.032573 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 334580us
I20260709 12:18:50.033569 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:49.973171 30126 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:50.035768 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50370.9us
I20260709 12:18:50.036976 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.037173 27732 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:49.973178 29873 tablet_copy_client.cc:323] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:18:49.988726 30163 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:50.015673 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.016472 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.023139 27662 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:50.043975 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1683) took 2163 ms. Trace:
I20260709 12:18:50.015121 27655 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:50.053261 27657 rpcz_store.cc:276] 0709 12:18:47.880891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.010464 (+2129573us) service_pool.cc:224] Handling call
0709 12:18:50.010769 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.011548 (+   779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.043900 (+ 32352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.056373 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1676) took 2177 ms. Trace:
I20260709 12:18:50.060276 27662 rpcz_store.cc:276] 0709 12:18:47.878711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.010520 (+2131809us) service_pool.cc:224] Handling call
0709 12:18:50.022959 (+ 12439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.056028 (+ 33069us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.056277 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.054672 27655 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:18:50.035581 30102 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:50.062671 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1701) took 1969 ms. Trace:
I20260709 12:18:49.930174 29945 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:50.057600 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58069.4us
I20260709 12:18:50.023450 30211 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:50.066826 27655 rpcz_store.cc:276] 0709 12:18:48.092967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.053786 (+1960819us) service_pool.cc:224] Handling call
0709 12:18:50.054164 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.062349 (+  8185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.062600 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:50.045941 29941 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.961s	user 0.012s	sys 0.000s
I20260709 12:18:50.084137 30224 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:50.065151 29998 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.118957 30224 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.121833 30224 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:50.122399 27732 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
W20260709 12:18:50.131436 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.131923 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:50.132525 30113 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:50.039301 29673 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.135398 29673 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:50.022409 29954 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.065s
I20260709 12:18:50.132706 27655 raft_consensus.cc:2376] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8. Re-sending same reply.
W20260709 12:18:50.137131 29972 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.000s
I20260709 12:18:50.137681 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.137599 29972 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.139856 27655 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260709 12:18:50.086076 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 139281us
I20260709 12:18:50.056456 30163 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:50.088546 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:50.085484 27701 leader_election.cc:336] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.799s (SENT)
W20260709 12:18:50.149722 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64301.7us
W20260709 12:18:50.147192 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 110296us
W20260709 12:18:50.150556 27701 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.813s (SENT)
I20260709 12:18:50.097263 27662 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:50.153774 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1727) took 1567 ms. Trace:
I20260709 12:18:50.154270 27662 rpcz_store.cc:276] 0709 12:18:48.585959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.061208 (+1475249us) service_pool.cc:224] Handling call
0709 12:18:50.061552 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.153367 (+ 91815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.153668 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.065672 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.157217 27553 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:50.157631 27659 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:18:50.159550 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.047750 30209 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.146445 27587 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:50.142831 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:50.166527 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:50.173211 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:50.085964 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 182667us
W20260709 12:18:50.174036 27701 leader_election.cc:336] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.974s (SENT)
W20260709 12:18:50.176468 27701 leader_election.cc:336] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.120s (SENT)
W20260709 12:18:50.178236 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.179350 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:50.179865 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.048372 30210 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:50.011284 29933 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:50.185763 27659 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:18:50.092911 30174 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.763s	user 0.009s	sys 0.002s
I20260709 12:18:50.188594 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.194298 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:50.196437 27732 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 3. Current term is 4. Ops: [1.2-3.3]
I20260709 12:18:50.056639 30200 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.117s
I20260709 12:18:50.157706 27553 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:50.190166 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 262) took 1168 ms. Trace:
W20260709 12:18:50.190248 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 155804us
I20260709 12:18:50.192617 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 290) took 3182 ms. Trace:
I20260709 12:18:50.215742 27658 rpcz_store.cc:276] 0709 12:18:47.010592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.985175 (+2974583us) service_pool.cc:224] Handling call
0709 12:18:49.985626 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.985884 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:50.032003 (+ 46119us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.191104 (+159101us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:50.192407 (+  1303us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:50.192437 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.192491 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.192572 (+    81us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3561,"prepare.queue_time_us":49928,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":4388,"thread_start_us":154542,"threads_started":1}
I20260709 12:18:50.217744 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.218323 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:50.190629 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47919.1us
I20260709 12:18:50.209699 27586 rpcz_store.cc:276] 0709 12:18:49.021520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.278779 (+257259us) service_pool.cc:224] Handling call
0709 12:18:49.300566 (+ 21787us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.300807 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:18:49.544206 (+243399us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.033935 (+489729us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.189876 (+155941us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:50.189920 (+    44us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:50.189958 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.190026 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.190112 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60784,"prepare.queue_time_us":325061,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":38384,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":11023,"spinlock_wait_cycles":64840576,"thread_start_us":822939,"threads_started":3}
I20260709 12:18:50.224421 27734 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:50.239571 27656 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:50.240116 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.240926 27731 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
W20260709 12:18:50.242056 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.242220 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.047159 30030 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.189630 30210 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.243109 27731 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:50.245095 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:50.248183 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.249050 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:50.250058 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:50.243839 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.264114 27738 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:50.267303 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.145534 29941 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.274470 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.139420 30211 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.259894 30113 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.278952 27734 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 3: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:50.301837 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.257805 30210 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:50.269346 29990 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.219799 30206 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:50.292588 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1709) took 2197 ms. Trace:
I20260709 12:18:50.274430 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet c596ddd50b894549b567a5b66ac1a31e from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:18:50.298122 28332 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:18:50.299052 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.314486 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:50.315125 27729 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
W20260709 12:18:50.288870 27701 leader_election.cc:336] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:50014 dropped due to backpressure: service queue is full with 50 items
I20260709 12:18:50.295495 29945 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:50.186399 30209 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.317592 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:50.307565 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.216089 30174 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.312697 27660 rpcz_store.cc:276] 0709 12:18:48.095067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.723099 (+1628032us) service_pool.cc:224] Handling call
0709 12:18:50.292371 (+569272us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54000e42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:50.292510 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580872064}
I20260709 12:18:50.321194 27660 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:50.315464 27587 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:50.300280 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.296124 30093 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:50.215071 30126 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.324568 27734 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:50.304440 30230 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:18:50.315748 30094 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.704s	user 0.008s	sys 0.004s
I20260709 12:18:50.307566 29972 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.335058 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.292124 30221 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:18:50.343286 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:50.323666 30170 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.648s	user 0.009s	sys 0.002s
I20260709 12:18:50.313412 30038 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.350900 27663 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:50.356354 27738 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:50.380779 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.381692 27738 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:18:50.382403 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.382885 27738 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:50.342801 27661 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:50.358587 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:50.373138 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.383293 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1735) took 1301 ms. Trace:
I20260709 12:18:50.383396 27734 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:50.383769 27661 rpcz_store.cc:276] 0709 12:18:49.082106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.766771 (+684665us) service_pool.cc:224] Handling call
0709 12:18:50.383231 (+616460us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.378521 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1767) took 1933 ms. Trace:
I20260709 12:18:50.384583 27588 rpcz_store.cc:276] 0709 12:18:48.444706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.207829 (+763123us) service_pool.cc:224] Handling call
0709 12:18:49.208122 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.985468 (+777346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.378189 (+392721us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:50.378444 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1772869888}
I20260709 12:18:50.334316 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet c596ddd50b894549b567a5b66ac1a31e from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-c596ddd50b894549b567a5b66ac1a31e
I20260709 12:18:50.308413 30113 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.390481 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:50.395711 27588 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:50.402911 27738 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:18:50.406711 27738 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:50.407724 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
W20260709 12:18:50.410197 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:50.410367 27627 connection.cc:256] RPC call timeout handler was delayed by -3.1429845939999268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260709 12:18:50.410908 27627 connection.cc:256] RPC call timeout handler was delayed by -3.1042845939998642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260709 12:18:50.411477 27627 connection.cc:256] RPC call timeout handler was delayed by -3.0187845939999534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20260709 12:18:50.411936 27627 connection.cc:256] RPC call timeout handler was delayed by -3.0169845939999504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 12:18:50.412309 27627 connection.cc:256] RPC call timeout handler was delayed by -3.0142845939999461s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
W20260709 12:18:50.412700 27627 connection.cc:256] RPC call timeout handler was delayed by -2.9602845939998588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260709 12:18:50.413556 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:50.413077 27627 connection.cc:256] RPC call timeout handler was delayed by -2.9305845939998107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
W20260709 12:18:50.416293 27627 connection.cc:256] RPC call timeout handler was delayed by -2.7694845939997776s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
W20260709 12:18:50.417109 27627 connection.cc:256] RPC call timeout handler was delayed by -2.7568845939999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20260709 12:18:50.395114 30138 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.760s	user 0.010s	sys 0.003s
I20260709 12:18:50.421485 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.428102 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:50.432492 27655 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
W20260709 12:18:50.418179 27627 connection.cc:256] RPC call timeout handler was delayed by -2.5318845939998482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
W20260709 12:18:50.440272 27627 connection.cc:256] RPC call timeout handler was delayed by -2.4058845939998719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2344s for a response
W20260709 12:18:50.440807 27627 connection.cc:256] RPC call timeout handler was delayed by -2.3338845939999828s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2424s for a response
W20260709 12:18:50.441259 27627 connection.cc:256] RPC call timeout handler was delayed by -2.1601845939999293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2617s for a response
W20260709 12:18:50.441677 27627 connection.cc:256] RPC call timeout handler was delayed by -1.7614845939999668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30600000000000005s for a response
W20260709 12:18:50.442085 27627 connection.cc:256] RPC call timeout handler was delayed by -1.7533845939999537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3069s for a response
W20260709 12:18:50.445109 27627 connection.cc:256] RPC call timeout handler was delayed by -1.7326845939999203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30920000000000003s for a response
W20260709 12:18:50.445547 27627 connection.cc:256] RPC call timeout handler was delayed by -1.7164845939998941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.311s for a response
W20260709 12:18:50.445959 27627 connection.cc:256] RPC call timeout handler was delayed by -1.7029845939998722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3125s for a response
W20260709 12:18:50.446358 27627 connection.cc:256] RPC call timeout handler was delayed by -1.5355845939998289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3311s for a response
W20260709 12:18:50.446781 27627 connection.cc:256] RPC call timeout handler was delayed by -1.4905845939999836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33610000000000007s for a response
I20260709 12:18:50.428977 27731 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:18:50.422266 30221 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:50.463132 27729 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 2. Current term is 3. Ops: []
I20260709 12:18:50.464170 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.409588 30195 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:50.344321 30093 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.364339 30094 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.357435 30229 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20260709 12:18:50.477862 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.483013 27730 raft_consensus.cc:606] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:50.483469 27730 raft_consensus.cc:614] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:50.483361 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:50.476227 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 158657us
I20260709 12:18:50.479491 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 295) took 2393 ms. Trace:
I20260709 12:18:50.481633 27734 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:50.484687 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:50.484500 27663 rpcz_store.cc:276] 0709 12:18:48.086459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.000980 (+1914521us) service_pool.cc:224] Handling call
0709 12:18:50.001344 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.379836 (+378492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.479156 (+ 99320us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:50.479407 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":803418880}
I20260709 12:18:50.354480 30230 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.488257 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:50.488924 30149 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.464958 30221 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:50.494177 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:50.495811 30149 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.498492 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1772) took 1983 ms. Trace:
I20260709 12:18:50.498927 27580 rpcz_store.cc:276] 0709 12:18:48.514973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.205124 (+690151us) service_pool.cc:224] Handling call
0709 12:18:49.205436 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:49.218714 (+ 13278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.218752 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:49.249403 (+ 30651us) log.cc:844] Serialized 26 byte log entry
0709 12:18:49.683958 (+434555us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b180042b6e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:18:50.170222 (+486264us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:50.170278 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.170354 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:50.498190 (+327836us) raft_consensus.cc:1685] finished
0709 12:18:50.498212 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.498431 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11875,"raft.queue_time_us":13783,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":13373,"spinlock_wait_cycles":8686848,"thread_start_us":484919,"threads_started":1,"wal-append.queue_time_us":1228016}
I20260709 12:18:50.486302 29933 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.463694 30138 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.492401 30214 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:18:50.571077 30231 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:18:50.571295 30242 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:18:50.572836 30244 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:50.573449 30244 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:50.575682 30215 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
W20260709 12:18:50.577454 30213 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
W20260709 12:18:50.578088 30021 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.031s	user 0.004s	sys 0.005s
I20260709 12:18:50.578575 30174 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:50.579667 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.580610 27660 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:50.581532 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.582060 27660 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:50.582767 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.583886 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 269) took 1016 ms. Trace:
I20260709 12:18:50.585605 29972 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.584378 27585 rpcz_store.cc:276] 0709 12:18:49.567194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.567426 (+   232us) service_pool.cc:224] Handling call
0709 12:18:49.583376 (+ 15950us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:49.583728 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.583764 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:49.593054 (+  9290us) log.cc:844] Serialized 26 byte log entry
0709 12:18:50.023804 (+430750us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:50.023859 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.023946 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:50.583655 (+559709us) raft_consensus.cc:1685] finished
0709 12:18:50.583685 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.583825 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":56541,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":430351,"threads_started":1,"wal-append.queue_time_us":699510}
W20260709 12:18:50.505797 30225 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:18:50.487723 30158 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:50.475155 30200 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:18:50.588840 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:50.507272 30212 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:18:50.504838 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:50.605445 27733 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:50.503401 29941 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.586163 29972 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.628770 30093 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:50.610441 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:50.629803 27660 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:50.636945 27663 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:50.638574 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.639552 30211 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:50.641971 30230 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.643620 30113 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.574607 30244 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.645161 30244 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:50.647209 30173 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.014s	user 0.013s	sys 0.000s
W20260709 12:18:50.614321 30045 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.672s	user 0.009s	sys 0.001s
I20260709 12:18:50.614830 30094 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:50.616050 30223 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:18:50.617568 30221 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.498696 30195 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:50.620460 30138 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:50.578518 30021 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.609211 30149 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:50.682227 27582 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.011s	sys 0.011s
W20260709 12:18:50.656697 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 173429us
I20260709 12:18:50.684181 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:50.681032 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 176277us
I20260709 12:18:50.648123 30209 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:50.682668 27582 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.011s	sys 0.011s
I20260709 12:18:50.690034 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:50.623188 27748 tablet_copy_source_session.cc:215] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 12:18:50.603423 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 384069us
I20260709 12:18:50.675788 30173 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.697793 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:50.712208 27657 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:50.709681 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.710391 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.714443 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.720470 27660 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:50.721300 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:50.680616 30195 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.682067 30214 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:50.680004 30236 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:18:50.715271 27663 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 2.
I20260709 12:18:50.711680 30221 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:50.706408 30149 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:50.680552 30045 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.726029 30245 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:18:50.733273 30110 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 12:18:50.746688 30112 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:50.731228 30138 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:50.726679 30195 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:50.739188 30149 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:50.752863 30214 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:18:50.760823 30246 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:18:50.805276 27587 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:50.816799 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:50.819628 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 109463us
W20260709 12:18:50.820067 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:50.821005 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.821352 27660 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:18:50.821585 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111118us
W20260709 12:18:50.818713 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 667122us
I20260709 12:18:50.823048 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.822994 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.825228 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:50.822775 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.827528 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:50.830717 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.833244 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:50.834337 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.843724 27585 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:50.839385 27580 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:50.841532 27589 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:50.846781 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.847772 27583 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:50.849702 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:50.851722 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:50.854051 27583 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:50.842641 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:50.856774 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.857862 27583 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:50.860085 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.861435 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:50.862236 27583 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:50.877367 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:50.878436 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:50.880193 29954 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:50.881611 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:50.881255 29954 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:50.884902 29954 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:50.728861 29972 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.888037 30030 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:50.888670 29998 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:18:50.889252 29933 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:50.889097 30030 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.889748 29998 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:50.890452 29933 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:50.888995 29990 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.892196 29998 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:50.892750 30030 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.893467 30030 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:50.894090 30021 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.894554 30038 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:50.880100 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:50.895629 30038 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:50.895983 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:50.896173 30038 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.896679 30038 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:50.895094 30021 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:50.897867 30021 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.898505 30021 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.890096 29985 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:50.901064 30021 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.904000 30093 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:50.905125 30093 consensus_queue.cc:1243] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:50.892923 29990 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:50.905911 30045 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:50.907228 30021 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:50.907675 29990 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.908265 30021 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:50.910084 29998 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:50.910388 30021 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:50.910559 30094 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:50.911597 30094 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:50.912840 30021 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.913077 30094 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.761822 30149 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:50.926817 27584 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:50.917392 30021 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:18:50.941088 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61813.5us
I20260709 12:18:50.894112 30030 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.925920 30173 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.754606 30138 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:50.975193 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132661us
I20260709 12:18:50.984965 30217 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:50.987623 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 260) took 1948 ms. Trace:
I20260709 12:18:50.988364 27582 rpcz_store.cc:276] 0709 12:18:49.020958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.271251 (+250293us) service_pool.cc:224] Handling call
0709 12:18:49.319190 (+ 47939us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:49.319711 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:18:49.607380 (+287669us) spinlock_profiling.cc:243] Waited 6.51 ms on lock 0x7b1800348280. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:18:49.984045 (+376665us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.803049 (+819004us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:50.804391 (+  1342us) log.cc:844] Serialized 26 byte log entry
0709 12:18:50.804689 (+   298us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:50.804738 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.804883 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:50.924832 (+119949us) raft_consensus.cc:1685] finished
0709 12:18:50.943537 (+ 18705us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.969753 (+ 26216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119371,"prepare.queue_time_us":347696,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":192442,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":6927,"spinlock_wait_cycles":14981888,"thread_start_us":1075200,"threads_started":2,"wal-append.queue_time_us":776313}
W20260709 12:18:50.990558 30210 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.000s
I20260709 12:18:50.992244 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:50.992884 30030 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:50.992688 30038 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:50.995662 29941 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:50.996616 29941 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:50.998888 30163 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.804s	user 0.012s	sys 0.002s
I20260709 12:18:51.008541 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.009845 29954 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.991148 30210 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:51.012660 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 283430us
I20260709 12:18:51.009871 30163 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.012982 27658 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:51.015834 30030 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:51.016264 30030 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.016625 30030 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:51.016867 30163 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.020393 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:51.020387 30210 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.021248 28330 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:51.023571 30217 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:51.023594 30138 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:51.010897 29954 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.024912 30217 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:51.031315 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47902.3us
I20260709 12:18:51.015179 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:51.033432 29954 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.034509 30254 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:51.030205 30030 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.042366 30254 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:51.042876 30173 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:51.071611 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:51.049914 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1723) took 2550 ms (client timeout 2934 ms). Trace:
I20260709 12:18:51.051502 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1756) took 1248 ms. Trace:
I20260709 12:18:51.051827 30038 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.053128 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1719) took 2934 ms. Trace:
I20260709 12:18:50.998651 29941 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.055266 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:51.057310 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:51.077741 27663 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:51.056979 30149 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:51.059198 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1494) took 1988 ms (client timeout 2055 ms). Trace:
W20260709 12:18:51.059685 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 229237us
I20260709 12:18:51.064123 27661 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:51.068526 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:51.047623 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 214645us
W20260709 12:18:51.072202 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211671us
I20260709 12:18:51.080101 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:51.081069 30254 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.076435 27655 rpcz_store.cc:276] 0709 12:18:49.802611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.251043 (+448432us) service_pool.cc:224] Handling call
0709 12:18:50.298792 (+ 47749us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.432927 (+134135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.050910 (+617983us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:51.051259 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307539072}
I20260709 12:18:51.084014 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:51.076792 27657 rpcz_store.cc:276] 0709 12:18:48.118571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.059146 (+1940575us) service_pool.cc:224] Handling call
0709 12:18:50.059546 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.712631 (+653085us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.052568 (+339937us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b54001e46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:51.052918 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1500472576}
I20260709 12:18:51.087539 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:51.087997 30214 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:51.078593 27664 rpcz_store.cc:269] 0709 12:18:49.070561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.713736 (+643175us) service_pool.cc:224] Handling call
0709 12:18:51.058774 (+1345038us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:51.059009 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792846592}
I20260709 12:18:51.092188 27736 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:18:51.091660 30244 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:51.091001 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:51.076056 27656 rpcz_store.cc:269] 0709 12:18:48.499117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.917541 (+1418424us) service_pool.cc:224] Handling call
0709 12:18:51.049376 (+1131835us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:51.049685 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641817856}
I20260709 12:18:51.101974 28330 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:51.106051 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 11) took 1315 ms. Trace:
I20260709 12:18:51.103842 30038 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:51.106506 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:51.110297 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.111937 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:51.114298 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.115556 27655 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:51.115897 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.122478 30210 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:51.124115 30210 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:51.124313 28328 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:18:51.124567 30210 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:51.135617 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.137022 30261 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:51.116940 27730 rpcz_store.cc:276] 0709 12:18:49.785771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.786026 (+   255us) service_pool.cc:224] Handling call
0709 12:18:51.087197 (+1301171us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54002a2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:51.100795 (+ 13598us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1595509120}
I20260709 12:18:51.137669 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.138562 27656 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:51.138862 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:51.139969 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:51.140237 30244 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:51.140497 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:51.140903 30244 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.145293 30244 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.145851 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:51.146780 27738 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:51.141356 30190 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:18:51.137667 30261 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:51.160707 30261 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.149832 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:51.152189 30095 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:18:51.153257 30247 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.553s	user 0.011s	sys 0.000s
I20260709 12:18:51.153483 28337 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:51.143764 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:51.154899 27730 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:51.128789 30210 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.157860 30149 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:51.170620 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99188.6us
I20260709 12:18:51.158095 30138 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.158800 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:51.159308 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.148520 27734 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:18:51.161972 30190 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:51.140789 27656 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:51.164038 27661 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:18:51.164197 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.165123 30229 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:51.165174 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:51.182550 30264 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:18:51.182955 27737 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:51.178530 30241 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:18:51.180754 27661 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:18:51.181903 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.167848 30095 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:51.190547 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.182935 30229 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:51.192408 30229 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:51.193789 30229 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.199736 30195 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.201678 30110 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:51.202164 30110 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:51.188265 27729 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:51.187286 30095 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.195221 30229 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:51.203307 30195 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.207219 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.239683 30190 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:51.207234 27737 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:51.240197 30190 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:51.203298 30110 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.207614 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.241896 30110 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:51.207670 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.244515 27588 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:51.208462 28328 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:18:51.245529 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 154688us
I20260709 12:18:51.209025 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:51.247493 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:51.240696 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1555) took 1022 ms. Trace:
I20260709 12:18:51.211629 30244 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:51.248600 27737 rpcz_store.cc:276] 0709 12:18:50.218572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.218723 (+   151us) service_pool.cc:224] Handling call
0709 12:18:51.164760 (+946037us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:18:51.240611 (+ 75851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3029888}
I20260709 12:18:51.229107 27582 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:18:51.232214 27586 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:51.235437 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.206017 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 308) took 1029 ms. Trace:
I20260709 12:18:51.234632 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:51.250546 30244 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:51.219985 27704 leader_election.cc:336] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.604s (SENT)
I20260709 12:18:51.211069 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.221692 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:51.215418 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51399.7us
I20260709 12:18:51.233075 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.263355 27656 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
W20260709 12:18:51.224259 28332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 475) took 2880 ms (client timeout 3220 ms). Trace:
W20260709 12:18:51.218350 30263 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:18:51.233954 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.214720 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.213605 30210 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.208395 30094 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:51.220669 30241 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:51.285281 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.288340 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.225100 30095 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:51.256111 27658 rpcz_store.cc:276] 0709 12:18:50.176881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.217476 (+ 40595us) service_pool.cc:224] Handling call
0709 12:18:51.205721 (+988245us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:51.205937 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1826476288}
W20260709 12:18:51.267660 28332 rpcz_store.cc:269] 0709 12:18:48.343875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.410873 (+ 66998us) service_pool.cc:224] Handling call
0709 12:18:51.223779 (+2812906us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54005eb854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:51.224049 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4069919616}
I20260709 12:18:51.290452 27583 raft_consensus.cc:2376] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3. Re-sending same reply.
I20260709 12:18:51.237161 30138 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.291173 30256 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:51.283360 30149 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:51.291893 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.291775 30195 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:51.292162 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72316.6us
I20260709 12:18:51.292327 30254 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:51.293406 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46074.5us
I20260709 12:18:51.287596 30173 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.240739 30190 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:51.297459 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82991.1us
I20260709 12:18:51.298243 29991 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:51.360913 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:51.361091 29991 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:51.361852 27583 raft_consensus.cc:2376] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5. Re-sending same reply.
W20260709 12:18:51.300226 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78627us
W20260709 12:18:51.302843 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:51.364063 27704 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.649s (SENT)
I20260709 12:18:51.290542 29965 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:51.365368 29965 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:51.365885 29965 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.366796 30195 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.369254 30195 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.369843 30195 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:51.323819 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.370414 30195 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.327796 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:51.372007 27581 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:51.372403 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:51.328568 27580 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:51.330072 27589 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:51.329913 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:51.331073 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:51.332094 30163 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.335626 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:51.343645 27660 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:51.364140 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.238677 27483 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:51.365758 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:51.383085 27664 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:51.362744 29991 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.384107 30254 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.385787 30173 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.388044 27584 raft_consensus.cc:2376] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7. Re-sending same reply.
I20260709 12:18:51.388643 30173 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.391450 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.393759 30038 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:18:51.228199 30030 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.315675 30241 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:51.322851 30021 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:51.394492 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:51.397883 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:51.439042 27581 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:18:51.444226 27588 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.445287 27586 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:51.445768 27582 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:18:51.457674 27589 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:51.457840 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:51.461217 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.461470 27585 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:51.460928 27483 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:51.462781 30241 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.395028 29954 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:18:51.395668 29941 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:51.462592 27483 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 4, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:18:51.475255 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 109636us
I20260709 12:18:51.446214 30149 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.490809 30149 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.477526 27483 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:18:51.478952 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93027.6us
I20260709 12:18:51.468417 30173 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:51.469108 30110 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:51.499414 30254 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.399685 30163 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.471061 29954 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.515620 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:51.517406 27483 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 6, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 6 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:18:51.521699 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81623.6us
I20260709 12:18:51.522809 30181 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260709 12:18:51.524825 30113 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:18:51.526291 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 75497.1us
I20260709 12:18:51.526350 30261 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:51.527295 30261 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:51.528687 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52294.8us
W20260709 12:18:51.532460 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138137us
I20260709 12:18:51.536875 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.528250 29954 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:51.541764 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:51.534291 27483 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.547785 27588 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:51.549746 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.530496 29965 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.557976 27588 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:51.546099 29941 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:51.551055 30203 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:51.559067 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.557600 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.560689 29941 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.552134 29991 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:51.566053 30030 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:18:51.567929 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.570380 27627 leader_election.cc:400] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7.
I20260709 12:18:51.570900 27627 leader_election.cc:403] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:51.571111 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:51.573369 30110 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:18:51.563377 30195 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.565841 29941 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:51.576778 30126 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
W20260709 12:18:51.578178 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47518us
I20260709 12:18:51.579385 30254 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:51.570361 30230 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:51.581585 30254 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.581740 30163 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.587064 30094 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7.
W20260709 12:18:51.591527 30269 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:18:51.587960 30230 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:51.595680 29965 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.603693 29945 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:18:51.597306 30163 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:51.585419 30261 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.609226 30254 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.609421 30261 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:51.609097 30230 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:51.613598 30254 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.614109 30254 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.615146 30149 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:51.615957 30241 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:51.615752 30149 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.599709 30195 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.595250 29991 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 12:18:51.618032 30163 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:51.626453 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:51.624990 30021 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:18:51.627060 30163 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.592736 30275 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:18:51.619637 30254 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.639734 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:51.638466 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67422.8us
I20260709 12:18:51.606848 29972 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:18:51.646595 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.628316 30030 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:51.646597 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.628361 30195 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.648394 30030 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.650139 30195 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.652518 30149 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.653200 30149 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.654304 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:51.653738 30149 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.654599 30138 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:51.656749 30195 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:51.658418 30138 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:18:51.659132 30138 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:51.659567 30138 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:51.674325 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:51.680318 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.684556 29941 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.690598 29954 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.687752 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.693634 29954 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.694110 29941 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.691215 30224 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260709 12:18:51.696113 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.689713 30149 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:51.662398 30138 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:51.702836 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:51.717756 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:51.719305 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85901.5us
W20260709 12:18:51.722790 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.707535 29991 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.720394 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:51.709601 30217 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:51.724207 30021 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:51.726226 29991 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.739703 30021 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.741154 29991 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.742359 30221 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 12:18:51.742837 29991 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:51.743269 30221 consensus_queue.cc:1243] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:18:51.725884 30149 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:18:51.743520 30021 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:51.743942 30221 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:51.744179 30149 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:51.744354 30221 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:51.744633 30149 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:51.743194 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:51.746716 30276 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:51.747149 30276 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:51.748209 30276 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:51.749630 30173 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:51.749845 30276 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:51.753273 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 108594us
I20260709 12:18:51.745908 30149 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.752817 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:51.764940 28332 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:51.747030 29991 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:51.762202 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:51.759946 30138 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:51.775166 29991 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.783314 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:51.783766 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:51.783700 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:51.784734 27655 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:18:51.783943 27733 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:51.788607 28336 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:18:51.790777 30275 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:51.791582 30224 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:18:51.795502 28334 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:51.785095 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:51.745491 30221 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:51.764880 30038 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:51.803589 27734 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:51.778868 30149 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.806063 30277 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
W20260709 12:18:51.806813 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 78775.9us
W20260709 12:18:51.815213 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.807369 30038 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.807554 30190 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.837321 30038 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.808284 30095 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:51.808815 27737 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:18:51.840672 30190 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:51.781459 30203 raft_consensus.cc:2764] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260709 12:18:51.842551 30221 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:51.823100 28337 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:51.843119 29991 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:51.853616 29991 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:52.802300 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.008s	user 0.019s	sys 0.011s
W20260709 12:18:52.865588 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800035da0 after lost signal to thread 26408
W20260709 12:18:52.978955 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.185s	user 0.020s	sys 0.013s
W20260709 12:18:52.979101 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800035b20 after lost signal to thread 26411
I20260709 12:18:52.979972 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 317) took 1206 ms. Trace:
I20260709 12:18:52.980481 27657 rpcz_store.cc:276] 0709 12:18:51.773701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.776252 (+  2551us) service_pool.cc:224] Handling call
0709 12:18:51.792860 (+ 16608us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:51.793272 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.793299 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:51.793778 (+   479us) log.cc:844] Serialized 26 byte log entry
0709 12:18:52.979512 (+1185734us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:52.979556 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:52.979658 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:52.979705 (+    47us) raft_consensus.cc:1685] finished
0709 12:18:52.979721 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:52.979881 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":294,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":1185280,"threads_started":1,"wal-append.queue_time_us":70204,"wal-append.run_cpu_time_us":2056,"wal-append.run_wall_time_us":1012234}
W20260709 12:18:52.881840 27733 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.078s	user 0.015s	sys 0.002s
W20260709 12:18:52.889091 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.910286 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.983632 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.983757 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.984035 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.985283 27733 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:18:52.985479 30181 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:52.985952 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 310) took 1213 ms. Trace:
I20260709 12:18:52.986086 30181 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:52.986312 27733 rpcz_store.cc:276] 0709 12:18:51.772863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.776300 (+  3437us) service_pool.cc:224] Handling call
0709 12:18:52.985869 (+1209569us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21248}
I20260709 12:18:52.988466 30181 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.001389 29945 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:53.001916 29945 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:53.002835 29945 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.004172 29945 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:53.007972 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.024130 30158 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.024892 30158 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:53.032047 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.2948e+06us
W20260709 12:18:53.034080 27701 leader_election.cc:336] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.631s (SENT)
I20260709 12:18:53.038605 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:53.038975 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.23541e+06us
W20260709 12:18:53.039875 27701 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.649s (SENT)
I20260709 12:18:53.039906 30291 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.040334 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.040529 30291 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:53.041343 27701 connection.cc:256] RPC call timeout handler was delayed by -1.2383412940000653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18330000000000002s for a response
I20260709 12:18:53.042851 30291 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.044329 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.047250 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.055230 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 487) took 1413 ms. Trace:
I20260709 12:18:53.059994 28337 rpcz_store.cc:276] 0709 12:18:51.641658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.641808 (+   150us) service_pool.cc:224] Handling call
0709 12:18:53.055009 (+1413201us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:53.055145 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":401973504}
I20260709 12:18:53.060498 30292 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:53.061117 30292 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.062539 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:53.063390 30292 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.063937 30292 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:53.064512 30292 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.066596 30292 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.066620 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 490) took 1368 ms. Trace:
W20260709 12:18:53.068244 27701 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.701s (SENT)
I20260709 12:18:53.067950 28334 rpcz_store.cc:276] 0709 12:18:51.698068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.705028 (+  6960us) service_pool.cc:224] Handling call
0709 12:18:51.705358 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.803203 (+ 97845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.066333 (+1263130us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:53.066530 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185711360}
I20260709 12:18:53.070708 29954 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:53.071167 29954 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:53.072005 27655 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.268s	user 0.011s	sys 0.006s
I20260709 12:18:53.072357 27655 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:18:53.067212 30292 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.073642 30292 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:53.074083 30292 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:53.074512 30292 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:53.072856 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 321) took 1292 ms. Trace:
I20260709 12:18:53.078064 27655 rpcz_store.cc:276] 0709 12:18:51.780582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.783403 (+  2821us) service_pool.cc:224] Handling call
0709 12:18:53.072803 (+1289400us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:53.100677 30210 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.101337 30210 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.112636 30210 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.113366 30210 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.113955 30210 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:53.115988 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.117882 30203 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.124965 30210 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:53.125828 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 1.33904e+06us
W20260709 12:18:53.126237 30221 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.283s	user 0.008s	sys 0.007s
I20260709 12:18:53.125816 30210 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:53.127527 30210 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:53.128772 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59147.3us
W20260709 12:18:53.129458 27701 leader_election.cc:336] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.743s (SENT)
I20260709 12:18:53.129853 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.129824 30283 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:18:53.131054 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.131235 30282 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.131758 30283 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:18:53.132125 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0684912549997989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3194s for a response
I20260709 12:18:53.132244 30283 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:53.132757 30277 consensus_queue.cc:1243] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:18:53.133066 30221 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:18:53.131853 30282 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:53.132591 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0349412939999638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20590000000000003s for a response
W20260709 12:18:53.141258 27701 connection.cc:256] RPC call timeout handler was delayed by -1.031341293999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20260709 12:18:53.150321 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 87853us
I20260709 12:18:53.151680 30296 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:18:53.151046 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 103927us
I20260709 12:18:53.152282 30296 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.153554 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.154958 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.156124 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.155467 30296 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.157642 30296 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:53.158322 30296 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:53.158882 30296 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.157021 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:53.161562 30298 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:53.162178 30298 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.162621 30298 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:53.164481 30282 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:53.165151 30285 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.319s	user 0.016s	sys 0.008s
I20260709 12:18:53.165061 30290 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:18:53.165953 30290 consensus_queue.cc:1243] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:18:53.166427 30290 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:53.166745 30290 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:53.160084 30221 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.171875 30221 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.159277 30296 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.173331 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.174075 30221 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.179653 30290 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.181126 30290 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.181900 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.176718 30298 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.133195 30283 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.185986 29985 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.186296 30283 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:53.186777 29985 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:18:53.188896 30292 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:53.195726 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.196950 30297 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.197543 30297 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.197956 30221 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.198734 30221 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.201514 30221 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:53.204706 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.38012e+06us
I20260709 12:18:53.206969 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:53.209540 29945 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:53.210067 29945 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:53.210145 30297 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:53.218426 30138 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.437s	user 0.007s	sys 0.025s
I20260709 12:18:53.219105 30293 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.219807 30293 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.220784 30181 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:18:53.227540 29945 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.233594 30181 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:18:53.223217 30149 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.380s	user 0.009s	sys 0.009s
I20260709 12:18:53.234880 30291 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.235383 30291 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.235587 30292 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.238890 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:53.241019 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.241858 27734 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 12 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:18:53.242031 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.242604 27733 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:53.243083 27738 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:53.244308 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.244879 27737 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:53.246142 27729 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:18:53.246657 28334 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:53.246733 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 308) took 1474 ms. Trace:
I20260709 12:18:53.247177 27729 rpcz_store.cc:276] 0709 12:18:51.772085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.776185 (+  4100us) service_pool.cc:224] Handling call
0709 12:18:53.245670 (+1469485us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:18:53.245847 (+   177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:53.246480 (+   633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.246659 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3853568}
W20260709 12:18:53.247704 27701 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.059s (SENT)
I20260709 12:18:53.248163 27730 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:53.248209 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.249233 30110 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.249852 27736 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:18:53.250448 27701 leader_election.cc:336] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.063s (SENT)
I20260709 12:18:53.250877 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.251756 30110 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.253712 27733 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:53.255761 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:53.256253 30110 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.261420 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:53.258008 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.275013 27730 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:53.275522 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.276739 27737 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:18:53.277316 30305 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:53.277899 30305 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:53.279235 30305 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.284144 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.284722 27733 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:53.284574 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.285405 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.285975 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.286564 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.287053 27738 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:18:53.288867 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107106us
I20260709 12:18:53.291980 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.295354 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.267944 30110 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:53.311583 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.282505 30149 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:53.314039 27733 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:53.315475 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:53.316469 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:53.319064 30298 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.283150 30296 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:53.323953 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 99445.8us
I20260709 12:18:53.322978 30293 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.325702 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.331022 30302 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:18:53.333483 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:53.299741 30230 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:53.334527 30302 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:53.334728 29873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c596ddd50b894549b567a5b66ac1a31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:53.326256 30095 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.335014 30302 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.334617 30230 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.336026 29941 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:18:53.336838 30110 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.336673 30297 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:53.337611 27737 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:53.283609 30292 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.326043 27738 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:18:53.341398 27730 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:18:53.337980 30291 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.345754 27730 pending_rounds.cc:85] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 6
I20260709 12:18:53.345567 27738 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 3
I20260709 12:18:53.347492 27730 pending_rounds.cc:107] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:18:53.349439 27730 raft_consensus.cc:2889] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:53.349694 27731 pending_rounds.cc:85] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 9
I20260709 12:18:53.349977 27731 pending_rounds.cc:107] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 12:18:53.350200 27731 raft_consensus.cc:2889] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:53.340709 30149 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:53.344422 30302 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.357843 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.347977 27738 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.4
I20260709 12:18:53.358667 27738 raft_consensus.cc:2917] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Aborting config change with OpId 4.4: Aborted: Op aborted by new leader
W20260709 12:18:53.355593 30301 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:18:53.360333 27738 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:53.362075 27701 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 3.194s (SENT)
I20260709 12:18:53.370767 27737 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 3.
I20260709 12:18:53.360044 30095 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:53.376621 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.375862 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.379375 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 320) took 1599 ms. Trace:
I20260709 12:18:53.359668 30292 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.380625 27662 rpcz_store.cc:276] 0709 12:18:51.780270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.784040 (+  3770us) service_pool.cc:224] Handling call
0709 12:18:51.805539 (+ 21499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:53.379004 (+1573465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.379033 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:53.379077 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:53.379119 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.379184 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:53.379303 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:53.381927 27734 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:53.382675 30292 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:53.387737 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:47062 (request call id 8) took 7896 ms. Trace:
I20260709 12:18:53.388366 27483 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:53.388164 27738 rpcz_store.cc:276] 0709 12:18:45.491059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.491182 (+   123us) service_pool.cc:224] Handling call
0709 12:18:45.570852 (+ 79670us) log.cc:844] Serialized 499 byte log entry
0709 12:18:47.997751 (+2426899us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b1800049ae0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c1142 00007f0e9a2a1eaa 00007f0e9a2a8d53
0709 12:18:48.404011 (+406260us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b54002998a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c1142 00007f0e9a2a1eaa 00007f0e9a2a8d53 00007f0e9a2a8cd9 00007f0e9a2a8c69 00007f0e9a2a8c31 00007f0e9a2a7dfd 0000000000388634 0000000000388479 00007f0e9725b475
0709 12:18:53.387618 (+4983607us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":226,"raft.queue_time_us":177724,"raft.run_cpu_time_us":25773,"raft.run_wall_time_us":2770949,"spinlock_wait_cycles":215857280,"thread_start_us":983010,"threads_started":2,"wal-append.queue_time_us":331377,"wal-append.run_cpu_time_us":11655,"wal-append.run_wall_time_us":1600371}
I20260709 12:18:53.390411 27483 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:18:53.396087 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:53.406217 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:53.413461 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.378868 30230 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.416365 30095 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:53.424546 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48836.6us
I20260709 12:18:53.426183 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:53.424561 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800035b40 after lost signal to thread 27521
W20260709 12:18:53.427301 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65289.1us
W20260709 12:18:53.427995 27518 debug-util.cc:398] Leaking SignalData structure 0x7b08000360c0 after lost signal to thread 27535
I20260709 12:18:53.404349 27483 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1), VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3), NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 3 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:18:53.432317 27701 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.248s (SENT)
I20260709 12:18:53.412683 30292 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:53.431572 29991 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:53.445402 27535 auto_rebalancer.cc:629] Failed to schedule move for tablet c596ddd50b894549b567a5b66ac1a31e: Aborted: Op aborted by new leader
I20260709 12:18:53.445295 29991 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.446583 30304 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:18:53.451908 28332 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 3 LEARNER]: Advancing to term 5
I20260709 12:18:53.453399 30305 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:53.455706 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.456122 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.456408 28337 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:53.439159 30149 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:53.460152 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.448992 30302 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.460584 30291 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:53.462347 30291 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:18:53.462730 30302 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.432195 30230 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:53.433457 30094 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:18:53.463565 30292 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:18:53.464294 30292 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:53.466207 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.464728 28332 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:53.472247 30296 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.474177 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.476737 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.441756 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 316) took 1673 ms. Trace:
I20260709 12:18:53.477700 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:53.482656 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.473455 30095 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.483786 27662 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:53.478139 27659 rpcz_store.cc:276] 0709 12:18:51.768731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.776625 (+  7894us) service_pool.cc:224] Handling call
0709 12:18:51.784526 (+  7901us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:51.822096 (+ 37570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.822127 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:53.220343 (+1398216us) log.cc:844] Serialized 26 byte log entry
0709 12:18:53.284502 (+ 64159us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:53.437906 (+153404us) log.cc:844] Serialized 14 byte log entry
0709 12:18:53.441354 (+  3448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.441454 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:53.441516 (+    62us) raft_consensus.cc:1685] finished
0709 12:18:53.441534 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:53.441700 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":176296,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":3246,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":1869,"thread_start_us":188692,"threads_started":2,"wal-append.queue_time_us":117595}
I20260709 12:18:53.484141 30256 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.484687 30256 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.486192 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.486143 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.485491 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.486788 30256 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.487120 27656 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:53.487421 27659 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:53.488003 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:53.488415 30306 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:18:53.488554 27656 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:53.489027 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.464856 30038 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.489737 30038 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.488278 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.489755 27660 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:18:53.472939 30296 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.493012 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.498454 27656 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:53.492616 30298 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.499959 30298 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.501924 27655 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:53.502508 30298 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.502856 30296 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.487457 30021 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:53.492350 30302 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:18:53.503680 30021 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.504981 27655 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:18:53.475577 29991 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.492866 30203 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.506071 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.508463 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:53.511140 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45039.2us
I20260709 12:18:53.490906 30256 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.514087 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.514829 27659 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:53.499580 30282 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:53.519176 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.521072 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.516263 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.521976 27660 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:53.517174 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.523720 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:53.524854 27664 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.483908 30095 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.526386 27655 raft_consensus.cc:2376] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6. Re-sending same reply.
I20260709 12:18:53.505795 30302 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:53.526968 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.534132 27660 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:53.535027 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.536253 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.536427 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:53.536909 27660 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:53.537860 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.538538 27660 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:53.538626 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:53.539016 27661 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:18:53.519192 30021 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.512986 30203 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.533265 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.544045 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.544728 28328 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:53.546489 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:53.546738 27655 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:53.547428 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.548763 27656 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:53.548846 27660 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:53.549646 30021 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:53.549731 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.550005 30313 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:53.550819 30302 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.543905 30296 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:53.560211 30296 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.560606 30296 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:53.561137 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 322) took 1779 ms. Trace:
I20260709 12:18:53.561508 27658 rpcz_store.cc:276] 0709 12:18:51.781977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.784773 (+  2796us) service_pool.cc:224] Handling call
0709 12:18:51.792526 (+  7753us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.014902 (+1222376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.014932 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:53.426440 (+411508us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:53.426878 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 12:18:53.477697 (+ 50819us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:53.477748 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.560839 (+ 83091us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:53.560884 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:53.560928 (+    44us) raft_consensus.cc:1685] finished
0709 12:18:53.560940 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:53.561078 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102736,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":3284665856,"thread_start_us":50430,"threads_started":1,"wal-append.queue_time_us":97324}
I20260709 12:18:53.555874 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.545694 30282 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:53.559593 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.521976 30256 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.563884 27661 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:18:53.559182 30095 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.564667 27660 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:18:53.578079 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.578770 27660 raft_consensus.cc:2376] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7. Re-sending same reply.
I20260709 12:18:53.556941 30021 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:53.579283 30021 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:53.579967 27660 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:53.579675 29954 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.581159 29954 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:53.582835 27664 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
W20260709 12:18:53.585809 27518 debug-util.cc:398] Leaking SignalData structure 0x7b080017cea0 after lost signal to thread 27613
W20260709 12:18:53.586163 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800237780 after lost signal to thread 27614
W20260709 12:18:53.587188 27518 debug-util.cc:398] Leaking SignalData structure 0x7b080036f8e0 after lost signal to thread 27688
I20260709 12:18:53.587939 30030 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.588497 30030 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.589568 30195 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:53.589968 30195 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:53.590660 30030 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.590814 30195 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.591486 29991 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:53.591846 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.592151 29991 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.594355 29991 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.544523 30305 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:18:53.570225 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51988.8us
I20260709 12:18:53.596385 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:53.603456 27583 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:53.603542 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.574472 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.576033 27655 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:53.595433 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:53.601140 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:53.624728 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86685.3us
I20260709 12:18:53.632627 27582 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 4.
I20260709 12:18:53.544440 30298 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.033s
W20260709 12:18:53.614390 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800401420 after lost signal to thread 27762
I20260709 12:18:53.593762 30021 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:53.636678 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800401220 after lost signal to thread 27763
W20260709 12:18:53.638383 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800315000 after lost signal to thread 28361
W20260709 12:18:53.639011 27518 debug-util.cc:398] Leaking SignalData structure 0x7b08002ce520 after lost signal to thread 30093
I20260709 12:18:53.640568 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:53.649837 30021 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.652479 28328 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:53.656662 27658 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:53.658998 27586 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:53.576882 29941 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.577133 30282 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.591868 30195 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.660041 27663 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 2: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:53.602320 30305 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.591248 30030 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.669860 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:53.604743 30038 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:53.657795 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:53.676092 27585 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:53.672845 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.674311 27582 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:18:53.653652 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:53.655834 30283 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:53.679244 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:53.683512 27582 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.674954 28336 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:18:53.683750 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.682148 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:53.681491 30296 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:53.685891 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:53.673316 30282 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:53.687291 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:53.695075 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:53.677134 30305 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.703029 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:53.718842 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:53.725589 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:53.685987 30283 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:53.692868 30230 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:53.673216 29941 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:53.758807 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.717793 30292 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:18:53.766497 27588 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:53.766068 30291 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:53.830657 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:53.831302 27581 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:53.733434 27658 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:18:53.769068 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83263.7us
I20260709 12:18:53.682710 30030 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.830308 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:53.825670 27701 leader_election.cc:336] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.691s (SENT)
W20260709 12:18:53.825513 30319 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 12:18:53.828620 30315 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
W20260709 12:18:53.768680 30261 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 2.159s	user 0.006s	sys 0.025s
I20260709 12:18:53.766566 30113 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.777822 30283 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:53.829356 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.842944 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:53.832152 30230 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:53.853494 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 150529us
I20260709 12:18:53.859941 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:53.861205 27659 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:53.858287 30317 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:18:53.857216 27586 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:53.830319 30291 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:53.859375 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:53.831493 30038 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:53.966380 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111179us
I20260709 12:18:53.970886 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:53.972074 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:53.972512 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.977377 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:18:53.849221 30292 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:18:53.979285 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:53.980352 30322 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
W20260709 12:18:53.984010 30323 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
W20260709 12:18:53.988147 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 128304us
I20260709 12:18:53.989259 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.858852 30158 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:53.995514 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:53.971103 30094 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.997223 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:54.006718 27581 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:53.978868 30290 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:54.010665 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:53.860780 30113 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.015820 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.005522 30094 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:53.993419 30203 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:54.012951 27663 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:53.993937 29991 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:18:54.017935 30321 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.251s	user 0.000s	sys 0.000s
I20260709 12:18:54.032112 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.027099 27659 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:54.041049 29873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58252 (request call id 481) took 4819 ms. Trace:
I20260709 12:18:54.008208 30290 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.045287 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:54.045748 27658 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:18:54.041571 29873 rpcz_store.cc:276] 0709 12:18:49.221351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.221860 (+   509us) service_pool.cc:224] Handling call
0709 12:18:49.608451 (+386591us) ts_tablet_manager.cc:934] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
0709 12:18:53.332738 (+3724287us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:53.502842 (+170104us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:53.502967 (+   125us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:54.010370 (+507403us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:54.019830 (+  9460us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:54.040806 (+ 20976us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:54.040914 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":49680,"tablet-copy.queue_time_us":242622,"thread_start_us":493288,"threads_started":2}
I20260709 12:18:54.137019 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:54.144781 30327 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:18:53.994609 30230 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:54.163391 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 255783us
I20260709 12:18:54.167834 27659 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:54.169682 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.157312 27585 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:18:54.176596 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:54.047331 30296 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:54.048641 30290 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:54.193209 27586 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:54.037686 30038 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.141s
W20260709 12:18:54.195271 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.197731 27663 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:18:54.198670 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.199663 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.200809 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:54.202486 27582 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.509s	user 0.006s	sys 0.002s
I20260709 12:18:54.202915 27582 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
W20260709 12:18:54.195623 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 165394us
I20260709 12:18:54.161959 29991 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:18:54.209903 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.206224 27483 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 6, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 6 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:18:54.208479 27585 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:54.211621 27658 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:54.215773 27483 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:54.217965 27483 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:54.220788 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.220731 27483 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 5, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:18:54.199088 30110 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:18:54.224632 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.227221 29954 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.646s	user 0.011s	sys 0.002s
I20260709 12:18:54.202768 29873 tablet_copy_client.cc:806] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
W20260709 12:18:54.227214 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 220) took 22169 ms (client timeout 15000 ms). Trace:
I20260709 12:18:54.227254 30256 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:54.168840 30291 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.243441 29941 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:54.213896 30030 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:54.247694 29873 tablet_copy_client.cc:670] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:54.235733 30113 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:54.248118 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.249585 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.250036 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:54.236773 30326 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:18:54.237416 30325 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.232s	user 0.001s	sys 0.000s
I20260709 12:18:54.162583 30292 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.188688 30283 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.241144 27663 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
W20260709 12:18:54.241448 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 192736us
I20260709 12:18:54.251040 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:54.260203 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:54.275516 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.276399 27581 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:54.274591 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:54.274554 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.281710 27585 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:54.262171 27588 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:18:54.255173 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:54.277959 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:54.283347 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:54.299441 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.289178 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:54.312634 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52563.5us
I20260709 12:18:54.312331 29945 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:54.312187 30332 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:54.323899 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:54.324458 30195 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.000s
I20260709 12:18:54.321452 29945 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:54.331351 29945 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:54.333837 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 508149us
I20260709 12:18:54.334544 30030 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:54.333947 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:54.312940 30203 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:18:54.292461 30230 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:54.346990 30094 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:54.347805 30195 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:54.348549 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.349282 27660 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:54.350777 27660 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:18:54.335101 30030 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:54.351881 30030 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.356449 27658 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:54.322978 30332 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:54.355255 29991 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:18:54.359710 30332 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:18:54.240520 27482 rpcz_store.cc:269] 0709 12:18:32.058142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.059588 (+  1446us) service_pool.cc:224] Handling call
0709 12:18:54.226999 (+22167411us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:45.339983 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:45.504596 (+164613us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:45.509767 (+  5171us) write_op.cc:435] Acquired schema lock
0709 12:18:45.516481 (+  6714us) tablet.cc:662] Decoding operations
0709 12:18:45.745379 (+228898us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:45.748406 (+  3027us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:45.758080 (+  9674us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:18:45.832199 (+ 74119us) tablet.cc:701] Row locks acquired
0709 12:18:45.832222 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:18:45.832373 (+   151us) write_op.cc:270] Start()
0709 12:18:45.832485 (+   112us) write_op.cc:276] Timestamp: P: 1783599525832324 usec, L: 0
0709 12:18:45.832508 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:18:46.078575 (+246067us) log.cc:844] Serialized 2069 byte log entry
0709 12:18:47.043269 (+964694us) op_driver.cc:464] REPLICATION: finished
0709 12:18:48.335014 (+1291745us) write_op.cc:301] APPLY: starting
0709 12:18:48.363349 (+ 28335us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:48.383476 (+ 20127us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:48.383639 (+   163us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:48.403251 (+ 19612us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:48.403583 (+   332us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0709 12:18:48.403661 (+    78us) write_op.cc:312] APPLY: finished
0709 12:18:48.404021 (+   360us) log.cc:844] Serialized 47 byte log entry
0709 12:18:49.057630 (+653609us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:49.077737 (+ 20107us) write_op.cc:454] Released schema lock
0709 12:18:49.079752 (+  2015us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:49.080225 (+   473us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:49.080258 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1003219,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":1257625,"apply.run_cpu_time_us":56425,"apply.run_wall_time_us":762893,"num_ops":5,"prepare.queue_time_us":219681,"prepare.run_cpu_time_us":25115,"prepare.run_wall_time_us":1208174,"raft.queue_time_us":353966,"raft.run_cpu_time_us":23953,"raft.run_wall_time_us":494344,"replication_time_us":1189111,"thread_start_us":1511948,"threads_started":5,"wal-append.queue_time_us":1084031,"wal-append.run_cpu_time_us":31774,"wal-append.run_wall_time_us":2228118}]]}
I20260709 12:18:54.360556 30290 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.355926 30230 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.361670 30230 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:54.361550 30291 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:18:54.363000 30290 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:54.355162 30203 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.364537 30203 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:54.369944 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:54.371316 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82252.2us
W20260709 12:18:54.375801 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.379395 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45546.8us
I20260709 12:18:54.379734 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:54.410498 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.411537 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.412227 27580 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:54.413009 27583 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:54.414189 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1582) took 1118 ms. Trace:
I20260709 12:18:54.414582 27734 rpcz_store.cc:276] 0709 12:18:53.295968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.296412 (+   444us) service_pool.cc:224] Handling call
0709 12:18:53.317342 (+ 20930us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:53.382261 (+ 64919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.413936 (+1031675us) spinlock_profiling.cc:243] Waited 64.2 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:54.414126 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":147751168}
I20260709 12:18:54.419992 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:54.364961 30110 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:18:54.429528 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 141406us
W20260709 12:18:54.416715 30339 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.107s	user 0.004s	sys 0.000s
W20260709 12:18:54.432610 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.433529 27586 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:18:54.433581 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.444357 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.445173 27580 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:54.446959 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.448577 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:54.452503 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.419050 29991 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:18:54.435214 30290 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.460966 30181 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:54.454939 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.470252 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.429180 30038 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
W20260709 12:18:54.453764 30282 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.724s	user 0.009s	sys 0.001s
I20260709 12:18:54.502730 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:18:54.505076 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.439486 30341 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.499958 30038 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:18:54.451421 30329 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 12:18:54.502038 30181 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:54.499426 30290 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.514405 27660 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:54.577862 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 323) took 1092 ms. Trace:
I20260709 12:18:54.578753 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:54.518589 30334 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:18:54.450457 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:18:54.599057 27734 raft_consensus.cc:606] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:54.531164 27580 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:54.600628 27734 raft_consensus.cc:614] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:54.516413 30283 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:54.584956 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:54.587371 27586 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:54.590185 27657 rpcz_store.cc:276] 0709 12:18:53.485189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.487544 (+  2355us) service_pool.cc:224] Handling call
0709 12:18:53.487821 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:53.506107 (+ 18286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.506138 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:53.506183 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:53.506215 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.577601 (+1071386us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:54.577678 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.577791 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41655040}
W20260709 12:18:54.591168 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 164468us
I20260709 12:18:54.592154 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
W20260709 12:18:54.521615 30305 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.792s	user 0.009s	sys 0.000s
I20260709 12:18:54.597223 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.577234 30292 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.635076 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.514303 30282 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.645351 27660 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:54.646385 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.516227 29945 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.661993 27589 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
W20260709 12:18:54.662703 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 252756us
I20260709 12:18:54.645854 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 315) took 1149 ms. Trace:
W20260709 12:18:54.684264 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.648412 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.685564 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:54.585819 30341 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.645962 30030 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20260709 12:18:54.649766 30349 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
W20260709 12:18:54.652202 30340 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
W20260709 12:18:54.655335 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 327195us
I20260709 12:18:54.655571 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 330) took 1022 ms. Trace:
I20260709 12:18:54.688019 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.716212 27659 rpcz_store.cc:276] 0709 12:18:53.633441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.194093 (+560652us) service_pool.cc:224] Handling call
0709 12:18:54.199660 (+  5567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:54.199871 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:54.269118 (+ 69247us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.654987 (+385869us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b180021aba0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:18:54.655155 (+   168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:54.655218 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:54.655267 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.655365 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.655510 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14404,"prepare.queue_time_us":78397,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":20732,"raft.queue_time_us":335238,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":83902720,"thread_start_us":80499,"threads_started":2,"wal-append.queue_time_us":57290}
I20260709 12:18:54.656579 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1768) took 1171 ms. Trace:
I20260709 12:18:54.657020 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:18:54.717592 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:54.717948 27662 rpcz_store.cc:276] 0709 12:18:53.484779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.485954 (+  1175us) service_pool.cc:224] Handling call
0709 12:18:54.656383 (+1170429us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:54.656516 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856636288}
W20260709 12:18:54.718531 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 270004us
I20260709 12:18:54.719117 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.703068 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.658326 30282 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:54.659494 30181 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.515255 30110 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.733019 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:54.733253 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:54.733845 28334 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:54.604806 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:54.645533 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.740s (SENT) [suppressed 32 similar messages]
I20260709 12:18:54.657793 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.735082 27655 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
W20260709 12:18:54.735734 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.735776 27585 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:54.720816 27734 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:18:54.721565 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:54.655310 30256 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:54.719739 27662 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:54.683651 30292 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.723282 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.735282 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.742659 27586 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
W20260709 12:18:54.723810 30353 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:18:54.752556 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.726629 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:54.656657 30210 raft_consensus.cc:1064] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:54.731688 27657 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:54.731786 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147111us
I20260709 12:18:54.754541 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.753226 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:18:54.755285 27738 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:54.644599 30305 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:54.738225 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:54.748816 28334 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:18:54.663686 27732 rpcz_store.cc:276] 0709 12:18:53.344997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.350921 (+  5924us) service_pool.cc:224] Handling call
0709 12:18:53.353076 (+  2155us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:53.353282 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:18:54.434003 (+1080721us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b1800423700. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:18:54.434136 (+   133us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.494466 (+ 60330us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:54.494522 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:54.494568 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.494632 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.494749 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45763,"prepare.queue_time_us":25825,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":671,"raft.queue_time_us":31424,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4721,"spinlock_wait_cycles":11684096,"thread_start_us":75559,"threads_started":2,"wal-append.queue_time_us":75634}
I20260709 12:18:54.714143 30291 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.760206 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:54.760999 27655 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 12:18:54.723886 30355 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:18:54.756004 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:54.740414 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94898.8us
I20260709 12:18:54.761441 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:54.757001 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152287us
I20260709 12:18:54.762302 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:54.762820 27582 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:18:54.763571 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:54.764374 29991 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:54.764716 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:54.764917 29991 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:54.765053 30030 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:54.765739 30360 raft_consensus.cc:2764] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:18:54.765826 30021 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:54.765904 29991 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.766630 30021 consensus_queue.cc:1243] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:18:54.766741 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:54.767498 29954 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:54.756956 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:54.767875 29954 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.768268 27587 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:54.759424 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:54.759528 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.769254 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:54.769320 27736 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:18:54.768272 29954 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:54.770519 30360 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:54.771025 30360 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.771474 30360 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:54.772401 27587 pending_rounds.cc:85] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 6
I20260709 12:18:54.772712 27587 pending_rounds.cc:107] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:18:54.772938 27587 raft_consensus.cc:2889] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:54.769958 27662 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:18:54.737332 30290 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:54.780777 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.781602 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:54.782538 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.809119 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.780826 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:54.810313 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:54.741832 30313 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:54.812239 27732 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:18:54.814106 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.815209 28336 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:54.815250 27655 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:18:54.815991 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.816752 27584 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:54.816226 27586 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:54.820642 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:54.824681 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:54.825260 27736 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:54.825557 27701 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.521s (SENT)
I20260709 12:18:54.757608 30282 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:54.754845 30256 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:54.741645 30297 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:18:54.975147 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 311) took 1638 ms. Trace:
I20260709 12:18:54.932861 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.579438 27730 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:54.964900 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.578887 27733 rpcz_store.cc:276] 0709 12:18:53.337027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.337195 (+   168us) service_pool.cc:224] Handling call
0709 12:18:53.337537 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.337945 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:18:54.017001 (+679056us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b18003e1520. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:18:54.026943 (+  9942us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.312006 (+285063us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:54.358201 (+ 46195us) log.cc:844] Serialized 26 byte log entry
0709 12:18:54.358360 (+   159us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:54.451484 (+ 93124us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.451717 (+   233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.451793 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.974937 (+523144us) raft_consensus.cc:1685] finished
0709 12:18:54.974958 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.975087 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21195,"prepare.queue_time_us":175767,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":93970,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":19614,"spinlock_wait_cycles":4337280,"thread_start_us":243535,"threads_started":4,"wal-append.queue_time_us":279107}
I20260709 12:18:55.580799 27737 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:54.859344 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.585114 27729 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:54.932396 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.956053 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:55.582396 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.012s	sys 0.017s
I20260709 12:18:54.933843 27655 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[5.2-7.3]   Dedup: 5.2->[7.3-7.3]
W20260709 12:18:55.586009 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.012s	sys 0.018s
W20260709 12:18:54.882706 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.586597 27655 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [5.2-7.3]
I20260709 12:18:54.833521 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:55.588027 27586 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:54.858870 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:55.588632 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.588860 27731 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:18:54.906774 27659 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:55.263269 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 335957us
I20260709 12:18:55.590854 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:55.591495 27581 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:55.058781 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 298) took 1458 ms. Trace:
W20260709 12:18:55.326766 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.045248 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:55.219729 27583 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:55.594739 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:55.595595 27582 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:55.047052 27701 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.571s (SENT)
W20260709 12:18:55.596931 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 549881us
W20260709 12:18:55.597666 27701 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.721s (SENT)
I20260709 12:18:55.598160 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:54.977156 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:54.875252 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.973860 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.599277 27588 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:55.576572 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:54.987648 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.600283 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:18:55.005265 27736 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
W20260709 12:18:55.322420 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.963030 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.592211 27589 rpcz_store.cc:276] 0709 12:18:53.600832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.612982 (+ 12150us) service_pool.cc:224] Handling call
0709 12:18:53.670550 (+ 57568us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:54.662442 (+991892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.058480 (+396038us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:55.058709 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2279623936}
I20260709 12:18:55.603981 27657 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:18:54.838954 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:54.823590 30030 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:18:55.046051 30353 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:18:55.507071 30230 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.145s	user 0.006s	sys 0.006s
W20260709 12:18:54.875692 30315 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260709 12:18:54.989410 30356 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.305s	user 0.000s	sys 0.001s
I20260709 12:18:55.611667 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.991780 29945 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:55.096040 30332 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.736s	user 0.009s	sys 0.001s
I20260709 12:18:54.988060 30346 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:55.605760 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:55.219482 30203 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.759s	user 0.005s	sys 0.005s
W20260709 12:18:54.988761 30359 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:18:55.615677 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.633524 27734 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:18:54.830812 30021 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:55.638768 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 334) took 1609 ms. Trace:
I20260709 12:18:55.639293 27656 rpcz_store.cc:276] 0709 12:18:54.029858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.302032 (+272174us) service_pool.cc:224] Handling call
0709 12:18:54.302593 (+   561us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:54.314341 (+ 11748us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:54.314377 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:54.344836 (+ 30459us) log.cc:844] Serialized 78 byte log entry
0709 12:18:54.460007 (+115171us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:54.460352 (+   345us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.727883 (+267531us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.728114 (+   231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.728226 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:55.638446 (+910220us) raft_consensus.cc:1685] finished
0709 12:18:55.638475 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:55.638694 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":580161,"prepare.queue_time_us":3292,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":73,"raft.queue_time_us":287185,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":16279,"thread_start_us":103505,"threads_started":2,"wal-append.queue_time_us":152050}
I20260709 12:18:55.642697 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.618364 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:55.619246 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:55.645226 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.617659 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:54.831228 30313 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:55.621306 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:55.622063 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:55.646499 27733 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:55.623202 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:55.096155 30158 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.990242 30292 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:54.989243 30341 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:54.785265 30291 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:55.092087 30328 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.904s	user 0.002s	sys 0.003s
I20260709 12:18:55.626334 27736 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:55.627820 27588 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:55.633055 30346 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:55.628845 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.666394 27730 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:18:55.632331 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:55.615696 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:55.635445 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:54.823873 30038 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.791s
I20260709 12:18:55.664119 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:55.669963 27589 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:55.651415 30364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.005s	sys 0.012s
I20260709 12:18:55.637513 27735 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:18:55.639562 27663 raft_consensus.cc:1217] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[5.6-7.7]   Dedup: 7.7->[]
I20260709 12:18:55.091043 30305 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:55.626019 30230 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:55.646548 27737 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:55.622754 30030 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:55.648459 27584 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:18:55.648923 30358 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.058110 29954 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:55.614593 30149 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:55.097591 30360 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:55.635524 30203 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:54.962711 30362 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.728s
I20260709 12:18:55.658700 27581 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:55.658936 27734 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:54.951401 30347 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.904s
I20260709 12:18:55.662139 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:55.672329 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.684207 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.671291 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:55.674391 27663 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260709 12:18:55.646674 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:18:55.675733 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:55.636189 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.009s	sys 0.007s
I20260709 12:18:55.676335 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:55.665825 30282 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.002s	sys 0.017s
I20260709 12:18:55.635418 30332 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.665241 30341 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:55.651047 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.680745 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.665896 29941 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.682919 27733 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:55.083467 30256 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:55.673767 30364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.006s	sys 0.013s
I20260709 12:18:55.685674 27584 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
W20260709 12:18:55.687047 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.009s	sys 0.007s
I20260709 12:18:55.688896 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:55.667584 30346 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:55.659742 30158 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.713176 27589 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:18:55.711304 27731 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:55.689123 27588 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:55.732582 27581 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:55.687150 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.733242 27734 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:55.712929 27730 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:55.691207 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:55.683501 30110 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.688122 30021 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:55.733628 29941 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.721972 30332 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:55.877379 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 260384us
W20260709 12:18:55.887835 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 290178us
I20260709 12:18:55.887852 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.888967 27730 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
W20260709 12:18:55.889326 27701 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.792s (SENT)
I20260709 12:18:55.889881 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:55.891012 30388 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:18:55.873279 30021 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:55.779565 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.898720 27655 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:55.780122 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:55.781100 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.900548 27734 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:18:55.688534 30282 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.002s	sys 0.018s
I20260709 12:18:55.721935 29954 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:55.784411 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.784494 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:55.904047 27731 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:55.904165 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1589) took 1075 ms. Trace:
I20260709 12:18:55.686470 30358 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:55.870661 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.012s	sys 0.009s
I20260709 12:18:55.712010 27656 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:18:55.871752 27737 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:18:55.720548 30377 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:55.896643 30371 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:55.731079 30149 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.788422 30313 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.901674 27732 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:55.902079 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.917490 27655 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:55.900136 30021 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:55.904582 27730 rpcz_store.cc:276] 0709 12:18:54.829057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.898786 (+1069729us) service_pool.cc:224] Handling call
0709 12:18:55.903655 (+  4869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.903823 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:55.903846 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:55.903891 (+    45us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:55.903918 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.903991 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:55.904109 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:55.919982 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:55.921320 27731 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:18:55.921945 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1590) took 1092 ms. Trace:
I20260709 12:18:55.921998 30371 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:55.922365 27731 rpcz_store.cc:276] 0709 12:18:54.829289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.904670 (+1075381us) service_pool.cc:224] Handling call
0709 12:18:55.918847 (+ 14177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.921702 (+  2855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.921889 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:55.922814 30358 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:55.921312 27730 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:18:55.924499 27582 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:55.690207 27658 raft_consensus.cc:2368] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7.
I20260709 12:18:55.906909 27735 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 3: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:18:55.907397 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.033s	user 0.013s	sys 0.009s
W20260709 12:18:55.906080 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 236539us
I20260709 12:18:55.909255 27734 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:55.914230 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.928158 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1591) took 1098 ms. Trace:
I20260709 12:18:55.928251 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
W20260709 12:18:55.915627 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 224458us
I20260709 12:18:55.928529 27734 rpcz_store.cc:276] 0709 12:18:54.829506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.908506 (+1079000us) service_pool.cc:224] Handling call
0709 12:18:55.908811 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.927910 (+ 19099us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.928106 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:55.928913 27737 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:18:55.929607 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1592) took 1099 ms. Trace:
I20260709 12:18:55.929896 27737 rpcz_store.cc:276] 0709 12:18:54.829750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.916094 (+1086344us) service_pool.cc:224] Handling call
0709 12:18:55.928621 (+ 12527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.929364 (+   743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.929538 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:55.918588 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.902151 30282 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:55.875770 30110 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.905470 30381 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:55.902650 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 1.06369e+06us
I20260709 12:18:55.932196 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:55.933099 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:55.875090 30030 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:55.913142 30377 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:55.936323 27656 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:18:55.936754 27588 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:55.937786 27735 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: [4.5-5.6]
I20260709 12:18:55.942842 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:18:55.942193 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52690.6us
I20260709 12:18:55.945812 27656 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:55.946835 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:18:55.949041 27701 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.833s (SENT)
I20260709 12:18:55.949693 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:55.931249 30384 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.954015 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1535) took 2410 ms. Trace:
I20260709 12:18:55.961364 27735 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:55.950650 30388 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:55.927402 30305 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:18:55.931368 30369 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:55.964936 27656 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:18:55.940737 30297 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:18:55.964473 27737 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:18:55.939915 30360 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:55.970819 27737 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:18:55.938231 30380 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:18:55.959126 27661 rpcz_store.cc:276] 0709 12:18:53.532605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.602133 (+ 69528us) service_pool.cc:224] Handling call
0709 12:18:53.616699 (+ 14566us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:53.617213 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.617254 (+    41us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:53.678050 (+ 60796us) log.cc:844] Serialized 78 byte log entry
0709 12:18:53.854169 (+176119us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:53.857608 (+  3439us) log.cc:844] Serialized 14 byte log entry
0709 12:18:53.859800 (+  2192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.859868 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.964184 (+1104316us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:55.938041 (+973857us) raft_consensus.cc:1685] finished
0709 12:18:55.942635 (+  4594us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:55.942801 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1948,"prepare.queue_time_us":3627,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":2965,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":9533,"spinlock_wait_cycles":8288640,"thread_start_us":95761,"threads_started":1,"wal-append.queue_time_us":332440}
I20260709 12:18:55.943269 30377 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:55.968727 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:55.984773 27581 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:18:55.980095 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.946844 30291 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:55.972280 27737 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 2
I20260709 12:18:55.966332 29873 tablet_copy_client.cc:538] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:55.983901 30203 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.000437 30391 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:56.001677 27737 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20260709 12:18:56.004614 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74551.5us
I20260709 12:18:56.004774 27737 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:56.005924 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:56.015812 27581 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:18:56.000485 30387 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:56.035928 29941 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.036214 30387 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:55.968413 30230 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:56.037994 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98366.2us
I20260709 12:18:56.040025 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:55.939091 30381 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.048503 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:56.042609 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54012.5us
I20260709 12:18:56.044008 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:56.004436 30390 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.051973 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:56.005300 30369 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:56.007719 30360 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.055202 30230 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:18:56.071723 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1532) took 2550 ms. Trace:
I20260709 12:18:56.072665 27664 rpcz_store.cc:276] 0709 12:18:53.521336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.614665 (+ 93329us) service_pool.cc:224] Handling call
0709 12:18:53.615093 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.615394 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.615435 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:53.661172 (+ 45737us) log.cc:844] Serialized 26 byte log entry
0709 12:18:53.721643 (+ 60471us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:53.721699 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.721794 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.060029 (+2338235us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b24002ac248. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:18:56.060797 (+   768us) raft_consensus.cc:1685] finished
0709 12:18:56.060822 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.071629 (+ 10807us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5561600,"thread_start_us":41624,"threads_started":1,"wal-append.queue_time_us":174407}
I20260709 12:18:55.970835 30282 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.086297 27664 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:18:56.087574 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1543) took 1013 ms. Trace:
I20260709 12:18:56.088369 27664 rpcz_store.cc:276] 0709 12:18:55.073890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.085609 (+1011719us) service_pool.cc:224] Handling call
0709 12:18:56.086066 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.087086 (+  1020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.087481 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.091794 27664 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:18:56.093130 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1544) took 1019 ms. Trace:
I20260709 12:18:56.093626 27664 rpcz_store.cc:276] 0709 12:18:55.074170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.090880 (+1016710us) service_pool.cc:224] Handling call
0709 12:18:56.091376 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.092667 (+  1291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.093064 (+   397us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.095424 27664 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:55.966660 30384 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.099558 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.056639 30393 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.076879 30367 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:56.039237 30380 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260709 12:18:56.099041 30256 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:56.111732 30256 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.052078 30394 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 12:18:56.101918 30391 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:18:56.076525 30360 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.099603 30400 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.108435 30367 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.110560 30203 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:56.061247 30390 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.107930 30399 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:56.116870 30297 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:56.119724 30149 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:56.122179 27664 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:18:56.149780 27664 pending_rounds.cc:85] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 2
I20260709 12:18:56.150151 27664 pending_rounds.cc:107] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:18:56.150470 27664 raft_consensus.cc:2889] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:56.131556 30369 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.140213 30256 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:56.140285 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 191281us
I20260709 12:18:56.119691 30407 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:56.155650 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:56.158228 27656 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:18:56.137315 30367 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:56.157191 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:56.160897 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:56.143865 30203 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.164780 30297 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.166790 30203 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:56.141937 30380 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:56.161818 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:56.076182 30388 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.148375 30158 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:56.109930 30393 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.167562 30380 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.164891 30404 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.168788 30256 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:18:56.124449 30392 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:18:56.167176 30292 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:56.147881 30394 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.227552 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1598) took 1095 ms. Trace:
I20260709 12:18:56.184460 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:56.136034 30395 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:18:56.242714 27735 rpcz_store.cc:276] 0709 12:18:55.132513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.964856 (+832343us) service_pool.cc:224] Handling call
0709 12:18:55.965112 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.000475 (+ 35363us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:18:56.115130 (+114655us) log.cc:844] Serialized 14 byte log entry
0709 12:18:56.227034 (+111904us) log.cc:844] Serialized 14 byte log entry
0709 12:18:56.227186 (+   152us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:56.227227 (+    41us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:56.227272 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.227359 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.227488 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":51272,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":164429,"threads_started":2}
I20260709 12:18:56.289701 27735 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:56.141264 30400 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.207333 30401 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.209913 30380 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.207149 30203 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:56.170048 30304 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:18:56.231519 30292 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:56.230070 30403 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:18:56.291059 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 191533us
I20260709 12:18:56.321486 30395 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:56.322214 30395 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:56.322995 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1596) took 1276 ms. Trace:
I20260709 12:18:56.323211 30395 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.323470 27732 rpcz_store.cc:276] 0709 12:18:55.046507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.927378 (+880871us) service_pool.cc:224] Handling call
0709 12:18:55.935790 (+  8412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:55.963582 (+ 27792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:55.963610 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:55.965428 (+  1818us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.098421 (+132993us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:56.098465 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.098534 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.322753 (+224219us) raft_consensus.cc:1685] finished
0709 12:18:56.322775 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.322932 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28018,"raft.queue_time_us":104002,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7481,"thread_start_us":121578,"threads_started":2,"wal-append.queue_time_us":170706}
I20260709 12:18:56.325219 30360 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.323491 30398 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.193s
I20260709 12:18:56.328503 30393 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:56.331387 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1601) took 1033 ms. Trace:
I20260709 12:18:56.331881 27737 rpcz_store.cc:276] 0709 12:18:55.297979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.971657 (+673678us) service_pool.cc:224] Handling call
0709 12:18:55.972024 (+   367us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:56.053439 (+ 81415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.053469 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:56.054041 (+   572us) log.cc:844] Serialized 52 byte log entry
0709 12:18:56.155948 (+101907us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:56.202324 (+ 46376us) log.cc:844] Serialized 14 byte log entry
0709 12:18:56.202520 (+   196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.202599 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.331084 (+128485us) raft_consensus.cc:1685] finished
0709 12:18:56.331110 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.331292 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21610,"prepare.queue_time_us":46503,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":76365,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3667,"spinlock_wait_cycles":4064128,"thread_start_us":94878,"threads_started":1,"wal-append.queue_time_us":220771}
I20260709 12:18:56.170444 30407 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.335503 30387 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:56.352131 30403 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:56.305706 30203 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:56.438927 30403 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.229306 30404 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:56.323138 30389 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:18:56.297149 30256 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:56.433561 29873 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:18:56.305089 30110 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:56.415087 30346 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.630s	user 0.004s	sys 0.006s
I20260709 12:18:56.355176 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:56.336915 30390 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.436033 30395 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:56.458041 27738 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.018s	sys 0.049s
I20260709 12:18:56.343216 30381 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:56.445998 30403 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:18:56.463697 27738 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.018s	sys 0.050s
I20260709 12:18:56.462821 30414 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.294245 30394 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:56.473192 30021 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.000s
I20260709 12:18:56.464653 30346 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.482860 30346 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:56.483117 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:56.483443 30394 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.484499 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:56.486080 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:56.411617 30401 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.471570 30384 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.466167 30292 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.496457 30292 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:56.496603 30407 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:56.498204 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 321) took 1405 ms. Trace:
I20260709 12:18:56.479866 30021 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.498668 27734 rpcz_store.cc:276] 0709 12:18:55.093144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.939452 (+846308us) service_pool.cc:224] Handling call
0709 12:18:55.939759 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.040149 (+100390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.040182 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.040651 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.447371 (+406720us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:56.447429 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.447521 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.497971 (+ 50450us) raft_consensus.cc:1685] finished
0709 12:18:56.497998 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.498145 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8143,"raft.queue_time_us":121051,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":27183,"thread_start_us":496051,"threads_started":2,"wal-append.queue_time_us":408897}
I20260709 12:18:56.478359 30414 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.516543 27655 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:18:56.516949 30375 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.632s	user 0.000s	sys 0.015s
I20260709 12:18:56.517091 30021 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.518869 30371 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.520331 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1824) took 1917 ms. Trace:
I20260709 12:18:56.520763 27580 rpcz_store.cc:276] 0709 12:18:54.602431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.602554 (+   123us) service_pool.cc:224] Handling call
0709 12:18:56.519703 (+1917149us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b5400087928. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea01255f4 00007f0e9f30e227 00007f0e9f30dab8 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6
0709 12:18:56.520060 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35380992}
I20260709 12:18:56.523152 30401 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:56.526669 30374 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.650s	user 0.003s	sys 0.000s
I20260709 12:18:56.528687 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:56.529824 28332 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:56.530675 30283 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:56.531633 28328 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:56.532413 29941 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:18:56.536765 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.528843 30397 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:56.539361 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:56.531107 30283 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:18:56.541414 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79235.4us
I20260709 12:18:56.490374 30377 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.532816 29941 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:56.545673 30400 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.545799 30203 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.547055 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1602) took 1222 ms. Trace:
I20260709 12:18:56.554486 27736 rpcz_store.cc:276] 0709 12:18:55.325018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.056712 (+731694us) service_pool.cc:224] Handling call
0709 12:18:56.078826 (+ 22114us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.222384 (+143558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.222421 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.223145 (+   724us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.293814 (+ 70669us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:56.293860 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.293935 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.546803 (+252868us) raft_consensus.cc:1685] finished
0709 12:18:56.546827 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.546985 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29015,"raft.queue_time_us":84358,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7070,"thread_start_us":183754,"threads_started":2,"wal-append.queue_time_us":255771}
I20260709 12:18:56.552325 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:56.552433 30404 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.489256 30256 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:56.565131 30210 raft_consensus.cc:1068] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:18:56.568946 29941 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.570284 30203 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:56.570519 30283 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.571733 30283 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.575927 30367 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.577335 28328 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:56.578495 30367 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.581354 27660 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:18:56.581754 27660 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 2
I20260709 12:18:56.581987 27660 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:18:56.582226 27660 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:56.582950 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1824) took 1004 ms. Trace:
I20260709 12:18:56.542464 30397 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.594388 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1783) took 2767 ms. Trace:
I20260709 12:18:56.594969 27663 rpcz_store.cc:276] 0709 12:18:53.827280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.285535 (+458255us) service_pool.cc:224] Handling call
0709 12:18:55.639422 (+1353887us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:55.686201 (+ 46779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.594126 (+907925us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001e41d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:56.594304 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3038860544}
I20260709 12:18:56.602425 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.583263 27660 rpcz_store.cc:276] 0709 12:18:55.578179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.580606 (+1002427us) service_pool.cc:224] Handling call
0709 12:18:56.580985 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.582681 (+  1696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.582893 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.604195 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:56.605155 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:56.605602 27660 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:56.606114 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:56.606544 27660 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:18:56.608173 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:56.614059 30380 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.600679 29941 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:56.608680 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:18:56.623721 27655 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:56.607826 30414 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.629189 30149 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:18:56.629678 30149 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:56.630180 30397 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:56.630487 30377 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:56.635767 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 280668us
I20260709 12:18:56.636854 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.637202 30292 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:56.637728 30292 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:56.638130 30292 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:56.639950 27660 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:18:56.640954 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.641721 30381 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:56.642144 30381 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.642566 30381 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:56.644739 28332 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:56.645078 30380 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:56.645815 30380 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:56.629750 30398 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:56.657310 27660 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:56.639067 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:56.661305 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:56.666127 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:56.667052 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64735.4us
I20260709 12:18:56.668390 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.669679 28337 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:56.670002 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:56.668339 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.676268 27733 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:18:56.677753 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:56.678411 27737 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:18:56.659724 30256 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:56.686765 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:56.682407 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.687438 27736 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:56.656730 30398 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:56.694461 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:56.672791 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:56.682902 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:56.699433 27737 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:56.684705 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:56.680011 30403 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:56.659598 30110 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.644850 30421 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:18:56.687942 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:56.700587 27734 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:56.688609 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:56.679409 30341 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:18:56.702433 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:56.702741 30403 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:56.703222 30403 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.703230 27734 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:56.681006 30392 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:56.707885 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:56.681097 30377 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:56.710703 30110 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.708240 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:56.734820 27736 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:56.713630 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:56.735549 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.739668 27737 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:56.735085 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.666617 30393 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 12:18:56.686978 30256 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.686872 30395 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:56.749411 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58403.4us
I20260709 12:18:56.681778 30363 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
W20260709 12:18:56.746829 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71308.5us
I20260709 12:18:56.683152 30302 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:18:56.681030 30380 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20260709 12:18:56.744774 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.687022 30367 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:18:56.736603 30414 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.752496 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.680917 30416 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:18:56.757112 30392 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:18:56.711584 30391 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:18:56.748651 30400 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:56.765502 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.769641 30438 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:56.758476 27737 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:18:56.758926 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.759519 30404 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:56.751592 30256 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:56.711683 30388 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:56.681942 30424 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260709 12:18:56.749716 30395 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:56.756253 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.743981 30210 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:56.782642 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:56.787674 30401 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:56.772181 30400 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:56.759932 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87640.2us
I20260709 12:18:56.784098 30404 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:56.800638 28336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 446) took 2083 ms (client timeout 1572 ms). Trace:
I20260709 12:18:56.801242 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:56.808988 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52802.6us
W20260709 12:18:56.816697 28336 rpcz_store.cc:269] 0709 12:18:54.717215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.717356 (+   141us) service_pool.cc:224] Handling call
0709 12:18:56.800440 (+2083084us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b54006bc6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:56.800569 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222169344}
I20260709 12:18:56.801733 30392 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:18:56.816148 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.815521 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:56.819144 30404 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:56.765208 30414 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.809888 30438 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:56.808641 30395 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.839454 30282 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.844568 30282 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:56.845141 30414 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.835151 30392 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:18:56.832731 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.818783 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.815099 30401 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.808838 30403 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.813252 30302 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:56.818118 30388 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:56.814841 30400 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:56.838918 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.849819 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:56.860140 27582 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:56.850008 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:56.851758 27657 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:18:56.862362 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:56.822624 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.853372 27580 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:56.821951 30377 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:56.868237 30388 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:18:56.868655 30377 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.839568 30203 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:18:56.868279 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.871538 30203 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:56.871932 30203 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.872260 30203 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.874768 30210 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:56.858073 30367 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:56.875439 30413 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.875993 30367 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:56.876111 30413 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.876966 30367 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.878391 30404 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.876180 29873 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:18:56.854374 30401 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:56.881867 29873 tablet_bootstrap.cc:492] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:18:56.833361 30110 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:56.863953 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.884244 29873 ts_tablet_manager.cc:1403] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.451s	user 0.035s	sys 0.012s
I20260709 12:18:56.884694 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.865298 27581 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:18:56.855039 27583 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:18:56.843977 30395 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.897694 30400 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.900072 30400 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.867467 30438 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:18:56.867566 30403 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.875497 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:56.847898 30382 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:18:56.857834 30302 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:56.933210 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48605.8us
I20260709 12:18:56.933454 30302 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:56.934142 30345 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:56.934628 30345 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:56.935499 30345 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.935698 29873 raft_consensus.cc:359] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.936549 30345 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.936312 30398 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.937235 29873 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:18:56.937786 30437 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:18:56.890661 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.938042 29873 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:18:56.854142 30282 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.940969 30341 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:18:56.941355 30341 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:56.942200 30341 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.868661 30388 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:56.944099 30388 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:18:56.896152 30404 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:56.813117 30292 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:56.934144 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.949175 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:56.950951 27583 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:18:56.946764 30381 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:56.951251 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:56.843792 30371 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.088s
I20260709 12:18:56.956776 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:18:56.954573 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83262.5us
I20260709 12:18:56.960635 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:56.956696 30398 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.910802 30438 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:56.899129 30203 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.024653 27737 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:56.957178 30400 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:57.023142 30387 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:57.022378 30438 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:56.971566 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:56.972455 30437 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:57.028882 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.069491 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:57.070124 27585 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:56.955508 30110 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:18:57.033948 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:56.970798 30424 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:18:57.041777 30149 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:57.044726 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1799) took 2594 ms (client timeout 1721 ms). Trace:
W20260709 12:18:57.075656 27658 rpcz_store.cc:269] 0709 12:18:54.450036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.450170 (+   134us) service_pool.cc:224] Handling call
0709 12:18:57.044378 (+2594208us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400210554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:57.044588 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2801835264}
I20260709 12:18:57.068531 30437 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:57.085353 30437 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:57.086581 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:57.057381 30400 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.973145 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:57.057029 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.953616 30441 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:18:57.087868 27733 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20260709 12:18:57.094020 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122687us
I20260709 12:18:57.094880 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:57.056907 30413 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:57.058229 30181 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:18:57.092155 30409 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:57.045477 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.099413 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.100528 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 312) took 1508 ms. Trace:
I20260709 12:18:57.153347 27733 raft_consensus.cc:2376] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5. Re-sending same reply.
I20260709 12:18:57.104758 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1822) took 1527 ms. Trace:
I20260709 12:18:57.106369 30282 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.107692 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.108666 27585 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:57.111506 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:57.112613 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 309) took 1522 ms. Trace:
W20260709 12:18:57.119038 30372 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.248s	user 0.002s	sys 0.002s
W20260709 12:18:57.121984 30418 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.664s	user 0.004s	sys 0.000s
I20260709 12:18:57.132031 30292 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:57.134729 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:57.003669 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:57.140091 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.142184 27583 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 3. Current term is 4. Ops: []
I20260709 12:18:57.146246 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.209575 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.257004 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.082772 30403 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.153729 27584 rpcz_store.cc:276] 0709 12:18:55.593291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.679411 (+ 86120us) service_pool.cc:224] Handling call
0709 12:18:55.682493 (+  3082us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.733362 (+ 50869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.100314 (+1366952us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5400186fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:57.100465 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6466816}
I20260709 12:18:57.267434 27589 rpcz_store.cc:276] 0709 12:18:55.590260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.677126 (+ 86866us) service_pool.cc:224] Handling call
0709 12:18:55.677442 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.731454 (+ 54012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.112324 (+1380870us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:57.112545 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81592576}
I20260709 12:18:57.081873 30392 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:57.265666 27582 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:57.271334 27581 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:57.274392 27734 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:57.081997 30387 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:57.278656 30158 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:57.258049 30443 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260709 12:18:57.285696 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:57.287635 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 176285us
I20260709 12:18:57.303501 27585 raft_consensus.cc:2368] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:57.291611 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.097841 30400 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:57.291034 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.294910 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.282894 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:57.265851 27655 rpcz_store.cc:276] 0709 12:18:55.577559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.518282 (+940723us) service_pool.cc:224] Handling call
0709 12:18:56.518616 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.624200 (+105584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.104529 (+480329us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:57.104699 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241258752}
I20260709 12:18:57.311983 27583 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:57.255880 30210 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:57.313985 27580 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:57.313045 30292 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:18:57.306015 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.318689 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.328922 27583 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:57.320947 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.330435 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.322450 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.332224 27582 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:57.344137 27733 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:57.267735 30424 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.096s
I20260709 12:18:57.266551 30409 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:57.408460 30347 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:57.409296 30347 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.411774 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.412787 30445 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:57.414464 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:57.415751 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:57.431703 30445 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.432349 30445 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.435849 30445 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:57.454806 30388 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.510s	user 0.003s	sys 0.005s
I20260709 12:18:57.457222 30381 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:57.458716 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:57.460816 29873 ts_tablet_manager.cc:1434] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.576s	user 0.010s	sys 0.001s
W20260709 12:18:57.461282 29873 ts_tablet_manager.cc:1467] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Tablet startup took 576ms
I20260709 12:18:57.281128 30282 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:57.283655 30403 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:57.481123 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.301344 30395 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.482366 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 318) took 2391 ms. Trace:
W20260709 12:18:57.483081 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.005s	sys 0.062s
I20260709 12:18:57.310418 30401 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.311097 30158 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:57.487612 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.006s	sys 0.062s
W20260709 12:18:57.492910 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 523008us
W20260709 12:18:57.482769 30367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.070s	sys 0.110s
I20260709 12:18:57.490629 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1705) took 2703 ms. Trace:
I20260709 12:18:57.508595 27738 rpcz_store.cc:276] 0709 12:18:54.781012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.781127 (+   115us) service_pool.cc:224] Handling call
0709 12:18:54.781472 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:54.781852 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:54.781879 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:54.808611 (+ 26732us) log.cc:844] Serialized 26 byte log entry
0709 12:18:55.732624 (+924013us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:55.918452 (+185828us) log.cc:844] Serialized 14 byte log entry
0709 12:18:56.467082 (+548630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.467189 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.118325 (+651136us) spinlock_profiling.cc:243] Waited 9.33 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:57.470291 (+351966us) raft_consensus.cc:1685] finished
0709 12:18:57.483932 (+ 13641us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.484141 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35501,"prepare.queue_time_us":233701,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":84262,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":13841,"spinlock_wait_cycles":99756672,"thread_start_us":1506509,"threads_started":3,"wal-append.queue_time_us":777352}
I20260709 12:18:57.329386 30210 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:57.489468 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.511025 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:57.487527 27731 rpcz_store.cc:276] 0709 12:18:55.091380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.923145 (+831765us) service_pool.cc:224] Handling call
0709 12:18:55.923471 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.071009 (+147538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.071059 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.082554 (+ 11495us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.154051 (+ 71497us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:56.154093 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.154172 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.121241 (+967069us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:57.470011 (+348770us) raft_consensus.cc:1685] finished
0709 12:18:57.470039 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.482275 (+ 12236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6427,"raft.queue_time_us":80416,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":11673472,"thread_start_us":171781,"threads_started":2,"wal-append.queue_time_us":205688}
I20260709 12:18:57.514602 27732 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:57.515249 30347 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:57.455570 30387 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.523592 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.525514 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:57.506263 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 410198us
I20260709 12:18:57.533751 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:57.582471 27731 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:18:57.526302 27738 raft_consensus.cc:2376] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5. Re-sending same reply.
I20260709 12:18:57.583904 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.584671 27738 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
W20260709 12:18:57.585008 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51382.5us
I20260709 12:18:57.585744 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:57.586009 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:57.588658 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:57.589669 27738 raft_consensus.cc:2376] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7. Re-sending same reply.
I20260709 12:18:57.590914 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.591777 27738 raft_consensus.cc:2376] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7. Re-sending same reply.
I20260709 12:18:57.592864 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.624300 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:57.630877 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.632496 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.633563 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:57.634255 27731 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:18:57.516812 30367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.072s	sys 0.110s
I20260709 12:18:57.478811 30388 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:57.646148 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:57.643208 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:57.670053 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:57.710081 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:57.508153 30345 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.134s
I20260709 12:18:57.710487 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:57.629897 30292 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:57.725001 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75565.9us
I20260709 12:18:57.728581 28336 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:18:57.623862 30203 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:57.644577 30381 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:57.738374 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:57.740289 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:57.676047 30445 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:18:57.736475 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.012s	sys 0.036s
I20260709 12:18:57.747174 27738 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:57.625494 30347 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260709 12:18:57.737352 30381 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:57.750617 30393 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:57.750914 30381 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:57.754447 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1542) took 2680 ms. Trace:
W20260709 12:18:57.736941 30324 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.020s	sys 0.029s
I20260709 12:18:57.754923 27656 rpcz_store.cc:276] 0709 12:18:55.073572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.988842 (+915270us) service_pool.cc:224] Handling call
0709 12:18:56.008015 (+ 19173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.158737 (+150722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.754129 (+1595392us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:57.754375 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344486784}
W20260709 12:18:57.755831 30324 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.020s	sys 0.029s
I20260709 12:18:57.758138 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.759557 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:18:57.759980 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.760231 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:57.761247 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.762095 27737 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:18:57.751221 30393 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.764101 30393 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:57.764180 30441 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.765004 30441 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.766811 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.770289 30441 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:57.770555 27737 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:57.771387 30367 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:18:57.771822 30367 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:57.747274 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.012s	sys 0.036s
I20260709 12:18:57.780670 30210 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:57.782264 30367 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:57.726200 30371 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:18:57.795529 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.030s	sys 0.057s
W20260709 12:18:57.796089 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.030s	sys 0.058s
W20260709 12:18:57.797034 30382 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.840s	user 0.011s	sys 0.001s
W20260709 12:18:57.818796 30448 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:18:57.820145 30158 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:57.741919 30413 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
W20260709 12:18:57.806565 30398 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.837s	user 0.010s	sys 0.002s
I20260709 12:18:57.833372 30409 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:57.808115 30371 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:57.834496 30395 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:57.821973 30382 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:18:57.835999 30388 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:57.814942 30367 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:57.839716 30282 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:57.839900 30302 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.906s	user 0.004s	sys 0.008s
I20260709 12:18:57.737970 30292 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:57.841591 30400 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.000s
W20260709 12:18:57.842273 30437 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.756s	user 0.008s	sys 0.012s
W20260709 12:18:57.844151 27628 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.047s	sys 0.025s
W20260709 12:18:57.880072 27628 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.047s	sys 0.025s
W20260709 12:18:57.880437 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 581513us
I20260709 12:18:57.881456 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:57.879900 27585 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.009s	sys 0.045s
W20260709 12:18:57.882664 27585 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.009s	sys 0.045s
I20260709 12:18:57.891796 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:57.892558 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:18:57.848546 27734 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.000s
I20260709 12:18:57.849491 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1836) took 2392 ms. Trace:
I20260709 12:18:57.851732 27733 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:57.852160 27738 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:18:57.862519 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:57.869161 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:57.872287 27587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.023s	sys 0.030s
W20260709 12:18:57.858867 30377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.023s	sys 0.060s
W20260709 12:18:57.897399 27587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.024s	sys 0.030s
I20260709 12:18:57.902486 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:57.904155 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:57.905427 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.908541 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:57.909066 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:57.897722 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.014s	sys 0.011s
I20260709 12:18:57.906136 27581 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
W20260709 12:18:57.910420 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s	user 0.014s	sys 0.011s
I20260709 12:18:57.900451 27734 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:57.873426 30413 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:18:57.894145 27586 rpcz_store.cc:276] 0709 12:18:55.457396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.605330 (+147934us) service_pool.cc:224] Handling call
0709 12:18:55.605711 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:55.606142 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:55.606201 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:57.099034 (+1492833us) spinlock_profiling.cc:243] Waited 7.21 ms on lock 0x7b2c000014c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:57.099478 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:18:57.773153 (+673675us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:57.773211 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.773571 (+   360us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.844031 (+ 70460us) raft_consensus.cc:1685] finished
0709 12:18:57.844058 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.849376 (+  5318us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":42553,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":16575360,"thread_start_us":673192,"threads_started":1,"wal-append.queue_time_us":691239}
I20260709 12:18:57.884150 30382 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.919943 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.887403 30401 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:57.886418 30388 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:57.924328 27586 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:18:57.872088 29941 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 12:18:57.873632 30409 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 12:18:57.921981 27581 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 12:18:57.461724 29873 ts_tablet_manager.cc:1469] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Trace:
0709 12:18:49.221351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.221860 (+   509us) service_pool.cc:224] Handling call
0709 12:18:49.608451 (+386591us) ts_tablet_manager.cc:934] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
0709 12:18:53.332738 (+3724287us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:53.502842 (+170104us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:53.502967 (+   125us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:54.010370 (+507403us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:54.019830 (+  9460us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:54.040806 (+ 20976us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:54.040914 (+   108us) inbound_call.cc:177] Queueing success response
0709 12:18:56.433175 (+2392261us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:18:56.884665 (+451490us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:18:56.896036 (+ 11371us) tablet_replica.cc:261] Starting instrumentation
0709 12:18:56.933741 (+ 37705us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":49680,"raft.queue_time_us":4227,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"tablet-copy.queue_time_us":242622,"tablet-download-pool.queue_time_us":533,"tablet-download-pool.run_cpu_time_us":1571,"tablet-download-pool.run_wall_time_us":1709987,"thread_start_us":493288,"threads_started":2}
W20260709 12:18:57.939265 27735 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.024s	sys 0.045s
I20260709 12:18:57.895530 30210 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:18:57.942154 27662 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:57.941215 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:57.879570 30302 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:57.885226 30400 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:18:57.944931 27735 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.024s	sys 0.046s
I20260709 12:18:57.956712 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 337) took 1136 ms. Trace:
I20260709 12:18:57.956825 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:57.917219 30402 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:57.958189 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:57.945883 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:57.913142 30377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.023s	sys 0.061s
I20260709 12:18:57.884349 30393 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.896981 30445 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.912935 30421 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:18:57.914537 30417 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:57.915741 30399 raft_consensus.cc:2764] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:18:57.915058 30181 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.975932 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:57.954691 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:57.896450 30158 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:18:57.922401 30347 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:18:57.957108 27735 rpcz_store.cc:276] 0709 12:18:56.820022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.820133 (+   111us) service_pool.cc:224] Handling call
0709 12:18:56.820379 (+   246us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.825662 (+  5283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.825713 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.826277 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:18:57.276154 (+449877us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:57.956154 (+680000us) log.cc:844] Serialized 14 byte log entry
0709 12:18:57.956382 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.956447 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.956488 (+    41us) raft_consensus.cc:1685] finished
0709 12:18:57.956507 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.956656 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4822,"raft.queue_time_us":14716,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":2164,"thread_start_us":1018126,"threads_started":2,"wal-append.queue_time_us":654733}
I20260709 12:18:57.986600 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:57.987800 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.966573 30203 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:18:57.968014 27729 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:18:57.967705 27581 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:18:57.969857 30445 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:57.970453 27655 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:57.973465 27657 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:18:57.887104 30437 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:57.959837 30401 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:57.963279 30291 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:18:57.983656 27584 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
W20260709 12:18:57.962997 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.018s	sys 0.025s
I20260709 12:18:57.985925 27731 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:18:57.928825 30382 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:57.919039 30380 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:18:57.944542 27734 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
W20260709 12:18:57.975888 30449 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 12:18:57.990859 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:57.993860 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:58.069399 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.018s	sys 0.025s
I20260709 12:18:58.069617 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1750) took 1026 ms. Trace:
I20260709 12:18:58.072417 30462 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.090106 27731 rpcz_store.cc:276] 0709 12:18:57.043706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.633321 (+589615us) service_pool.cc:224] Handling call
0709 12:18:58.069548 (+436227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:58.073885 30468 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:57.927436 30371 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.131700 30468 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:58.076341 28336 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 12:18:58.134470 28336 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 8
I20260709 12:18:58.134764 28336 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 12:18:58.135032 28336 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:58.078707 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.014s	sys 0.015s
W20260709 12:18:58.135931 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.014s	sys 0.015s
W20260709 12:18:58.136278 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 625297us
I20260709 12:18:58.137137 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:58.099305 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:58.100035 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-c596ddd50b894549b567a5b66ac1a31e received from {username='slave'} at 127.0.0.1:47076
I20260709 12:18:58.100963 30400 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:58.101212 30302 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.102718 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:58.108777 30292 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:58.109257 30414 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.111936 27732 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
W20260709 12:18:58.112222 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 121432us
I20260709 12:18:58.149860 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:58.112583 30158 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:58.113160 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:18:58.115569 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.142987 30462 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:58.118295 30381 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.119849 30341 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:58.123183 30398 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:18:57.992465 30417 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:58.125422 30437 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:58.128482 30438 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.042s	user 0.011s	sys 0.000s
I20260709 12:18:58.138423 27734 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:58.153357 30158 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.146940 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-c596ddd50b894549b567a5b66ac1a31e on tablet c596ddd50b894549b567a5b66ac1a31e with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:58.144776 30388 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:58.147583 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:57.985709 30181 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:58.133836 30468 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:58.118197 30454 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.180s	user 0.001s	sys 0.001s
I20260709 12:18:58.154505 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:58.237422 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82981.2us
I20260709 12:18:58.159781 30400 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:18:58.212424 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 466044us
I20260709 12:18:58.143680 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:58.242269 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 98651.5us
I20260709 12:18:58.225927 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.226819 27731 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:18:58.227658 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.244632 27580 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:58.229529 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:58.249358 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.250859 27588 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:58.236478 30388 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.285725 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.287515 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.178953 30398 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:58.289424 30398 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.290928 30398 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:18:58.274014 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.213495 30158 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:58.160037 30414 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.301327 30414 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.277078 27734 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
W20260709 12:18:58.277705 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.007s	sys 0.033s
W20260709 12:18:58.303243 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.007s	sys 0.033s
I20260709 12:18:58.254976 30465 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:58.235994 30210 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:58.304840 30437 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:58.304734 30210 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:58.305590 30437 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:58.307533 30210 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:58.308938 30437 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:18:58.312737 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.313489 30377 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.252028 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.319629 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:58.320832 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:58.274518 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6436293270000988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 12:18:58.313723 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:58.265908 30404 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:58.265230 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.277410 30400 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:58.214267 30438 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:58.322184 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6373293270000886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20260709 12:18:58.292774 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:58.324914 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 334265us
W20260709 12:18:58.330273 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5149293270001181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
I20260709 12:18:58.331687 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1809) took 3560 ms. Trace:
W20260709 12:18:58.331887 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4006293270001606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260709 12:18:58.332154 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 339579us
I20260709 12:18:58.333405 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:58.332398 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3889293270001417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
I20260709 12:18:58.334183 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.338107 27583 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:18:58.314951 30388 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:58.339896 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.342058 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.342831 27583 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:58.319190 30437 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:58.343613 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3133293270000195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
I20260709 12:18:58.347641 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:18:58.349032 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3034293270000035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
I20260709 12:18:58.327281 30404 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:58.349592 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1927293270000519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
I20260709 12:18:58.348838 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:58.351851 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0577293270000609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
W20260709 12:18:58.352483 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0046293269999751s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20260709 12:18:58.351835 27735 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:18:58.332100 27587 rpcz_store.cc:276] 0709 12:18:54.771723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.771867 (+   144us) service_pool.cc:224] Handling call
0709 12:18:54.772214 (+   347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:54.773573 (+  1359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:54.773603 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:56.800166 (+2026563us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b2c000014c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:56.814069 (+ 13903us) log.cc:844] Serialized 52 byte log entry
0709 12:18:56.959402 (+145333us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:57.902173 (+942771us) log.cc:844] Serialized 14 byte log entry
0709 12:18:58.326481 (+424308us) log.cc:844] Serialized 14 byte log entry
0709 12:18:58.326971 (+   490us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.327126 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.331464 (+  4338us) raft_consensus.cc:1685] finished
0709 12:18:58.331484 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.331630 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6079,"prepare.queue_time_us":1114643,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":200907,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":163,"spinlock_wait_cycles":21145856,"thread_start_us":1376444,"threads_started":3,"wal-append.queue_time_us":236824}
I20260709 12:18:58.354377 27660 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:18:58.353113 27588 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:58.354636 27704 leader_election.cc:400] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:58.357663 27704 leader_election.cc:403] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:18:58.357028 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:58.354837 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:58.398224 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:58.413133 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57172.3us
I20260709 12:18:58.413686 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:58.414813 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:58.414951 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:58.350257 30471 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:18:58.428831 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74324us
I20260709 12:18:58.345791 30453 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:58.323740 30377 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.437924 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:58.358666 30404 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:58.441047 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.443404 27587 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:18:58.444327 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:58.444521 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.445510 27587 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:58.446180 30472 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:18:58.455497 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:18:58.427868 30446 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:18:58.427994 30256 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.615s	user 0.003s	sys 0.011s
I20260709 12:18:58.358459 30461 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:58.420228 30382 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.456795 27587 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:18:58.476578 28336 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:18:58.463084 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:58.477648 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62782.8us
I20260709 12:18:58.478494 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.480374 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.481699 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.437294 30413 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:58.435222 30453 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:58.439227 30467 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:58.351500 30437 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.421916 30475 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:58.495151 30457 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.006s
W20260709 12:18:58.505144 30474 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:18:58.505379 30484 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:58.512807 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:58.421471 30445 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.476192 30461 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.505223 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:58.550266 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.499387 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.559244 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.569486 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 339) took 1718 ms. Trace:
I20260709 12:18:58.549502 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.518430 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:18:58.597857 27729 rpcz_store.cc:276] 0709 12:18:56.851272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.854855 (+  3583us) service_pool.cc:224] Handling call
0709 12:18:56.859578 (+  4723us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:57.989623 (+1130045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.569238 (+579615us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:58.569425 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2548450048}
I20260709 12:18:58.568408 30416 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.477878 30256 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.498203 30471 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:58.599778 30487 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:58.565660 30403 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.558961 30392 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:58.616011 30392 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:58.616950 30256 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.617722 30471 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:58.618010 30469 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.543s	user 0.007s	sys 0.003s
I20260709 12:18:58.618180 30471 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:58.618251 30392 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:58.601236 30181 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.619418 30416 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:58.622233 30450 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.686s	user 0.000s	sys 0.002s
I20260709 12:18:58.624465 27588 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 4. Current term is 5. Ops: []
W20260709 12:18:58.627451 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.630156 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:58.630566 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.631567 30403 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:58.632102 30403 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.632885 30403 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:58.635746 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 342) took 1576 ms. Trace:
I20260709 12:18:58.635886 30438 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:58.636170 27732 rpcz_store.cc:276] 0709 12:18:57.059281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.876610 (+817329us) service_pool.cc:224] Handling call
0709 12:18:57.891050 (+ 14440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.148603 (+257553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.635496 (+486893us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:58.635666 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461483776}
I20260709 12:18:58.637964 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 338) took 1787 ms. Trace:
I20260709 12:18:58.638621 27736 rpcz_store.cc:276] 0709 12:18:56.850893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.854154 (+  3261us) service_pool.cc:224] Handling call
0709 12:18:56.854473 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.860138 (+  5665us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:56.860176 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.886069 (+ 25893us) log.cc:844] Serialized 26 byte log entry
0709 12:18:57.912678 (+1026609us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:57.912738 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.914387 (+  1649us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.617277 (+702890us) spinlock_profiling.cc:243] Waited 7.55 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:58.637655 (+ 20378us) raft_consensus.cc:1685] finished
0709 12:18:58.637674 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.637893 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1155,"raft.queue_time_us":77,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":10689,"spinlock_wait_cycles":17359232,"thread_start_us":941876,"threads_started":1,"wal-append.queue_time_us":1121446}
I20260709 12:18:58.643676 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1850) took 1496 ms. Trace:
I20260709 12:18:58.495340 30468 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:18:58.644174 27585 rpcz_store.cc:276] 0709 12:18:57.147026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.309749 (+162723us) service_pool.cc:224] Handling call
0709 12:18:57.310117 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:57.883359 (+573242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:57.897398 (+ 14039us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:57.911501 (+ 14103us) log.cc:844] Serialized 26 byte log entry
0709 12:18:58.343831 (+432330us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:18:58.343871 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.343935 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.643403 (+299468us) raft_consensus.cc:1685] finished
0709 12:18:58.643431 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.643593 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10084,"raft.queue_time_us":647726,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":9339,"thread_start_us":929856,"threads_started":2,"wal-append.queue_time_us":493264}
I20260709 12:18:58.508410 30453 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:58.574182 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.600829 30149 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:18:58.572499 30377 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:58.600883 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101665us
I20260709 12:18:58.699690 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.491683 30458 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:18:58.597038 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84726.8us
W20260709 12:18:58.519369 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63848.8us
I20260709 12:18:58.656955 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:18:58.518621 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.713114 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.441015 30292 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.162s
I20260709 12:18:58.714147 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:18:58.568619 30462 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:18:58.493340 30383 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.164s
W20260709 12:18:58.657822 30371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.004s	sys 0.012s
I20260709 12:18:58.544498 30381 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:58.721021 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:18:58.596556 30461 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:58.725101 27732 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:18:58.656656 30468 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:58.563333 30451 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.221s
I20260709 12:18:58.587818 30158 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.743453 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1751) took 1699 ms. Trace:
I20260709 12:18:58.744005 27737 rpcz_store.cc:276] 0709 12:18:57.044044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.771509 (+727465us) service_pool.cc:224] Handling call
0709 12:18:57.771937 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:57.775941 (+  4004us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:57.775972 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:57.776472 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 12:18:58.218825 (+442353us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:58.218882 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.218978 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.743175 (+524197us) raft_consensus.cc:1685] finished
0709 12:18:58.743196 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.743364 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":364,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":441905,"threads_started":1,"wal-append.queue_time_us":560824}
I20260709 12:18:58.747215 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.491395 30400 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.742340 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:18:58.751304 27732 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:58.721040 30282 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:18:58.723944 30371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.004s	sys 0.013s
W20260709 12:18:58.758731 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59149.6us
I20260709 12:18:58.718981 30424 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:18:58.770756 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.764266 27729 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:18:58.749046 30468 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:58.748399 30458 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:58.728837 30462 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:18:58.778715 30479 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260709 12:18:58.785230 30466 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.712s	user 0.006s	sys 0.000s
I20260709 12:18:58.727205 30409 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:18:58.776974 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.792991 30445 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:58.799930 30386 raft_consensus.cc:1064] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:58.807828 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.812528 30452 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:58.812806 30387 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:18:58.814826 30371 raft_consensus.cc:1064] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:18:58.815019 27659 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:18:58.816524 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:58.822470 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1740) took 1786 ms (client timeout 1611 ms). Trace:
W20260709 12:18:58.853585 27733 rpcz_store.cc:269] 0709 12:18:57.035913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.345058 (+309145us) service_pool.cc:224] Handling call
0709 12:18:58.822191 (+1477133us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54000a5fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:58.822347 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1009605504}
W20260709 12:18:58.821141 27580 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.548s	user 0.009s	sys 0.001s
I20260709 12:18:58.854707 27580 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:18:58.824716 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1752) took 1780 ms (client timeout 2226 ms). Trace:
I20260709 12:18:58.857929 27585 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:18:58.858248 27738 rpcz_store.cc:269] 0709 12:18:57.044420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.645853 (+601433us) service_pool.cc:224] Handling call
0709 12:18:58.824457 (+1178604us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:58.824643 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230618368}
I20260709 12:18:58.823938 27734 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:18:58.830242 30416 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:58.832460 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 594119us
I20260709 12:18:58.866564 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:58.844409 27582 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:18:58.874213 30416 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:58.874663 27582 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:18:58.875206 27582 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:58.876225 27582 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.877390 27582 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:58.879422 30494 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:18:58.884503 30499 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260709 12:18:58.892416 30491 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
W20260709 12:18:58.893680 30481 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:18:58.896306 27582 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:18:58.896705 30387 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:18:58.898999 30490 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:18:58.901178 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:18:58.908113 30387 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.908610 30387 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:58.911644 30488 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:18:58.899992 30256 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:58.915009 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:58.900599 30181 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:18:58.846192 30382 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.846923 30473 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:58.904299 27582 pending_rounds.cc:85] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:18:58.934841 27582 pending_rounds.cc:107] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:18:58.935209 27582 raft_consensus.cc:2889] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:58.917065 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:18:58.917554 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1851) took 1770 ms. Trace:
I20260709 12:18:58.916563 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:18:58.917085 30341 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.708s	user 0.006s	sys 0.004s
I20260709 12:18:58.915470 30256 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:58.931169 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.938498 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:58.937575 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:58.859402 30386 raft_consensus.cc:1068] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:18:58.852530 30458 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:58.853495 30421 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.855615 30452 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:58.939913 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:18:58.861049 30437 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:58.945183 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44114.8us
I20260709 12:18:58.910749 30387 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:58.856915 30477 raft_consensus.cc:2764] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:18:58.948829 27731 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
W20260709 12:18:58.951223 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 172225us
I20260709 12:18:58.926891 30181 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:18:58.963734 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 239767us
I20260709 12:18:58.965811 27731 pending_rounds.cc:85] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:18:58.966145 27731 pending_rounds.cc:107] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 12:18:58.966488 27731 raft_consensus.cc:2889] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:58.967084 30398 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.674s	user 0.011s	sys 0.000s
I20260709 12:18:58.874878 30416 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.948343 30495 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:18:58.948179 30445 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:58.986109 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1713) took 3343 ms. Trace:
I20260709 12:18:58.979197 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:58.998656 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:59.000854 27582 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:18:58.985944 27735 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:18:59.001456 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:18:58.967374 30110 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.967437 30392 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:18:59.008927 30501 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:59.009407 30501 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:59.009754 30501 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:59.011880 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:18:58.986275 30367 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.114s	user 0.013s	sys 0.000s
I20260709 12:18:59.013973 28336 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:18:59.014434 30452 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.015892 30452 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:59.016300 30158 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.017068 30158 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:18:58.969982 30473 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:18:59.000165 30445 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.939729 30382 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:59.020298 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 153795us
I20260709 12:18:59.000315 27730 rpcz_store.cc:276] 0709 12:18:55.642409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.169150 (+526741us) service_pool.cc:224] Handling call
0709 12:18:56.169458 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:57.265343 (+1095885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:57.265375 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:57.276793 (+ 11418us) log.cc:844] Serialized 26 byte log entry
0709 12:18:57.886721 (+609928us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:58.285630 (+398909us) log.cc:844] Serialized 14 byte log entry
0709 12:18:58.286907 (+  1277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.784534 (+497627us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:58.816660 (+ 32126us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:58.816713 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.985798 (+169085us) raft_consensus.cc:1685] finished
0709 12:18:58.985826 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.986025 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106006,"prepare.queue_time_us":323523,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":168052,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":5030,"spinlock_wait_cycles":2181303808,"thread_start_us":867825,"threads_started":2,"wal-append.queue_time_us":753840}
I20260709 12:18:59.007722 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:59.038141 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:18:59.007184 30504 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:18:58.966655 30421 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:58.980150 30457 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 12:18:59.046501 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:59.050809 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47331.7us
I20260709 12:18:59.053359 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:59.054076 30513 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:18:59.061537 30371 raft_consensus.cc:1068] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:18:59.064622 30417 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.856s	user 0.001s	sys 0.010s
I20260709 12:18:59.066164 30485 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:18:59.066927 30485 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:18:59.065209 30417 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:58.977458 30387 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.061s
I20260709 12:18:59.029536 30495 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:59.072587 30416 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:59.072438 30495 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.074154 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 354) took 1132 ms. Trace:
I20260709 12:18:59.074538 27663 rpcz_store.cc:276] 0709 12:18:57.941760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.941867 (+   107us) service_pool.cc:224] Handling call
0709 12:18:57.954757 (+ 12890us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.956282 (+  1525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:57.956310 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:57.956349 (+    39us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:57.956376 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.073927 (+1117551us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:59.073970 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.074097 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3072256}
I20260709 12:18:59.078491 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:59.029687 30445 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:59.020961 30110 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.088775 30158 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:59.096599 30393 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:18:59.101136 30388 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.743s	user 0.010s	sys 0.000s
I20260709 12:18:59.027814 30256 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.101883 30399 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.102438 30501 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.104279 30468 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.104876 30387 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:59.019642 30473 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.108186 30509 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:59.027801 30451 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:18:59.135768 30256 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:59.114168 30158 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:59.159432 27585 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:18:59.040112 30181 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.135623 30494 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:59.135810 30421 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:59.131589 30388 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:59.164183 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 156529us
W20260709 12:18:59.156142 30403 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.523s	user 0.006s	sys 0.006s
I20260709 12:18:59.142509 30387 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:59.135936 30399 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.190953 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1754) took 1858 ms. Trace:
I20260709 12:18:59.191370 27734 rpcz_store.cc:276] 0709 12:18:57.332700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.302395 (+969695us) service_pool.cc:224] Handling call
0709 12:18:58.823478 (+521083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.859630 (+ 36152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.190745 (+331115us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b54000a5fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:59.190896 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1197611264}
I20260709 12:18:59.192783 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 356) took 1249 ms. Trace:
I20260709 12:18:59.193147 27659 rpcz_store.cc:276] 0709 12:18:57.942935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.943074 (+   139us) service_pool.cc:224] Handling call
0709 12:18:58.814682 (+871608us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.853144 (+ 38462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.192614 (+339470us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b540005fd50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:18:59.192732 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1984456960}
I20260709 12:18:59.045204 30458 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:59.180974 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 108318us
I20260709 12:18:59.196277 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:18:59.182803 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:18:59.172287 30158 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:59.157931 30392 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:18:59.203775 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:59.183247 30494 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:18:59.185336 30383 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:59.209631 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 156343us
I20260709 12:18:59.195050 30387 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:59.202957 30181 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:18:59.256050 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59446.9us
W20260709 12:18:59.253160 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46863.1us
I20260709 12:18:59.201490 30341 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:18:59.208205 30467 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:18:59.205374 30421 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:18:59.254633 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:59.258154 30461 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.517s	user 0.005s	sys 0.011s
I20260709 12:18:59.210583 30416 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:18:59.188403 30479 consensus_queue.cc:1175] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 9 status { last_received { term: 9 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 9 index: 8 } }
I20260709 12:18:59.194590 30403 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:59.257555 30453 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.536s	user 0.022s	sys 0.015s
W20260709 12:18:59.258785 30413 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.749s	user 0.008s	sys 0.004s
I20260709 12:18:59.282055 30386 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:18:59.286100 30413 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:18:59.286262 30386 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:59.252781 30510 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:18:59.286721 30386 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:59.288229 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1836) took 1028 ms. Trace:
I20260709 12:18:59.288611 27588 rpcz_store.cc:276] 0709 12:18:58.259743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.625243 (+365500us) service_pool.cc:224] Handling call
0709 12:18:58.625509 (+   266us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:58.731248 (+105739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:58.731277 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:58.732355 (+  1078us) log.cc:844] Serialized 26 byte log entry
0709 12:18:58.947284 (+214929us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:58.947355 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.076882 (+129527us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001248d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:59.076934 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:59.287907 (+210973us) raft_consensus.cc:1685] finished
0709 12:18:59.287930 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.288167 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4556,"raft.queue_time_us":58411,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9263,"spinlock_wait_cycles":242361600,"thread_start_us":167994,"threads_started":1,"wal-append.queue_time_us":263940}
I20260709 12:18:59.291029 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:59.292033 27588 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:18:59.292711 30453 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.294982 30453 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.255524 30509 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:18:59.279258 30416 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:18:59.283344 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:59.262456 30503 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:18:59.303280 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:59.285473 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:18:59.336978 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51622.8us
I20260709 12:18:59.337961 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:59.264986 30467 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:18:59.254223 30383 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.349387 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:59.319054 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:18:59.344625 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.283751 30382 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.347676 27656 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:18:59.263859 30392 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.286736 30501 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.350592 27730 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:59.264109 30341 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:18:59.346832 30488 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:18:59.335218 30416 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:18:59.385486 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.280435 30461 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.350239 30467 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:18:59.349869 30417 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:18:59.320559 30452 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:18:59.369848 30400 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:18:59.405740 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67855.7us
W20260709 12:18:59.414850 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 119691us
I20260709 12:18:59.416069 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:18:59.417464 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:18:59.419220 30523 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
W20260709 12:18:59.431612 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 163076us
I20260709 12:18:59.410280 30468 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.411314 30400 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:18:59.369452 30516 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260709 12:18:59.442482 30537 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:18:59.442402 30387 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.443280 27656 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:18:59.443761 30532 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:18:59.448628 30445 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.451555 30383 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:18:59.454061 30404 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.979s	user 0.006s	sys 0.004s
I20260709 12:18:59.454514 30404 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.441057 30400 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:18:59.416211 30417 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:18:59.469376 27701 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.727s (SENT)
W20260709 12:18:59.475510 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.438136 30371 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:18:59.431965 30388 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.496466 30537 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:18:59.497174 30537 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.498968 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:59.484109 30387 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.516997 30388 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.517230 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:18:59.517904 27663 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:18:59.517612 30388 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.518951 30497 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:18:59.519414 30497 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:18:59.519769 27661 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:18:59.520344 30497 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:18:59.457043 30457 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
W20260709 12:18:59.515067 30475 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.858s	user 0.012s	sys 0.001s
I20260709 12:18:59.514464 30438 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.554875 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:18:59.514477 30371 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:18:59.438602 30533 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20260709 12:18:59.521754 30497 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:18:59.582230 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:59.581338 27733 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:18:59.574635 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1862) took 1321 ms. Trace:
I20260709 12:18:59.583451 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1758) took 1684 ms. Trace:
I20260709 12:18:59.599243 27735 rpcz_store.cc:276] 0709 12:18:57.898661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.626662 (+728001us) service_pool.cc:224] Handling call
0709 12:18:58.626896 (+   234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.002041 (+375145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.583174 (+581133us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:18:59.583378 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":824967552}
I20260709 12:18:59.596175 27586 rpcz_store.cc:276] 0709 12:18:58.253446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.453978 (+200532us) service_pool.cc:224] Handling call
0709 12:18:58.454276 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.454473 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:18:58.455117 (+   644us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.574154 (+1119037us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b1800391660. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287
0709 12:18:59.574267 (+   113us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:59.574324 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:18:59.574375 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.574448 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.574575 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1582,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":39853,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":16813,"spinlock_wait_cycles":79387008,"thread_start_us":343196,"threads_started":2,"wal-append.queue_time_us":283536}
I20260709 12:18:59.614745 27663 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:18:59.579941 30387 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:59.573104 30471 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.954s	user 0.007s	sys 0.007s
W20260709 12:18:59.664078 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.561434 30475 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:18:59.563917 30403 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:18:59.595293 30504 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:18:59.563477 30438 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.778066 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:18:59.777805 27580 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:18:59.770936 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.782828 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:18:59.800374 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:59.803300 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.673755 30341 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:18:59.577030 30400 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.675292 30387 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.563812 30417 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.694800 30471 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:18:59.807862 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:59.877599 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:18:59.840652 27580 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:19:00.023072 27730 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.000s
W20260709 12:19:00.261140 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:59.917510 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.851727 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:19:00.634176 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:00.627785 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.21027e+06us
W20260709 12:19:00.323580 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:00.624272 27701 leader_election.cc:336] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.727s (SENT)
I20260709 12:19:00.638774 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:00.639457 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:00.641121 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:00.642397 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:00.643004 27587 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:00.646998 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:00.630566 27730 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:00.648857 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 359) took 1336 ms. Trace:
I20260709 12:19:00.649319 27730 rpcz_store.cc:276] 0709 12:18:59.312602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.318817 (+  6215us) service_pool.cc:224] Handling call
0709 12:19:00.648776 (+1329959us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:00.274369 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:00.650950 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1606) took 1069 ms. Trace:
I20260709 12:19:00.651504 27737 rpcz_store.cc:276] 0709 12:18:59.581389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.583598 (+  2209us) service_pool.cc:224] Handling call
0709 12:19:00.650874 (+1067276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13440}
I20260709 12:19:00.653483 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:00.654179 27581 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:00.654464 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:00.688203 27736 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 9. Ops: []
W20260709 12:19:00.370680 30398 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.263s	user 0.019s	sys 0.008s
W20260709 12:19:00.446111 30540 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.328s	user 0.001s	sys 0.000s
W20260709 12:19:00.182911 30181 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.924s	user 0.009s	sys 0.000s
W20260709 12:18:59.924316 30522 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.498s	user 0.001s	sys 0.000s
W20260709 12:19:00.151983 30386 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.865s	user 0.013s	sys 0.005s
W20260709 12:18:59.971880 30494 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.718s	user 0.004s	sys 0.007s
W20260709 12:19:00.647174 30467 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.222s	user 0.011s	sys 0.001s
I20260709 12:18:59.879890 30392 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:00.653564 30416 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.246s	user 0.012s	sys 0.004s
W20260709 12:19:00.657547 30345 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.012s	sys 0.022s
I20260709 12:18:59.969872 30445 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:00.705416 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.008s	sys 0.004s
W20260709 12:19:00.774318 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.008s	sys 0.007s
I20260709 12:19:00.646473 30341 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:00.701795 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:00.775518 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:00.695190 30398 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.588s	user 0.021s	sys 0.008s
I20260709 12:19:00.725898 27581 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
W20260709 12:19:00.734278 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.851s (SENT) [suppressed 21 similar messages]
W20260709 12:19:00.777645 27701 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.861s (SENT)
I20260709 12:19:00.778445 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:00.700361 30525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.222s	user 0.000s	sys 0.012s
I20260709 12:19:00.741185 30494 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:00.725348 30452 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.009s	sys 0.008s
W20260709 12:19:00.701933 30473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.008s	sys 0.003s
I20260709 12:19:00.786113 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:00.794360 30452 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.010s	sys 0.013s
I20260709 12:19:00.794041 30341 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:00.751497 30392 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.784996 30438 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:00.782864 30404 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:00.773046 30345 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.014s	sys 0.035s
I20260709 12:19:00.730683 30537 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:00.781651 30525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.344s	user 0.000s	sys 0.013s
I20260709 12:19:00.725342 30548 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:00.701862 30513 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.100s	user 0.002s	sys 0.010s
I20260709 12:18:59.949613 30371 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.729532 30386 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.742233 30467 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:00.916147 30158 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.706s	user 0.016s	sys 0.005s
I20260709 12:19:00.941948 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 343) took 3804 ms. Trace:
I20260709 12:19:00.777246 30416 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:00.952042 30513 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.003s	sys 0.018s
W20260709 12:19:00.921497 30549 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.126s	user 0.003s	sys 0.004s
I20260709 12:19:00.961333 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:00.923679 30398 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.913400 30554 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.913727 30445 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:00.919266 30473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.219s	user 0.008s	sys 0.012s
I20260709 12:19:00.965611 27731 rpcz_store.cc:276] 0709 12:18:57.137597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.255625 (+1118028us) service_pool.cc:224] Handling call
0709 12:18:58.255996 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.966897 (+710901us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:00.941606 (+1974709us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:00.941869 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1592237440}
I20260709 12:19:00.972739 27581 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:00.913486 30512 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.960742 30525 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:00.962508 30537 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:00.942198 30438 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:00.963984 30404 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:00.983445 30445 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.999238 27587 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:01.003630 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 341) took 1200 ms. Trace:
I20260709 12:19:00.980903 30458 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.013439 27587 rpcz_store.cc:276] 0709 12:18:59.803045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.642136 (+839091us) service_pool.cc:224] Handling call
0709 12:19:01.003569 (+361433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.014424 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:00.973737 30158 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.963388 30548 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.996609 30560 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.000941 30537 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:01.003489 30473 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.021854 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 343) took 1218 ms. Trace:
I20260709 12:19:01.123149 27587 rpcz_store.cc:276] 0709 12:18:59.803688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.020937 (+1217249us) service_pool.cc:224] Handling call
0709 12:19:01.021294 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:01.021497 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:01.021526 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:01.021578 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:01.021616 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.021679 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.021795 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.004503 30551 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:01.178787 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:01.242221 30547 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.323s	user 0.000s	sys 0.000s
I20260709 12:19:01.006848 30400 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.242538 30537 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.247550 30562 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.247996 30473 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.074909 30387 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.255028 30416 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.006984 30563 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:00.999935 30438 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.016213 30561 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.979950 30554 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.258689 27581 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:01.261281 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:00.986699 30512 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:00.976059 30398 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.020570 30494 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.015245 30471 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.272805 27656 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:01.314064 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.236756 30458 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:01.243999 30539 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
W20260709 12:19:01.007510 30553 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
W20260709 12:19:01.372640 30543 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:19:01.373972 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 368) took 2070 ms. Trace:
I20260709 12:19:01.374353 27659 rpcz_store.cc:276] 0709 12:18:59.303175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.303292 (+   117us) service_pool.cc:224] Handling call
0709 12:18:59.315569 (+ 12277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:59.316121 (+   552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.316147 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:59.333881 (+ 17734us) log.cc:844] Serialized 26 byte log entry
0709 12:18:59.477572 (+143691us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:59.477623 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.477699 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.373749 (+1896050us) raft_consensus.cc:1685] finished
0709 12:19:01.373769 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.373912 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":178,"raft.queue_time_us":637,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":25562,"thread_start_us":98219,"threads_started":1,"wal-append.queue_time_us":1651601}
I20260709 12:19:01.348285 30562 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.269335 30345 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.007128 30404 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:01.379904 30550 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.418s	user 0.000s	sys 0.001s
W20260709 12:19:01.380843 30541 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.462s	user 0.000s	sys 0.001s
I20260709 12:19:01.382035 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 357) took 2070 ms. Trace:
I20260709 12:19:01.384148 30345 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.387908 30467 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.244269 30560 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.390501 30560 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.391391 30386 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.392966 30341 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:01.393522 30529 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.977s	user 0.019s	sys 0.029s
I20260709 12:19:01.395542 30158 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.397624 30468 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:01.398056 30468 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:01.400929 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1604) took 1820 ms. Trace:
I20260709 12:19:01.401402 27736 rpcz_store.cc:276] 0709 12:18:59.580800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.581489 (+   689us) service_pool.cc:224] Handling call
0709 12:18:59.581769 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:00.688929 (+1107160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.400656 (+711727us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540017f2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:01.400863 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2543896448}
I20260709 12:19:01.382422 27734 rpcz_store.cc:276] 0709 12:18:59.311850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.318150 (+  6300us) service_pool.cc:224] Handling call
0709 12:18:59.318452 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:59.318671 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.318694 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:59.344931 (+ 26237us) log.cc:844] Serialized 26 byte log entry
0709 12:18:59.477203 (+132272us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:59.477253 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.477318 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.381822 (+1904504us) raft_consensus.cc:1685] finished
0709 12:19:01.381842 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.381976 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":69342,"threads_started":1,"wal-append.queue_time_us":1376249}
W20260709 12:19:01.451645 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.004s	sys 0.012s
W20260709 12:19:01.452183 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.004s	sys 0.012s
W20260709 12:19:01.453552 27730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.009s	sys 0.009s
W20260709 12:19:01.454032 27730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.009s	sys 0.009s
W20260709 12:19:01.456082 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.004s	sys 0.013s
W20260709 12:19:01.456517 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.004s	sys 0.013s
W20260709 12:19:01.457477 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:01.458058 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.011s	sys 0.005s
W20260709 12:19:01.458524 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.011s	sys 0.005s
W20260709 12:19:01.467028 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.000s	sys 0.016s
W20260709 12:19:01.467658 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.001s	sys 0.016s
W20260709 12:19:01.468438 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 690782us
W20260709 12:19:01.469390 27701 connection.cc:256] RPC call timeout handler was delayed by -1.017002482999942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 12:19:01.473868 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.015s	sys 0.056s
W20260709 12:19:01.474767 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.016s	sys 0.056s
I20260709 12:19:01.402922 30566 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.398912 30468 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.427747 30467 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:01.485889 30557 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:19:01.273018 30551 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.437143 30345 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:01.484565 30497 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.891s	user 0.004s	sys 0.014s
I20260709 12:19:01.474303 30560 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.478451 30392 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.363169 30561 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.505976 30554 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.506598 30548 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.486933 30562 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:01.502023 30453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.978s	user 0.029s	sys 0.050s
I20260709 12:19:01.510250 30561 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.493178 30556 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:01.508168 30494 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.518718 30473 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.519209 30551 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.489007 30566 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.523904 30494 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.525441 30512 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.509418 30560 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.528038 30560 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.526010 30566 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:01.561489 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.013s	sys 0.036s
W20260709 12:19:01.562515 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.928s	user 0.013s	sys 0.036s
W20260709 12:19:01.565025 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.566651 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1851) took 2488 ms. Trace:
I20260709 12:18:58.936803 27589 rpcz_store.cc:276] 0709 12:18:57.147261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.316531 (+169270us) service_pool.cc:224] Handling call
0709 12:18:57.399249 (+ 82718us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:57.885911 (+486662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:57.885946 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:58.571938 (+685992us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b2c000014c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:18:58.572491 (+   553us) log.cc:844] Serialized 52 byte log entry
0709 12:18:58.835868 (+263377us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:18:58.869353 (+ 33485us) log.cc:844] Serialized 14 byte log entry
0709 12:18:58.869661 (+   308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.869736 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.917286 (+ 47550us) raft_consensus.cc:1685] finished
0709 12:18:58.917367 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.917495 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18772,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":20765,"raft.queue_time_us":644305,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":98938,"spinlock_wait_cycles":5096704,"thread_start_us":746955,"threads_started":2,"wal-append.queue_time_us":297771}
W20260709 12:19:01.567456 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.929s	user 0.008s	sys 0.006s
W20260709 12:19:01.576175 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.008s	sys 0.007s
W20260709 12:19:01.567835 27582 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.007s	sys 0.007s
W20260709 12:19:01.578276 27582 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.007s	sys 0.008s
W20260709 12:19:01.578531 27552 connection.cc:256] RPC call timeout handler was delayed by -3.2258637519998956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 12:19:01.566905 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.000s	sys 0.016s
I20260709 12:19:01.568929 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1875) took 1786 ms. Trace:
I20260709 12:19:01.567509 27583 rpcz_store.cc:276] 0709 12:18:59.077694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.110842 (+ 33148us) service_pool.cc:224] Handling call
0709 12:18:59.111238 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:59.125493 (+ 14255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.125541 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:59.132745 (+  7204us) log.cc:844] Serialized 52 byte log entry
0709 12:18:59.206673 (+ 73928us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:18:59.206714 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.206779 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.392919 (+2186140us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:01.428176 (+ 35257us) raft_consensus.cc:1685] finished
0709 12:19:01.428208 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.566180 (+137972us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3537,"raft.queue_time_us":72011,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":16162,"spinlock_wait_cycles":4177664,"thread_start_us":34829,"threads_started":1,"wal-append.queue_time_us":142291}
I20260709 12:19:01.580639 27588 rpcz_store.cc:276] 0709 12:18:59.782725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.782849 (+   124us) service_pool.cc:224] Handling call
0709 12:18:59.783149 (+   300us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:59.783526 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.783564 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:00.648453 (+864889us) log.cc:844] Serialized 52 byte log entry
0709 12:19:01.452691 (+804238us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:01.453109 (+   418us) log.cc:844] Serialized 14 byte log entry
0709 12:19:01.453329 (+   220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.453435 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.568073 (+114638us) raft_consensus.cc:1685] finished
0709 12:19:01.568177 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.568657 (+   480us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74734,"prepare.queue_time_us":79487,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":55636,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":37894,"thread_start_us":732006,"threads_started":1,"wal-append.queue_time_us":842594}
W20260709 12:19:01.583487 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.000s	sys 0.017s
W20260709 12:19:01.579201 27552 connection.cc:256] RPC call timeout handler was delayed by -3.1142637519999425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
I20260709 12:19:01.572517 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1608) took 1990 ms. Trace:
W20260709 12:19:01.587669 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 933070us
W20260709 12:19:01.588513 27586 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.002s	sys 0.014s
W20260709 12:19:01.589869 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.003s	sys 0.014s
W20260709 12:19:01.590411 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.003s	sys 0.014s
I20260709 12:19:01.490902 30404 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.588102 27730 rpcz_store.cc:276] 0709 12:18:59.581878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.650310 (+1068432us) service_pool.cc:224] Handling call
0709 12:19:00.650723 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:01.454479 (+803756us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:01.454524 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:01.455149 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 12:19:01.492872 (+ 37723us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:01.492926 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.493050 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.571699 (+ 78649us) raft_consensus.cc:1685] finished
0709 12:19:01.571752 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.572155 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":314190,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":10972,"thread_start_us":840455,"threads_started":2,"wal-append.queue_time_us":78139}
W20260709 12:19:01.588060 27552 connection.cc:256] RPC call timeout handler was delayed by -3.0098637519997737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
W20260709 12:19:01.598747 27552 connection.cc:256] RPC call timeout handler was delayed by -2.8226637519999258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
W20260709 12:19:01.589320 27586 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.002s	sys 0.015s
I20260709 12:19:01.601348 30445 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.602802 30458 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:01.603860 27552 connection.cc:256] RPC call timeout handler was delayed by -2.8163637519999156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20260709 12:19:01.604262 27552 connection.cc:256] RPC call timeout handler was delayed by -2.71556375199998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21200000000000002s for a response
I20260709 12:19:01.498241 30467 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.606835 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 339) took 1834 ms. Trace:
I20260709 12:19:01.607414 27586 rpcz_store.cc:276] 0709 12:18:59.771987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.772148 (+   161us) service_pool.cc:224] Handling call
0709 12:18:59.772506 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.772753 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:18:59.845173 (+ 72420us) log.cc:844] Serialized 14 byte log entry
0709 12:19:01.606118 (+1760945us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:01.606267 (+   149us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:01.606381 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.606506 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.606770 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":635331,"prepare.queue_time_us":74549,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":574717,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":13052,"thread_start_us":1760194,"threads_started":2,"wal-append.queue_time_us":1411559}
W20260709 12:19:01.609884 27552 connection.cc:256] RPC call timeout handler was delayed by -2.2889637519999724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2594s for a response
I20260709 12:19:01.610183 30467 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:01.610411 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1962637519998225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2697s for a response
I20260709 12:19:01.619987 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.620887 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:01.621693 27659 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:01.611083 30458 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:01.611671 30461 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.509764 30438 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.624411 30560 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:01.512377 30453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.989s	user 0.029s	sys 0.050s
I20260709 12:19:01.625187 30573 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:19:01.625598 30560 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.628604 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:01.612799 30581 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.613375 30341 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.630771 30341 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.634506 30341 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.614521 30445 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:01.635780 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.636168 30445 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.637276 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.605170 30566 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:01.638221 27661 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:01.638455 30566 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.640038 30445 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:01.640679 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49912.7us
I20260709 12:19:01.641510 30581 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.644541 27655 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:19:01.644969 27663 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:19:01.646533 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:01.647100 27662 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:19:01.647379 27663 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:01.648389 27659 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:01.642179 30566 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.617599 30512 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.649560 27660 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:01.649760 30512 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.651124 30560 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.654102 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1842) took 3382 ms. Trace:
I20260709 12:19:01.654543 27580 rpcz_store.cc:276] 0709 12:18:58.271790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.860369 (+588579us) service_pool.cc:224] Handling call
0709 12:18:58.860706 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:00.629596 (+1768890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.653523 (+1023927us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:01.653890 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2108613888}
I20260709 12:19:01.635251 30341 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:01.619158 30525 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.003s
I20260709 12:19:01.623028 30458 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.642258 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:01.674737 30525 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:19:01.679267 27584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1872) took 2761 ms (client timeout 2050 ms). Trace:
I20260709 12:19:01.680037 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
W20260709 12:19:01.679876 27584 rpcz_store.cc:269] 0709 12:18:58.914900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.937281 (+ 22381us) service_pool.cc:224] Handling call
0709 12:19:01.664455 (+2727174us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b54000898d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:01.676025 (+ 11570us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4702015360}
I20260709 12:19:01.682276 27664 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
W20260709 12:19:01.683976 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55733.4us
I20260709 12:19:01.687530 30341 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.690997 30341 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.694466 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:01.696506 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54507.4us
I20260709 12:19:01.712991 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.730396 30110 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.678741 30573 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:01.692246 30512 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.695746 30581 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.702783 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1868) took 3213 ms. Trace:
I20260709 12:19:01.693539 30458 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.724436 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.681216 30467 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.696808 30461 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:01.708936 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 358) took 3373 ms. Trace:
I20260709 12:19:01.710675 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.744071 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.753671 28337 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:01.755182 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.753686 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:01.769280 28331 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:01.770094 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.770483 27583 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:01.770916 27583 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:01.771073 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:01.772074 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:01.772424 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:01.773891 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.770504 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.775913 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:01.776264 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.777249 30341 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:01.777640 30341 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.778072 30341 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:01.781213 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:01.782562 27586 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:01.760516 27585 rpcz_store.cc:276] 0709 12:18:58.489150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.858754 (+369604us) service_pool.cc:224] Handling call
0709 12:18:58.859041 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.160268 (+301227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.702226 (+2541958us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:01.702613 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":690377984}
I20260709 12:19:01.787734 30467 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.789912 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:01.791255 27581 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:01.796927 30467 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:01.800174 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 113524us
I20260709 12:19:01.808704 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:01.810422 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:01.761062 30573 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:01.762248 27658 rpcz_store.cc:276] 0709 12:18:58.335568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.335715 (+   147us) service_pool.cc:224] Handling call
0709 12:18:58.348104 (+ 12389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:59.089287 (+741183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.089335 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:59.089893 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:18:59.182069 (+ 92176us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:18:59.182126 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.708174 (+2526048us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:01.708262 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.708334 (+    72us) raft_consensus.cc:1685] finished
0709 12:19:01.708347 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.708716 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46998,"raft.queue_time_us":32627,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3150,"spinlock_wait_cycles":1635981056,"thread_start_us":83822,"threads_started":1,"wal-append.queue_time_us":108889,"wal-append.run_cpu_time_us":2996,"wal-append.run_wall_time_us":1210871}
I20260709 12:19:01.815260 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:01.816893 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.803798 30256 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:19:01.818248 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.818835 30256 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:01.802867 30512 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:19:01.787144 27583 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.819271 27659 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:01.823899 30110 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:01.824174 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1554) took 2325 ms (client timeout 1504 ms). Trace:
I20260709 12:19:01.824389 30494 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.824623 30110 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.827452 30573 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:01.827834 28333 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:01.828132 30551 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.828848 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1755) took 3947 ms. Trace:
I20260709 12:19:01.828953 30551 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.829495 30110 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:01.821296 30507 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 2.698s	user 0.011s	sys 0.038s
I20260709 12:19:01.820215 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:01.822674 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:01.824628 27656 rpcz_store.cc:269] 0709 12:18:59.498575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.498727 (+   152us) service_pool.cc:224] Handling call
0709 12:19:01.823743 (+2325016us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b5400229054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:01.823976 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4077052672}
I20260709 12:19:01.832578 27658 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:01.818351 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:01.834913 28337 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:01.827348 30461 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.835907 30461 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.836517 30404 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.837397 30404 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.838079 30399 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.839795 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:01.841216 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:01.841931 27588 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:01.842852 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:01.843983 30551 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.840061 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:01.821185 30518 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:01.846127 30518 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.804957 30504 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:01.846664 30518 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:01.846752 30388 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.847157 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:01.848414 27584 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:01.847994 30573 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.836318 30512 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.833228 27664 raft_consensus.cc:2368] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:19:01.822486 27583 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:01.838173 30461 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.857842 27733 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:19:01.858639 28328 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:01.860524 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1607) took 2278 ms. Trace:
I20260709 12:19:01.829383 27729 rpcz_store.cc:276] 0709 12:18:57.881742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.618131 (+736389us) service_pool.cc:224] Handling call
0709 12:18:58.618595 (+   464us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:58.621541 (+  2946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:58.621574 (+    33us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:58.623445 (+  1871us) log.cc:844] Serialized 551 byte log entry
0709 12:18:58.764087 (+140642us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:18:59.195950 (+431863us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.208880 (+ 12930us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.209083 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.827409 (+2618326us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b18002dfd00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:01.828271 (+   862us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b540017edd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:01.828326 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.828389 (+    63us) raft_consensus.cc:1685] finished
0709 12:19:01.828400 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.828631 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":957,"prepare.queue_time_us":13351,"raft.queue_time_us":5679,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":47764,"spinlock_wait_cycles":20777984,"thread_start_us":140218,"threads_started":1,"wal-append.queue_time_us":344029,"wal-append.run_cpu_time_us":4858,"wal-append.run_wall_time_us":1457698}
I20260709 12:19:01.862553 30398 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.864317 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:01.862088 27733 rpcz_store.cc:276] 0709 12:18:59.581654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.667782 (+ 86128us) service_pool.cc:224] Handling call
0709 12:19:01.834226 (+2166444us) spinlock_profiling.cc:243] Waited 7.64 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:01.834692 (+   466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:01.859457 (+ 24765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.860453 (+   996us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17580288}
I20260709 12:19:01.864337 30512 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.868057 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 463) took 3446 ms. Trace:
I20260709 12:19:01.868767 30512 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.846781 30504 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:01.869151 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.870098 27658 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:01.870261 30387 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.871500 27589 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:01.871063 30387 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:01.879707 30399 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.880643 30399 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.882983 30404 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.884119 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1876) took 2101 ms. Trace:
I20260709 12:19:01.894894 27588 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:01.883459 27734 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:19:01.875161 30387 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.860948 30467 raft_consensus.cc:2764] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 12:19:01.915303 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105027us
I20260709 12:19:01.915776 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:01.883147 30377 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:01.917833 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:01.920538 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:01.922873 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:01.950219 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.952916 28329 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:01.918375 27733 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:01.956121 27736 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:01.956895 27664 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:19:01.909951 30398 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.963944 30398 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:01.965672 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.966241 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:01.968736 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:01.969769 30525 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:01.970477 30525 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:01.970791 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:01.971676 30404 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:01.971567 30371 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:01.971892 30525 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:01.975266 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:01.975615 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.977332 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:01.978022 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 241359us
W20260709 12:19:01.979727 30586 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.204s	user 0.000s	sys 0.002s
I20260709 12:19:01.981070 30398 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:01.890703 27582 rpcz_store.cc:276] 0709 12:18:59.783056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.783673 (+   617us) service_pool.cc:224] Handling call
0709 12:18:59.818229 (+ 34556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.818476 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:01.578963 (+1760487us) log.cc:844] Serialized 14 byte log entry
0709 12:19:01.762580 (+183617us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:01.773448 (+ 10868us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:01.815972 (+ 42524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.858278 (+ 42306us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.884063 (+ 25785us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39392,"prepare.queue_time_us":874081,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":14392,"raft.queue_time_us":82871,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":271,"thread_start_us":1006917,"threads_started":2,"wal-append.queue_time_us":92533}
I20260709 12:19:01.987901 30573 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:01.988318 27732 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:19:01.989799 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.989327 30377 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.039805 27655 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:02.041689 30377 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:01.990810 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:02.039284 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.042520 27731 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:02.042758 27659 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
I20260709 12:19:01.999924 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:01.868563 28332 rpcz_store.cc:276] 0709 12:18:58.421597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.421727 (+   130us) service_pool.cc:224] Handling call
0709 12:18:58.455830 (+ 34103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:58.730368 (+274538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:58.730767 (+   399us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:58.740105 (+  9338us) log.cc:844] Serialized 26 byte log entry
0709 12:18:58.868280 (+128175us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:18:58.868350 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.868842 (+   492us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.806316 (+2937474us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:01.867493 (+ 61177us) raft_consensus.cc:1685] finished
0709 12:19:01.867523 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.867816 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":49669,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"spinlock_wait_cycles":45173120,"thread_start_us":391937,"threads_started":2,"wal-append.queue_time_us":337676}
I20260709 12:19:02.043888 27733 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:02.000000 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:01.985208 30341 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.045046 27662 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:02.046190 27658 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:02.046711 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.046883 30341 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:02.047619 27663 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260709 12:19:02.048455 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58660.1us
I20260709 12:19:02.050256 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:01.924286 30399 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:02.038015 30512 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:02.051457 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:02.051967 30512 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.052196 27584 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:02.052600 30512 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:02.054822 30566 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:02.056591 27581 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:02.003940 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:02.005131 30149 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:02.006219 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:02.007928 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:02.027925 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.024735 27656 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
W20260709 12:19:02.029649 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58935.5us
I20260709 12:19:02.028790 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:01.908917 30110 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.041844 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.067000 27586 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:02.071565 27664 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:02.025141 30404 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.062153 30573 raft_consensus.cc:2764] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20260709 12:19:02.082175 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 74326.1us
I20260709 12:19:02.096225 30399 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:19:02.101894 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 79279.3us
I20260709 12:19:02.100553 30399 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.070953 30458 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:02.105137 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:02.104338 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.111567 27735 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:02.111795 27583 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 12:19:02.112248 27583 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 3
I20260709 12:19:02.086401 30404 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:02.108217 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.108827 28337 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
W20260709 12:19:02.113945 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.112900 27658 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
W20260709 12:19:02.114172 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49509.5us
I20260709 12:19:02.118007 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.117468 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.103895 30399 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:02.078001 30467 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:19:02.128520 27587 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
W20260709 12:19:02.123953 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1609) took 2534 ms (client timeout 2365 ms). Trace:
I20260709 12:19:02.051985 30535 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 12:19:02.134852 27655 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:19:02.140628 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.142400 27658 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:02.136296 27735 rpcz_store.cc:269] 0709 12:18:59.582128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.623472 (+ 41344us) service_pool.cc:224] Handling call
0709 12:19:02.006011 (+2382539us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223f8c 00007f0e97223ef6 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:19:02.116549 (+110538us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16430720}
I20260709 12:19:02.113322 27583 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:19:02.144856 27583 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:02.157886 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52820.2us
I20260709 12:19:02.158409 27664 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:02.163940 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:02.169955 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:02.119544 30581 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:19:02.173353 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.171993 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:02.186383 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.158179 30149 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:02.162801 30438 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:02.138552 30110 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.161620 30573 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:19:02.172959 30377 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:02.208483 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64362us
I20260709 12:19:02.211282 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:02.210744 28295 connection.cc:256] RPC call timeout handler was delayed by -1.4590444230002504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20260709 12:19:02.218885 28295 connection.cc:256] RPC call timeout handler was delayed by -1.2349444230001154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
W20260709 12:19:02.219372 28295 connection.cc:256] RPC call timeout handler was delayed by -1.0918444230001114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
I20260709 12:19:02.223464 30371 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:19:02.223932 30591 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:19:02.231390 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.233007 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.233888 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:02.206804 30525 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:02.235622 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 508) took 3017 ms. Trace:
I20260709 12:19:02.235744 30525 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:02.237047 30525 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:02.240793 30468 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:02.245488 30556 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.246012 30545 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:02.246694 30545 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.246305 30556 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.247212 30545 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:02.248805 30341 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:02.249249 30341 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:02.249282 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.250197 30341 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.248817 30377 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:02.256361 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:02.204900 30149 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:19:02.259131 30402 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:02.259578 30402 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.259979 30402 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:02.268405 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:02.298471 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:02.304811 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63884.1us
I20260709 12:19:02.306690 28332 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:02.335472 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.343055 27584 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
W20260709 12:19:02.350100 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.350924 27664 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:02.347306 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.423842 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.476595 30424 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:02.304090 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:02.483574 30149 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:02.484340 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 207835us
I20260709 12:19:02.236156 28334 rpcz_store.cc:276] 0709 12:18:59.197258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.197492 (+   234us) service_pool.cc:224] Handling call
0709 12:18:59.197854 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:59.400899 (+203045us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.400933 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:59.401403 (+   470us) log.cc:844] Serialized 26 byte log entry
0709 12:18:59.546679 (+145276us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:18:59.546734 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:02.183584 (+2636850us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:02.189143 (+  5559us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:02.194671 (+  5528us) raft_consensus.cc:1685] finished
0709 12:19:02.194690 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:02.214102 (+ 19412us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2121,"raft.queue_time_us":94,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4482,"spinlock_wait_cycles":465924992,"thread_start_us":144905,"threads_started":1,"wal-append.queue_time_us":572832,"wal-append.run_cpu_time_us":22257,"wal-append.run_wall_time_us":1911950}
I20260709 12:19:02.347932 30341 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:02.471885 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:02.473868 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 128999us
I20260709 12:19:02.352634 27656 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:02.471100 27584 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:19:02.492915 27663 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:02.496914 27655 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:19:02.500384 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.503113 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.506178 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.511942 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:02.514817 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:02.515221 30282 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:02.479152 30424 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:02.531523 27657 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6. Re-sending same reply.
I20260709 12:19:02.552799 27589 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:02.553295 27589 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:02.554148 27589 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:02.555133 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:02.555167 27589 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:02.569445 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.570560 28333 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:02.559932 27661 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:19:02.579836 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.571914 30282 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:19:02.588100 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73443.7us
I20260709 12:19:02.589119 30487 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:02.589515 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:02.594515 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:02.604918 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48743.4us
I20260709 12:19:02.595101 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.544876 30424 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:02.630918 27658 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:02.628053 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:02.628702 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.646483 27660 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:02.648185 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:02.627669 30518 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.780s	user 0.016s	sys 0.003s
I20260709 12:19:02.651053 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.657955 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.658190 27664 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:02.629983 30573 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.664418 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.661170 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:02.702693 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48679.9us
I20260709 12:19:02.650008 30377 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:02.645815 30581 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:19:02.705222 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48699.2us
I20260709 12:19:02.701529 30324 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:19:02.713099 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.712947 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.688434 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:02.720984 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:02.721745 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.665601 30556 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:02.729056 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.731242 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:02.731603 30324 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:02.738543 27701 leader_election.cc:336] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.555s (SENT)
I20260709 12:19:02.739518 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:02.718333 30403 raft_consensus.cc:2764] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:19:02.746959 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:02.655963 30518 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.762094 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.763200 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
W20260709 12:19:02.765153 28292 connection.cc:256] RPC call timeout handler was delayed by -1.8712948590000451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20260709 12:19:02.765554 28292 connection.cc:256] RPC call timeout handler was delayed by -1.8172948589999578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1569s for a response
W20260709 12:19:02.765830 28292 connection.cc:256] RPC call timeout handler was delayed by -1.7375757209999847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
W20260709 12:19:02.766152 28292 connection.cc:256] RPC call timeout handler was delayed by -1.4455948590000389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
W20260709 12:19:02.766465 28292 connection.cc:256] RPC call timeout handler was delayed by -1.3771948589999283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
W20260709 12:19:02.766774 28292 connection.cc:256] RPC call timeout handler was delayed by -1.3771948589999283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
I20260709 12:19:02.749809 30581 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:02.776194 30581 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:02.778335 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.757006 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:02.750356 27663 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:02.777729 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:02.767074 28292 connection.cc:256] RPC call timeout handler was delayed by -1.2367948589999287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260709 12:19:02.781999 28292 connection.cc:256] RPC call timeout handler was delayed by -1.2367948589999287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
I20260709 12:19:02.780921 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:02.787664 27584 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:02.727970 30573 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.804549 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:02.805594 27663 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:02.807518 27582 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.023s	sys 0.063s
W20260709 12:19:02.807938 27582 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.023s	sys 0.063s
I20260709 12:19:02.809755 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:02.761054 30504 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.891s	user 0.014s	sys 0.003s
W20260709 12:19:02.789214 30404 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.004s
I20260709 12:19:02.817022 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:02.821424 30532 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:02.822588 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:02.824396 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.825336 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:02.827214 30504 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:02.828912 30532 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:02.832377 30532 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.832754 30532 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:02.789731 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67892us
I20260709 12:19:02.835242 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:02.836601 30181 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:02.838043 27585 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:02.831619 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:02.745029 30377 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:19:02.852221 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:02.853049 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 114561us
I20260709 12:19:02.812251 30556 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:19:02.824023 30599 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:02.846374 30404 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.868011 30324 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:02.870041 30404 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:02.844975 30467 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260709 12:19:02.871043 27656 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:02.871691 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.788866 30110 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:02.868618 30573 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:02.858424 30600 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:02.872291 27657 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 3.
W20260709 12:19:02.876410 27701 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.621s (SENT)
W20260709 12:19:02.892031 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.060s	sys 0.085s
I20260709 12:19:02.895361 30555 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:02.895372 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56957.8us
I20260709 12:19:02.895659 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.875708 30600 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.876356 30404 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:02.901794 27657 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:02.896368 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:02.907034 27701 leader_election.cc:336] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.695s (SENT)
I20260709 12:19:02.907521 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:02.908219 30545 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.660s	user 0.006s	sys 0.007s
I20260709 12:19:02.909075 30556 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:02.904528 30600 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:02.909919 30556 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:02.914548 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83015.9us
I20260709 12:19:02.916122 27584 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:02.910502 30556 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:02.908766 30545 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:02.901263 30399 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.744s	user 0.012s	sys 0.005s
I20260709 12:19:02.912618 30404 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.902311 30555 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.927265 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:02.922865 27587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.017s	sys 0.022s
W20260709 12:19:02.930509 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.017s	sys 0.023s
W20260709 12:19:02.902376 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.060s	sys 0.085s
I20260709 12:19:02.943118 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:02.944257 27657 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:02.915307 27582 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 12:19:02.947175 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:02.947950 27582 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:02.949462 27582 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:19:02.979522 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:02.982460 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44339.6us
I20260709 12:19:02.936805 30404 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:02.943903 30438 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.053s	sys 0.058s
W20260709 12:19:02.937321 30551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.011s	sys 0.030s
I20260709 12:19:02.936992 30555 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:02.988183 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.002954 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:02.987852 30438 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.054s	sys 0.058s
I20260709 12:19:02.936025 30399 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:02.998669 30551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.012s	sys 0.030s
W20260709 12:19:03.011129 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.942050 30556 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.016291 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.017393 27657 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:03.006899 30525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.004s	sys 0.013s
I20260709 12:19:03.006356 30518 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.008916 30504 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:03.009337 30438 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:03.026144 30525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.004s	sys 0.014s
I20260709 12:19:03.032958 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.040818 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:03.042030 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.048372 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:03.049916 30603 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:03.050459 30603 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:03.055172 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:03.063458 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 484721us
I20260709 12:19:03.043294 30551 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:03.072351 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:03.065793 30512 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.012s	user 0.014s	sys 0.001s
I20260709 12:19:03.079658 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.082816 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:03.106032 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:03.119593 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.124589 30542 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:03.125425 30542 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:03.081182 30603 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:03.128110 30606 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:19:03.125607 27582 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:03.129678 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 351) took 1067 ms. Trace:
I20260709 12:19:03.130123 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:03.132941 27589 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.572s	user 0.010s	sys 0.001s
I20260709 12:19:03.182552 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.160488 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:03.183472 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 344912us
I20260709 12:19:03.158787 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:03.159531 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:03.186317 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 344635us
I20260709 12:19:03.130127 27586 rpcz_store.cc:276] 0709 12:19:02.062741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.077014 (+ 14273us) service_pool.cc:224] Handling call
0709 12:19:02.099683 (+ 22669us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:02.164764 (+ 65081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:02.164802 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:02.208422 (+ 43620us) log.cc:844] Serialized 26 byte log entry
0709 12:19:02.939455 (+731033us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:02.939520 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:02.939617 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:03.129413 (+189796us) raft_consensus.cc:1685] finished
0709 12:19:03.129439 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:03.129604 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":50023,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":10891,"thread_start_us":684210,"threads_started":1,"wal-append.queue_time_us":767616}
I20260709 12:19:03.187891 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:03.191813 27729 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:03.196305 27589 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:19:03.201440 27589 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:03.206204 27589 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:19:03.206684 27589 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:03.207094 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.207746 27658 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
W20260709 12:19:03.208886 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.079092 30551 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.164624 30556 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.211277 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:03.134016 30399 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.083009 30512 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.171476 30545 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.193112 30525 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:03.214303 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.223619 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.224433 27736 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:03.226029 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.228096 27734 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:19:03.229441 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.236940 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.172241 30600 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.126782 30542 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.287832 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.243294 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.248585 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.243450 30615 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:03.252724 30324 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:03.245311 30600 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:03.241899 30341 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.674s	user 0.013s	sys 0.000s
I20260709 12:19:03.254204 27733 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:03.241585 30468 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.000s	user 0.015s	sys 0.000s
I20260709 12:19:03.260661 30551 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.261225 30617 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:19:03.278712 27583 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:03.278586 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:03.280974 30110 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:03.172996 30438 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.282019 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:03.255664 30424 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:03.370906 28336 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:03.312775 27663 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:03.254477 30532 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.313513 27589 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:19:03.315584 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.315845 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.324718 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:03.327672 30626 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.084s	user 0.005s	sys 0.000s
I20260709 12:19:03.328229 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:03.328513 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:03.332221 30402 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.072s	user 0.021s	sys 0.002s
I20260709 12:19:03.334312 27736 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:03.375949 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:03.376883 27736 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:03.336006 27582 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:03.337011 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.339958 27733 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:03.344723 30618 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.083s	user 0.000s	sys 0.002s
I20260709 12:19:03.348358 27580 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:19:03.382781 27660 raft_consensus.cc:2368] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7.
I20260709 12:19:03.378319 27662 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:19:03.380807 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.395638 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.396490 27582 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:03.398651 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:03.399466 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:03.401209 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:03.401834 27735 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:03.394863 30555 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.403654 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.404790 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.406052 30424 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:03.405928 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.408205 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.409336 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.410390 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.390794 30402 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:03.413713 30402 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.414593 27655 raft_consensus.cc:2368] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11.
I20260709 12:19:03.415652 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.416831 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.417874 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.418864 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.370093 30110 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:03.420866 30532 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.420562 30110 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:03.421456 30609 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:19:03.422011 30110 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:03.422744 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.423900 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.424067 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:03.424441 27656 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:03.425179 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:03.425966 27584 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:03.425199 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.381134 30512 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:03.431349 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:03.453585 27663 raft_consensus.cc:2376] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6. Re-sending same reply.
W20260709 12:19:03.466380 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.469122 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:03.478096 28336 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:03.369855 30404 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:03.447940 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 375665us
W20260709 12:19:03.439527 30611 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:19:03.486568 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:03.492202 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:03.493889 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 271217us
I20260709 12:19:03.493335 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:03.555752 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:03.759845 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 301826us
I20260709 12:19:03.761183 27733 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:03.500898 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:03.761904 27701 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.851s (SENT)
I20260709 12:19:03.757615 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:03.765997 28336 raft_consensus.cc:2368] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:03.649495 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:03.758607 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:03.760339 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:03.733126 30581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.912s	user 0.008s	sys 0.008s
I20260709 12:19:03.762441 27656 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:03.636051 30402 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20260709 12:19:03.776507 27736 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:03.766603 27589 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:03.777067 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:03.636365 27657 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:03.516673 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.762878 27733 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:03.781749 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.782450 27661 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:03.773754 30581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.960s	user 0.008s	sys 0.008s
I20260709 12:19:03.787271 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.789812 27656 raft_consensus.cc:2376] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7. Re-sending same reply.
I20260709 12:19:03.792258 30404 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:03.793273 30555 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:03.794870 27733 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:19:03.795733 30402 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:03.796671 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:03.829176 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50590us
I20260709 12:19:03.860858 27656 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:03.876748 27661 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:03.888936 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.889235 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.889736 27662 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:19:03.890832 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.892534 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:03.893040 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 461643us
I20260709 12:19:03.893289 27661 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6. Re-sending same reply.
I20260709 12:19:03.860028 30402 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:03.891824 27662 raft_consensus.cc:2376] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9. Re-sending same reply.
I20260709 12:19:03.652369 30512 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.247s
I20260709 12:19:03.913249 27586 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:03.914414 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.916402 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.942458 27661 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:03.916488 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.923233 30399 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:03.861687 30424 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:03.925371 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.925019 27487 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 1 to 7, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 7 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:03.652004 30630 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.242s
I20260709 12:19:03.935416 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:03.931941 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 170058us
I20260709 12:19:03.917140 27729 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 7. Ops: []
I20260709 12:19:03.916586 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:03.898568 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:03.917562 30611 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:19:03.941604 27662 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:03.984043 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.992331 27658 raft_consensus.cc:2376] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9. Re-sending same reply.
I20260709 12:19:03.992655 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.990039 27656 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:03.986418 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.994295 27661 raft_consensus.cc:2376] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5. Re-sending same reply.
I20260709 12:19:03.899215 30149 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:04.000006 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.000914 27584 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:04.002025 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.002537 27584 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:04.007717 30424 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:04.008246 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.009773 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.010872 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.011652 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:04.012722 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.013625 27663 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:19:04.014494 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.033084 27663 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:04.010957 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:03.957265 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1883) took 1132 ms. Trace:
I20260709 12:19:04.037918 27583 rpcz_store.cc:276] 0709 12:19:02.781529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.781679 (+   150us) service_pool.cc:224] Handling call
0709 12:19:03.913864 (+1132185us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:03.914013 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1076276992}
I20260709 12:19:04.043849 27662 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:04.120086 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.121465 27661 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:04.045513 27736 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:19:04.046980 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:04.127384 27658 raft_consensus.cc:2368] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:19:04.047744 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.049388 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1866) took 2212 ms. Trace:
I20260709 12:19:04.128201 27585 rpcz_store.cc:276] 0709 12:19:01.837353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.858522 (+ 21169us) service_pool.cc:224] Handling call
0709 12:19:04.049135 (+2190613us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:04.049318 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2236224256}
I20260709 12:19:04.051877 27587 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
W20260709 12:19:04.037401 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 207148us
I20260709 12:19:04.130015 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:04.134536 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:04.134465 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.135107 27735 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:04.135459 27662 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:04.141973 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.152904 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.155644 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.158277 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.161223 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.161984 27583 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:04.168270 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.159348 27663 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:04.169858 27662 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 12:19:04.170006 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.171768 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:04.175172 27734 raft_consensus.cc:2368] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
W20260709 12:19:04.175765 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.183758 27589 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:04.168651 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.191960 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:04.197367 27701 leader_election.cc:336] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.632s (SENT)
I20260709 12:19:04.197973 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.199530 27656 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 12:19:04.202886 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 287981us
I20260709 12:19:04.174974 30623 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:04.203429 27663 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:04.205680 27658 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:19:04.207964 27663 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:19:04.211426 27663 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:04.212723 27589 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:04.204241 30110 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.782s	user 0.011s	sys 0.000s
I20260709 12:19:04.216029 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:04.220034 27658 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:19:04.220780 27585 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:04.205081 30623 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:04.226336 27589 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:04.227010 27589 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:04.232995 27661 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:19:04.232877 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1884) took 1432 ms (client timeout 1826 ms). Trace:
W20260709 12:19:04.234544 27580 rpcz_store.cc:269] 0709 12:19:02.800824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.801085 (+   261us) service_pool.cc:224] Handling call
0709 12:19:04.232493 (+1431408us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:04.232794 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1122895232}
I20260709 12:19:04.233682 27589 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.238304 27663 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 7. Ops: []
I20260709 12:19:04.240998 27583 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:04.244638 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:04.247105 27583 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:04.245889 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:04.253895 27663 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:04.254827 27589 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:04.256232 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.230036 30623 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:04.261736 27580 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:04.221338 30149 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:04.256202 27585 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:04.232291 30399 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:04.259997 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.285672 27663 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:04.286700 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.275600 27580 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:04.277460 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80115.3us
I20260709 12:19:04.278561 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.279872 27583 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:19:04.278352 27585 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:04.299000 27589 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260709 12:19:04.299851 27589 pending_rounds.cc:85] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
I20260709 12:19:04.300787 27663 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:04.253683 30625 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:04.301800 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1845) took 2324 ms. Trace:
I20260709 12:19:04.302251 27589 pending_rounds.cc:107] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:19:04.300885 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:04.285481 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:04.306572 27589 raft_consensus.cc:2889] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:04.309898 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:04.311148 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:04.311939 28334 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:04.318405 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116407us
I20260709 12:19:04.307152 27585 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.321781 27589 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:19:04.323518 27580 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:04.325377 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:04.326695 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:04.322067 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:04.308162 30625 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:04.335223 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.304507 27659 rpcz_store.cc:276] 0709 12:19:02.014286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.043445 (+ 29159us) service_pool.cc:224] Handling call
0709 12:19:02.044445 (+  1000us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:02.127804 (+ 83359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:02.127926 (+   122us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:02.168100 (+ 40174us) log.cc:844] Serialized 52 byte log entry
0709 12:19:02.646514 (+478414us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:02.650755 (+  4241us) log.cc:844] Serialized 14 byte log entry
0709 12:19:03.297153 (+646398us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b1800302260. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:19:04.155039 (+857886us) log.cc:844] Serialized 14 byte log entry
0709 12:19:04.280684 (+125645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.280771 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.301587 (+ 20816us) raft_consensus.cc:1685] finished
0709 12:19:04.301608 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.301743 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25230,"prepare.queue_time_us":452561,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":4758,"raft.queue_time_us":459905,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":9099,"spinlock_wait_cycles":10117376,"thread_start_us":1407370,"threads_started":4,"wal-append.queue_time_us":790179,"wal-append.run_cpu_time_us":2842,"wal-append.run_wall_time_us":1146796}
I20260709 12:19:04.338910 27585 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:04.339111 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.341187 27661 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:04.342363 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.338661 28329 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:04.343775 30649 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:04.344408 30634 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:19:04.344738 30559 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.344658 30633 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:04.346030 30643 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:19:04.346202 30650 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:04.343010 27658 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:04.349465 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 371) took 1744 ms. Trace:
I20260709 12:19:04.349870 27664 rpcz_store.cc:276] 0709 12:19:02.602857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.668207 (+ 65350us) service_pool.cc:224] Handling call
0709 12:19:02.713719 (+ 45512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:03.081600 (+367881us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:03.081632 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:03.186405 (+104773us) log.cc:844] Serialized 26 byte log entry
0709 12:19:03.476471 (+290066us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:03.476512 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:03.476578 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.347044 (+870466us) raft_consensus.cc:1685] finished
0709 12:19:04.347070 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.347217 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":330318,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4881,"thread_start_us":585917,"threads_started":2,"wal-append.queue_time_us":940990}
I20260709 12:19:04.338335 30625 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:04.346685 30650 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:04.345685 30633 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:04.356130 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58048.8us
I20260709 12:19:04.333844 30110 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:19:04.345458 30559 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.359961 27660 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:04.360673 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:04.362038 27655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1866) took 1316 ms (client timeout 1510 ms). Trace:
W20260709 12:19:04.362381 27655 rpcz_store.cc:269] 0709 12:19:03.042960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.043430 (+   470us) service_pool.cc:224] Handling call
0709 12:19:04.359012 (+1315582us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54001e3cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:04.359165 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":824470912}
I20260709 12:19:04.358510 30650 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:04.368829 30651 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:04.346594 30149 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.381856 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1634) took 1097 ms. Trace:
I20260709 12:19:04.382349 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:04.366179 30630 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:04.369256 30647 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:04.391762 30647 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:04.392200 30647 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:04.392544 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1873443820002194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260709 12:19:04.392937 27552 connection.cc:256] RPC call timeout handler was delayed by -1.7967443820000426s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
W20260709 12:19:04.393251 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5816443820001496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2177s for a response
I20260709 12:19:04.393482 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1637) took 1108 ms. Trace:
W20260709 12:19:04.393564 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4124443820001034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23650000000000004s for a response
I20260709 12:19:04.393602 30402 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.393944 27737 rpcz_store.cc:276] 0709 12:19:03.285264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.922496 (+637232us) service_pool.cc:224] Handling call
0709 12:19:03.922788 (+   292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:04.213433 (+290645us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:04.213467 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:04.220164 (+  6697us) log.cc:844] Serialized 26 byte log entry
0709 12:19:04.359771 (+139607us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:04.361832 (+  2061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.361910 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.393166 (+ 31256us) raft_consensus.cc:1685] finished
0709 12:19:04.393183 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.393317 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101283,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":402283,"threads_started":2,"wal-append.queue_time_us":148029}
I20260709 12:19:04.380939 30650 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:19:04.400626 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 355) took 1130 ms. Trace:
I20260709 12:19:04.401069 27588 rpcz_store.cc:276] 0709 12:19:03.270089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.270400 (+   311us) service_pool.cc:224] Handling call
0709 12:19:03.270749 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:03.270948 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:03.271706 (+   758us) log.cc:844] Serialized 14 byte log entry
0709 12:19:04.400280 (+1128574us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:04.400332 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:04.400376 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.400442 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.400572 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1127721,"prepare.queue_time_us":190,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":68}
I20260709 12:19:04.387015 30559 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.366220 30646 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:04.387957 30651 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:04.388640 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1876) took 2134 ms. Trace:
I20260709 12:19:04.389164 30623 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.389839 30633 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:04.394954 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.394894 30402 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:04.410171 27585 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:04.409827 30399 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:04.382709 27733 rpcz_store.cc:276] 0709 12:19:03.284295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.888916 (+604621us) service_pool.cc:224] Handling call
0709 12:19:03.916997 (+ 28081us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:04.243285 (+326288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:04.243316 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:04.249052 (+  5736us) log.cc:844] Serialized 26 byte log entry
0709 12:19:04.356870 (+107818us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:04.356920 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.357062 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.381617 (+ 24555us) raft_consensus.cc:1685] finished
0709 12:19:04.381637 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.381771 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":65732,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":58,"thread_start_us":288059,"threads_started":2,"wal-append.queue_time_us":64971}
I20260709 12:19:04.404508 27581 rpcz_store.cc:276] 0709 12:19:02.254218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.309756 (+ 55538us) service_pool.cc:224] Handling call
0709 12:19:02.492035 (+182279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:02.600539 (+108504us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:19:02.649693 (+ 49154us) log.cc:844] Serialized 14 byte log entry
0709 12:19:03.057068 (+407375us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800442000. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:03.110305 (+ 53237us) log.cc:844] Serialized 14 byte log entry
0709 12:19:04.388173 (+1277868us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b1800442000. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:19:04.388317 (+   144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:04.388366 (+    49us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:04.388412 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.388478 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.388586 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1273787,"prepare.queue_time_us":25169,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":1318330,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":218,"spinlock_wait_cycles":177287040,"thread_start_us":348620,"threads_started":1,"wal-append.queue_time_us":598676}
W20260709 12:19:04.431222 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48977.8us
I20260709 12:19:04.432367 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:04.435177 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1904) took 1099 ms. Trace:
I20260709 12:19:04.437592 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.438692 30468 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.439182 30468 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.439736 30650 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:04.440316 30630 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:04.441169 30468 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.415886 30399 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:04.441824 30399 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:04.435663 27582 rpcz_store.cc:276] 0709 12:19:03.347433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.427170 (+ 79737us) service_pool.cc:224] Handling call
0709 12:19:03.442870 (+ 15700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:03.645030 (+202160us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:03.757356 (+112326us) log.cc:844] Serialized 14 byte log entry
0709 12:19:04.434810 (+677454us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:04.434872 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:04.434923 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.434995 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.435122 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":408941,"prepare.queue_time_us":111913,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":6594,"thread_start_us":268014,"threads_started":1,"wal-append.queue_time_us":455006}
I20260709 12:19:04.443717 30617 raft_consensus.cc:1064] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:19:04.445889 30617 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.400771 30516 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:19:04.447872 30630 consensus_queue.cc:1243] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:19:04.451682 30533 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:04.452438 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:04.452907 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.453857 30581 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:04.452018 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55112.7us
I20260709 12:19:04.455996 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:04.457062 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:04.457435 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:04.458227 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:04.459653 30525 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:04.460161 30525 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:04.460420 30632 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:04.461179 30525 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.436642 30534 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.463896 30534 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.464386 30632 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:04.464941 30632 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.466116 30534 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.467119 30632 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:04.467969 30632 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.468636 27553 leader_election.cc:400] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:04.468456 30632 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:04.468953 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.406931 30282 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 12:19:04.468962 27553 leader_election.cc:403] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:19:04.470677 30631 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:04.470409 30632 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.407588 30646 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:04.463919 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:04.476755 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:04.477118 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.477401 27589 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:04.427650 30110 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:04.474407 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109698us
I20260709 12:19:04.481070 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:04.474888 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.484844 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:04.437192 30651 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:04.491012 27582 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:04.492434 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:04.483722 30650 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:04.436381 30457 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:04.495826 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:04.498610 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.499663 27588 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:04.499872 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:04.472637 30600 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:04.501049 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.494534 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.476591 30555 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:04.479444 30532 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:04.516377 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.518802 27589 raft_consensus.cc:2368] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10.
I20260709 12:19:04.485687 30533 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:04.470489 30656 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:19:04.474782 30404 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:19:04.475165 30581 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.536252 30658 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:04.491918 30632 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:19:04.525283 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.523053 30282 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:04.529313 30533 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:04.530159 30534 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:04.531924 30646 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.534584 30625 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.516646 30653 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:19:04.535619 30650 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:04.501163 30110 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.514017 30649 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:19:04.553079 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57893.5us
I20260709 12:19:04.518325 30600 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.575420 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 522) took 1275 ms. Trace:
I20260709 12:19:04.541333 30581 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.517877 30516 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:04.493808 30651 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:04.582887 30651 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:04.584226 30647 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.500066 30542 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:19:04.577965 28332 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:19:04.578332 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49745.6us
W20260709 12:19:04.540305 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 64667.6us
I20260709 12:19:04.586477 30647 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:04.524171 30457 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.545284 30399 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.576018 28330 rpcz_store.cc:276] 0709 12:19:03.277650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.277769 (+   119us) service_pool.cc:224] Handling call
0709 12:19:03.278105 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:03.310374 (+ 32269us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:03.310678 (+   304us) log.cc:844] Serialized 14 byte log entry
0709 12:19:03.776083 (+465405us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:03.776137 (+    54us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:03.776179 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.553066 (+776887us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:04.553110 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.553282 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11429,"prepare.queue_time_us":3104,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":42,"raft.queue_time_us":64749,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":29966,"spinlock_wait_cycles":73568512,"thread_start_us":453421,"threads_started":1,"wal-append.queue_time_us":708787}
I20260709 12:19:04.605120 28333 raft_consensus.cc:1217] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Deduplicated request from leader. Original: 7.7->[9.8-9.8]   Dedup: 9.8->[]
I20260709 12:19:04.605021 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:04.605808 28333 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:19:04.607270 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:04.608078 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 112303us
W20260709 12:19:04.596836 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95213.3us
I20260709 12:19:04.608290 27589 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:04.609637 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.609961 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:04.610935 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:04.616709 30282 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.612326 30656 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:19:04.617592 30656 consensus_queue.cc:1243] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:19:04.617393 30282 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.616731 30646 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:04.595423 30650 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:04.619901 28334 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:04.610102 30555 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:04.642860 30533 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:04.643357 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:04.636102 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:04.623050 30653 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:04.622658 30623 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:04.651032 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 156566us
I20260709 12:19:04.611522 30402 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:19:04.655151 30555 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.661463 30670 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:04.661957 30670 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:04.662366 30670 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:04.644389 30668 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:04.668124 30600 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.668324 30669 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:04.669948 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.653216 30282 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:04.670307 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.623924 30581 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:19:04.647357 30653 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:04.623026 30532 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:04.678164 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:04.682857 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.683027 30625 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:04.683527 30650 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:04.658855 30667 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:04.687848 27704 leader_election.cc:400] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10.
I20260709 12:19:04.641566 30660 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:19:04.679173 30669 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:04.686899 30600 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:04.704751 30600 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:04.705053 28330 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:19:04.704782 30647 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:04.705823 30600 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.708076 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.708789 30653 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.709910 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:04.711026 30650 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:04.702387 30667 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:04.713299 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.688714 30670 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.712294 30667 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:04.717308 30670 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:04.707267 30600 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:04.721890 30532 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.724086 30669 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:04.737079 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1877) took 1439 ms. Trace:
I20260709 12:19:04.737542 27662 rpcz_store.cc:276] 0709 12:19:03.297237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:04.159988 (+862751us) service_pool.cc:224] Handling call
0709 12:19:04.169459 (+  9471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:04.170300 (+   841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.736880 (+566580us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b54001e46d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:04.737021 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20926208}
I20260709 12:19:04.719179 30647 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:04.744001 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:04.747105 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 153781us
I20260709 12:19:04.726382 30110 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:04.740955 30668 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:04.757827 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1892) took 1752 ms. Trace:
I20260709 12:19:04.762055 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:04.758214 27587 rpcz_store.cc:276] 0709 12:19:03.004972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.005145 (+   173us) service_pool.cc:224] Handling call
0709 12:19:04.051618 (+1046473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:04.137150 (+ 85532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.757622 (+620472us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001fe0d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:04.757766 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2371130752}
I20260709 12:19:04.871410 27662 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:04.869913 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 352) took 1611 ms. Trace:
I20260709 12:19:04.874255 27586 rpcz_store.cc:276] 0709 12:19:03.258491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.259570 (+  1079us) service_pool.cc:224] Handling call
0709 12:19:03.912998 (+653428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:03.941474 (+ 28476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.869578 (+928104us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54000f00d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:04.869790 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1502002944}
W20260709 12:19:04.890812 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:04.893852 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 149931us
I20260709 12:19:04.757052 30649 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:04.901854 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:04.917831 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 4, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 4 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:04.926577 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:04.927268 27660 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:04.929529 27658 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:04.945420 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:04.955293 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 311993us
I20260709 12:19:04.955632 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.920856 30399 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:04.965696 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:04.922259 30649 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:04.970790 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:04.921070 30551 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:19:04.982681 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87892.5us
I20260709 12:19:05.001569 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:05.009217 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.020687 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.022326 27730 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:05.023023 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.023789 27734 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:05.024547 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.025101 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:05.025882 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.026352 27734 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:05.027618 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.028033 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.028714 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.029414 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.030265 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:05.030628 30619 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.782s	user 0.009s	sys 0.036s
I20260709 12:19:04.981288 30649 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:19:05.037933 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.039605 27737 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:05.018837 27733 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:05.052507 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.053220 27733 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:05.054226 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.055500 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:05.047576 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 146079us
W20260709 12:19:05.056953 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73538.2us
W20260709 12:19:05.065361 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58284.2us
I20260709 12:19:05.011566 30647 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:05.070992 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:04.984440 30581 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:19:05.073078 27736 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:05.050987 30621 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.789s	user 0.014s	sys 0.017s
I20260709 12:19:05.072157 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.072780 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:05.080438 27730 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:04.979984 30551 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:05.083115 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.083891 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1635) took 1793 ms. Trace:
I20260709 12:19:05.084110 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.084324 27735 rpcz_store.cc:276] 0709 12:19:03.284613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.919034 (+634421us) service_pool.cc:224] Handling call
0709 12:19:04.126234 (+207200us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:04.145296 (+ 19062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.078248 (+932952us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54002e8bd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:05.078374 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475548032}
I20260709 12:19:05.086416 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:05.086782 30682 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:19:05.086977 27735 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 3.
I20260709 12:19:05.087944 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.088943 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.089918 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.090415 27735 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:05.113503 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.117301 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:05.127084 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.127946 27730 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 7
W20260709 12:19:05.128062 30678 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:19:05.128746 27735 raft_consensus.cc:2410] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:19:05.129627 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:05.130478 30680 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:19:05.132261 30551 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.135082 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.136551 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.137262 27736 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:05.138458 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:05.141284 27734 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:05.143211 27736 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:05.048725 30651 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.151510 27734 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:05.155093 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77731us
W20260709 12:19:05.158236 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44795.6us
I20260709 12:19:05.159605 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:05.161199 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.170773 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.124228 30686 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:19:05.153715 30621 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:19:05.175931 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.137825 30534 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:19:05.179663 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.167258 30282 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:05.181816 27733 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:05.144410 30149 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:05.179384 30516 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:05.183588 30676 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:19:05.184065 30679 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:19:05.184482 27736 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:05.190618 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:05.178447 30581 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:05.196218 27735 pending_rounds.cc:85] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 1
W20260709 12:19:05.198220 30662 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.544s	user 0.000s	sys 0.007s
I20260709 12:19:05.196223 30664 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:19:05.199872 27657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 1865) took 2174 ms (client timeout 2134 ms). Trace:
I20260709 12:19:05.202848 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.143162 30399 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.210690 27730 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:05.174888 30667 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:05.219018 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62156.9us
I20260709 12:19:05.201998 30551 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:05.224783 27735 pending_rounds.cc:107] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:19:05.227265 28332 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:05.238713 27735 raft_consensus.cc:2889] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:05.231851 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.216976 30686 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.237552 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:05.228430 30676 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:05.229120 27657 rpcz_store.cc:269] 0709 12:19:03.025316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.025436 (+   120us) service_pool.cc:224] Handling call
0709 12:19:05.199658 (+2174222us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:05.199801 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1379756800}
I20260709 12:19:05.210314 30534 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:05.246706 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.226881 30581 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:05.228591 30282 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:05.238574 30650 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.000s
I20260709 12:19:05.230918 30525 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:05.253744 28332 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
W20260709 12:19:05.255385 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 351456us
I20260709 12:19:05.207598 30647 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.228375 30664 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:05.267972 30468 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:05.268455 30468 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:05.268703 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.269311 30468 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.270411 30468 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:05.272012 30676 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.251277 30581 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:05.230365 30399 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.279975 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.281459 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.282080 27730 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:05.282618 30457 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:05.261308 30650 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:19:05.284825 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94373us
I20260709 12:19:05.288647 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.284770 27731 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:05.255095 30149 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:05.309274 30664 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.281545 30399 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:05.310971 30534 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.292207 30461 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:05.300112 30110 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:05.311280 27584 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:19:05.330107 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44255us
I20260709 12:19:05.330911 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:05.332280 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:05.332667 30668 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:05.333606 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:05.334123 30653 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:05.320461 30521 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:05.320988 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 366) took 2693 ms (client timeout 1569 ms). Trace:
I20260709 12:19:05.338181 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:05.337906 30521 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.338958 28332 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:05.340749 30521 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:05.254313 30686 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.344098 30686 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:05.323161 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:05.334748 30600 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.614s	user 0.007s	sys 0.004s
I20260709 12:19:05.326673 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1586) took 1130 ms. Trace:
I20260709 12:19:05.336391 27585 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:19:05.346304 27656 rpcz_store.cc:276] 0709 12:19:04.196093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:04.209306 (+ 13213us) service_pool.cc:224] Handling call
0709 12:19:05.326434 (+1117128us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400211454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:05.326609 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":819195136}
I20260709 12:19:05.243156 30667 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.335144 27732 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:05.336391 27582 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:19:05.336506 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:05.317608 30651 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:05.316134 30461 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.329692 30649 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:05.337930 27738 rpcz_store.cc:269] 0709 12:19:02.627301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.627461 (+   160us) service_pool.cc:224] Handling call
0709 12:19:05.320767 (+2693306us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:05.320911 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5694054656}
I20260709 12:19:05.373708 27731 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:05.357550 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:05.351269 30632 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:05.376542 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:05.351666 30525 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:05.378279 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:05.321192 30646 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.703s	user 0.005s	sys 0.005s
W20260709 12:19:05.377002 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 201602us
I20260709 12:19:05.403113 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:05.399715 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:05.323493 30670 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
W20260709 12:19:05.406911 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73386.3us
I20260709 12:19:05.402225 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:05.354271 30633 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:05.409732 27729 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:05.410799 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.351948 30630 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:05.376178 30399 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:05.312511 30534 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:05.415236 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.412559 27738 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:05.373369 30149 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:05.417326 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:05.417740 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129138us
I20260709 12:19:05.421214 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:05.398219 30632 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:05.404809 30525 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:05.424892 27730 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:05.427023 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:05.427788 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.408684 30660 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:05.417479 30665 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:05.432298 30665 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:05.432449 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.432735 30665 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:05.432863 30683 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.289s	user 0.000s	sys 0.003s
I20260709 12:19:05.435232 30698 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:05.414146 30635 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:05.435868 30698 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:05.436165 30633 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:05.436376 30698 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:05.433084 30461 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:05.419302 30181 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:19:05.447242 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:05.419559 30617 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.463198 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.429095 30632 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:05.469130 27738 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 12:19:05.473493 30667 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:05.474747 30660 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:05.476558 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.478349 30581 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.440347 30525 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:05.480515 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:05.496083 27701 leader_election.cc:400] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6.
I20260709 12:19:05.500957 30181 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:05.501420 30181 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:05.501780 30181 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:05.464008 30646 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:05.503763 30646 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:05.504740 30646 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.513825 30671 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:05.481492 30617 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.491475 30668 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:05.518651 30545 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:05.532223 27738 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
W20260709 12:19:05.536620 30690 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
W20260709 12:19:05.536972 30702 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:19:05.538383 30660 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.549985 30649 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:05.551222 30149 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.561969 27730 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:05.567354 30677 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:05.567950 30677 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:05.567718 30181 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:05.568998 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.568907 30677 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.570163 30677 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:05.496871 30670 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:19:05.571506 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:05.572093 27587 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:05.581276 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:05.584570 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:05.609462 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.622030 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:05.625954 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:05.632436 27730 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:19:05.635871 28330 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:19:05.638128 27738 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:05.639617 30181 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:05.642571 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111295us
W20260709 12:19:05.644995 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 153951us
I20260709 12:19:05.646003 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:05.649467 30668 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:05.656373 30684 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.588s	user 0.000s	sys 0.003s
I20260709 12:19:05.659638 30457 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:05.660576 30457 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:05.660923 30457 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:05.661845 30457 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.636329 30402 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:19:05.670885 30707 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:19:05.636497 30545 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:05.672132 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:05.674583 30617 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:05.675701 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:05.633870 30633 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:05.692417 27730 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:19:05.693799 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50413.8us
I20260709 12:19:05.676568 28334 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
W20260709 12:19:05.679273 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 257499us
W20260709 12:19:05.683667 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:05.696432 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.637908 30651 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:19:05.697301 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.663079 30457 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:05.720675 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.713505 27731 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:05.654762 30632 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.714841 27737 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:05.715128 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:05.715934 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70040.9us
I20260709 12:19:05.730593 27655 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:05.735610 27737 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:05.635039 30686 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
W20260709 12:19:05.741890 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 324612us
I20260709 12:19:05.745250 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:05.747722 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:05.748442 27584 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:05.748857 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:05.750056 27582 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:05.742888 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.753638 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.754324 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:05.717501 30651 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:05.756562 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:05.711577 30668 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:05.747879 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:05.762833 27585 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:05.719137 30603 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.763713 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.770109 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:05.771006 27659 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:05.717304 30633 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:05.697726 30650 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:19:05.714386 30545 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:05.736799 27737 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.795609 27737 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:05.798102 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.760208 30600 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:05.777637 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:05.764906 30651 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:05.761070 30551 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.516s	user 0.009s	sys 0.002s
W20260709 12:19:05.792375 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59479.5us
I20260709 12:19:05.759123 27731 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:05.776823 27659 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:19:05.806922 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.825606 27738 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:05.739611 30617 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:19:05.782095 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:05.739475 30468 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.045s
I20260709 12:19:05.821346 27661 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:05.767062 30461 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.824306 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:05.809633 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:05.760882 30282 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 12:19:05.856108 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.857122 27731 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:05.834860 27660 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:19:05.826745 27729 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:05.836010 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:05.841341 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.748546 30387 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:19:05.846040 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:05.868681 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.874917 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.878413 27659 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:05.824460 30668 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:05.782387 30603 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.894162 27657 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:19:05.893918 27737 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:19:05.895437 27737 pending_rounds.cc:85] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:19:05.895742 27737 pending_rounds.cc:107] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:19:05.896047 27737 raft_consensus.cc:2889] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:05.870301 30110 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.000s
I20260709 12:19:05.897408 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.851517 30581 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:05.902029 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56076.7us
I20260709 12:19:05.903501 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:05.904670 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.904637 27657 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:05.827765 30698 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:05.941713 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.908272 27663 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:05.960678 30533 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:05.972884 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:05.973256 28336 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
W20260709 12:19:05.985299 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 240110us
I20260709 12:19:05.866704 30461 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:05.986927 27737 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:05.992249 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:05.923516 30110 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:05.874547 30665 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.904940 30710 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:19:06.020779 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.897259 30668 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.050114 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:05.990782 30533 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:06.051687 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:06.044400 27487 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:06.076746 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84570.2us
I20260709 12:19:06.054138 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:19:06.083666 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115753us
W20260709 12:19:06.089174 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.939761 30545 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:05.952860 30633 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.106668 27487 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:06.015214 30700 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.330s
W20260709 12:19:06.109619 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.115213 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.053632 30110 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
W20260709 12:19:06.120095 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 235) took 23108 ms (client timeout 15000 ms). Trace:
I20260709 12:19:06.123874 27487 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 9, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:06.127007 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.105528 30632 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.111497 30704 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:06.145639 27487 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 3 to 4, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 4 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:06.121589 30725 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:06.120397 30282 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.151360 30282 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.151856 30732 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:06.153637 30282 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:06.154577 30282 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.155490 30282 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.156216 30581 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:06.156642 30581 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:06.157508 30581 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:06.158396 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.158865 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.157888 30282 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:06.160962 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3465453879998677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
W20260709 12:19:06.161594 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2259453879999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 12:19:06.162113 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1593453879997924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
I20260709 12:19:06.162242 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:06.162482 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1215453879999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
I20260709 12:19:06.161505 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:06.164077 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.110926 30653 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:19:06.169835 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:06.158797 30581 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:06.132287 30704 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.174327 27583 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:06.174916 30716 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
W20260709 12:19:06.175786 30717 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 12:19:06.176326 30632 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:19:06.176644 30719 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:19:06.177311 30698 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.176618 30660 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:19:06.178287 30712 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 12:19:06.130226 30726 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:19:06.201787 30646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.002s	sys 0.003s
W20260709 12:19:06.202746 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 299335us
I20260709 12:19:06.219349 30704 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:06.172065 27487 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:06.218143 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91246.2us
I20260709 12:19:06.252064 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:06.224543 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 373) took 1011 ms. Trace:
I20260709 12:19:06.201804 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:06.160004 30725 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.198982 30402 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:06.199386 30623 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:06.246886 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:06.203357 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.230803 30534 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.803s	user 0.007s	sys 0.007s
W20260709 12:19:06.244719 30525 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.743s	user 0.016s	sys 0.000s
I20260709 12:19:06.252956 27736 rpcz_store.cc:276] 0709 12:19:05.213145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.585964 (+372819us) service_pool.cc:224] Handling call
0709 12:19:05.586296 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.586505 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:05.757705 (+171200us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.224050 (+466345us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.224106 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:06.224149 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.224347 (+   198us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.224474 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":182939,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":600206,"threads_started":2}
I20260709 12:19:06.194609 30533 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:06.551493 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 349865us
I20260709 12:19:06.552516 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:06.394471 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.005s	sys 0.057s
W20260709 12:19:06.553723 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.012s	sys 0.057s
W20260709 12:19:06.554088 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 840784us
I20260709 12:19:06.289296 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 389) took 1025 ms. Trace:
I20260709 12:19:06.558213 27664 rpcz_store.cc:276] 0709 12:19:05.263886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.264562 (+   676us) service_pool.cc:224] Handling call
0709 12:19:05.264840 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.323757 (+ 58917us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.323792 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:05.323838 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:05.323874 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.289039 (+965165us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:06.289089 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.289232 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135033216}
W20260709 12:19:06.123103 27483 rpcz_store.cc:269] 0709 12:18:43.003040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.003341 (+   301us) service_pool.cc:224] Handling call
0709 12:19:06.103923 (+23100582us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:54.989601 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:54.990207 (+   606us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:54.990260 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:18:54.990279 (+    19us) tablet.cc:662] Decoding operations
0709 12:18:54.991788 (+  1509us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:54.991898 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:54.991938 (+    40us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:18:54.993481 (+  1543us) tablet.cc:701] Row locks acquired
0709 12:18:54.993547 (+    66us) write_op.cc:260] PREPARE: finished
0709 12:18:54.993751 (+   204us) write_op.cc:270] Start()
0709 12:18:54.993923 (+   172us) write_op.cc:276] Timestamp: P: 1783599534993675 usec, L: 0
0709 12:18:54.993953 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 12:18:54.994994 (+  1041us) log.cc:844] Serialized 5386 byte log entry
0709 12:18:56.525060 (+1530066us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:18:56.693458 (+168398us) op_driver.cc:464] REPLICATION: finished
0709 12:18:56.757221 (+ 63763us) write_op.cc:301] APPLY: starting
0709 12:18:56.757311 (+    90us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:56.757577 (+   266us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:56.757614 (+    37us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:56.759297 (+  1683us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:56.759994 (+   697us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0709 12:18:56.760025 (+    31us) write_op.cc:312] APPLY: finished
0709 12:18:56.761207 (+  1182us) log.cc:844] Serialized 95 byte log entry
0709 12:18:56.780520 (+ 19313us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:56.783558 (+  3038us) write_op.cc:454] Released schema lock
0709 12:18:56.785213 (+  1655us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:56.816193 (+ 30980us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:56.820795 (+  4602us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2770987,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":63431,"apply.run_cpu_time_us":3952,"apply.run_wall_time_us":53022,"num_ops":13,"prepare.queue_time_us":275388,"prepare.run_cpu_time_us":31293,"prepare.run_wall_time_us":907794,"raft.queue_time_us":126946,"raft.run_cpu_time_us":14140,"raft.run_wall_time_us":56393,"replication_time_us":1699343,"spinlock_wait_cycles":15054080,"thread_start_us":979450,"threads_started":4,"wal-append.queue_time_us":799452,"wal-append.run_cpu_time_us":17765,"wal-append.run_wall_time_us":1917994}]]}
I20260709 12:19:06.347672 30727 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.563898 30727 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.566807 30727 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.567449 30525 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:06.349401 30722 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
W20260709 12:19:06.333806 30743 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 12:19:06.246068 30646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.002s	sys 0.003s
W20260709 12:19:06.339143 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.351925 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 495) took 1023 ms. Trace:
I20260709 12:19:06.579604 28330 rpcz_store.cc:276] 0709 12:19:05.329017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.329191 (+   174us) service_pool.cc:224] Handling call
0709 12:19:05.329555 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.636271 (+306716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.351655 (+715384us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:06.351859 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":703634816}
W20260709 12:19:06.316255 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110377us
W20260709 12:19:06.581467 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.333165 30714 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
W20260709 12:19:06.350337 30745 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:19:06.456959 30461 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:06.350944 30724 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:19:06.298399 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:06.599727 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:06.242038 30726 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.583262 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.569404 30623 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:06.455930 30719 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:19:06.353327 30665 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.260824 30725 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:06.620842 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 374011us
I20260709 12:19:06.261201 30603 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:06.348430 30705 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:19:06.425666 30402 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.628782 27658 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:19:06.634230 27487 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 8, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 8 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:06.638728 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55531.6us
I20260709 12:19:06.639710 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.304464 30728 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:06.599116 30110 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:06.639854 27487 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 4 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:06.646559 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:06.637701 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:06.655112 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:06.628635 30461 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:06.663225 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 372) took 1450 ms. Trace:
I20260709 12:19:06.621661 30726 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:06.309242 30521 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.328s
W20260709 12:19:06.672127 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.615666 30650 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:19:06.685230 27583 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:06.685696 30727 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:06.686156 30686 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:06.686547 30525 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.638176 30705 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:06.686643 30646 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:06.687413 30743 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:19:06.687880 27588 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:06.685232 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:06.737589 27584 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:06.689126 30738 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.691639 30730 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:06.652931 27487 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:06.692368 30715 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.491s	user 0.001s	sys 0.000s
W20260709 12:19:06.694535 30457 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.955s	user 0.006s	sys 0.004s
W20260709 12:19:06.697773 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.651216 30402 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.699096 30623 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.644083 30728 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:06.673550 27733 rpcz_store.cc:276] 0709 12:19:05.212883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.572086 (+359203us) service_pool.cc:224] Handling call
0709 12:19:05.572440 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:05.583550 (+ 11110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.583581 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:05.613927 (+ 30346us) log.cc:844] Serialized 26 byte log entry
0709 12:19:05.743985 (+130058us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:05.852635 (+108650us) log.cc:844] Serialized 14 byte log entry
0709 12:19:05.905138 (+ 52503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.905206 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:06.662993 (+757787us) raft_consensus.cc:1685] finished
0709 12:19:06.663024 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.663161 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57342,"prepare.queue_time_us":94833,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":235668,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":20121,"thread_start_us":231535,"threads_started":3,"wal-append.queue_time_us":338561}
I20260709 12:19:06.700865 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 379) took 1116 ms. Trace:
I20260709 12:19:06.803686 27587 rpcz_store.cc:276] 0709 12:19:05.584044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.592425 (+  8381us) service_pool.cc:224] Handling call
0709 12:19:05.592860 (+   435us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:05.650734 (+ 57874us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.650781 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:05.721190 (+ 70409us) log.cc:844] Serialized 52 byte log entry
0709 12:19:05.855770 (+134580us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:05.855827 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.855900 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:06.700602 (+844702us) raft_consensus.cc:1685] finished
0709 12:19:06.700620 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.700800 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49144,"raft.queue_time_us":51658,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":23072,"thread_start_us":87665,"threads_started":1,"wal-append.queue_time_us":268150}
I20260709 12:19:06.805495 30729 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:06.702812 30468 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:19:06.705883 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66249.2us
I20260709 12:19:06.806332 30468 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:06.806613 27731 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:19:06.806813 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.807395 30468 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.807928 30525 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:06.808892 30746 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:19:06.650815 30534 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:06.809527 30534 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:06.810283 30534 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:06.655380 30665 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:06.709830 30660 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.712574 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 394) took 1021 ms. Trace:
I20260709 12:19:06.715749 30149 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:06.746266 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:06.747759 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164201us
I20260709 12:19:06.698904 30704 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:06.735862 30651 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.896s	user 0.007s	sys 0.005s
I20260709 12:19:06.734761 30705 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:06.806231 30729 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.808832 30468 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:06.809453 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.741343 30646 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.722947 30698 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.814378 30149 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:06.811471 27660 rpcz_store.cc:276] 0709 12:19:05.690976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.691196 (+   220us) service_pool.cc:224] Handling call
0709 12:19:05.714168 (+ 22972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.867932 (+153764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.712326 (+844394us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:06.712517 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":273246592}
W20260709 12:19:06.814759 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68589.7us
I20260709 12:19:06.787439 30686 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:06.807868 30755 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.835852 30755 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.812772 30728 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:06.840164 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:06.727468 27583 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:06.782724 30650 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:19:06.827982 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:06.814592 30665 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:06.727082 30700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.005s	sys 0.021s
W20260709 12:19:06.842078 30700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.006s	sys 0.021s
I20260709 12:19:06.841837 27583 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.829312 30651 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.842895 27583 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:06.842449 30665 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.843351 27736 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:06.843488 30744 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:06.844018 30744 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:06.844450 30744 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:06.844461 30727 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:06.845295 30727 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:06.846298 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:06.846266 30727 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.847152 27587 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:19:06.847446 30741 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:06.848255 30752 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:06.848856 30752 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.851161 30461 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:06.851470 30371 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:06.834563 30729 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.851478 30753 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:19:06.853750 30651 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:06.830190 30698 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.858952 30698 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:06.853792 30461 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:06.854135 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:06.834334 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 281918us
I20260709 12:19:06.788506 30725 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.861814 30461 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:06.863646 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.864475 27732 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:06.864920 30461 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:06.865175 30521 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:19:06.865864 30581 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.651s	user 0.007s	sys 0.004s
I20260709 12:19:06.851637 30757 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:06.866694 30461 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:06.866992 30757 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:19:06.867785 30402 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:06.870039 30757 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.870368 30402 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:06.863821 30725 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:06.873039 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.819145 30660 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.875346 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.876240 27587 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:06.876611 30757 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:06.876852 30402 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:06.885296 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:06.886420 30725 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:06.886994 30725 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:06.877379 30660 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:06.878952 30461 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:06.879895 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.887951 30725 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.888723 30461 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:06.888304 30149 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.889521 30149 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:06.890771 30735 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:06.834416 30686 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.892292 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.765630 30738 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:06.881656 27552 leader_election.cc:400] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:06.839012 30755 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:06.831492 30734 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.832733 30525 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:06.844043 30665 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:06.906455 28334 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:06.904824 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.908282 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:06.911041 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.900269 30461 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:19:06.913911 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.904063 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.005s	sys 0.005s
I20260709 12:19:06.915349 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.915608 30741 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:06.908070 30581 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:06.925791 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:06.909391 30752 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:06.913954 30676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.000s	sys 0.011s
W20260709 12:19:06.922334 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.005s	sys 0.005s
I20260709 12:19:06.924937 30700 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:06.930115 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:06.911337 30729 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 12:19:06.913388 30755 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.020s
W20260709 12:19:06.932701 30676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.000s	sys 0.012s
I20260709 12:19:06.907637 30734 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.930346 30752 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:06.966361 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:06.967705 27587 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:19:06.974167 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:06.984045 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 528) took 1012 ms. Trace:
I20260709 12:19:06.984521 28334 rpcz_store.cc:276] 0709 12:19:05.971867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.972044 (+   177us) service_pool.cc:224] Handling call
0709 12:19:05.972393 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.909198 (+936805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.983766 (+ 74568us) spinlock_profiling.cc:243] Waited 934 ms on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:06.983972 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2147558144}
I20260709 12:19:06.985958 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:06.986375 30747 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:19:06.988420 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58384.9us
I20260709 12:19:06.990687 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:06.949455 30755 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:06.999018 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:06.999732 28332 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:06.933243 30581 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:06.986763 30534 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:07.010227 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 190569us
I20260709 12:19:07.012887 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.014467 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.012502 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.018016 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.019544 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.020449 30668 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:07.022655 30677 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.452s	user 0.010s	sys 0.007s
I20260709 12:19:07.001724 30704 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.014725 30630 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.024341 30738 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:07.024089 30630 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.026258 30630 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:07.027657 30686 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:07.011384 30762 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:07.029503 30762 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:07.029560 30402 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:07.029901 30762 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:07.031513 28330 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:07.035524 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.036357 27736 raft_consensus.cc:2368] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7.
I20260709 12:19:07.037019 30534 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:07.038198 27662 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:07.038794 30646 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:07.041204 30752 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:07.010571 30512 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.041733 30752 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.042279 30512 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.044411 30752 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.044605 30512 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:07.046509 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.020450 30734 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.051223 30282 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:07.051674 30282 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:07.052604 30282 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.053259 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:07.054005 30282 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:07.053784 30762 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.054514 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.060801 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.065634 27664 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:07.066418 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:07.037127 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:07.068620 27736 raft_consensus.cc:2376] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7. Re-sending same reply.
W20260709 12:19:07.069533 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.071498 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.072088 27658 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:07.073657 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.076473 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.077550 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:07.078740 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.079485 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:07.090932 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.091176 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.092990 27659 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:19:07.093937 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 103328us
I20260709 12:19:07.094202 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:06.986779 30747 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:07.096247 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:07.113497 27589 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:07.114569 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.115561 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.116119 27660 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:07.117007 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:07.117835 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.120316 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:07.120885 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47020.8us
I20260709 12:19:07.120250 27585 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:07.118197 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 392) took 1428 ms. Trace:
I20260709 12:19:07.151054 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.151707 27659 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:07.150738 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:07.137765 27656 rpcz_store.cc:276] 0709 12:19:05.689341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.689477 (+   136us) service_pool.cc:224] Handling call
0709 12:19:05.689801 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.690032 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:05.755276 (+ 65244us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.721172 (+965896us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b180042ed40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:07.117874 (+396702us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:07.117930 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:07.117969 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.118024 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.118141 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2022,"prepare.queue_time_us":52829,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":13458,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":2088,"spinlock_wait_cycles":6923136,"thread_start_us":396426,"threads_started":2,"wal-append.queue_time_us":1238437}
I20260709 12:19:07.156018 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.157029 27656 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:07.127033 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.168275 27655 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:07.129855 30752 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:07.170895 30752 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:07.172124 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.172277 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.172906 27664 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:07.173015 27734 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:07.175625 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:07.182040 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:07.119845 30600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.253s	user 0.006s	sys 0.026s
I20260709 12:19:07.110662 30371 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.130s
I20260709 12:19:07.112342 30424 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.138s
I20260709 12:19:07.110440 30756 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.137s
I20260709 12:19:07.186513 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.248677 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.296049 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:07.267686 27660 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:19:07.323366 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.169565 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.275753 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.261869 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.333544 27734 raft_consensus.cc:2376] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7. Re-sending same reply.
I20260709 12:19:07.110280 30646 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:07.313907 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.114852 30740 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:07.310621 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.338517 27664 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:07.171411 30752 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:07.119673 30623 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:07.349152 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73515.2us
I20260709 12:19:07.338848 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:07.310787 30726 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.583s	user 0.012s	sys 0.000s
I20260709 12:19:07.239312 30729 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:07.201865 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.344184 27627 leader_election.cc:400] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7.
I20260709 12:19:07.230832 30534 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.119939 30671 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:07.351459 27656 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:07.353051 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.356876 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:07.338078 30600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.480s	user 0.009s	sys 0.037s
W20260709 12:19:07.211292 30732 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.058s	user 0.007s	sys 0.020s
I20260709 12:19:07.248330 30741 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.326792 30424 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:19:07.367928 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:07.381979 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.389185 27660 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:07.391739 27662 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
W20260709 12:19:07.393121 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.396040 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:07.310582 30734 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:19:07.352298 30630 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:07.172168 30633 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.208s
I20260709 12:19:07.404338 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:07.408667 27663 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:07.353034 30651 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:07.387717 30732 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.235s	user 0.009s	sys 0.020s
I20260709 12:19:07.415405 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:07.353919 30686 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:07.419519 28334 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:19:07.388576 30738 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:07.388136 30600 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:07.424006 27662 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:07.405860 30677 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:07.430152 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73334.3us
I20260709 12:19:07.392947 30755 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:07.430948 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.388623 30671 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:07.405084 30757 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.406877 30399 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:07.419452 30512 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:07.443455 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:07.445050 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:07.356784 30646 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.380331 30726 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.406869 30630 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:07.407572 30660 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:07.447249 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.420044 30762 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:07.389863 30729 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.392030 30534 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:07.450508 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1648) took 1663 ms (client timeout 1797 ms). Trace:
I20260709 12:19:07.438665 30581 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.452132 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:07.455891 27738 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:07.460285 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:07.514333 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54172.2us
I20260709 12:19:07.515657 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.470958 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:07.445466 30525 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.537s	user 0.009s	sys 0.001s
W20260709 12:19:07.448541 30744 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.603s	user 0.012s	sys 0.000s
I20260709 12:19:07.450986 28332 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:19:07.445881 30402 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:19:07.488600 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.448513 30757 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.491066 27733 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:07.443473 30399 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:07.404865 30741 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.475626 30651 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:07.476550 27737 rpcz_store.cc:269] 0709 12:19:05.781211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.897073 (+115862us) service_pool.cc:224] Handling call
0709 12:19:07.444102 (+1547029us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b540017e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:07.444194 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103834496}
I20260709 12:19:07.450914 30424 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:07.492990 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.541136 27735 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:07.457010 30650 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:07.477198 30660 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.504813 30534 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:07.505497 30785 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:19:07.506031 30774 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:19:07.440999 30677 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.440490 30671 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:07.512010 28333 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:07.517781 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
W20260709 12:19:07.519939 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.486256 30784 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:07.526954 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:07.524900 30755 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:19:07.521827 30744 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.531662 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:07.522226 30525 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:19:07.532876 30782 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:19:07.533126 30630 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.534330 30399 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.536566 30741 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.536921 30651 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:07.537401 30665 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.626s	user 0.004s	sys 0.005s
I20260709 12:19:07.521943 30668 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:07.537684 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:07.576192 27737 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:07.527105 30740 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:07.512732 30773 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:19:07.434504 30686 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.548031 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:07.580154 27588 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:07.549216 30720 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.354s	user 0.007s	sys 0.002s
I20260709 12:19:07.518410 30738 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.581976 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:07.563066 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:07.522418 30542 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.563606 27660 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[5.2-9.4]   Dedup: 5.2->[7.3-9.4]
W20260709 12:19:07.568972 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.585126 27660 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [5.2-9.4]
I20260709 12:19:07.565157 30784 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:07.545639 30650 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:07.577450 28330 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:07.584918 27735 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:07.575903 30651 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:07.521965 30402 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:07.573963 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:07.589147 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.554627 30424 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 12:19:07.564060 30600 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:07.579602 30781 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:19:07.594007 27738 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:07.577489 30740 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:07.595705 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:07.604048 30792 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.001s	sys 0.001s
I20260709 12:19:07.591449 30542 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.617743 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:07.594328 30650 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:07.622373 27729 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:07.588301 30738 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.626175 27737 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:07.598897 30399 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:07.632146 30744 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:07.632920 30776 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:07.633452 30776 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.636188 30757 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:07.636421 30729 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.637207 30402 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:07.637909 30784 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:07.638444 30402 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:07.639204 30784 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:07.631264 27735 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260709 12:19:07.638921 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 669406us
W20260709 12:19:07.642376 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 121156us
I20260709 12:19:07.642813 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:07.643152 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.643960 27588 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:19:07.645175 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.646006 27737 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:19:07.646466 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:07.647692 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.647969 27588 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:07.647924 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:07.651880 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:07.625991 30728 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.786s	user 0.009s	sys 0.003s
I20260709 12:19:07.654611 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:07.654796 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.660494 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.661320 27738 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:07.626987 30646 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.764068 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:07.764796 27737 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:07.822562 27738 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:07.651656 30726 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.662154 30728 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:07.852603 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 203162us
I20260709 12:19:07.983376 27738 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:07.979238 27737 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:08.620698 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1805) took 1715 ms. Trace:
I20260709 12:19:08.621060 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1809) took 1707 ms. Trace:
I20260709 12:19:08.621515 27737 rpcz_store.cc:276] 0709 12:19:06.914009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.978602 (+1064593us) service_pool.cc:224] Handling call
0709 12:19:07.979018 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.620807 (+641789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.621007 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:07.880208 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:08.626298 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:08.344813 27729 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.720s	user 0.011s	sys 0.000s
I20260709 12:19:08.627381 27729 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:08.628019 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1662) took 1277 ms. Trace:
I20260709 12:19:08.628427 27729 rpcz_store.cc:276] 0709 12:19:07.350225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.600138 (+249913us) service_pool.cc:224] Handling call
0709 12:19:08.627946 (+1027808us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:08.621311 27738 rpcz_store.cc:276] 0709 12:19:06.905289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.862868 (+957579us) service_pool.cc:224] Handling call
0709 12:19:07.936828 (+ 73960us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.620292 (+683464us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.620565 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:07.911063 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:08.630899 27588 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.981s	user 0.011s	sys 0.000s
I20260709 12:19:08.632906 27588 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:07.904680 30726 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.730s
I20260709 12:19:08.634645 27586 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:08.637276 27730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.009s	sys 0.014s
W20260709 12:19:08.642724 27730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.009s	sys 0.015s
W20260709 12:19:08.638144 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:08.025743 30791 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.401s	user 0.000s	sys 0.000s
I20260709 12:19:08.651535 27738 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:19:08.659683 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1819) took 1699 ms. Trace:
I20260709 12:19:08.645314 27588 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:08.028388 30796 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 12:19:07.988735 30709 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:08.681345 27737 rpcz_store.cc:276] 0709 12:19:06.960232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.635497 (+1675265us) service_pool.cc:224] Handling call
0709 12:19:08.644817 (+  9320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.659159 (+ 14342us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:08.659205 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:08.659267 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:08.659312 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.659436 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.659606 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13835}
W20260709 12:19:08.026535 30795 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.263s	user 0.000s	sys 0.000s
I20260709 12:19:08.065639 30677 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:08.492105 30755 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.013s	sys 0.024s
W20260709 12:19:08.640779 30623 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.281s	user 0.011s	sys 0.002s
I20260709 12:19:08.691527 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1821) took 1729 ms. Trace:
I20260709 12:19:07.990545 30660 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:08.619432 30686 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:07.851305 30729 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:08.456871 30468 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.627s	user 0.014s	sys 0.000s
I20260709 12:19:07.791307 30646 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:08.032733 30802 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:19:07.892482 30745 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 12:19:08.402400 30705 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.571s	user 0.013s	sys 0.000s
I20260709 12:19:08.656540 30668 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:19:08.705247 27738 rpcz_store.cc:276] 0709 12:19:06.961885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.635408 (+1673523us) service_pool.cc:224] Handling call
0709 12:19:08.651249 (+ 15841us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.691135 (+ 39886us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.691452 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:07.897075 30757 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:08.709755 27588 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:07.863185 30776 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:08.731602 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1941) took 1088 ms. Trace:
I20260709 12:19:08.732159 27588 rpcz_store.cc:276] 0709 12:19:07.643353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.644821 (+1001468us) service_pool.cc:224] Handling call
0709 12:19:08.645181 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.731171 (+ 85990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.731521 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:07.935941 30282 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.881s	user 0.008s	sys 0.005s
W20260709 12:19:08.708346 30764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.000s	sys 0.033s
W20260709 12:19:08.699967 30755 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.014s	sys 0.029s
W20260709 12:19:08.722131 30399 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.090s	user 0.009s	sys 0.003s
I20260709 12:19:08.690932 30709 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:08.717056 30802 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:08.739526 27588 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:08.738169 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:08.741539 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1942) took 1098 ms. Trace:
I20260709 12:19:08.741920 27588 rpcz_store.cc:276] 0709 12:19:07.643602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.738748 (+1095146us) service_pool.cc:224] Handling call
0709 12:19:08.739094 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.741259 (+  2165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.741471 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:08.743589 27588 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:08.744372 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1943) took 1100 ms. Trace:
I20260709 12:19:08.744805 27588 rpcz_store.cc:276] 0709 12:19:07.643824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.743001 (+1099177us) service_pool.cc:224] Handling call
0709 12:19:08.743375 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.744083 (+   708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.744310 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:08.749409 30671 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.182s	user 0.012s	sys 0.000s
I20260709 12:19:08.749080 30709 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.749879 30671 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:08.750277 30709 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:08.727666 30534 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.746659 30282 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:08.751682 30534 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:08.751951 30282 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:08.753774 30752 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.397s	user 0.005s	sys 0.008s
I20260709 12:19:08.754575 30812 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.754312 30752 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.755107 30812 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:08.756785 30650 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.116s	user 0.009s	sys 0.002s
I20260709 12:19:08.757366 30812 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:08.757212 30650 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:08.758494 27735 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:08.734324 30764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.000s	sys 0.034s
I20260709 12:19:08.756611 30752 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:08.760079 30402 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.121s	user 0.012s	sys 0.000s
I20260709 12:19:08.760197 30728 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:08.754305 30282 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:08.761555 30650 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:08.764727 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1926) took 1689 ms. Trace:
W20260709 12:19:08.762524 30651 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.162s	user 0.011s	sys 0.000s
W20260709 12:19:08.765050 30740 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.156s	user 0.012s	sys 0.000s
I20260709 12:19:08.747437 30802 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:08.759462 30533 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:08.765743 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1641) took 3006 ms. Trace:
I20260709 12:19:08.767987 27730 rpcz_store.cc:276] 0709 12:19:05.759101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.762091 (+  2990us) service_pool.cc:224] Handling call
0709 12:19:05.783703 (+ 21612us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:06.201288 (+417585us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.201325 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:07.656611 (+1455286us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:19:07.666135 (+  9524us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.643625 (+977490us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:19:08.643662 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.643738 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:08.765450 (+121712us) raft_consensus.cc:1685] finished
0709 12:19:08.765470 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.765688 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27491,"raft.queue_time_us":202794,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":28380,"spinlock_wait_cycles":35308928,"thread_start_us":1099415,"threads_started":2,"wal-append.queue_time_us":876285}
I20260709 12:19:08.765179 27657 rpcz_store.cc:276] 0709 12:19:07.074953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.075116 (+   163us) service_pool.cc:224] Handling call
0709 12:19:07.119502 (+ 44386us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:07.149872 (+ 30370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:07.149923 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:07.201440 (+ 51517us) log.cc:844] Serialized 52 byte log entry
0709 12:19:07.420132 (+218692us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:07.448434 (+ 28302us) log.cc:844] Serialized 14 byte log entry
0709 12:19:07.464286 (+ 15852us) log.cc:844] Serialized 14 byte log entry
0709 12:19:07.473084 (+  8798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.473160 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:08.764515 (+1291355us) raft_consensus.cc:1685] finished
0709 12:19:08.764536 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.764670 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25051,"prepare.queue_time_us":74769,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":62,"raft.queue_time_us":26245,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":83819,"spinlock_wait_cycles":3261696,"thread_start_us":124544,"threads_started":1,"wal-append.queue_time_us":327105}
W20260709 12:19:08.763212 30784 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.124s	user 0.007s	sys 0.003s
I20260709 12:19:08.765494 30651 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:08.731823 30705 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:08.732662 30729 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:08.789917 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.842s	user 0.038s	sys 0.059s
I20260709 12:19:08.765568 30740 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.736706 30757 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.795480 30671 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:08.802642 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.038s	sys 0.059s
I20260709 12:19:08.813508 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:08.816809 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.16488e+06us
I20260709 12:19:08.818842 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:08.775238 30600 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.000s	sys 0.012s
I20260709 12:19:08.821029 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:08.819972 27588 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:08.830715 30757 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:08.830977 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 395) took 2040 ms. Trace:
I20260709 12:19:08.831552 27733 rpcz_store.cc:276] 0709 12:19:06.790244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.817914 (+ 27670us) service_pool.cc:224] Handling call
0709 12:19:06.825075 (+  7161us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.524606 (+699531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.830695 (+1306089us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:08.830904 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1531156224}
I20260709 12:19:08.832835 30630 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:08.832959 30705 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:08.835440 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:08.826004 30600 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.000s	sys 0.012s
I20260709 12:19:08.838680 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:08.839689 27588 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:08.839514 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:08.840572 30686 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:08.836807 30806 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:08.847079 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:08.847491 27588 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:08.848322 30625 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.850477 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:08.843278 30686 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:08.848961 30650 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:19:08.850847 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.026s	sys 0.036s
W20260709 12:19:08.871148 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.026s	sys 0.036s
I20260709 12:19:08.871479 27588 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:08.872020 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:08.874766 30600 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:08.875492 30600 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.876488 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1916) took 1814 ms. Trace:
I20260709 12:19:08.876999 27587 rpcz_store.cc:276] 0709 12:19:07.062076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.063641 (+  1565us) service_pool.cc:224] Handling call
0709 12:19:07.115229 (+ 51588us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:07.349035 (+233806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:07.349077 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:07.349532 (+   455us) log.cc:844] Serialized 26 byte log entry
0709 12:19:07.526754 (+177222us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:07.526813 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.875954 (+1349141us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400089dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:08.876020 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:08.876106 (+    86us) raft_consensus.cc:1685] finished
0709 12:19:08.876127 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.876414 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3497,"raft.queue_time_us":3650,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":10250,"spinlock_wait_cycles":512425856,"thread_start_us":119382,"threads_started":1,"wal-append.queue_time_us":211201}
I20260709 12:19:08.881342 27587 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:19:08.849659 30784 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:08.898551 27584 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.028s	sys 0.028s
W20260709 12:19:08.918533 27584 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.028s	sys 0.028s
I20260709 12:19:08.854723 30623 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:08.916847 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45886.1us
I20260709 12:19:08.919169 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1940) took 1276 ms. Trace:
I20260709 12:19:08.867733 30399 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:08.920701 30399 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:08.920055 27584 rpcz_store.cc:276] 0709 12:19:07.643067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.657088 (+ 14021us) service_pool.cc:224] Handling call
0709 12:19:07.657356 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.658638 (+  1282us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:07.765825 (+107187us) log.cc:844] Serialized 14 byte log entry
0709 12:19:08.072336 (+306511us) log.cc:844] Serialized 14 byte log entry
0709 12:19:08.918852 (+846516us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:08.918900 (+    48us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:08.918939 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.918995 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.919118 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":133837,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":118,"raft.queue_time_us":200134,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":161175,"spinlock_wait_cycles":129822848,"thread_start_us":1151910,"threads_started":2,"wal-append.queue_time_us":963077}
I20260709 12:19:08.921334 30399 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:08.917802 30784 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.919602 30651 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:08.922539 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50593.9us
I20260709 12:19:08.854944 30649 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:08.879139 30600 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:08.927423 27587 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:19:08.931001 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:08.907932 30757 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.909761 30705 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.945020 27588 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:08.945705 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:08.947664 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.856078 30625 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.962285 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:08.966495 28333 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:08.926810 30726 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:08.908401 30731 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
W20260709 12:19:08.949508 27627 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.633s (SENT) [suppressed 45 similar messages]
I20260709 12:19:08.870657 30804 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260709 12:19:08.912374 30545 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:08.931761 30649 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:08.919684 30623 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.944494 30793 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:08.949472 30709 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.999502 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:08.995505 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:08.940593 30650 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260709 12:19:08.972985 30814 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.966174 30651 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:08.993129 30649 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:09.003786 30545 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.018041 30545 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.018975 30545 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:09.019625 30545 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:08.946331 30704 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:19:09.020205 30545 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:08.974805 30675 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:09.021649 30675 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.992815 30729 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:09.009917 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61504.7us
W20260709 12:19:09.023442 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.950886 30807 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:19:08.957185 30757 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.963090 30705 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:08.989567 30646 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260709 12:19:09.041725 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 92240.4us
I20260709 12:19:09.044390 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:09.046131 30813 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:19:09.049954 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 498) took 1655 ms. Trace:
I20260709 12:19:09.050467 28330 rpcz_store.cc:276] 0709 12:19:07.394564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.394721 (+   157us) service_pool.cc:224] Handling call
0709 12:19:07.420123 (+ 25402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.593249 (+173126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.049664 (+1456415us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:09.049877 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":359042432}
I20260709 12:19:09.052392 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1929) took 2401 ms. Trace:
I20260709 12:19:09.011137 30815 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:09.052765 27586 rpcz_store.cc:276] 0709 12:19:06.651896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.653231 (+  1335us) service_pool.cc:224] Handling call
0709 12:19:06.653652 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.642071 (+1988419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.052178 (+410107us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400186fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:09.052335 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4555213696}
I20260709 12:19:09.051424 30822 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:09.063656 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.063823 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.064359 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
W20260709 12:19:09.055348 27627 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.647s (SENT)
I20260709 12:19:08.984915 30686 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.065075 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:09.066542 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71134.1us
I20260709 12:19:09.065460 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.007853 30709 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.073467 28333 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:09.079586 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:09.079823 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.081063 28337 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:09.084028 27586 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:09.008158 30793 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.084066 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.085273 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:09.085361 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.086689 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.087625 30830 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.088701 30816 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:09.091965 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.092649 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1823) took 2122 ms. Trace:
I20260709 12:19:09.027238 30651 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.013051 30814 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.107722 30832 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:19:09.108508 30806 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:09.112354 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57127.7us
I20260709 12:19:09.101526 27586 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
W20260709 12:19:09.122826 27627 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.668s (SENT)
I20260709 12:19:09.120127 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1820) took 2159 ms. Trace:
W20260709 12:19:09.123929 27627 leader_election.cc:336] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.679s (SENT)
W20260709 12:19:09.125034 27627 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.683s (SENT)
I20260709 12:19:09.125905 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:09.124011 27729 rpcz_store.cc:276] 0709 12:19:06.961194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.634765 (+1673571us) service_pool.cc:224] Handling call
0709 12:19:08.635144 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.662437 (+ 27293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.662484 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.681796 (+ 19312us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.944421 (+262625us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:09.075814 (+131393us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.118777 (+ 42963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.119771 (+   994us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.119819 (+    48us) raft_consensus.cc:1685] finished
0709 12:19:09.119832 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.120068 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35456,"prepare.queue_time_us":133509,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":5159,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":320445,"threads_started":3,"wal-append.queue_time_us":376150}
I20260709 12:19:09.114909 27738 rpcz_store.cc:276] 0709 12:19:06.970393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.726198 (+1755805us) service_pool.cc:224] Handling call
0709 12:19:08.738039 (+ 11841us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.738374 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.738403 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.752898 (+ 14495us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.972897 (+219999us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:09.019970 (+ 47073us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.063505 (+ 43535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.064453 (+   948us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.092410 (+ 27957us) raft_consensus.cc:1685] finished
0709 12:19:09.092429 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.092568 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20892,"prepare.queue_time_us":105485,"prepare.run_cpu_time_us":18,"prepare.run_wall_time_us":19,"raft.queue_time_us":88335,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":23829,"thread_start_us":275136,"threads_started":3,"wal-append.queue_time_us":276187}
I20260709 12:19:09.108762 30755 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.097412 30726 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:09.128911 27586 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:19:09.135056 30726 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.119568 30807 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.146308 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1824) took 2140 ms. Trace:
I20260709 12:19:09.145641 27738 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:09.147147 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1654) took 2047 ms. Trace:
I20260709 12:19:09.147706 27738 rpcz_store.cc:276] 0709 12:19:07.099893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.131471 (+2031578us) service_pool.cc:224] Handling call
0709 12:19:09.145452 (+ 13981us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.146909 (+  1457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.147087 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:09.128007 30828 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.156133 30828 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.157043 30793 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.146780 27730 rpcz_store.cc:276] 0709 12:19:07.005574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.770083 (+1764509us) service_pool.cc:224] Handling call
0709 12:19:08.775446 (+  5363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.775801 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.775828 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.779007 (+  3179us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.962411 (+183404us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:09.102384 (+139973us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.145889 (+ 43505us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.145990 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.146032 (+    42us) raft_consensus.cc:1685] finished
0709 12:19:09.146042 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.146251 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28901,"prepare.queue_time_us":115794,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":1841,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":3001,"thread_start_us":295276,"threads_started":3,"wal-append.queue_time_us":227610}
I20260709 12:19:09.074079 30815 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.133235 30650 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:09.075793 30822 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.151513 30807 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.149303 30461 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:19:09.152824 30811 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:19:09.116483 30830 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.149111 27580 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:09.159952 27586 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:09.154944 30534 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.163375 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.179302 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:09.133132 30806 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.163939 30828 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.166769 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:09.167042 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:09.168453 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:09.139264 30755 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.172983 27738 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:09.185426 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1655) took 2085 ms. Trace:
I20260709 12:19:09.185786 27738 rpcz_store.cc:276] 0709 12:19:07.100251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.148846 (+2048595us) service_pool.cc:224] Handling call
0709 12:19:09.167459 (+ 18613us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.185166 (+ 17707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.185367 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:09.181704 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:09.174147 27730 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:09.180071 27655 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:09.181533 27658 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:09.186853 27662 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:09.182044 27656 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:09.182314 27659 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:09.169543 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.189038 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1656) took 2088 ms. Trace:
I20260709 12:19:09.189461 27730 rpcz_store.cc:276] 0709 12:19:07.100576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.159381 (+2058805us) service_pool.cc:224] Handling call
0709 12:19:09.173904 (+ 14523us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.187958 (+ 14054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.188978 (+  1020us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:09.177845 30762 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:09.253058 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56658.3us
I20260709 12:19:09.177724 30623 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.242522 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.224315 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:09.259135 27586 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:09.192212 30838 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:09.273639 30649 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.276229 30814 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.276515 30755 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.277245 30649 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.272004 27730 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:09.274076 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.279418 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1661) took 1935 ms. Trace:
W20260709 12:19:09.281757 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.280216 27730 rpcz_store.cc:276] 0709 12:19:07.343759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.242059 (+1898300us) service_pool.cc:224] Handling call
0709 12:19:09.271776 (+ 29717us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.278876 (+  7100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.279267 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:09.289800 27658 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:09.277804 30840 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:09.198065 30822 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.203162 30399 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.301461 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1919) took 2238 ms. Trace:
W20260709 12:19:09.298964 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1806) took 2392 ms (client timeout 2848 ms). Trace:
W20260709 12:19:09.328120 27735 rpcz_store.cc:269] 0709 12:19:06.906416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.654580 (+748164us) service_pool.cc:224] Handling call
0709 12:19:09.298705 (+1644125us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002a0654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:09.298889 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2536337152}
I20260709 12:19:09.314210 30670 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:09.318703 27738 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:09.274751 30838 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:09.326328 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.328049 27581 rpcz_store.cc:276] 0709 12:19:07.062901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.064987 (+  2086us) service_pool.cc:224] Handling call
0709 12:19:07.115596 (+ 50609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.115787 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:07.332029 (+216242us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.297134 (+1965105us) spinlock_profiling.cc:243] Waited 6.77 ms on lock 0x7b18004427e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a
0709 12:19:09.301030 (+  3896us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:09.301101 (+    71us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:09.301163 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.301253 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.301386 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3539,"prepare.queue_time_us":288486,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":3247,"raft.queue_time_us":4644,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"spinlock_wait_cycles":15576320,"thread_start_us":262473,"threads_started":2,"wal-append.queue_time_us":216901,"wal-append.run_cpu_time_us":2017,"wal-append.run_wall_time_us":1055438}
I20260709 12:19:09.335121 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1660) took 2159 ms. Trace:
I20260709 12:19:09.337764 27738 rpcz_store.cc:276] 0709 12:19:07.175834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.186648 (+2010814us) service_pool.cc:224] Handling call
0709 12:19:09.318438 (+131790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.329578 (+ 11140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.335056 (+  5478us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:09.272255 30762 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.312146 27656 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:09.344611 27662 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:09.293433 30757 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:09.337284 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.334867 30670 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:09.352587 30399 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.192953 30835 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.142s
W20260709 12:19:09.368230 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.368407 30625 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.373481 27586 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:09.362861 30664 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:09.365738 30707 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.403419 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.405190 27586 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:09.242456 30747 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.364738 30705 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.413699 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:09.421211 28330 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:09.404372 30461 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:09.415220 30742 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.799s	user 0.007s	sys 0.006s
I20260709 12:19:09.420994 30748 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.411242 30686 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.412031 30707 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:09.423898 30737 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 3.270s	user 0.005s	sys 0.023s
I20260709 12:19:09.438954 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.442639 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 390) took 3245 ms. Trace:
I20260709 12:19:09.423468 30399 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.443105 27732 rpcz_store.cc:276] 0709 12:19:06.197734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.198354 (+   620us) service_pool.cc:224] Handling call
0709 12:19:06.232617 (+ 34263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.864888 (+632271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.442375 (+2577487us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:09.442568 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1452620032}
W20260709 12:19:09.433215 30802 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.659s	user 0.013s	sys 0.000s
I20260709 12:19:09.427673 30815 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.443614 30747 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:09.424229 30670 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.459877 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.464759 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1597) took 3675 ms. Trace:
I20260709 12:19:09.465197 27661 rpcz_store.cc:276] 0709 12:19:05.788990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.838027 (+ 49037us) service_pool.cc:224] Handling call
0709 12:19:05.870196 (+ 32169us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:06.110308 (+240112us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.110329 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:06.114801 (+  4472us) log.cc:844] Serialized 26 byte log entry
0709 12:19:06.200115 (+ 85314us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:19:06.200224 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.200370 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.309702 (+3109332us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:09.464444 (+154742us) raft_consensus.cc:1685] finished
0709 12:19:09.464466 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.464696 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":121102,"raft.queue_time_us":209354,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":20670,"spinlock_wait_cycles":9608576,"thread_start_us":162115,"threads_started":2,"wal-append.queue_time_us":307300}
W20260709 12:19:09.467228 30848 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:19:09.468667 30833 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.178s	user 0.001s	sys 0.010s
I20260709 12:19:09.469403 30707 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.438704 30748 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.470592 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1822) took 2503 ms. Trace:
I20260709 12:19:09.471004 27737 rpcz_store.cc:276] 0709 12:19:06.967552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.702075 (+1734523us) service_pool.cc:224] Handling call
0709 12:19:08.718189 (+ 16114us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.718896 (+   707us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:08.973345 (+254449us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.120818 (+147473us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:09.121363 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.121634 (+   271us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:09.121969 (+   335us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.122185 (+   216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.122292 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.470373 (+348081us) raft_consensus.cc:1685] finished
0709 12:19:09.470394 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.470535 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31822,"prepare.queue_time_us":311167,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":3616,"raft.queue_time_us":106807,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":10365,"spinlock_wait_cycles":12336384,"thread_start_us":340488,"threads_started":3,"wal-append.queue_time_us":287233}
I20260709 12:19:09.430156 30461 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.472540 30748 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.476059 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.474893 30461 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.486907 30670 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:09.488098 27586 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:09.489953 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.492312 27588 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:09.491567 30651 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.497465 30766 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.498103 30766 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.457404 30807 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.503003 30830 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.454432 30802 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.509428 30802 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:09.493402 30843 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:19:09.463187 30806 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.449803 30399 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.505281 30461 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.511370 27582 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:09.511492 30461 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.512738 30399 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.514569 27729 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:09.516937 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1653) took 2417 ms. Trace:
I20260709 12:19:09.517267 27729 rpcz_store.cc:276] 0709 12:19:07.099551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.133826 (+2034275us) service_pool.cc:224] Handling call
0709 12:19:09.147617 (+ 13791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.515174 (+367557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.516718 (+  1544us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54001d15d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:09.516879 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843438464}
I20260709 12:19:09.441383 30686 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.514510 30461 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.523617 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.524386 28330 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:09.523250 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:09.526472 27582 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:09.527491 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.528682 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:09.528910 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:09.521360 30545 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.005s
I20260709 12:19:09.531188 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:09.530580 27582 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:09.540223 30664 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:09.545769 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.545378 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:09.546443 28328 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:09.533021 30670 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:19:09.549834 27702 connection.cc:256] RPC call timeout handler was delayed by -1.0618649779999032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17920000000000003s for a response
I20260709 12:19:09.550494 30807 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.550395 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.550987 30784 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.545392 30823 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:19:09.552305 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:09.535228 30461 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:09.553530 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:09.543337 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:09.558336 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:09.558624 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:09.555408 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.559427 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:09.560573 30651 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:09.563035 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:09.560972 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1812) took 2620 ms (client timeout 1679 ms). Trace:
I20260709 12:19:09.563899 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:09.561321 30674 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:19:09.563841 27734 rpcz_store.cc:269] 0709 12:19:06.940595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.560767 (+2620172us) service_pool.cc:211] Skipping call since client already timed out
0709 12:19:09.560903 (+   136us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:19:09.564774 27736 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:09.565092 30651 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:09.573770 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:09.574402 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.564849 30784 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:09.573802 30651 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:09.575089 30784 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.575616 30784 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:09.575798 30828 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:09.540915 30625 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:19:09.574570 27735 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:09.575381 30651 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:09.575498 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.579700 28330 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:19:09.580286 27730 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:09.540005 30181 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 12:19:09.579241 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.552934 30670 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.536053 30545 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.557619 30461 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.595363 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.564673 30735 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:09.593801 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:09.596706 27734 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:09.597162 27736 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:09.555711 30630 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:19:09.599721 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.601691 27729 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:09.602869 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.603832 27729 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:09.580227 30674 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.589128 30670 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:09.598169 30748 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:19:09.561578 30807 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.609581 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.589574 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.626576 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:09.628011 30856 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:19:09.634878 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.628087 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.631441 30812 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.631516 30815 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:09.636991 27729 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:09.637877 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:09.638638 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.606550 30674 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:09.643851 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:09.644953 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:09.646018 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:09.646459 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88214.7us
I20260709 12:19:09.603677 30461 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:09.649430 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.650296 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:09.648231 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:09.658918 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.660120 27729 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:09.661453 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.663277 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:09.665689 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.666786 27729 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:09.638986 30638 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.672111 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.673596 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.647194 30630 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.684422 27729 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:09.684664 27738 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:09.652256 30815 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:09.685758 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:09.686246 28332 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:09.687987 30859 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:09.692656 27736 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4.
I20260709 12:19:09.688191 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:09.688706 27737 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:19:09.688771 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.652604 30647 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:09.653170 30830 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:09.653033 30806 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:09.698422 27737 pending_rounds.cc:85] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:19:09.694654 27731 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:09.698971 27737 pending_rounds.cc:107] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:19:09.699383 27737 raft_consensus.cc:2889] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:09.665375 30840 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:19:09.703853 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55692.4us
W20260709 12:19:09.709694 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.644610 30812 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.698091 30686 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.715723 30542 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.716435 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 396) took 2670 ms. Trace:
I20260709 12:19:09.667090 30705 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:19:09.716606 30807 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.691361 30638 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.696658 30859 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.719637 30830 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:09.719141 30670 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.696744 30630 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.721696 30847 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.656203 30735 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:19:09.723557 30647 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:09.724056 30545 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.725348 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:09.724069 30806 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:09.733747 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 531) took 2310 ms. Trace:
I20260709 12:19:09.736877 27733 rpcz_store.cc:276] 0709 12:19:07.045736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.832618 (+1786882us) service_pool.cc:224] Handling call
0709 12:19:08.832980 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.835139 (+  2159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.835182 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.837204 (+  2022us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.618957 (+781753us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b180037aec0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:09.673756 (+ 54799us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:09.673817 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.674015 (+   198us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.716128 (+ 42113us) raft_consensus.cc:1685] finished
0709 12:19:09.716153 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.716374 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3392,"raft.queue_time_us":3387,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4273,"spinlock_wait_cycles":10987776,"thread_start_us":53602,"threads_started":1,"wal-append.queue_time_us":852325}
W20260709 12:19:09.738674 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.751885 28334 rpcz_store.cc:276] 0709 12:19:07.423686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.424228 (+   542us) service_pool.cc:224] Handling call
0709 12:19:07.438016 (+ 13788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.569855 (+131839us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:07.569885 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:07.569921 (+    36us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:07.569950 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.733417 (+2163467us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:09.733486 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.733661 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302781696}
I20260709 12:19:09.704265 30840 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:09.757114 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:09.757869 27584 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:09.745602 30865 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.766552 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:09.745296 30859 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:09.772069 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:09.772783 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52529.5us
I20260709 12:19:09.773531 27581 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:09.760433 30840 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:09.763863 30545 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:09.780544 27586 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:09.789597 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:09.756636 30847 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.807188 30675 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.809028 30828 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:09.809417 30816 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index 2 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:09.813184 30670 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.815429 30867 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:19:09.818374 30630 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.819301 30630 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:09.819846 30630 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:09.820865 30674 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.821102 30630 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.823036 30674 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.796110 30534 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.825127 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.829947 28334 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:09.803838 30647 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:09.825937 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.791090 30830 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.771409 30865 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.823110 30630 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:09.842114 27581 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:19:09.824970 30812 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.766505 30815 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.828701 30638 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:09.854620 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65193.2us
I20260709 12:19:09.834369 30534 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:09.854622 30784 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.796311 30806 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:09.872635 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95360.3us
I20260709 12:19:09.883893 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:09.899197 30651 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:09.900813 30534 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.901263 28329 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:09.901441 30534 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:09.901672 30651 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.892962 30812 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:09.907881 30812 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.908447 30812 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:09.916729 30871 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:09.955925 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74367.8us
I20260709 12:19:09.965502 30675 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:19:09.965868 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 266846us
I20260709 12:19:09.967598 30865 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:09.968909 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:09.969666 28329 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:09.970315 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:09.970773 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:09.971477 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:09.970943 27660 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:09.978706 30762 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.979633 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:09.980377 27663 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:09.993053 30757 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:09.995579 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:09.996567 27661 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:09.998711 30784 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:09.985834 30651 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.003849 30651 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.006362 30651 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:09.987411 30762 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:10.012820 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.017609 27660 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:10.019601 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.020565 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:10.021821 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.022760 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:10.025825 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.026710 27660 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:10.027994 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.054725 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.062024 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.062309 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:10.014448 30762 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:10.070429 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.071218 27663 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:10.071973 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:10.075510 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.076164 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.076969 27660 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:10.080668 27584 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:10.081516 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.082899 27660 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:10.083791 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1924) took 2987 ms (client timeout 1633 ms). Trace:
I20260709 12:19:10.084100 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.085438 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.085831 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:10.084338 27585 rpcz_store.cc:269] 0709 12:19:07.096206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.250117 (+153911us) service_pool.cc:224] Handling call
0709 12:19:10.083526 (+2833409us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54000bdbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:10.083695 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3498991872}
I20260709 12:19:10.087311 27663 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:19:10.087306 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:10.093930 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:10.094873 27589 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:10.095927 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:10.090886 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:10.108181 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:10.111660 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.114995 30461 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.116173 30876 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.131435 30876 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.129695 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:10.081614 30647 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:10.134647 27584 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:10.141422 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:10.142056 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 177148us
I20260709 12:19:10.142560 27737 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:10.082376 30668 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:10.143035 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:10.081550 30816 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:10.080547 30651 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:10.116271 30807 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:10.078859 30812 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.154819 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:10.159025 28332 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:10.116555 30875 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.161199 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.161326 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:10.161405 30859 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.114269 30830 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:10.161367 30840 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.143445 30461 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.173365 27731 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:10.136082 30686 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:10.143241 30647 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.133136 30638 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:10.173911 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.118543 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.162218 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.147150 30816 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:10.148062 30874 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:19:10.149555 30651 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.178375 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.158183 30807 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.155305 30668 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:10.183243 27661 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:10.188410 27731 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:19:10.183614 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:10.187584 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.174397 30830 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.182709 30686 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.145032 30762 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.209959 30830 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.187175 30647 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.186648 30748 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:10.220208 30748 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:10.220768 30812 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:10.221556 30748 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.186363 30747 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:10.181553 30884 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:10.203281 27738 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:10.201658 27589 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:10.198312 30651 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.215674 27660 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:10.201253 30807 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.214478 30686 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.226827 30762 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.189215 30638 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.177047 30875 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.133126 30635 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
W20260709 12:19:10.275529 28292 connection.cc:256] RPC call timeout handler was delayed by -1.1269120130000374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23420000000000002s for a response
I20260709 12:19:10.276139 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:10.200881 30880 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 12:19:10.281279 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:10.282534 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:10.293128 27738 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:10.285373 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 123352us
I20260709 12:19:10.288446 27663 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[5.2-9.4]   Dedup: 5.2->[7.3-9.4]
I20260709 12:19:10.294111 27663 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [5.2-9.4]
I20260709 12:19:10.294202 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:10.294499 27589 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.248828 30747 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.296710 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:10.302179 27660 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:10.322129 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.328884 30866 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.329484 30866 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.332716 30866 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.332789 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:10.333645 28332 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:10.333817 30898 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:10.334576 30898 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:10.286749 30876 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:10.342166 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 223713us
I20260709 12:19:10.280746 30884 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 12:19:10.248333 30816 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.344264 30647 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.353660 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.341892 27589 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:10.354524 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.356338 30747 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.356494 30646 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:10.357985 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:10.360286 27589 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.371832 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.372712 27729 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:10.373498 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:10.374387 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:10.359244 30878 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
I20260709 12:19:10.293709 30668 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.377981 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.378156 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:10.378834 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.343482 30892 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:10.375528 30747 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:10.401922 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:10.368168 30891 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:10.396347 27737 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:10.399387 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:10.405766 28328 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:10.399958 30870 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:10.406472 30870 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:10.406615 30888 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:19:10.406913 30870 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:10.408434 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.409210 27662 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:10.409704 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.410324 27664 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.411657 30793 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:19:10.411828 30885 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:19:10.412060 30793 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:10.412303 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:10.412890 30793 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.414222 30751 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:10.416172 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.417029 27659 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:10.417896 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:10.421185 30890 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:19:10.380699 30727 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:10.423619 30727 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:10.423919 30889 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:19:10.426092 30638 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:10.426415 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.427048 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:10.427340 30875 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:10.428009 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.428545 27655 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:10.384073 30869 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:10.402799 30871 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:10.430178 30870 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.416165 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:10.434635 30883 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
I20260709 12:19:10.380234 30859 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:10.400589 30881 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260709 12:19:10.439724 30892 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:10.367831 30646 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.404510 30747 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.441075 30747 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:10.444172 30870 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.444556 30738 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.451403 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:10.405200 30891 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.452188 30891 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:10.452602 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:10.453222 27587 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:10.403929 27732 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:10.453838 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.400321 30840 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.424008 30727 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:10.429975 30869 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:10.440194 30892 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.389590 27589 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:10.467175 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 170501us
W20260709 12:19:10.473351 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.366488 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.617363 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 1938) took 2974 ms. Trace:
I20260709 12:19:10.760933 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:10.770725 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:10.612913 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.781440 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:10.765641 27586 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:10.783406 28328 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:10.652146 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.538498 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:10.474449 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:10.711063 27664 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
W20260709 12:19:10.782104 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.730898 27738 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
W20260709 12:19:10.435225 30630 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.556s	user 0.008s	sys 0.000s
I20260709 12:19:10.756045 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.588903 27656 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:10.631165 27663 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:19:10.781947 27583 rpcz_store.cc:276] 0709 12:19:07.642468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.642579 (+   111us) service_pool.cc:224] Handling call
0709 12:19:10.617071 (+2974492us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:10.617270 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2517492992}
I20260709 12:19:10.801901 27655 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:10.802649 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.792585 27587 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:19:10.790150 30738 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.796911 30674 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:10.589478 30807 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.806217 30738 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:19:10.534416 30686 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.789340 30892 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:10.808430 30738 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:10.805655 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45408.8us
W20260709 12:19:10.794857 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 428441us
I20260709 12:19:10.794775 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:10.770752 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 352677us
I20260709 12:19:10.791723 27737 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:19:10.584461 30675 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:10.517227 30859 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:10.759759 30827 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:19:10.503764 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.812309 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.767732 30647 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.759509 30847 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.589730 30876 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:10.808741 30686 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.877871 30807 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.822463 27664 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:10.818403 27662 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.520581 30638 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:10.789193 30646 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.764276 30399 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:10.880802 30399 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.800825 30731 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:10.871949 30738 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:10.870862 30876 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:10.883764 30731 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.885495 30731 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:10.887564 30740 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.884675 30876 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.888458 30876 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:10.888655 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:10.889266 28332 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.794296 30884 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:10.873716 30869 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.875444 30893 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:10.838455 30859 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.794634 30668 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:10.799341 30830 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.583s	user 0.015s	sys 0.002s
I20260709 12:19:10.818619 30674 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:10.825893 30651 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.001s
I20260709 12:19:10.899659 27662 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:10.697984 30747 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.826124 30675 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:10.797086 30896 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
W20260709 12:19:10.905627 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 85683.8us
I20260709 12:19:10.883285 30399 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.835498 30827 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:10.916926 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:10.919478 27737 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:10.893587 30901 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:10.904613 30830 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.902637 28332 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:10.920442 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:10.925035 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:10.923643 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85143us
I20260709 12:19:10.928499 27729 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:10.931160 27737 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:10.904505 30738 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:10.934612 30670 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:10.935079 30670 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:10.935454 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.935521 30670 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.933947 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.934041 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.936689 27663 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:19:10.935164 30738 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.819280 30903 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.117s
I20260709 12:19:10.941699 27738 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:10.945022 27663 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:10.900627 30638 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:10.939805 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70604.3us
I20260709 12:19:10.951516 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.955915 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.953490 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.956664 27663 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:19:10.956982 27656 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:10.947022 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:10.960309 27627 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.633s (SENT)
I20260709 12:19:10.961436 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:10.962210 27732 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:10.968361 27655 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:10.968830 30675 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.969997 30675 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:10.970253 30892 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.908349 30651 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:10.972735 30747 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.973806 30651 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:10.974772 27731 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:19:10.978641 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:10.979799 27588 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:19:10.980320 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 1943) took 1507 ms. Trace:
I20260709 12:19:10.980701 27588 rpcz_store.cc:276] 0709 12:19:09.473363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.493009 (+ 19646us) service_pool.cc:224] Handling call
0709 12:19:10.978437 (+1485428us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:19:10.980251 (+  1814us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18783616}
I20260709 12:19:10.986200 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:10.904788 30668 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:10.989295 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:10.984777 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:10.993499 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:10.997953 30747 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:19:10.997452 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72429us
I20260709 12:19:10.994647 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:10.995631 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:11.004163 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.998566 30747 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:11.006395 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 408) took 2039 ms. Trace:
W20260709 12:19:11.007056 30891 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.554s	user 0.003s	sys 0.007s
I20260709 12:19:11.006847 30747 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.006779 27657 rpcz_store.cc:276] 0709 12:19:08.966691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.966868 (+   177us) service_pool.cc:224] Handling call
0709 12:19:08.989315 (+ 22447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.016595 (+ 27280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:09.016630 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:09.016676 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:09.016709 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.006168 (+1989459us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b5400041354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:11.006214 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.006336 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62191488}
I20260709 12:19:11.007601 30891 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.009219 30727 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.009759 30891 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.009653 30461 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:10.996343 30896 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:11.012465 30668 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:10.996920 30651 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 12:19:11.013661 27659 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 7. Ops: []
I20260709 12:19:10.944235 30784 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:11.017159 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:11.017549 27655 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:11.017810 28332 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:11.020206 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.020246 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.021485 27656 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.023252 27658 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:11.025552 27663 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:11.026360 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.031158 28332 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
W20260709 12:19:11.044001 27552 connection.cc:256] RPC call timeout handler was delayed by -3.5541572790000373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
I20260709 12:19:11.054041 27663 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:11.088845 27582 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:11.065016 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.074229 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.091627 27656 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:10.996610 30674 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.010368 30461 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:11.093276 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.016470 30727 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:11.074474 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.100001 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.101759 27658 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.102353 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:11.103945 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.104913 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.092339 30651 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:11.084828 30859 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.087074 30638 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.106500 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.110543 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.157275 27662 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.081674 30896 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:11.154477 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:11.112569 27552 connection.cc:256] RPC call timeout handler was delayed by -2.4382954969998991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 12:19:11.172107 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:11.160702 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64932.4us
I20260709 12:19:11.112061 30776 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:19:11.112501 30762 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.831s	user 0.010s	sys 0.002s
I20260709 12:19:11.175184 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:11.174292 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3311954969999533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
I20260709 12:19:11.097182 30521 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
W20260709 12:19:11.209318 27552 connection.cc:256] RPC call timeout handler was delayed by -2.183595496999942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18320000000000003s for a response
W20260709 12:19:11.207868 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:11.210306 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.106290 30830 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:11.207742 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:11.210904 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0818954969997776s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
I20260709 12:19:11.211444 30918 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:11.244498 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3519954969999617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2756s for a response
W20260709 12:19:11.245471 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3015954969998802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2812s for a response
I20260709 12:19:11.154562 30871 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.246177 28332 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:11.246240 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:11.247625 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:11.249061 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:11.156360 30884 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.107000 30901 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:11.246250 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.154659 30917 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.250458 28328 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:11.252614 27658 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
W20260709 12:19:11.249457 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.160154 30806 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:11.191042 27662 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:11.263684 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:11.262221 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.160421 30727 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:11.257851 28332 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:11.267200 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:11.201895 30892 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:11.268074 27627 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.634s (SENT)
I20260709 12:19:11.264698 27658 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.268648 30402 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:11.275718 30921 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:11.276150 30402 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:11.277235 30402 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.275738 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:11.280661 30638 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:11.270711 30727 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.282862 30727 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.258971 30461 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.283737 30461 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.283545 30896 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.285826 30766 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:11.262459 30884 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.272841 30534 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:11.184064 30913 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.100s
I20260709 12:19:11.276185 30921 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.260480 30762 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:11.260313 30871 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.289208 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.314930 27662 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.290512 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.292827 27658 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:11.230422 30776 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:11.300734 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.265926 30912 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:19:11.320693 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.325374 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.356622 27658 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:19:11.295013 30534 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:11.372433 30534 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.373733 30534 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:11.357402 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.376924 27658 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:11.317541 30762 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.378472 30762 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.325596 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.325847 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.354384 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.259312 30918 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.380543 27737 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:11.381505 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.382277 27737 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:11.382493 30918 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.287906 30806 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.325742 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.385120 30806 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.363426 30901 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.354156 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.386373 27732 raft_consensus.cc:2410] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:19:11.359373 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1930) took 4700 ms. Trace:
I20260709 12:19:11.364634 30884 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.365800 27662 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:11.387380 27580 rpcz_store.cc:276] 0709 12:19:06.654252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.659776 (+  5524us) service_pool.cc:224] Handling call
0709 12:19:06.660131 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.177680 (+2517549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.326162 (+2148482us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b540020cbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:11.354562 (+ 28400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5713533824}
I20260709 12:19:11.353915 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.365962 30776 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.266996 30917 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.379720 27733 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:11.380411 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:11.326565 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.385242 27730 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.390535 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:11.365252 30897 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.036s	user 0.002s	sys 0.000s
W20260709 12:19:11.324973 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 399060us
I20260709 12:19:11.369287 30731 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.383219 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.297302 30892 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:11.408936 27734 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:11.410773 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.420147 27733 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:11.411775 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:11.411871 30772 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:11.414333 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.428156 28329 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:11.428669 30871 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.414979 30806 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:19:11.420300 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.430917 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:11.312549 30921 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.421008 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.435685 27731 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:11.315644 30668 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.413165 30521 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:11.357077 30670 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.403560 30912 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.437052 30892 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:11.439390 30901 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:11.438932 30668 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.443632 27730 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:11.448823 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.459208 27737 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
W20260709 12:19:11.450042 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 201037us
I20260709 12:19:11.461320 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:11.461373 27731 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:19:11.440877 30923 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:11.464634 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:11.450829 30912 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.467597 30871 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:11.452713 30901 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:11.476337 30670 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.479957 30896 raft_consensus.cc:2764] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260709 12:19:11.481051 30918 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:11.481837 28329 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:11.481612 30891 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:11.482338 30876 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:11.483126 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:11.488431 30917 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.489840 30827 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.490392 30668 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:11.491535 30892 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.486984 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 413) took 1148 ms. Trace:
I20260709 12:19:11.510963 27589 rpcz_store.cc:276] 0709 12:19:10.338718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.799635 (+460917us) service_pool.cc:224] Handling call
0709 12:19:10.800036 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.892812 (+ 92776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:10.892845 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:10.892889 (+    44us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:10.892934 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.486584 (+593650us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b54002b98d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:11.486660 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.486888 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":212903168}
I20260709 12:19:11.462779 30923 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:11.503459 27657 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:11.502276 30901 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.514806 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:11.483222 30815 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:19:11.510802 30731 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:11.546650 30918 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:11.550429 30827 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:11.560999 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46268.8us
I20260709 12:19:11.562935 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:11.552860 30815 consensus_queue.cc:1243] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:11.565126 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.568626 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:11.569875 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.565462 30918 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:11.571209 30916 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.306s	user 0.001s	sys 0.000s
I20260709 12:19:11.571846 30884 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:11.571508 30918 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.572242 30884 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:11.572623 30884 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:11.575044 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.576327 27589 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:11.576354 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:11.563306 30917 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:11.598392 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.599248 27662 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:11.599967 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:11.602896 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.604379 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.604645 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:11.605283 30812 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:11.605638 30402 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:11.606259 30776 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:11.606652 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.606805 27662 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:11.606977 30776 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:11.580343 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.567786 30815 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:19:11.581785 30877 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.403s	user 0.005s	sys 0.008s
I20260709 12:19:11.608076 30776 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.608753 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.582865 30816 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:11.609993 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.573979 30923 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.585577 30838 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.586467 30675 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:11.587471 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.617240 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:11.589154 30709 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.590453 27583 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:11.592535 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.619379 27731 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.592792 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.620750 27737 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:11.593273 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.593462 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.562081 30896 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:19:11.596390 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.623041 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.576565 30880 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:11.627153 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.631915 27702 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.632887 30668 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:11.633399 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.634243 30668 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:11.601743 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.605522 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.638517 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:11.639614 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.573849 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:11.575923 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.641227 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.642020 27581 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:11.621999 30932 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:19:11.615759 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.647356 27663 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:19:11.647585 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.647969 27589 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:11.621539 27730 raft_consensus.cc:2368] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:19:11.621996 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.624141 27737 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:11.624408 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.607812 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.649430 27662 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.650507 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1851) took 1487 ms. Trace:
I20260709 12:19:11.651791 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.652544 28329 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:11.650853 27736 rpcz_store.cc:276] 0709 12:19:10.163115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.163287 (+   172us) service_pool.cc:224] Handling call
0709 12:19:10.163758 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:10.175102 (+ 11344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:10.175148 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:10.176197 (+  1049us) log.cc:844] Serialized 26 byte log entry
0709 12:19:10.318928 (+142731us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:11.594820 (+1275892us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b18000ffa60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:19:11.594980 (+   160us) log.cc:844] Serialized 14 byte log entry
0709 12:19:11.650129 (+ 55149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.650208 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:11.650247 (+    39us) raft_consensus.cc:1685] finished
0709 12:19:11.650336 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.650445 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":477288,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":10904,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":9705,"spinlock_wait_cycles":6024064,"thread_start_us":213752,"threads_started":3,"wal-append.queue_time_us":195018,"wal-append.run_cpu_time_us":1561,"wal-append.run_wall_time_us":1041022}
I20260709 12:19:11.654130 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:11.654835 28329 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:19:11.640041 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:11.657311 27662 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:11.659298 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:11.660838 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:11.659298 27662 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 12:19:11.661978 27663 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:11.662766 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.663514 27589 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:11.665844 27589 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:11.605669 30812 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:11.667436 27582 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:11.669611 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:11.670243 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:11.670758 27582 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.667649 27663 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 9. Ops: [7.4-7.4]
W20260709 12:19:11.667982 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94196.4us
I20260709 12:19:11.622378 30675 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:11.636188 30880 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:11.574263 30921 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.668426 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:11.678459 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.681015 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:11.683437 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.641000 30709 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:11.685441 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:11.694983 28328 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:11.696038 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:11.635263 30766 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:19:11.660101 30402 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:11.678114 30896 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:11.743800 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:11.744607 27662 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:11.744812 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:11.745481 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.660830 30838 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.691730 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.693106 27702 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:19:11.684181 30880 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.689357 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:11.753486 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61874.8us
W20260709 12:19:11.754079 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.754319 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.677935 30638 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:11.677497 30812 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:11.715035 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.741413 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.762157 27730 raft_consensus.cc:2376] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 4. Re-sending same reply.
I20260709 12:19:11.746965 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:11.700132 30149 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.750375 27702 leader_election.cc:403] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
W20260709 12:19:11.766703 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 76966us
I20260709 12:19:11.766535 30149 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.693009 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:11.767690 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74761.1us
I20260709 12:19:11.768371 30647 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:11.768865 30149 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:11.769901 30923 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:11.689771 30668 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.690557 30806 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:11.770882 30806 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.772938 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:11.690289 30900 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:11.690287 30686 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.774293 30149 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:11.683241 30912 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.774730 30686 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.775570 27587 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:11.776764 30686 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.772954 30806 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:11.777948 30912 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.778597 30912 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:11.684463 30892 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260709 12:19:11.785598 30647 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:19:11.765734 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.690436 30847 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.786477 30847 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.787039 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.786707 30675 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.788517 30675 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.789237 30847 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:11.773599 30668 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.794657 30668 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:11.785213 30762 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.797442 30762 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:11.804093 30933 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:19:11.749764 30402 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:11.815846 30402 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:11.740383 30542 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:19:11.821267 30542 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:11.821800 30542 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:11.823669 30638 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:11.825011 30638 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.831784 30823 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:11.832417 30823 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:11.833649 30823 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.839396 30751 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.840126 30751 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:11.846251 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.680012 30674 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:19:11.850059 30921 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.851955 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.853241 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.853777 27733 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:11.856397 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.764743 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:11.859357 27662 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:11.859258 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 190948us
W20260709 12:19:11.867642 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.758513 30709 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:11.872128 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:11.877888 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 210010us
I20260709 12:19:11.764060 30917 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:11.881505 27731 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:11.884809 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.886341 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.898978 30876 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:11.899600 30876 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:11.900532 30876 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:11.905627 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:11.930356 30934 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:19:11.921474 30823 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:11.936264 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:11.885082 30917 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:11.937523 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.937950 30917 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:11.938259 27734 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:11.939106 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:11.940019 27734 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:11.822924 30542 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:11.947887 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.948690 30542 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:11.843092 30751 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:11.978935 30896 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:12.004776 30896 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.012189 30923 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:12.053858 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:12.056883 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.058561 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:12.059931 27583 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:12.053856 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.061373 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:12.064468 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 200690us
W20260709 12:19:12.068003 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.069943 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:12.070567 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.073697 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:11.907196 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:12.076406 27737 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:12.077239 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:12.071142 27662 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:12.063256 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:12.104426 27582 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:12.106069 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.107678 30869 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.107836 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.108932 30869 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.109928 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.111840 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.112331 30942 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:12.112603 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:12.113065 27582 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:12.113824 27662 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:12.115408 30880 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.116871 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:12.118008 27582 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:12.121882 30884 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.000s
I20260709 12:19:12.123479 30884 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.127122 30884 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:12.111840 30940 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:19:12.146118 30871 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.644s	user 0.003s	sys 0.009s
I20260709 12:19:12.146822 30871 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:12.148590 27582 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:12.149741 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.150911 27582 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:19:12.151620 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:12.154727 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56031.5us
I20260709 12:19:12.155737 30870 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:12.156402 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:12.152292 30923 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.157486 30940 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:12.157981 30940 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:12.158473 30940 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:12.107610 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:12.161940 27583 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:12.083740 30868 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
W20260709 12:19:12.164032 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87414.7us
I20260709 12:19:12.166574 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:12.167647 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:12.166718 28328 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:19:12.168548 27737 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:19:12.170531 27737 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:12.171957 30651 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.172991 30651 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:12.173775 30651 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.119038 30880 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:12.155303 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52522.9us
I20260709 12:19:12.184289 30898 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.184631 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.185381 30898 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.187407 27734 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:12.192148 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:12.193365 27734 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
W20260709 12:19:12.200018 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 445635us
I20260709 12:19:12.203311 27734 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:12.208563 27583 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:12.201468 27552 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7.
I20260709 12:19:12.126849 30917 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.223582 27581 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:12.223883 27583 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:12.225983 27583 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:12.226472 27583 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:12.227680 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.227893 27583 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.228578 27587 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:12.229626 27583 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:12.230827 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.231794 30901 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.684s	user 0.021s	sys 0.000s
W20260709 12:19:12.234172 30816 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.625s	user 0.010s	sys 0.005s
I20260709 12:19:12.239866 30651 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:12.240607 27736 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:12.241160 28328 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:19:12.241245 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:12.242503 30869 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:12.243052 27582 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:12.243433 30802 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.244973 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:12.245530 30898 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:12.245935 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:12.246901 27737 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:12.247957 28330 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:19:12.216878 30868 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:19:12.251717 30542 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:12.257396 30827 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.693s	user 0.008s	sys 0.007s
I20260709 12:19:12.259263 30812 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:12.260875 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59728us
W20260709 12:19:12.269213 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.271840 30534 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:12.277486 27582 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:12.287352 27587 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:12.278664 30793 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:12.285982 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 118499us
I20260709 12:19:12.286767 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:12.253382 30402 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.278143 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:12.305668 28334 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
W20260709 12:19:12.310509 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78815.8us
I20260709 12:19:12.311137 30880 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:12.312107 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.311877 30880 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:12.313774 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 153184us
I20260709 12:19:12.314220 30880 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:12.317221 30402 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:12.318368 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.319459 30917 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:12.320061 30828 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.320353 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:12.307598 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:12.291548 30534 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:12.270701 30901 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.338933 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:12.339546 30638 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20260709 12:19:12.343672 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.346659 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:12.347100 28329 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:19:12.351575 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.342715 30940 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.333215 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:12.364382 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:12.378422 27658 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:12.379810 27731 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:19:12.328012 30542 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.381278 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:12.381325 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55568us
I20260709 12:19:12.382643 30940 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:12.383632 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74834.2us
I20260709 12:19:12.374264 30880 raft_consensus.cc:2764] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:19:12.385780 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 424) took 1096 ms. Trace:
I20260709 12:19:12.387406 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1633) took 1286 ms. Trace:
I20260709 12:19:12.389436 27580 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:19:12.393287 30876 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:12.396102 30534 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.356508 30917 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
W20260709 12:19:12.405272 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.418318 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57969.4us
W20260709 12:19:12.419914 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101687us
W20260709 12:19:12.422821 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76316.6us
I20260709 12:19:12.426069 30542 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:12.426668 28328 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:12.427389 27661 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:12.429737 30534 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:12.431449 30827 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:12.431874 30827 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:12.432231 30827 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:12.435469 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 411) took 2132 ms. Trace:
I20260709 12:19:12.435911 27659 rpcz_store.cc:276] 0709 12:19:10.256460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.479198 (+222738us) service_pool.cc:224] Handling call
0709 12:19:10.756583 (+277385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.014173 (+257590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.388093 (+1373920us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:12.388395 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":590796160}
I20260709 12:19:12.438026 27659 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:12.438561 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1984) took 1062 ms. Trace:
I20260709 12:19:12.417563 30812 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:12.439165 27659 rpcz_store.cc:276] 0709 12:19:11.376247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.437115 (+1060868us) service_pool.cc:224] Handling call
0709 12:19:12.437449 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.438336 (+   887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.438514 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.441813 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.442847 30892 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:12.443480 27552 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:12.443797 27552 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:12.444511 30892 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:19:12.405841 27588 rpcz_store.cc:276] 0709 12:19:11.289272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.290310 (+  1038us) service_pool.cc:224] Handling call
0709 12:19:12.385317 (+1095007us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.385564 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":782827264}
I20260709 12:19:12.406370 27657 rpcz_store.cc:276] 0709 12:19:11.101104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.173571 (+ 72467us) service_pool.cc:224] Handling call
0709 12:19:12.387052 (+1213481us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.387211 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649521920}
I20260709 12:19:11.374802 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:19:12.446707 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 413) took 2189 ms (client timeout 1673 ms). Trace:
W20260709 12:19:12.490850 27660 rpcz_store.cc:269] 0709 12:19:10.257058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.321897 (+ 64839us) service_pool.cc:224] Handling call
0709 12:19:12.446477 (+2124580us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.446624 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4472669056}
I20260709 12:19:12.453912 30912 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:12.493140 30912 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.456786 27657 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260709 12:19:12.469399 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.495298 30912 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:12.470840 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.472139 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:12.472414 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:12.445747 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.486732 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1975) took 1465 ms. Trace:
I20260709 12:19:12.478287 30947 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.456749 30647 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:19:12.481568 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:12.467645 30738 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:19:12.456461 30949 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.113s	user 0.003s	sys 0.000s
I20260709 12:19:12.451820 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:12.471628 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.463050 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.466934 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 406) took 2088 ms. Trace:
I20260709 12:19:12.508337 27658 rpcz_store.cc:276] 0709 12:19:11.021666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.615868 (+594202us) service_pool.cc:224] Handling call
0709 12:19:11.616227 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.379863 (+763636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.486420 (+106557us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:12.486638 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1622092160}
I20260709 12:19:12.545954 27729 rpcz_store.cc:276] 0709 12:19:10.378389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.378627 (+   238us) service_pool.cc:224] Handling call
0709 12:19:12.466535 (+2087908us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400316d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.466756 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1219329408}
I20260709 12:19:12.553069 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.499750 30876 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.554966 30876 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.557385 30900 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:12.557881 27627 leader_election.cc:400] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:19:12.558261 27627 leader_election.cc:403] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:19:12.469655 30815 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:12.504335 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 424) took 1067 ms. Trace:
I20260709 12:19:12.467152 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1622) took 1478 ms. Trace:
I20260709 12:19:12.474447 27663 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
W20260709 12:19:12.510851 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.488915 27655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1624) took 1495 ms (client timeout 1865 ms). Trace:
I20260709 12:19:12.464257 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1667) took 1164 ms. Trace:
I20260709 12:19:12.487179 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 423) took 1050 ms. Trace:
W20260709 12:19:12.445756 30757 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:19:12.559798 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:12.627648 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.455204 30901 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:12.651659 30668 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:19:12.651870 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 206225us
I20260709 12:19:12.653504 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:12.657258 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.660244 30635 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:12.558760 30912 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:12.664069 30942 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:19:12.557346 30876 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:12.669253 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.627705 27657 rpcz_store.cc:276] 0709 12:19:11.437142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.456085 (+1018943us) service_pool.cc:224] Handling call
0709 12:19:12.456458 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.500030 (+ 43572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.504276 (+  4246us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:12.636123 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78428.4us
I20260709 12:19:12.636961 30827 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.639827 27730 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:12.641635 27658 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 4. Current term is 5. Ops: []
W20260709 12:19:12.646139 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.647810 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184910us
I20260709 12:19:12.648067 30900 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:12.649377 30947 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.460299 30751 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.191s
I20260709 12:19:12.707829 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:12.708319 27585 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:12.490167 30830 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.161s
I20260709 12:19:12.653594 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:12.628548 27661 rpcz_store.cc:276] 0709 12:19:10.988869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.989040 (+   171us) service_pool.cc:224] Handling call
0709 12:19:12.466910 (+1477870us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.467065 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3085413760}
W20260709 12:19:12.626796 27655 rpcz_store.cc:269] 0709 12:19:10.993121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.993265 (+   144us) service_pool.cc:224] Handling call
0709 12:19:12.488576 (+1495311us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.488726 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46026112}
I20260709 12:19:12.712212 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1637) took 1032 ms. Trace:
I20260709 12:19:12.684065 27581 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:12.685631 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.712764 27658 rpcz_store.cc:276] 0709 12:19:11.679761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.546361 (+866600us) service_pool.cc:224] Handling call
0709 12:19:12.641390 (+ 95029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.711862 (+ 70472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.712139 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.719306 27580 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:12.681107 30901 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.749150 30891 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:12.627643 27659 rpcz_store.cc:276] 0709 12:19:11.436776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.486093 (+1049317us) service_pool.cc:224] Handling call
0709 12:19:12.486580 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.486840 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.486874 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.486914 (+    40us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:12.486950 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.487008 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.487104 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.626502 30815 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:12.706315 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:12.724498 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.627760 27732 rpcz_store.cc:276] 0709 12:19:11.300236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.300389 (+   153us) service_pool.cc:224] Handling call
0709 12:19:12.463788 (+1163399us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.464067 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":154952448}
I20260709 12:19:12.708525 27663 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:12.747140 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:12.753338 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49102.2us
I20260709 12:19:12.754824 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.755630 30776 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.756825 27663 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:12.757284 30635 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.760897 30830 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:12.761709 30830 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.764928 30830 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:12.760066 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:12.767129 30635 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:12.767807 30635 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:12.769109 30635 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.711100 30947 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:12.798614 30647 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.799713 30827 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:12.806748 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 335188us
W20260709 12:19:12.807936 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 122) took 19757 ms (client timeout 15000 ms). Trace:
I20260709 12:19:12.808341 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:12.810331 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:12.810518 27664 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
W20260709 12:19:12.815933 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 343974us
I20260709 12:19:12.817072 30869 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:12.818003 30869 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:12.799356 30647 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.819943 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:12.782727 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.828053 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:12.830250 30776 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:19:12.831879 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49561.4us
I20260709 12:19:12.835613 30776 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.837575 30776 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.840081 30776 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:12.862979 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 177530us
I20260709 12:19:12.789780 30534 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:12.854758 27588 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:12.886798 27583 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:19:12.858656 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.886955 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:12.888758 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:12.892576 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82304.4us
I20260709 12:19:12.896548 27588 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:12.850594 30815 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.906123 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:12.912672 30766 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.912794 28335 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:12.913306 30766 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:12.902393 30830 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:19:12.919939 30891 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:12.920435 30891 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:12.922781 30181 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:12.923750 30181 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:12.924145 30181 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:12.942543 27581 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:12.942974 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51748.3us
I20260709 12:19:12.943900 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:12.944562 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1682) took 1260 ms (client timeout 1605 ms). Trace:
W20260709 12:19:12.944980 27730 rpcz_store.cc:269] 0709 12:19:11.684045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.786766 (+102721us) service_pool.cc:224] Handling call
0709 12:19:12.944367 (+1157601us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b54001d10d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:12.944482 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1514725120}
I20260709 12:19:12.948442 30647 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:12.949287 30912 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:12.950289 30912 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:12.950640 30912 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:12.950089 27485 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:12.951532 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.952167 27586 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:12.952514 30912 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:12.956269 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:12.964184 30830 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:12.966691 30900 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:12.967154 30900 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:12.967549 30900 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:12.973762 30912 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:12.975435 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.977172 27587 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:12.978224 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:12.979882 27581 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:12.983747 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:12.991204 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:12.939733 30892 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:12.993543 28335 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20260709 12:19:12.993942 27656 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.036s	sys 0.068s
W20260709 12:19:12.994423 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.995509 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48247.8us
W20260709 12:19:12.996106 27656 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.036s	sys 0.068s
I20260709 12:19:12.997756 27588 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:19:12.999926 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.986409 30815 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:12.998200 30876 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.006763 28329 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:19:12.985167 30766 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:13.001329 27485 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 6, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 6 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:12.993448 30873 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.000985 30751 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:19:13.027379 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:13.004739 30901 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:13.033445 30901 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.033934 30901 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:13.034387 30876 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.032572 30892 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:19:13.037192 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194830us
W20260709 12:19:12.808493 27487 rpcz_store.cc:269] 0709 12:18:52.989471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.989770 (+   299us) service_pool.cc:224] Handling call
0709 12:19:12.720432 (+19730662us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:07.383965 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:07.385049 (+  1084us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:07.385121 (+    72us) write_op.cc:435] Acquired schema lock
0709 12:19:07.385148 (+    27us) tablet.cc:662] Decoding operations
0709 12:19:07.386873 (+  1725us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:07.386983 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:07.387029 (+    46us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:19:07.610655 (+223626us) tablet.cc:701] Row locks acquired
0709 12:19:07.610708 (+    53us) write_op.cc:260] PREPARE: finished
0709 12:19:07.627619 (+ 16911us) write_op.cc:270] Start()
0709 12:19:07.659788 (+ 32169us) write_op.cc:276] Timestamp: P: 1783599547622326 usec, L: 0
0709 12:19:07.662047 (+  2259us) op_driver.cc:348] REPLICATION: starting
0709 12:19:08.286847 (+624800us) log.cc:844] Serialized 5470 byte log entry
0709 12:19:09.273311 (+986464us) op_driver.cc:464] REPLICATION: finished
0709 12:19:09.560613 (+287302us) write_op.cc:301] APPLY: starting
0709 12:19:09.560758 (+   145us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:09.561189 (+   431us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:09.561341 (+   152us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:09.690174 (+128833us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:09.690540 (+   366us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 12:19:09.690583 (+    43us) write_op.cc:312] APPLY: finished
0709 12:19:09.712345 (+ 21762us) log.cc:844] Serialized 101 byte log entry
0709 12:19:09.712570 (+   225us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:09.712741 (+   171us) write_op.cc:454] Released schema lock
0709 12:19:09.712856 (+   115us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:09.713333 (+   477us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:09.713364 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6677033,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":286861,"apply.run_cpu_time_us":6116,"apply.run_wall_time_us":152767,"num_ops":14,"prepare.queue_time_us":435287,"prepare.run_cpu_time_us":69321,"prepare.run_wall_time_us":1405990,"raft.queue_time_us":137176,"raft.run_cpu_time_us":32401,"raft.run_wall_time_us":348770,"replication_time_us":1595029,"spinlock_wait_cycles":6528,"thread_start_us":862903,"threads_started":4,"wal-append.queue_time_us":508538,"wal-append.run_cpu_time_us":30534,"wal-append.run_wall_time_us":1809363}]]}
I20260709 12:19:13.007874 30757 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:13.043805 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:13.055765 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 175552us
I20260709 12:19:13.055064 27583 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:13.066978 30947 raft_consensus.cc:2764] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:19:13.070861 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1955) took 2823 ms. Trace:
I20260709 12:19:13.040009 30876 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.046301 30757 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.065265 27485 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:13.083429 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:13.083778 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:13.081427 27662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.004s	sys 0.018s
I20260709 12:19:13.086468 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:13.087165 27662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.005s	sys 0.018s
I20260709 12:19:13.071372 27584 rpcz_store.cc:276] 0709 12:19:10.246868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.772826 (+525958us) service_pool.cc:224] Handling call
0709 12:19:10.773197 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:10.773644 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:10.773690 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:10.774334 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:19:12.355904 (+1581570us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b180039b620. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:12.748692 (+392788us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:19:12.748749 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.749952 (+  1203us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.070373 (+320421us) raft_consensus.cc:1685] finished
0709 12:19:13.070397 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.070660 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":8176,"spinlock_wait_cycles":33653504,"thread_start_us":358128,"threads_started":1,"wal-append.queue_time_us":2293780}
I20260709 12:19:13.031075 30942 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:19:13.088359 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:13.077943 30923 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.016s	sys 0.012s
I20260709 12:19:13.079711 30757 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:13.089643 27485 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:13.106081 28330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.013s	sys 0.025s
W20260709 12:19:13.113691 28330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.013s	sys 0.025s
W20260709 12:19:13.103685 30923 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.016s	sys 0.012s
I20260709 12:19:13.115012 27485 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:13.071745 30873 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.082237 30812 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:19:13.112771 30647 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:13.143496 30638 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:13.144177 30638 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:13.145430 30638 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:13.154027 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91112.6us
W20260709 12:19:13.157346 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78007us
I20260709 12:19:13.171038 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:13.133389 30942 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:19:13.184043 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:13.185144 30892 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:13.185323 30873 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:13.185640 30892 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:13.186133 30892 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:13.172374 30869 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:13.188342 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:13.188563 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:13.189496 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:13.196381 27589 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.006s	sys 0.021s
W20260709 12:19:13.196866 27589 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.007s	sys 0.021s
I20260709 12:19:13.197960 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 1987) took 1154 ms. Trace:
W20260709 12:19:13.199700 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 122620us
I20260709 12:19:13.204795 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.205456 27733 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:13.204672 27485 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 4, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:13.207476 30534 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:13.210012 30891 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.176514 30638 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:13.210361 27485 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 3 to 4, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:13.212352 30181 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.214618 30181 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.213817 27485 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 4 to 6, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 6 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:13.175346 30900 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.216310 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.216931 27730 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:13.217902 30900 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:13.222863 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.009s	sys 0.074s
W20260709 12:19:13.223260 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.009s	sys 0.074s
W20260709 12:19:13.226686 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.198400 27589 rpcz_store.cc:276] 0709 12:19:12.043532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.252132 (+208600us) service_pool.cc:224] Handling call
0709 12:19:12.252634 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.253286 (+   652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.253336 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:12.253970 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:19:12.375568 (+121598us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:13.197288 (+821720us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.197565 (+   277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.197676 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.197731 (+    55us) raft_consensus.cc:1685] finished
0709 12:19:13.197748 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.197894 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":752,"prepare.queue_time_us":42113,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":134,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":205,"thread_start_us":902606,"threads_started":2,"wal-append.queue_time_us":215655}
W20260709 12:19:13.261075 27701 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.020s	sys 0.026s
W20260709 12:19:13.267015 27701 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.021s	sys 0.026s
W20260709 12:19:13.267382 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 761011us
I20260709 12:19:13.267939 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:13.242651 30812 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:13.270082 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 418) took 1073 ms. Trace:
I20260709 12:19:13.264503 27730 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:13.286190 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.287676 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:13.288192 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.288338 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:13.288899 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:13.291725 27664 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:13.288295 27731 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:13.312572 27737 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:19:13.318504 27733 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:13.321998 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.323704 27658 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:19:13.294030 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:13.325742 28337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.008s	sys 0.031s
I20260709 12:19:13.273159 27738 rpcz_store.cc:276] 0709 12:19:12.193406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.243237 (+ 49831us) service_pool.cc:224] Handling call
0709 12:19:12.464418 (+221181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.464877 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:12.465273 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.241187 (+775914us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:13.241257 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:13.241319 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.241420 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.262067 (+ 20647us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54002e95d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:13.266539 (+  4472us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":899,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":17383,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":26684,"spinlock_wait_cycles":418246912,"thread_start_us":758065,"threads_started":1}
W20260709 12:19:13.337898 28337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.008s	sys 0.032s
I20260709 12:19:13.329582 27659 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:13.342412 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:13.342600 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:13.350643 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.313913 30647 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:19:13.384902 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 113315us
I20260709 12:19:13.371559 27736 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
W20260709 12:19:13.378014 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.378624 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:13.389863 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 432064us
I20260709 12:19:13.390671 27735 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:13.379727 30963 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:19:13.389813 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.383153 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:13.399338 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.364962 30812 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:13.396818 28332 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.013s	sys 0.012s
W20260709 12:19:13.417277 28332 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.013s	sys 0.013s
I20260709 12:19:13.366956 30876 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.422401 27736 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:19:13.412683 30901 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.430727 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.431717 27663 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:19:13.431799 30534 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:13.432531 30534 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.433670 27733 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:13.435650 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.436344 27658 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
W20260709 12:19:13.438903 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48100.2us
I20260709 12:19:13.444680 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:13.428391 30876 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.448945 30876 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.416034 30947 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:13.418893 30738 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.435185 30534 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.477121 28332 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:13.449834 30891 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.471748 30876 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.480343 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:13.483193 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:13.464066 30668 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:13.483852 30635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.030s	sys 0.053s
I20260709 12:19:13.527911 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:13.524544 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.531576 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.532434 27664 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:13.525861 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:13.525101 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:13.538834 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:13.495844 30534 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.468897 30910 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:19:13.544121 27738 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:13.544936 27737 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:13.467402 30947 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.472117 30812 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:13.529515 30635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.030s	sys 0.054s
W20260709 12:19:13.552230 27729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.026s	sys 0.052s
W20260709 12:19:13.556767 27729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.027s	sys 0.052s
I20260709 12:19:13.478963 30757 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.558636 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 555) took 1002 ms. Trace:
I20260709 12:19:13.537998 30181 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:13.506522 30876 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.570118 27657 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:19:13.511904 30738 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.560842 28330 rpcz_store.cc:276] 0709 12:19:12.556283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.556423 (+   140us) service_pool.cc:224] Handling call
0709 12:19:12.556794 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.557160 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.557188 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:12.557715 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:19:13.115362 (+557647us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:13.115419 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.115512 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.558416 (+442904us) raft_consensus.cc:1685] finished
0709 12:19:13.558435 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.558576 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4258,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":33946,"thread_start_us":557198,"threads_started":1,"wal-append.queue_time_us":969812}
W20260709 12:19:13.580967 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:13.576427 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.016s	sys 0.032s
W20260709 12:19:13.586603 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.016s	sys 0.032s
I20260709 12:19:13.560735 30827 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:19:13.586942 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 549497us
I20260709 12:19:13.567723 30181 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:13.560262 30892 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:13.590168 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 495974us
W20260709 12:19:13.591183 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.038s	sys 0.056s
W20260709 12:19:13.591429 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.009s	sys 0.034s
I20260709 12:19:13.576469 30635 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:13.609041 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.010s	sys 0.034s
W20260709 12:19:13.610417 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.038s	sys 0.056s
I20260709 12:19:13.612251 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1638) took 1912 ms. Trace:
I20260709 12:19:13.612661 27657 rpcz_store.cc:276] 0709 12:19:11.700306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.785194 (+1084888us) service_pool.cc:224] Handling call
0709 12:19:12.806323 (+ 21129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.572765 (+766442us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.612018 (+ 39253us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54001e5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:13.612195 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1755906816}
I20260709 12:19:13.569717 30534 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:13.614697 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 443733us
I20260709 12:19:13.586484 30668 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:13.609952 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1981) took 2432 ms. Trace:
W20260709 12:19:13.588519 30751 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.008s	sys 0.034s
I20260709 12:19:13.591820 30181 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:13.589093 30827 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.590284 30891 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:13.626703 30751 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.008s	sys 0.035s
W20260709 12:19:13.616611 30912 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.642s	user 0.013s	sys 0.001s
I20260709 12:19:13.610085 30870 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:13.615406 30910 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:13.648777 30968 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:19:13.623847 27662 rpcz_store.cc:276] 0709 12:19:11.177050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.114888 (+937838us) service_pool.cc:224] Handling call
0709 12:19:12.209786 (+ 94898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.247566 (+ 37780us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:12.307772 (+ 60206us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.591532 (+1283760us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:13.591621 (+    89us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:13.591683 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.607840 (+ 16157us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.609887 (+  2047us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5276,"prepare.queue_time_us":204306,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":15310,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":18691,"thread_start_us":1273854,"threads_started":3,"wal-append.queue_time_us":931864}
W20260709 12:19:13.650849 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.634531 30827 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:13.634828 30901 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.656710 28330 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:13.639958 30891 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.642958 30668 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:13.681443 27661 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:13.636386 30635 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.644752 30638 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:19:13.639814 30942 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.712321 30876 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:13.713378 30901 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:13.655825 30910 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:13.733248 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117666us
I20260709 12:19:13.737172 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:13.738188 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:13.748183 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:13.748066 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:13.750155 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:13.731841 30912 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.724404 30947 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:13.755579 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 239009us
I20260709 12:19:13.757113 27733 raft_consensus.cc:2376] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8. Re-sending same reply.
I20260709 12:19:13.761366 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:13.755859 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:13.750125 30980 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.761701 27736 raft_consensus.cc:2376] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9. Re-sending same reply.
I20260709 12:19:13.791141 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:13.792379 28334 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:13.792946 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56045.5us
W20260709 12:19:13.788622 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 260882us
I20260709 12:19:13.794363 27735 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:13.767467 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:13.795835 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:13.799981 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184457us
I20260709 12:19:13.762501 30812 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.804078 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:13.763936 30776 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:13.759227 30757 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:13.788486 30947 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:19:13.805569 30815 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.787s	user 0.009s	sys 0.004s
I20260709 12:19:13.748690 30942 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:13.809311 30972 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
W20260709 12:19:13.812573 30976 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:19:13.814284 27659 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:13.814874 30534 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.806866 30668 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.815594 30751 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:13.816680 30738 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:13.818310 30974 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:13.819157 30891 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.821381 30975 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:19:13.822381 30973 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:19:13.822925 30766 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:13.823877 30977 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:13.825603 27582 raft_consensus.cc:2410] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:19:13.843806 30986 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:13.872437 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.844754 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:13.844839 30910 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.825733 30892 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:13.807689 30980 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.805866 30983 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:13.847108 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 557) took 1128 ms. Trace:
I20260709 12:19:13.847994 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:13.761298 30912 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.854842 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:13.764402 30981 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
I20260709 12:19:13.860772 30751 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:13.900060 30751 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:13.901275 30751 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:13.861528 30993 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:13.861776 30738 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:13.904045 30738 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:13.905277 30738 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.903033 30751 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:13.912729 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.876204 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.879631 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:13.883590 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58257.2us
I20260709 12:19:13.885046 27657 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
W20260709 12:19:13.882073 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 128901us
I20260709 12:19:13.915055 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:13.894644 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.898204 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.914073 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:13.919112 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110077us
I20260709 12:19:13.919270 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:13.924577 27586 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:13.930835 27658 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:13.895843 28328 rpcz_store.cc:276] 0709 12:19:12.718692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.718830 (+   138us) service_pool.cc:224] Handling call
0709 12:19:12.788233 (+ 69403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.788614 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.788655 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:12.795658 (+  7003us) log.cc:844] Serialized 26 byte log entry
0709 12:19:13.366573 (+570915us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:13.366634 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.366746 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.846754 (+480008us) raft_consensus.cc:1685] finished
0709 12:19:13.846773 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.846974 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":587,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":5929,"thread_start_us":480238,"threads_started":1,"wal-append.queue_time_us":776919}
I20260709 12:19:13.881253 30977 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.942822 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.898030 30983 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.923527 30988 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:19:13.966998 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53002.5us
I20260709 12:19:13.933658 30635 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:13.966358 27655 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:13.940784 30900 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:13.937081 30977 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:13.981262 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.944096 30892 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:19:13.983387 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 138710us
I20260709 12:19:13.958915 30983 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:13.965417 30988 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:13.987967 27664 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:13.990826 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75910.4us
I20260709 12:19:13.920635 30986 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:13.992266 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:13.993578 27655 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:13.993707 27663 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:19:13.994657 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:13.997876 30668 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:13.984817 30738 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:13.988840 30635 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:14.000361 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 422) took 1286 ms. Trace:
I20260709 12:19:14.001619 27729 rpcz_store.cc:276] 0709 12:19:12.710131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.710284 (+   153us) service_pool.cc:224] Handling call
0709 12:19:12.710607 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.711004 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.711040 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:12.789602 (+ 78562us) log.cc:844] Serialized 26 byte log entry
0709 12:19:13.557212 (+767610us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:13.557257 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.557321 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.996015 (+438694us) raft_consensus.cc:1685] finished
0709 12:19:13.996034 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.996164 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":835,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":71229,"thread_start_us":666360,"threads_started":1,"wal-append.queue_time_us":817646}
I20260709 12:19:14.003183 30992 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:19:14.004674 30986 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.005611 27580 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:19:14.007690 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:14.008354 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:14.009176 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:14.010555 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:14.011582 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:13.995942 30900 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:14.037060 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45048.7us
I20260709 12:19:14.041415 27663 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:14.002293 30635 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:14.004678 30992 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:14.016594 31005 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.023679 30668 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:14.066497 30999 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:19:14.066804 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:14.068554 30989 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:14.069953 30635 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.072088 30668 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.053500 30942 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:14.078243 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61212.6us
W20260709 12:19:14.078310 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86108us
I20260709 12:19:14.054270 30992 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:14.052906 30986 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.085587 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:14.083436 30993 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:14.063809 31005 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.121771 30181 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.087425 30992 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:14.136123 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.139986 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:14.092464 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:14.145172 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105754us
W20260709 12:19:14.145203 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.140755 28330 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
W20260709 12:19:14.137460 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52050.5us
I20260709 12:19:14.174408 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:14.143616 30635 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:14.176038 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 102089us
I20260709 12:19:14.098384 30912 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:14.132408 31017 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.138151 31005 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:14.156443 30873 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:14.142078 30923 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:14.137795 30181 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.171363 30738 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:14.201365 31017 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.065271 30988 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.197501 30891 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:14.204857 30976 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.317s	user 0.001s	sys 0.002s
I20260709 12:19:14.138391 30986 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.153928 30989 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:19:14.214867 28334 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
W20260709 12:19:14.263913 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88883.3us
I20260709 12:19:14.206529 30873 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:14.270089 30873 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.270699 31017 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:14.265041 30534 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.271661 30534 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.273841 30534 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.210043 30827 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.205484 30751 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:14.213302 31005 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:14.207899 30923 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:14.263617 30891 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:14.318416 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55479.4us
W20260709 12:19:14.297111 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.280436 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:14.271136 30873 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:14.354059 30534 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:14.354547 31012 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.365549 31012 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.366189 30827 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:14.353807 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 73432.6us
I20260709 12:19:14.136205 30993 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.218487 30912 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.101s
W20260709 12:19:14.339521 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 140858us
W20260709 12:19:14.442902 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73147.1us
W20260709 12:19:14.392058 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:14.445880 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:14.354547 31008 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:19:14.279793 31018 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.204s
I20260709 12:19:14.344412 30891 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:14.459406 31013 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.343843 30751 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.281219 31004 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.354s
I20260709 12:19:14.344259 31015 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:14.371825 30989 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:14.369956 30942 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:14.502918 27660 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:14.514871 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 1983) took 3120 ms. Trace:
I20260709 12:19:14.496855 30638 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:14.493312 30977 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.000s
W20260709 12:19:14.526543 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.526777 30986 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.527842 30942 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.531754 31004 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:14.516682 30534 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:14.535074 31000 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:19:14.536327 31012 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:14.536777 31001 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.425s	user 0.001s	sys 0.000s
I20260709 12:19:14.513542 31007 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:14.549793 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89280.9us
I20260709 12:19:14.522771 27656 rpcz_store.cc:276] 0709 12:19:11.375924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.276866 (+900942us) service_pool.cc:224] Handling call
0709 12:19:12.277479 (+   613us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.291627 (+ 14148us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:12.998628 (+707001us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.736493 (+737865us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:13.760467 (+ 23974us) log.cc:844] Serialized 26 byte log entry
0709 12:19:13.760655 (+   188us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:13.760734 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.761193 (+   459us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:14.204582 (+443389us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b24002ac1b8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:19:14.489239 (+284657us) raft_consensus.cc:1685] finished
0709 12:19:14.495980 (+  6741us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:14.496209 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4940,"prepare.queue_time_us":143523,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":166388,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":34793,"spinlock_wait_cycles":26769152,"thread_start_us":1326999,"threads_started":3,"wal-append.queue_time_us":609616}
I20260709 12:19:14.572187 30977 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.550971 30989 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.548906 31015 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.600406 31004 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:14.602627 30974 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:14.602358 31021 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.142s
I20260709 12:19:14.603533 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:14.480319 30738 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.513362 31013 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.628676 28335 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:19:14.607442 30986 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:14.630646 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 135689us
I20260709 12:19:14.550254 30638 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.518919 30923 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.647570 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:14.656792 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:14.660347 31024 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:14.638195 31031 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:14.642146 31012 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:14.670009 27731 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
W20260709 12:19:14.670517 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 145806us
I20260709 12:19:14.624414 31022 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.033s
I20260709 12:19:14.663488 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:14.647226 30942 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.673098 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:14.676221 27732 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:14.674917 27580 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:19:14.676658 27580 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:14.678013 27737 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:19:14.679593 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:14.681381 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:14.682610 27589 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:14.656886 30923 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:14.636545 30986 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.689661 27583 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:19:14.676931 27580 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:19:14.690522 27580 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:14.623778 31007 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.693009 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:14.625351 30974 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.694419 27738 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:14.693110 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:14.699925 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:14.690224 27583 pending_rounds.cc:85] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:14.702414 27583 pending_rounds.cc:107] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:19:14.702740 27583 raft_consensus.cc:2889] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:14.635749 30534 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.706612 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:14.714542 27586 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:14.716213 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:14.716866 27588 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:14.730546 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.720603 30751 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.738348 31007 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.736784 31034 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:14.761096 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [7.5-10.6]
I20260709 12:19:14.762770 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:14.763725 27738 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:14.700628 31012 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.684777 31031 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.739126 31036 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:14.769878 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74214us
I20260709 12:19:14.759727 31034 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:14.777031 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:14.767201 31018 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:14.781226 27736 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:14.788930 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:14.774595 30977 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:14.816648 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:14.833148 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:14.834312 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:14.838002 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:14.781983 31042 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:14.832981 28295 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 2.583s (SENT) [suppressed 26 similar messages]
W20260709 12:19:14.831091 31010 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.524s	user 0.000s	sys 0.003s
I20260709 12:19:14.845808 27738 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:14.776166 31036 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.850914 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:14.849809 31036 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:14.836280 31007 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:14.836071 30942 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:14.881614 27736 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:19:14.874727 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 202002us
I20260709 12:19:14.853111 31015 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:14.883143 30983 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.883s	user 0.002s	sys 0.011s
I20260709 12:19:14.900341 31031 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:14.900022 30983 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.904219 31041 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:14.904954 31041 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:14.906414 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:14.906106 31041 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:14.907137 31022 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:14.908165 31022 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:14.908115 31046 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:14.909675 31022 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:14.910040 31022 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:14.910396 31022 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:14.910466 31046 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:14.911392 31038 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.911911 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:14.891076 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.927909 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:14.902971 30983 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.983566 31048 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.984061 31048 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.986249 31048 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:14.898077 30974 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:14.917039 30986 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:14.970767 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58979.7us
I20260709 12:19:14.992987 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:14.911054 31046 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:14.892390 31045 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:19:14.889604 31035 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:19:14.899447 30989 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:14.889261 31040 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
W20260709 12:19:14.972546 31039 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
W20260709 12:19:14.981343 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147116us
I20260709 12:19:14.883620 31042 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:15.000735 31050 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:14.982223 31019 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:19:14.971649 30992 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.834s	user 0.012s	sys 0.000s
I20260709 12:19:15.001390 31050 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:15.001868 30992 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.002708 31052 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:15.003218 31052 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.003584 31052 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:15.003940 27732 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:15.004846 30992 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.006232 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.007931 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.008458 31055 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:15.009223 31055 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.009740 31054 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 12:19:15.009907 31055 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:14.931095 31034 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.012317 31034 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:15.014699 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 440) took 1210 ms. Trace:
I20260709 12:19:15.015535 31017 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:14.980772 31015 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:15.017446 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1636) took 3339 ms. Trace:
I20260709 12:19:14.883126 30942 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:15.017840 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184870us
I20260709 12:19:15.009171 27733 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:15.017845 27660 rpcz_store.cc:276] 0709 12:19:11.678533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.491802 (+813269us) service_pool.cc:224] Handling call
0709 12:19:14.496338 (+2004536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.525512 (+ 29174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.017212 (+491700us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b54000e5bd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:15.017390 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4608961408}
I20260709 12:19:14.970543 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:14.897300 31012 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:14.874564 31007 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.020635 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:15.015221 27662 rpcz_store.cc:276] 0709 12:19:13.804121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:13.806627 (+  2506us) service_pool.cc:224] Handling call
0709 12:19:13.811844 (+  5217us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:13.988223 (+176379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:13.988256 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:13.993378 (+  5122us) log.cc:844] Serialized 26 byte log entry
0709 12:19:14.516983 (+523605us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:14.517056 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.014178 (+497122us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:15.014249 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:15.014315 (+    66us) raft_consensus.cc:1685] finished
0709 12:19:15.014336 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.014624 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59639,"raft.queue_time_us":142241,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":125,"spinlock_wait_cycles":288055040,"thread_start_us":499719,"threads_started":2,"wal-append.queue_time_us":573589}
I20260709 12:19:15.024490 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:14.898795 30891 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.026064 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.028797 27738 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:15.033885 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:15.035727 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:15.036026 27734 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:15.036386 27661 raft_consensus.cc:2376] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7. Re-sending same reply.
I20260709 12:19:15.033613 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.034263 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.040143 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:15.041455 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:15.041788 27733 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 12:19:15.042142 27733 pending_rounds.cc:85] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:19:15.042348 27733 pending_rounds.cc:107] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:19:15.042606 27733 raft_consensus.cc:2889] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:15.042815 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.043746 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:15.027709 30983 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.036526 31048 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:14.912017 31038 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.047758 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:15.050130 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:15.033795 31017 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:15.046190 27662 raft_consensus.cc:2376] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9. Re-sending same reply.
I20260709 12:19:15.031581 30992 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.052068 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.054725 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.055078 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.057559 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:15.059206 27663 raft_consensus.cc:2376] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9. Re-sending same reply.
I20260709 12:19:15.062922 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.082995 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.084038 27664 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:15.036182 31054 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 12:19:15.049628 30986 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:15.026834 31015 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.092804 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.101233 27734 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:15.035717 31042 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:19:15.112355 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:15.113767 27733 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:15.055712 31059 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:15.115437 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:15.124140 27656 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:15.051631 31048 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.124382 28334 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:15.137738 31048 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.060837 30992 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.116132 31054 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:15.105301 30980 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.086248 30983 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.123888 31059 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:15.159698 31061 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:19:15.161082 30900 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:15.162210 30873 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.725s	user 0.009s	sys 0.003s
I20260709 12:19:15.167555 30891 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.114921 28336 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22597fcae204491f84839e680c083ecc"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:15.168207 31007 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.168498 31015 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.170231 30992 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.171540 28330 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:15.172232 28336 raft_consensus.cc:606] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:15.172767 28336 raft_consensus.cc:614] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:15.157518 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:15.178042 27733 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:15.178313 27656 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
W20260709 12:19:15.180336 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 120184us
I20260709 12:19:15.199283 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:15.148284 31048 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:15.202128 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:15.203078 28328 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:15.168242 30891 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:15.172197 31059 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:19:15.201920 30635 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.001s	user 0.003s	sys 0.010s
I20260709 12:19:15.196363 30977 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.201236 31048 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:15.221663 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.222288 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:15.222570 31056 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:19:15.200098 30989 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:15.249869 30989 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.225396 30891 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:15.252668 30891 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:15.253274 30891 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:15.254330 30891 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.226095 30942 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.227543 30983 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.256191 31038 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.261428 30989 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.228327 31052 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.202523 31043 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:15.267980 31043 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.220261 31042 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:15.270315 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 436) took 2296 ms (client timeout 2583 ms). Trace:
I20260709 12:19:15.270627 31043 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.244529 31066 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:15.271070 30977 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.271306 31066 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.273392 31066 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:15.273470 31052 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.246503 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:15.248092 30923 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.553s	user 0.007s	sys 0.003s
I20260709 12:19:15.270156 31042 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.274075 31048 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.243683 31036 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.275336 31048 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:15.275604 31036 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.276563 31066 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:15.277112 31066 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:15.277791 31036 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:15.278026 31066 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.279860 31042 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.280619 31042 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:15.210443 30534 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 12:19:15.281116 31042 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.282155 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.283759 27588 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:15.284874 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.285497 27585 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:15.288486 31057 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:19:15.288290 31022 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.283182 31042 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.292297 30638 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.294510 30638 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.299652 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.300602 27586 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:15.304672 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:15.270762 27582 rpcz_store.cc:269] 0709 12:19:12.973803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.977984 (+  4181us) service_pool.cc:224] Handling call
0709 12:19:15.270105 (+2292121us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b540011c954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:15.270241 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1946393856}
I20260709 12:19:15.305495 27580 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:15.317952 27588 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:15.321250 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.322760 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.323817 27583 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260709 12:19:15.341010 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.352596 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:15.295009 30638 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.297317 30977 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.356671 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.361163 27582 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:15.361788 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 179135us
I20260709 12:19:15.362785 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.363607 27580 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:15.317456 31015 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:15.375123 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.381645 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.382737 27585 raft_consensus.cc:2410] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:19:15.390478 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.356385 30983 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:15.390478 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.359572 31034 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:15.356104 30942 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:15.394845 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:15.396744 27582 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:15.400890 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.374256 30986 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:15.409981 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.366848 31055 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.385360 31022 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:15.410782 27585 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:15.422346 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:15.425292 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.425942 27733 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:15.442533 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:15.387032 31015 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.463229 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.463707 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:15.463937 27732 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:15.464534 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:15.465159 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:15.403463 31036 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:19:15.463296 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.468484 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.404203 30983 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.406011 30942 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.481988 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.482870 27738 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:15.459035 30923 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:15.492456 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.495174 27733 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:15.497573 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:15.498021 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.502559 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:15.503135 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.504765 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:15.473147 30635 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.509109 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.509949 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:15.503695 31016 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:15.511001 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:15.513677 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:15.514292 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.510967 31016 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.527225 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:15.531213 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.531890 27734 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:15.499061 30923 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:15.511533 30900 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:15.516659 30977 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:15.524617 30873 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.524400 31042 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.523847 30980 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.539922 30992 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:15.567611 27585 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:15.529755 31052 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.528353 30534 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.571177 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.571272 27587 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:15.539108 30923 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:15.503118 30986 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.525460 30635 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.602375 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:15.897507 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.808785 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.777688 27735 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:19:15.898334 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 356567us
I20260709 12:19:15.611778 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:15.778403 31085 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:19:15.908507 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
W20260709 12:19:15.910297 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 307989us
I20260709 12:19:15.544368 30900 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:15.711130 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.891569 30987 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:15.601238 30992 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.548283 30977 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.912514 27664 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:15.848196 31074 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.359s	user 0.000s	sys 0.001s
I20260709 12:19:15.931818 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:15.851956 31083 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:19:15.913290 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.610419 31059 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.933997 27583 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
I20260709 12:19:15.934952 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.935144 31059 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:15.935732 27583 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:15.912005 30534 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:15.936631 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.937448 27583 raft_consensus.cc:2410] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:19:15.937599 31016 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:15.929844 31055 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 12:19:15.938473 27583 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
W20260709 12:19:15.906581 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.939582 30534 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.917062 30776 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:15.921523 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:15.921847 31063 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.447s	user 0.000s	sys 0.001s
I20260709 12:19:15.926126 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:15.616596 31042 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.926710 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.928277 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.890108 31013 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.909176 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.932969 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:15.936180 31086 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:19:15.928632 30900 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:15.601563 30873 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:15.935168 31080 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:19:15.940248 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.920236 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:15.613984 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:15.946969 30891 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:15.948746 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.948618 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.949857 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:15.955214 31059 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:15.956885 27589 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.000s
I20260709 12:19:15.611284 31012 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.319s
I20260709 12:19:15.938975 30988 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:19:15.831496 31069 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.425s	user 0.001s	sys 0.000s
I20260709 12:19:15.891567 31015 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:15.910835 31052 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:15.723376 30980 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.124246 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:15.712299 30638 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:16.125311 30873 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.125478 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.126123 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:15.993499 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:16.073280 27731 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.073444 27730 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:16.127200 27734 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:16.074700 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:19:16.007131 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:16.073613 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 153926us
I20260709 12:19:15.999881 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:16.018062 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:16.081760 27589 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:16.082800 27587 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:16.081161 27660 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:16.087818 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.130995 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.131212 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:16.132735 27588 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:16.019502 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:16.015008 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:15.931202 30992 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:15.964809 30977 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:16.138942 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.139751 27664 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:16.074076 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.141690 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:16.071907 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.142271 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.134235 27583 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:15.928148 30987 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:16.136858 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.087675 31055 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:16.135766 30942 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:16.148425 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.149138 27730 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:16.015146 30978 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.138639 31041 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:16.139057 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.140631 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:16.153333 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.153702 27664 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:16.130115 28333 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:16.015028 31067 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.215s
I20260709 12:19:16.143821 27655 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.145063 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:16.128055 28328 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:16.073247 30776 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:16.160032 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 142103us
I20260709 12:19:16.145393 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.147017 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:16.162031 27734 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:16.145710 30912 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.676s	user 0.007s	sys 0.019s
I20260709 12:19:16.163451 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.133317 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.164150 27588 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:16.164484 27587 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:16.163220 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.164680 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:16.165052 27660 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:16.167151 27730 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:19:16.167944 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.168860 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:16.171384 27660 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:19:16.120042 31059 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.179869 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.172997 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:16.196523 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 479192us
I20260709 12:19:16.156181 31041 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:16.191030 27584 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:16.166116 30912 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.697s	user 0.007s	sys 0.020s
I20260709 12:19:16.119192 30891 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.172384 30534 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:19:16.163939 30776 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:16.226320 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.225477 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.230782 27588 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:16.227623 27587 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:19:16.246356 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:16.223825 31095 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:16.252410 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.253093 30891 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.253255 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:16.254230 31042 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:16.326799 31090 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
I20260709 12:19:16.254355 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:16.227653 30986 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:16.267028 31059 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.308903 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:16.318594 27660 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.324441 27664 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:16.159471 30978 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.190013 31015 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.098s
I20260709 12:19:16.348902 27655 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:16.377826 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.378698 27655 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:16.379703 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.380395 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2016) took 1007 ms. Trace:
I20260709 12:19:16.380620 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:16.380816 27655 rpcz_store.cc:276] 0709 12:19:15.372919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.379434 (+1006515us) service_pool.cc:224] Handling call
0709 12:19:16.380309 (+   875us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:16.328014 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73725.7us
W20260709 12:19:16.382781 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 237471us
I20260709 12:19:16.382826 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.389081 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.390465 27589 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:16.391067 27589 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:19:16.391453 27589 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:16.308524 31095 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:16.335058 31105 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:19:16.335018 30974 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.367429 30983 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:16.362670 31094 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:16.419618 31094 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:16.420128 31094 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:16.406529 31012 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:19:16.424553 31012 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:16.424954 31012 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:16.425837 31012 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.428150 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.428588 30181 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:19:16.429446 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81225.1us
I20260709 12:19:16.431946 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:16.384557 30942 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:16.413846 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 492) took 2561 ms. Trace:
I20260709 12:19:16.434347 28335 rpcz_store.cc:276] 0709 12:19:13.852106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:13.852391 (+   285us) service_pool.cc:224] Handling call
0709 12:19:13.860795 (+  8404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.629473 (+768678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.410866 (+1781393us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:16.413742 (+  2876us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1765335936}
I20260709 12:19:16.414003 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:16.414258 31100 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 12:19:16.415589 30970 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 2.782s	user 0.022s	sys 0.025s
W20260709 12:19:16.416620 31103 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:19:16.403978 30873 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.410079 31095 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:16.430414 30974 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.453606 30974 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:16.454514 28334 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:16.455474 31052 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.455894 30978 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.458743 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2013) took 1315 ms. Trace:
W20260709 12:19:16.460002 30923 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.000s
I20260709 12:19:16.460649 30923 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.461719 28332 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.462154 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 443) took 1020 ms. Trace:
I20260709 12:19:16.464736 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.465976 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.466547 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:16.471022 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:16.471863 31095 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:16.472652 31078 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.663s	user 0.002s	sys 0.000s
I20260709 12:19:16.412846 31067 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 12:19:16.474195 31095 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.474354 31067 consensus_queue.cc:1243] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:16.408298 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:16.478327 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.479827 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:16.481506 30980 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:16.411420 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.438272 27656 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:16.438735 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56020.3us
I20260709 12:19:16.482052 31071 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:16.482671 31071 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:16.483042 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.483816 30638 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:16.483575 31071 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.484012 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.484200 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.484509 31004 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:16.484820 27656 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:16.484926 31004 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:16.485235 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.485827 31004 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.487346 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.487908 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:16.488211 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2012) took 1357 ms. Trace:
I20260709 12:19:16.440093 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.427119 31012 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:16.437768 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:16.442436 31110 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:16.462531 27738 rpcz_store.cc:276] 0709 12:19:15.441416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.491743 (+ 50327us) service_pool.cc:224] Handling call
0709 12:19:15.507045 (+ 15302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:15.507274 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:15.518390 (+ 11116us) log.cc:844] Serialized 14 byte log entry
0709 12:19:16.461688 (+943298us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b18000a78c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:19:16.461833 (+   145us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:16.461880 (+    47us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:16.461922 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.461984 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:16.462099 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15213,"prepare.queue_time_us":17707,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":945127,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":48,"spinlock_wait_cycles":12002560}
I20260709 12:19:16.459147 27659 rpcz_store.cc:276] 0709 12:19:15.147652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.356596 (+208944us) service_pool.cc:224] Handling call
0709 12:19:15.365829 (+  9233us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:15.385987 (+ 20158us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:15.934833 (+548846us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b18000fe080. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:19:16.123733 (+188900us) log.cc:844] Serialized 14 byte log entry
0709 12:19:16.271944 (+148211us) log.cc:844] Serialized 14 byte log entry
0709 12:19:16.286604 (+ 14660us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:16.308479 (+ 21875us) log.cc:844] Serialized 52 byte log entry
0709 12:19:16.308633 (+   154us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:16.308665 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.308732 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:16.458488 (+149756us) raft_consensus.cc:1685] finished
0709 12:19:16.458511 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:16.458681 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":228099,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":83,"raft.queue_time_us":36578,"raft.run_cpu_time_us":475,"raft.run_wall_time_us":64962,"spinlock_wait_cycles":25267712,"thread_start_us":317654,"threads_started":2,"wal-append.queue_time_us":312773}
I20260709 12:19:16.443533 31113 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:16.432991 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:16.488760 27657 rpcz_store.cc:276] 0709 12:19:15.135991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.137680 (+  1689us) service_pool.cc:224] Handling call
0709 12:19:15.138027 (+   347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:15.200224 (+ 62197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:15.200254 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:15.216649 (+ 16395us) log.cc:844] Serialized 52 byte log entry
0709 12:19:15.427293 (+210644us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:19:15.427344 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.943558 (+516214us) spinlock_profiling.cc:243] Waited 61.8 ms on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:15.943631 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:16.471905 (+528274us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:16.487936 (+ 16031us) raft_consensus.cc:1685] finished
0709 12:19:16.487966 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:16.488140 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1277,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":5523,"spinlock_wait_cycles":148974720,"thread_start_us":205289,"threads_started":1,"wal-append.queue_time_us":318790}
W20260709 12:19:16.493849 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56123.7us
I20260709 12:19:16.429272 30181 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:16.494185 27660 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.494974 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:16.495839 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:16.495913 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59798.8us
I20260709 12:19:16.498610 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.518165 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:16.519757 27731 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
I20260709 12:19:16.520006 28295 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
I20260709 12:19:16.439079 30635 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:16.535184 27657 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:19:16.533478 27580 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:19:16.544157 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2019) took 1139 ms. Trace:
I20260709 12:19:16.544649 27657 rpcz_store.cc:276] 0709 12:19:15.404518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.494336 (+1089818us) service_pool.cc:224] Handling call
0709 12:19:16.534879 (+ 40543us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.543771 (+  8892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.544085 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:16.527666 27660 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:16.546265 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1660) took 1003 ms. Trace:
I20260709 12:19:16.546658 27660 rpcz_store.cc:276] 0709 12:19:15.542787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.432764 (+889977us) service_pool.cc:224] Handling call
0709 12:19:16.546192 (+113428us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:16.549722 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.550915 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.551496 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.551977 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.499951 31024 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.450956 30900 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:16.554078 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73386.6us
I20260709 12:19:16.522814 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.484241 30638 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:16.481531 31000 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:16.560190 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.570391 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:16.560945 30181 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.564203 30776 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.565840 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.576951 27731 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:16.497442 30986 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:16.568073 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.580538 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:16.568982 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46234us
I20260709 12:19:16.567224 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 531) took 4010 ms. Trace:
I20260709 12:19:16.462793 30923 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.497506 30978 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:16.585355 30978 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:16.585843 30978 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:16.587458 30827 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:16.500211 31067 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:16.534386 31059 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.584045 28337 rpcz_store.cc:276] 0709 12:19:12.556602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.556818 (+   216us) service_pool.cc:224] Handling call
0709 12:19:12.557188 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.557639 (+   451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.557674 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:12.648355 (+ 90681us) log.cc:844] Serialized 26 byte log entry
0709 12:19:12.709740 (+ 61385us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b180008c2c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:13.338478 (+628738us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:13.338538 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.338639 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:16.415037 (+3076398us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b2400318128. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:19:16.566964 (+151927us) raft_consensus.cc:1685] finished
0709 12:19:16.566992 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:16.567153 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3915,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":6227,"spinlock_wait_cycles":82013184,"thread_start_us":628512,"threads_started":1,"wal-append.queue_time_us":966633}
I20260709 12:19:16.500285 30974 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:16.571687 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.588012 30827 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.493340 30873 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.560250 31024 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.631007 31024 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.577621 31000 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.590622 31097 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:16.594354 31070 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:16.633535 31113 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:16.633396 31070 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.634282 31113 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:16.635669 31070 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:16.598058 30776 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:16.594911 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.599435 30892 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:16.600070 27664 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:16.600245 30900 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.637513 30900 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:16.637850 30900 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:16.596199 31067 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.639187 31067 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.579409 30986 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.639780 30986 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.638675 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:16.642398 27731 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:16.610262 30978 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.565953 31107 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:16.611809 30923 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:16.614913 30638 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.615521 31065 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:16.500005 30977 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:16.626514 27701 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.501s (SENT)
I20260709 12:19:16.627954 30827 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:16.651244 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56434.5us
I20260709 12:19:16.638583 30992 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.676137 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.695549 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.696512 27655 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
W20260709 12:19:16.704268 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.653205 30974 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:16.676154 30980 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.643492 31059 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:16.786849 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160350us
I20260709 12:19:16.790921 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:16.666720 31065 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:16.701989 30638 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:16.838840 30912 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.839217 31065 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.839473 30912 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.840342 31065 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:16.667053 31066 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.118s
I20260709 12:19:16.844192 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:16.658461 30923 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.658790 31107 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:16.852207 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.844666 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.846889 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:16.845583 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.638644 30900 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.771838 30647 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:16.860711 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:16.861634 27701 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.506s (SENT)
I20260709 12:19:16.866102 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.851819 31066 consensus_queue.cc:1243] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:19:16.901014 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:16.862586 27655 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:16.875633 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.877183 27581 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:16.902519 27581 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:16.878335 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.879894 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.888337 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:16.890537 27737 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:16.892230 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.846302 31110 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:16.910332 27583 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:16.887081 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:16.911396 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:16.911931 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50339.4us
I20260709 12:19:16.927037 27658 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:16.915257 30873 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.915808 30974 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.843751 30980 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.931447 30980 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.935273 31110 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:16.911290 31052 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:16.939311 31052 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:16.939805 31052 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:16.943850 31071 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:16.948103 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 560) took 1918 ms. Trace:
I20260709 12:19:16.948505 28330 rpcz_store.cc:276] 0709 12:19:15.029920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.030112 (+   192us) service_pool.cc:224] Handling call
0709 12:19:16.947845 (+1917733us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:16.947997 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292134272}
I20260709 12:19:16.950728 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.952224 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:16.952914 27587 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:16.954813 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.916765 30923 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.862561 31042 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:16.919577 31000 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:16.920909 30977 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:16.921629 31094 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.501s	user 0.011s	sys 0.000s
W20260709 12:19:16.919598 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.865923 30988 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.743s	user 0.006s	sys 0.006s
I20260709 12:19:16.927063 27655 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:16.907039 30912 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.956030 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:16.957708 27657 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:16.958274 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:16.954917 27581 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.973402 27581 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:16.974426 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:16.963744 31107 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.984205 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:16.980782 27627 connection.cc:256] RPC call timeout handler was delayed by -1.8725173220000215s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
W20260709 12:19:16.986073 27627 connection.cc:256] RPC call timeout handler was delayed by -1.8113173219999226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
W20260709 12:19:16.986238 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 401920us
W20260709 12:19:16.986873 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.987217 27627 connection.cc:256] RPC call timeout handler was delayed by -1.7537173220000568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
W20260709 12:19:16.987778 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:16.988353 27627 connection.cc:256] RPC call timeout handler was delayed by -1.46211732200004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19970000000000002s for a response
W20260709 12:19:16.988749 27627 connection.cc:256] RPC call timeout handler was delayed by -1.3388173220000681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260709 12:19:16.989125 27627 connection.cc:256] RPC call timeout handler was delayed by -1.3055173220000142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
W20260709 12:19:16.989494 27627 connection.cc:256] RPC call timeout handler was delayed by -1.2749173219999648s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22050000000000003s for a response
W20260709 12:19:16.989868 27627 connection.cc:256] RPC call timeout handler was delayed by -1.0184173220000048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24900000000000003s for a response
W20260709 12:19:16.990228 27627 connection.cc:256] RPC call timeout handler was delayed by -1.0178905279999526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
I20260709 12:19:16.993259 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.971832 30991 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.974563 31000 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:16.997311 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.001108 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:17.002705 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:17.005116 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:16.985591 30978 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:16.971082 31094 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:16.998826 31070 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:16.993419 30873 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:16.996424 30980 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.993796 30988 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:16.998574 31054 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:17.093575 27655 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:17.092532 30766 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:17.100112 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 109111us
I20260709 12:19:17.103070 30992 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.006989 31022 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:17.013684 31018 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:17.105777 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2018) took 1710 ms. Trace:
I20260709 12:19:17.006502 30827 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:17.107626 27658 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:17.115494 27581 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:19:17.230638 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.006s	sys 0.027s
W20260709 12:19:17.360042 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.006s	sys 0.028s
I20260709 12:19:17.137293 30980 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.364777 30980 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:17.248083 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 347114us
W20260709 12:19:17.250337 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:17.367408 31117 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.276s	user 0.003s	sys 0.006s
I20260709 12:19:17.008347 30991 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.091398 31012 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:19:17.370883 30986 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.731s	user 0.006s	sys 0.010s
I20260709 12:19:17.121451 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:17.372097 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2017) took 1979 ms. Trace:
I20260709 12:19:17.372565 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:17.372481 27656 rpcz_store.cc:276] 0709 12:19:15.393343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.485517 (+1092174us) service_pool.cc:224] Handling call
0709 12:19:16.485878 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.486207 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.486236 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.486633 (+   397us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.771002 (+284369us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:17.004401 (+233399us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.025295 (+ 20894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.025411 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.371869 (+346458us) raft_consensus.cc:1685] finished
0709 12:19:17.371890 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.372028 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":44819,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":7219,"thread_start_us":450786,"threads_started":2,"wal-append.queue_time_us":493573}
I20260709 12:19:17.373805 27655 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:17.374500 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:17.376277 31046 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.377372 31046 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:17.378130 31026 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 12:19:17.375108 27656 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:17.367264 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.156710 31059 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:17.270850 31120 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.177s	user 0.000s	sys 0.008s
I20260709 12:19:17.120671 30873 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:17.359185 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1661) took 1811 ms (client timeout 1680 ms). Trace:
W20260709 12:19:17.418668 27658 rpcz_store.cc:269] 0709 12:19:15.552688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.886666 (+1333978us) service_pool.cc:224] Handling call
0709 12:19:17.359100 (+472434us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.420420 30873 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.420629 27658 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:17.421540 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2022) took 1998 ms. Trace:
I20260709 12:19:17.421340 31052 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.422009 27658 rpcz_store.cc:276] 0709 12:19:15.423087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.419773 (+1996686us) service_pool.cc:224] Handling call
0709 12:19:17.420150 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.421203 (+  1053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.421457 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.252240 30988 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.424094 31052 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:17.425241 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.425314 30988 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.361819 30827 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:17.426647 30827 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:17.385833 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1993) took 2058 ms. Trace:
W20260709 12:19:17.351809 31127 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.196s	user 0.002s	sys 0.005s
I20260709 12:19:17.357957 27659 rpcz_store.cc:276] 0709 12:19:15.395607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.492600 (+1096993us) service_pool.cc:224] Handling call
0709 12:19:16.493027 (+   427us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:16.518200 (+ 25173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.518232 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:16.520817 (+  2585us) log.cc:844] Serialized 52 byte log entry
0709 12:19:16.690234 (+169417us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:16.690296 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.690401 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.105460 (+415059us) raft_consensus.cc:1685] finished
0709 12:19:17.105540 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.105706 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6437,"raft.queue_time_us":6044,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":14470,"thread_start_us":157152,"threads_started":1,"wal-append.queue_time_us":466546}
I20260709 12:19:17.410080 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:17.358278 31116 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.268s	user 0.004s	sys 0.004s
I20260709 12:19:17.405997 31070 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:17.455371 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.455655 31070 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.428079 31107 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:17.460058 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 1988) took 2181 ms. Trace:
I20260709 12:19:17.460602 27584 rpcz_store.cc:276] 0709 12:19:15.278175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.286191 (+  8016us) service_pool.cc:224] Handling call
0709 12:19:15.286477 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:16.234009 (+947532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.459755 (+1225746us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:17.459981 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2037167488}
I20260709 12:19:17.461596 31070 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:17.352766 30766 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.470273 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2005) took 2067 ms. Trace:
I20260709 12:19:17.380533 30986 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.472081 27586 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:17.417286 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2010) took 1446 ms. Trace:
I20260709 12:19:17.475871 27589 rpcz_store.cc:276] 0709 12:19:15.970487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.181902 (+211415us) service_pool.cc:224] Handling call
0709 12:19:16.390250 (+208348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.432546 (+ 42296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.432574 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.435807 (+  3233us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.907754 (+471947us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:16.907806 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.907898 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.416976 (+509078us) raft_consensus.cc:1685] finished
0709 12:19:17.417002 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.417208 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":392625,"threads_started":1,"wal-append.queue_time_us":934390}
W20260709 12:19:17.477424 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.477843 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:17.478894 27589 raft_consensus.cc:2376] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6. Re-sending same reply.
I20260709 12:19:17.479970 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:17.480075 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.480805 27587 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:17.481554 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:17.482451 27589 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:17.417994 31119 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:19:17.484941 31052 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:17.487635 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 445) took 1472 ms. Trace:
I20260709 12:19:17.492477 31107 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:17.493100 31107 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.495836 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.498782 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2012) took 1092 ms. Trace:
I20260709 12:19:17.499142 31131 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.500129 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.498461 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:17.501057 27589 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 5: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:17.488023 27662 rpcz_store.cc:276] 0709 12:19:16.014913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.486792 (+1471879us) service_pool.cc:224] Handling call
0709 12:19:17.487110 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.487290 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.487317 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.487381 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:17.487414 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.487466 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.487582 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.507767 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2016) took 1516 ms. Trace:
I20260709 12:19:17.510206 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 500) took 1046 ms. Trace:
I20260709 12:19:17.505836 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.462383 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:17.510681 28334 rpcz_store.cc:276] 0709 12:19:16.400952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.401135 (+   183us) service_pool.cc:224] Handling call
0709 12:19:16.409256 (+  8121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:16.454908 (+ 45652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.447032 (+992124us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b54006bc6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:17.447290 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103194880}
W20260709 12:19:17.526578 27580 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.065s	sys 0.069s
I20260709 12:19:17.508262 27588 rpcz_store.cc:276] 0709 12:19:15.991841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.336393 (+344552us) service_pool.cc:224] Handling call
0709 12:19:16.641369 (+304976us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:16.641630 (+   261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.642100 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:16.642459 (+   359us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.360584 (+718125us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:17.382521 (+ 21937us) log.cc:844] Serialized 26 byte log entry
0709 12:19:17.382715 (+   194us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:17.382763 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.382870 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.428899 (+ 46029us) raft_consensus.cc:1685] finished
0709 12:19:17.506324 (+ 77425us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.507690 (+  1366us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":940,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":2720,"raft.queue_time_us":3966,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":6261,"spinlock_wait_cycles":83837312,"thread_start_us":717618,"threads_started":1,"wal-append.queue_time_us":772271}
I20260709 12:19:17.528668 27659 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:17.499385 27585 rpcz_store.cc:276] 0709 12:19:16.406042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.870248 (+464206us) service_pool.cc:224] Handling call
0709 12:19:16.870575 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.901669 (+ 31094us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.901722 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.968075 (+ 66353us) log.cc:844] Serialized 26 byte log entry
0709 12:19:17.317921 (+349846us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:17.317991 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.318099 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.498509 (+180410us) raft_consensus.cc:1685] finished
0709 12:19:17.498538 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.498708 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30618,"raft.queue_time_us":31150,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":14081,"thread_start_us":317993,"threads_started":1,"wal-append.queue_time_us":513815}
I20260709 12:19:17.353137 31075 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.427922 27581 rpcz_store.cc:276] 0709 12:19:15.301440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.315004 (+ 13564us) service_pool.cc:224] Handling call
0709 12:19:15.356184 (+ 41180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:16.877084 (+1520900us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400204a20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:19:17.359527 (+482443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.359812 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066878336}
I20260709 12:19:17.385635 30534 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:17.379597 31018 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.448726 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1670) took 1460 ms. Trace:
I20260709 12:19:17.539472 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 446) took 1524 ms. Trace:
I20260709 12:19:17.553742 27662 rpcz_store.cc:276] 0709 12:19:16.015286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.507498 (+1492212us) service_pool.cc:224] Handling call
0709 12:19:17.538831 (+ 31333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.539066 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.539103 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.539156 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:17.539201 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.539268 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.539406 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.470818 27582 rpcz_store.cc:276] 0709 12:19:15.402861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.411356 (+  8495us) service_pool.cc:224] Handling call
0709 12:19:15.424200 (+ 12844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:15.424401 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:15.459010 (+ 34609us) log.cc:844] Serialized 14 byte log entry
0709 12:19:15.856747 (+397737us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b180032b9c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:17.469561 (+1612814us) spinlock_profiling.cc:243] Waited 6.14 ms on lock 0x7b180032b9c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287
0709 12:19:17.469789 (+   228us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.469861 (+    72us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:17.469936 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.470044 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.470200 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10286,"prepare.queue_time_us":38952,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":6391,"raft.queue_time_us":21495,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":51676,"spinlock_wait_cycles":21215872,"thread_start_us":368467,"threads_started":2,"wal-append.queue_time_us":876353,"wal-append.run_cpu_time_us":1931,"wal-append.run_wall_time_us":1111900}
W20260709 12:19:17.535493 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 188030us
W20260709 12:19:17.454983 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:17.530110 27580 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.065s	sys 0.069s
I20260709 12:19:17.531299 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:17.532344 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.551098 27664 rpcz_store.cc:276] 0709 12:19:15.988752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.407138 (+418386us) service_pool.cc:224] Handling call
0709 12:19:17.448444 (+1041306us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:17.448614 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287146752}
I20260709 12:19:17.563181 27664 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:17.563884 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 448) took 1547 ms. Trace:
I20260709 12:19:17.564301 27664 rpcz_store.cc:276] 0709 12:19:16.016076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.562719 (+1546643us) service_pool.cc:224] Handling call
0709 12:19:17.563020 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.563641 (+   621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.563833 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.532677 27584 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
I20260709 12:19:17.524907 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:17.438535 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.543032 30986 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.488514 31052 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.543958 30766 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.530123 31107 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:17.546229 27702 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.011s	sys 0.064s
I20260709 12:19:17.534343 31113 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:17.431098 27658 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:19:17.384893 31124 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.241s	user 0.000s	sys 0.009s
I20260709 12:19:17.552196 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:17.565834 27664 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:17.536984 31075 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.594928 28329 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:17.567952 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.620697 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.521677 31059 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:17.521762 31071 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:17.628105 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 129703us
I20260709 12:19:17.572525 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.629424 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:17.576259 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.578043 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:17.578526 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53721.6us
I20260709 12:19:17.579263 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.522418 31097 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:17.499754 31131 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.624926 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2023) took 2189 ms. Trace:
I20260709 12:19:17.633358 27658 rpcz_store.cc:276] 0709 12:19:15.435892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.423003 (+1987111us) service_pool.cc:224] Handling call
0709 12:19:17.430706 (+  7703us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.624590 (+193884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.624863 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.595366 28334 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:17.568737 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:17.621111 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1707) took 2167 ms (client timeout 2147 ms). Trace:
I20260709 12:19:17.566416 31018 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:17.625550 27589 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:17.625886 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 453) took 1608 ms. Trace:
I20260709 12:19:17.566473 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2025) took 2077 ms. Trace:
I20260709 12:19:17.639178 27664 rpcz_store.cc:276] 0709 12:19:16.017661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.565261 (+1547600us) service_pool.cc:224] Handling call
0709 12:19:17.565618 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.624682 (+ 59064us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.625832 (+  1150us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:17.592327 27702 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.012s	sys 0.065s
W20260709 12:19:17.640549 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 796922us
I20260709 12:19:17.632086 28295 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
I20260709 12:19:17.641299 28295 leader_election.cc:403] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:19:17.599095 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.598541 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.644284 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1722) took 1697 ms. Trace:
I20260709 12:19:17.644933 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.644644 27730 rpcz_store.cc:276] 0709 12:19:15.946812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.174784 (+227972us) service_pool.cc:224] Handling call
0709 12:19:16.174997 (+   213us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.548779 (+373782us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.548828 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.576042 (+ 27214us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.860967 (+284925us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:16.861312 (+   345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.862365 (+  1053us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.644074 (+781709us) raft_consensus.cc:1685] finished
0709 12:19:17.644102 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.644231 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":375632,"raft.queue_time_us":352411,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":20040,"thread_start_us":226815,"threads_started":1,"wal-append.queue_time_us":461036}
I20260709 12:19:17.636691 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:17.647660 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:17.648373 27587 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:17.647214 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:17.654659 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:17.595916 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.634806 28330 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
W20260709 12:19:17.634754 27735 rpcz_store.cc:269] 0709 12:19:15.453969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.473003 (+ 19034us) service_pool.cc:224] Handling call
0709 12:19:17.620759 (+2147756us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:17.620960 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641697024}
I20260709 12:19:17.588001 31052 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:17.639302 27659 rpcz_store.cc:276] 0709 12:19:15.489067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.436380 (+1947313us) service_pool.cc:224] Handling call
0709 12:19:17.528426 (+ 92046us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.566140 (+ 37714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.566404 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.627293 30986 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:17.717634 30986 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.718179 30986 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:17.722690 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.631946 27588 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:17.724012 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.724261 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.725406 27588 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:17.726970 27588 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:17.659446 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.727671 28329 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
W20260709 12:19:17.728693 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.729970 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:17.730724 30766 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:17.731076 30766 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:17.731464 30766 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:17.732143 27663 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:19:17.732800 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1675) took 1227 ms. Trace:
I20260709 12:19:17.733111 27663 rpcz_store.cc:276] 0709 12:19:16.505050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.731583 (+1226533us) service_pool.cc:224] Handling call
0709 12:19:17.731902 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:17.732536 (+   634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.732731 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.734611 27663 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[5.2-10.5]   Dedup: 5.2->[7.3-10.5]
I20260709 12:19:17.660648 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 454) took 1162 ms. Trace:
I20260709 12:19:17.661002 27664 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:19:17.737756 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74099.2us
I20260709 12:19:17.738355 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1671) took 1239 ms. Trace:
I20260709 12:19:17.728729 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.738777 27664 rpcz_store.cc:276] 0709 12:19:16.499194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.640489 (+1141295us) service_pool.cc:224] Handling call
0709 12:19:17.640880 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.738114 (+ 97234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.738304 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.573382 30635 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:19:17.724838 27583 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:17.598912 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:17.652674 31110 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.033s	sys 0.067s
I20260709 12:19:17.596848 30987 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:17.741050 31110 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.034s	sys 0.067s
I20260709 12:19:17.741325 30987 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:17.742612 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:17.668529 30873 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:17.671521 30975 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:17.677049 30827 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.599606 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:17.678510 31125 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.322s	user 0.000s	sys 0.001s
I20260709 12:19:17.679121 31021 raft_consensus.cc:2764] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:19:17.679522 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:17.679832 27702 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:17.680743 31075 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.634002 31113 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:17.691749 27585 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:17.692915 31131 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:17.628741 31107 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:17.659435 30978 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
W20260709 12:19:17.704648 31067 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.065s	user 0.007s	sys 0.010s
I20260709 12:19:17.705368 31059 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:17.637396 31071 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.706719 31012 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:17.619796 31123 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:17.708612 31064 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:17.711268 30900 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:17.665534 27659 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:19:17.782099 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1672) took 1280 ms. Trace:
I20260709 12:19:17.782554 27659 rpcz_store.cc:276] 0709 12:19:16.501860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.664771 (+1162911us) service_pool.cc:224] Handling call
0709 12:19:17.665100 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.781734 (+116634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.782023 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.740237 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.743809 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.744294 27663 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [5.2-10.5]
I20260709 12:19:17.784392 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1678) took 1275 ms. Trace:
I20260709 12:19:17.784988 27663 rpcz_store.cc:276] 0709 12:19:16.509072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.733873 (+1224801us) service_pool.cc:224] Handling call
0709 12:19:17.734343 (+   470us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:19:17.784000 (+ 49657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.784310 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:17.749212 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:17.752456 27658 rpcz_store.cc:276] 0709 12:19:16.497784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.634323 (+1136539us) service_pool.cc:224] Handling call
0709 12:19:17.660159 (+ 25836us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.660331 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.660356 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.660401 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:17.660431 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.660487 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.660594 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:17.787376 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47166.2us
I20260709 12:19:17.733848 31094 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:17.791373 27658 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:17.793043 27658 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:17.795817 27658 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:17.796662 30827 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:17.806991 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1720) took 1860 ms. Trace:
I20260709 12:19:17.807444 27734 rpcz_store.cc:276] 0709 12:19:15.946278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.169790 (+223512us) service_pool.cc:224] Handling call
0709 12:19:16.170139 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.578431 (+408292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.578475 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.579095 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.970295 (+391200us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:16.970335 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.970402 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.806785 (+836383us) raft_consensus.cc:1685] finished
0709 12:19:17.806806 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.806931 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":425542,"raft.queue_time_us":418419,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6386,"spinlock_wait_cycles":3968,"thread_start_us":381744,"threads_started":1,"wal-append.queue_time_us":524684}
W20260709 12:19:17.810606 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61479.3us
I20260709 12:19:17.766374 31031 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:17.811887 31064 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:17.812960 28292 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10.
I20260709 12:19:17.812865 31064 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.813977 27584 pending_rounds.cc:85] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 2
I20260709 12:19:17.814136 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:17.816819 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.818097 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:17.762420 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 82715.7us
I20260709 12:19:17.836094 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:17.837575 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:17.839267 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.845160 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:17.763068 27589 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:17.765439 27588 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:17.736738 30988 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:19:17.760130 30975 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:17.740629 30638 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.894s	user 0.013s	sys 0.000s
I20260709 12:19:17.818992 31012 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:17.819576 31121 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:17.821569 28332 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:17.811017 30991 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:17.870824 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.871675 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:17.870745 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.775650 31067 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:17.811888 31016 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:17.829061 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2020) took 2419 ms. Trace:
I20260709 12:19:17.849146 31094 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:17.882923 27655 rpcz_store.cc:276] 0709 12:19:15.410124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.374612 (+1964488us) service_pool.cc:224] Handling call
0709 12:19:17.374997 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.429356 (+ 54359us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:17.429737 (+   381us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.764574 (+334837us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.764650 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:17.764715 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.828058 (+ 63343us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b18000c0040. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:17.828785 (+   727us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:17.828837 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.828982 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6305,"prepare.queue_time_us":131,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":21972,"raft.queue_time_us":9992,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":8992,"spinlock_wait_cycles":128767488,"thread_start_us":324442,"threads_started":1}
W20260709 12:19:17.885531 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.888689 27587 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:17.814325 27584 pending_rounds.cc:107] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:19:17.890725 27587 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:17.891007 27584 raft_consensus.cc:2889] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:17.849799 31054 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:17.852200 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.853816 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.855162 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.864996 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.869812 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:17.902145 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.870793 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.822881 31071 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:17.878489 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.907954 27729 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:17.879724 27738 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:19:17.832497 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 394050us
W20260709 12:19:17.911185 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.899827 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.895236 27589 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:17.861809 31031 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:17.915068 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:17.902096 31046 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.004s
I20260709 12:19:17.904122 27732 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:19:17.923213 28330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:17.941905 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.942548 28336 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:17.948359 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:17.942839 28337 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:17.944784 28328 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:17.905581 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.867609 30873 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:17.954843 31071 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:17.958065 31071 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:17.941752 30975 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.910962 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:17.871201 30983 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:19:17.900174 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:17.917268 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2027) took 1687 ms. Trace:
I20260709 12:19:17.868327 30668 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:17.918782 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1676) took 1412 ms. Trace:
I20260709 12:19:17.920058 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2026) took 1690 ms. Trace:
I20260709 12:19:17.920817 27663 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:19:17.892241 31016 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.943189 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:17.904910 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:17.912012 27737 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:17.904107 31054 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.893392 31121 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:17.903573 31050 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:17.991976 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:17.962559 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51693.6us
I20260709 12:19:17.962976 27733 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:17.972831 27586 rpcz_store.cc:276] 0709 12:19:16.230107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.869226 (+639119us) service_pool.cc:224] Handling call
0709 12:19:16.869592 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.475559 (+605967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.917043 (+441484us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:17.917208 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385036160}
I20260709 12:19:17.978091 27661 rpcz_store.cc:276] 0709 12:19:16.506021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.875287 (+369266us) service_pool.cc:224] Handling call
0709 12:19:17.918602 (+1043315us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:17.918721 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123383552}
I20260709 12:19:17.978858 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:18.015170 27586 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:17.912305 30638 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:17.972206 28334 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:19:18.015607 27586 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:17.943975 31138 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:19:17.881402 30766 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.917536 30892 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:17.978157 27580 rpcz_store.cc:276] 0709 12:19:16.229762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.589938 (+360176us) service_pool.cc:224] Handling call
0709 12:19:16.644525 (+ 54587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:16.644779 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:16.992661 (+347882us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.567559 (+574898us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.916109 (+348550us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.916174 (+    65us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:17.916234 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.916321 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.919990 (+  3669us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":104271,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":81,"raft.queue_time_us":97914,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"thread_start_us":1233438,"threads_started":3,"wal-append.queue_time_us":776694}
I20260709 12:19:17.987628 31016 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:18.033602 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:18.016914 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:18.018105 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 113281us
I20260709 12:19:18.021086 27589 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 4. Current term is 5. Ops: []
W20260709 12:19:18.021311 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.977869 30668 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:17.982301 31026 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:18.021867 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.040237 27737 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:17.912950 30978 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.076s
W20260709 12:19:18.023520 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.026592 27586 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:17.959199 31071 consensus_queue.cc:1243] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:19:17.915475 31143 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260709 12:19:18.011425 31054 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:18.016198 31072 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.046928 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:17.989404 31046 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:19:18.024302 31005 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.051520 27737 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:19:18.022368 30647 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:18.028417 31139 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.039958 31026 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.056422 31143 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:18.056855 31026 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:18.056874 31143 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:18.057353 31052 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.058151 31143 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.059764 31143 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:18.059633 31121 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.066428 31067 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:18.054497 31123 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:19:18.018111 31050 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:19:18.071647 31005 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:18.072279 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1709) took 2581 ms (client timeout 1613 ms). Trace:
W20260709 12:19:18.072688 27733 rpcz_store.cc:269] 0709 12:19:15.491020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.496043 (+  5023us) service_pool.cc:224] Handling call
0709 12:19:18.072071 (+2576028us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b54002e81d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:18.072211 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5641340672}
I20260709 12:19:18.073724 31050 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:18.073817 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:18.074570 31072 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.075577 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:18.076686 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.077242 27733 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:18.078019 31000 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:18.078305 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.078428 31000 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:18.079735 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.079427 31000 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.080701 31000 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:18.070990 27737 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:18.077064 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.073979 31005 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:18.085981 27586 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:18.084045 27734 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:18.086776 27737 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:18.040671 30668 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:18.022954 30988 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:19:18.040452 31107 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:18.047134 31043 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:18.042995 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:17.924322 31113 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.020269 31018 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:18.101862 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 471502us
I20260709 12:19:18.044672 31075 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
W20260709 12:19:18.101369 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58511.6us
I20260709 12:19:18.077178 30647 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:18.153640 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.093034 30999 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:18.156038 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:18.134769 31107 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:18.094499 30983 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:18.141168 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:18.102412 31059 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:18.096698 31046 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.171977 27580 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:18.160377 27731 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:18.162513 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:18.067248 31139 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.136166 31021 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.177831 27589 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:18.179944 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:18.158530 30999 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:18.185988 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 280447us
I20260709 12:19:18.190133 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:18.191638 31140 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:18.197546 31140 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:18.199867 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:18.201352 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:18.202739 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:18.227835 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 86763.2us
W20260709 12:19:18.179955 27662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.034s	sys 0.031s
W20260709 12:19:18.236038 27662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.035s	sys 0.031s
I20260709 12:19:18.171622 31043 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:18.233924 27734 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: []
I20260709 12:19:18.249852 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 447) took 2234 ms. Trace:
I20260709 12:19:18.238022 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:18.254815 27734 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:19:18.180560 30983 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:18.260344 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:18.261477 27589 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:18.265328 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2035) took 1349 ms. Trace:
I20260709 12:19:18.265738 27582 rpcz_store.cc:276] 0709 12:19:16.915724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.803572 (+887848us) service_pool.cc:224] Handling call
0709 12:19:17.803872 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.804066 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:17.804340 (+   274us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.264980 (+460640us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:18.265037 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:18.265086 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.265157 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.265274 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74998,"prepare.queue_time_us":139,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"spinlock_wait_cycles":2432,"thread_start_us":379221,"threads_started":1}
I20260709 12:19:18.269455 31018 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:18.269256 31107 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.270099 31018 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:18.270447 31018 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:18.182062 31050 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 12:19:18.273272 28335 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:18.278921 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.250279 27662 rpcz_store.cc:276] 0709 12:19:16.015727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.554934 (+1539207us) service_pool.cc:224] Handling call
0709 12:19:17.589869 (+ 34935us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.590068 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:17.629423 (+ 39355us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.249513 (+620090us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:18.249571 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:18.249615 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.249675 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.249797 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13020,"prepare.queue_time_us":43122,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":39,"thread_start_us":606638,"threads_started":1}
I20260709 12:19:18.180814 31046 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.249698 31043 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:18.271708 30999 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.297346 27586 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:19:18.302762 27586 pending_rounds.cc:85] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 8
I20260709 12:19:18.251586 31144 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:18.300905 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.303023 27586 pending_rounds.cc:107] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:19:18.304188 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2036) took 1366 ms. Trace:
I20260709 12:19:18.262090 31145 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:18.307950 27586 raft_consensus.cc:2889] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:18.308002 27657 rpcz_store.cc:276] 0709 12:19:16.938007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.974105 (+ 36098us) service_pool.cc:224] Handling call
0709 12:19:18.303990 (+1329885us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:18.304113 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2061261696}
I20260709 12:19:18.274469 30766 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:18.251502 30988 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:19:18.287030 31052 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:18.316545 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 324817us
W20260709 12:19:18.321419 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 368977us
W20260709 12:19:18.315586 31141 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.297s	user 0.001s	sys 0.001s
I20260709 12:19:18.185861 31021 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.316766 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:18.335402 27552 connection.cc:256] RPC call timeout handler was delayed by -1.344185687999925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.168s for a response
W20260709 12:19:18.315189 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.302335 31043 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:18.335261 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 142751us
W20260709 12:19:18.318387 30986 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.600s	user 0.012s	sys 0.000s
I20260709 12:19:18.338111 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:18.341997 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:18.311239 31145 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:18.343912 27664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.024s	sys 0.034s
W20260709 12:19:18.316704 31131 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
W20260709 12:19:18.348707 27664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.024s	sys 0.034s
I20260709 12:19:18.346988 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:18.271245 31018 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.355895 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:18.356788 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:18.350513 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:18.359897 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.326444 30999 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:18.358935 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 156284us
I20260709 12:19:18.362437 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:18.363776 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:18.360611 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:18.365733 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.317190 31059 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:19:18.343497 30992 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.368263 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:18.369136 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.370395 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:18.372836 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 337203us
I20260709 12:19:18.374133 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:18.375715 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2031) took 1498 ms. Trace:
I20260709 12:19:18.355470 30635 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:19:18.409941 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:18.414386 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52545us
I20260709 12:19:18.350899 31145 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:18.442232 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1682) took 1893 ms. Trace:
I20260709 12:19:18.367216 31018 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:18.314175 31144 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.442675 27659 rpcz_store.cc:276] 0709 12:19:16.548510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.785270 (+1236760us) service_pool.cc:224] Handling call
0709 12:19:17.831119 (+ 45849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.831404 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:19:17.862332 (+ 30928us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.441758 (+579426us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:18.441865 (+   107us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:18.441951 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.442038 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.442171 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31269,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4640,"thread_start_us":578890,"threads_started":2,"wal-append.queue_time_us":505477}
I20260709 12:19:18.436004 30912 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:18.378623 27583 rpcz_store.cc:276] 0709 12:19:16.877504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.763885 (+886381us) service_pool.cc:224] Handling call
0709 12:19:17.777703 (+ 13818us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.777944 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:17.789754 (+ 11810us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.375218 (+585464us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:18.375298 (+    80us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:18.375381 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.375476 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.375638 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42443,"prepare.queue_time_us":12292,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":73684,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":6458,"thread_start_us":542424,"threads_started":2,"wal-append.queue_time_us":556797}
I20260709 12:19:18.367286 31149 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:18.364883 30647 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:18.357957 30986 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:18.499931 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.439303 31156 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:19:18.530684 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90367.2us
W20260709 12:19:18.526644 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 158432us
I20260709 12:19:18.446903 31075 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:18.452292 30534 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:18.455597 30912 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.454689 30988 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:18.445734 31155 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:18.445824 31012 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.097s
W20260709 12:19:18.549634 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 185962us
I20260709 12:19:18.552196 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:18.454948 31149 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:18.557070 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:18.554216 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 227212us
I20260709 12:19:18.560526 27659 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:18.560544 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:18.565908 27658 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
W20260709 12:19:18.566568 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.453279 31059 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:18.415103 30992 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.572891 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:18.575840 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:18.556646 31146 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.307s	user 0.001s	sys 0.000s
I20260709 12:19:18.587563 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:18.598387 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:18.547996 31075 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:18.608253 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:18.561558 30912 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:18.573618 31031 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:18.638688 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54423.1us
I20260709 12:19:18.643225 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:18.643887 27659 raft_consensus.cc:2376] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11. Re-sending same reply.
I20260709 12:19:18.575053 30534 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:18.588186 31147 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.588335 31149 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:18.591919 31059 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.662743 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:18.663812 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:18.625507 31075 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:18.619848 30988 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:18.639746 31052 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:18.668326 31107 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260709 12:19:18.669375 31151 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.309s	user 0.000s	sys 0.000s
I20260709 12:19:18.642403 31156 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:18.679883 31148 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.312s	user 0.001s	sys 0.000s
I20260709 12:19:18.660480 30980 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:18.681870 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:18.729570 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47781.6us
I20260709 12:19:18.682482 31155 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:18.686427 31004 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:18.665431 31059 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:18.746918 31016 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.703s	user 0.009s	sys 0.000s
I20260709 12:19:18.747704 30534 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.677619 30988 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:18.752063 30988 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.753304 30988 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:18.683105 31021 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:18.692849 31046 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:18.667909 27659 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:19:18.756592 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:18.757020 31026 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.700s	user 0.009s	sys 0.003s
I20260709 12:19:18.757184 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:18.757429 31026 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.648277 31005 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:18.709690 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:18.758560 27580 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:19:18.758836 27655 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:18.655227 31031 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.619619 31162 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260709 12:19:18.738696 31176 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:18.739442 30980 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.740417 31155 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:18.740504 30986 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:18.743477 31174 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 12:19:18.743799 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:18.666716 31147 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.644655 31159 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 12:19:18.647828 31173 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:19:18.727275 31143 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.662s	user 0.006s	sys 0.006s
I20260709 12:19:18.766752 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2021) took 3347 ms. Trace:
I20260709 12:19:18.802507 27656 rpcz_store.cc:276] 0709 12:19:15.419070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.381794 (+1962724us) service_pool.cc:224] Handling call
0709 12:19:17.393793 (+ 11999us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:17.394324 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.394389 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:17.428011 (+ 33622us) log.cc:844] Serialized 26 byte log entry
0709 12:19:17.866945 (+438934us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:17.866989 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.867066 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:18.766451 (+899385us) raft_consensus.cc:1685] finished
0709 12:19:18.766480 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.766678 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":291027,"threads_started":1,"wal-append.queue_time_us":607858}
I20260709 12:19:18.709095 30766 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:18.759189 31139 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:18.775779 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 191766us
I20260709 12:19:18.717900 31161 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.100s
I20260709 12:19:18.818243 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:18.825110 31021 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.819067 31165 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 12:19:18.819345 30766 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.830061 30766 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:18.729429 31052 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.830978 31155 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.831840 31052 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.833016 31145 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.820839 31139 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:19:18.836917 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2036) took 1912 ms. Trace:
I20260709 12:19:18.837378 27584 rpcz_store.cc:276] 0709 12:19:16.924934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.813451 (+888517us) service_pool.cc:224] Handling call
0709 12:19:17.813796 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:17.906935 (+ 93139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.906972 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:17.929177 (+ 22205us) log.cc:844] Serialized 52 byte log entry
0709 12:19:18.312419 (+383242us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:19:18.312474 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.314998 (+  2524us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:18.836683 (+521685us) raft_consensus.cc:1685] finished
0709 12:19:18.836704 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.836854 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":323,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":10892,"thread_start_us":290171,"threads_started":1,"wal-append.queue_time_us":784595}
W20260709 12:19:18.789539 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 181629us
I20260709 12:19:18.839190 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1680) took 2327 ms. Trace:
I20260709 12:19:18.839838 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:18.840939 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:18.839617 27664 rpcz_store.cc:276] 0709 12:19:16.511467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.739680 (+1228213us) service_pool.cc:224] Handling call
0709 12:19:17.739992 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:17.748279 (+  8287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.748326 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:17.760494 (+ 12168us) log.cc:844] Serialized 26 byte log entry
0709 12:19:18.351431 (+590937us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:18.351468 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.351538 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:18.838981 (+487443us) raft_consensus.cc:1685] finished
0709 12:19:18.839000 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.839122 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":802,"raft.queue_time_us":1333,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":8050,"thread_start_us":542136,"threads_started":1,"wal-append.queue_time_us":1042525}
I20260709 12:19:18.805691 27655 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:18.844080 31147 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:18.844659 31147 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:18.845587 31156 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:18.791476 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 447645us
W20260709 12:19:18.847983 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 464) took 1307 ms (client timeout 1507 ms). Trace:
I20260709 12:19:18.830106 30992 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:18.827523 31059 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.802774 31026 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:18.857921 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:18.773528 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 210362us
W20260709 12:19:18.848419 27585 rpcz_store.cc:269] 0709 12:19:17.540870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.567694 (+ 26824us) service_pool.cc:224] Handling call
0709 12:19:18.847836 (+1280142us) spinlock_profiling.cc:243] Waited 65.3 ms on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:18.847930 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150083328}
W20260709 12:19:18.799356 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 425259us
I20260709 12:19:18.842006 30912 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:18.808509 28330 raft_consensus.cc:606] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:18.785470 31176 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:18.761425 31016 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.835106 31145 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:18.861068 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:18.847916 27583 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:19:18.868985 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:18.783372 30980 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:18.877895 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:18.878836 27582 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:19:18.869964 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:18.891286 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:18.894608 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:18.896727 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:18.897500 27664 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:18.863759 30995 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:18.914471 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:18.820199 30668 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.715s	user 0.009s	sys 0.002s
I20260709 12:19:18.909730 28333 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:18.871480 31021 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:19:18.921023 27656 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:19:18.817353 31172 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20260709 12:19:18.888571 30992 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:19:18.931921 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 241314us
I20260709 12:19:18.888171 31144 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:18.934207 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:18.979552 27730 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:18.921751 30983 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.000s
I20260709 12:19:18.861881 31165 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:19:18.994982 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:18.995155 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 135230us
I20260709 12:19:18.911283 30995 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:18.891809 31166 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:18.995745 27730 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:18.887887 31187 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:18.997843 27664 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
W20260709 12:19:18.997752 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.010035 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:18.930743 31145 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:19.019977 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:18.990212 30983 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:18.936463 30992 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.027611 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.028680 27589 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:18.926367 31121 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:19:19.031296 27664 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:19.036849 27659 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:19.040274 27655 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:19.042928 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:19.011562 30995 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.000849 31162 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.933147 30668 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.018448 31187 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:19.033458 31054 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.946s	user 0.006s	sys 0.006s
I20260709 12:19:19.018092 31145 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:19.050755 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.021791 31147 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:19.061584 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.081229 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.083305 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.084064 27656 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.074141 27730 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:19.032318 30992 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:19.046943 30975 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:19:18.998351 31005 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:19:19.102275 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.101289 27736 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
W20260709 12:19:19.103737 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 248587us
I20260709 12:19:19.109175 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.067497 30974 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:19.078104 31145 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:19.078526 31187 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.086078 31143 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:19.102560 31189 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:19.125633 31188 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:19:19.128026 31059 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:19.129666 31018 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.607s	user 0.009s	sys 0.003s
I20260709 12:19:19.132220 31075 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:19.155872 31181 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:19:19.082540 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:19.105679 31184 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:19.115090 31031 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.199608 31184 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.202425 31184 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.163163 31199 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:19.152750 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.204012 27730 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:19.203846 31199 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.205030 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:19.205674 31000 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.125s	user 0.010s	sys 0.003s
I20260709 12:19:19.206156 31199 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.206346 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.206534 27733 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:19:19.206918 27733 pending_rounds.cc:85] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 6
I20260709 12:19:19.206996 27730 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:19.207116 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.150938 31026 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:19.207727 30980 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.207937 27664 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 5.
I20260709 12:19:19.208123 31026 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:19.082218 31054 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.157819 31185 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:19:19.212911 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.009s	sys 0.014s
I20260709 12:19:19.151561 31143 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.063797 31162 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.051904 31176 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.170405 27734 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:19:19.172925 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:19.174742 31183 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:19:19.175199 31204 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:19.175812 30912 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.177826 31157 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.226684 31054 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:19.227890 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1907) took 1374 ms (client timeout 1525 ms). Trace:
W20260709 12:19:19.182644 31043 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.823s	user 0.009s	sys 0.000s
I20260709 12:19:19.194945 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:19.196559 30668 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:19.198698 30999 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.833s	user 0.006s	sys 0.002s
I20260709 12:19:19.203217 27737 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:19.159443 30974 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:19.142997 27656 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:19.208858 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.207222 27733 pending_rounds.cc:107] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:19:19.236510 27733 raft_consensus.cc:2889] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:19.208123 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:19.154515 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.236552 27664 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:19.212132 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.238227 27655 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:19.190892 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.157598 31189 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.245662 31189 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.247745 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.249260 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.249898 27655 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:19.214139 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.009s	sys 0.014s
I20260709 12:19:19.251801 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:19.252620 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 462) took 2147 ms. Trace:
I20260709 12:19:19.252985 27588 rpcz_store.cc:276] 0709 12:19:17.105689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.863366 (+757677us) service_pool.cc:224] Handling call
0709 12:19:17.863701 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.863909 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:19:17.864251 (+   342us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.695263 (+831012us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b180033f6a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:19.252263 (+557000us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:19.252321 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:19.252362 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.252432 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:19.252566 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":115,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":4518,"spinlock_wait_cycles":9135360,"thread_start_us":556757,"threads_started":1}
I20260709 12:19:19.210122 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:19.254698 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44668.7us
I20260709 12:19:19.238890 31043 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.218092 31143 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.223570 31075 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.257488 31043 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.220219 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.224462 31162 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.260217 30975 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:19.260762 30975 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.261178 30975 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:19.261606 30974 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.225668 27735 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:19.226164 31157 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:19.228523 27731 rpcz_store.cc:269] 0709 12:19:17.853398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.853529 (+   131us) service_pool.cc:224] Handling call
0709 12:19:19.227618 (+1374089us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.227809 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":594375936}
I20260709 12:19:19.233613 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.238231 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.251456 31026 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.202859 31149 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.334524 30892 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:19.230784 31204 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.235747 30827 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.335889 30827 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.338095 27664 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:19.334677 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101153us
I20260709 12:19:19.339883 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.340828 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:19.343254 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.269855 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:19.326058 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:19.346613 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 76889us
I20260709 12:19:19.346992 27731 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:19.306851 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.238540 30912 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:19.236212 31187 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.409907 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:19.411520 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68390.4us
I20260709 12:19:19.410858 27656 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:19.412624 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.416967 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.341070 30892 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.350764 31156 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:19.288110 31162 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:19.420042 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.426851 27664 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:19.428043 27655 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:19.377017 31097 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:19.381678 31204 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.465502 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1734) took 1187 ms. Trace:
I20260709 12:19:19.465342 31162 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.465941 27737 rpcz_store.cc:276] 0709 12:19:18.278500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.278641 (+   141us) service_pool.cc:224] Handling call
0709 12:19:19.465342 (+1186701us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b540017f2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.465446 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2123891072}
I20260709 12:19:19.425782 31174 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:19.467834 31162 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.447742 27659 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:19.425475 31131 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:19.469350 31131 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.470268 31145 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:19.437840 31192 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.334s	user 0.003s	sys 0.000s
I20260709 12:19:19.472476 31097 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:19.473467 27656 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:19.473896 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.449537 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.464226 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.475502 27658 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:19.459766 28329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 567) took 1846 ms (client timeout 2190 ms). Trace:
I20260709 12:19:19.457957 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.407096 30912 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.412582 31075 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:19:19.478931 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100995us
I20260709 12:19:19.467911 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.489501 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.489811 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.490120 27658 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:19.480433 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.481278 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.467039 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.491655 27737 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:19.476945 28329 rpcz_store.cc:269] 0709 12:19:17.613161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.743548 (+130387us) service_pool.cc:224] Handling call
0709 12:19:19.458922 (+1715374us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b54006133a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34be18 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6
0709 12:19:19.459585 (+   663us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.459711 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2753037312}
I20260709 12:19:19.483911 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.495550 27656 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:19.496511 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.497138 27656 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:19.499097 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.499608 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:19.500375 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.501291 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:19.501883 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:19.502061 28336 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:19.421947 30986 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:19.439281 31133 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.811s	user 0.004s	sys 0.004s
I20260709 12:19:19.507714 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.507794 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:19.505169 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.450404 31072 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.510848 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.456082 30635 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:19.512741 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.448674 31187 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.478466 31174 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.454192 31147 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:19.512964 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 549) took 1914 ms (client timeout 1677 ms). Trace:
W20260709 12:19:19.523111 28335 rpcz_store.cc:269] 0709 12:19:17.598738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.598832 (+    94us) service_pool.cc:224] Handling call
0709 12:19:19.512775 (+1913943us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b5400611054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.512890 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1448618240}
I20260709 12:19:19.513384 27731 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:19.510321 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:19.515658 31191 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:19.427682 31201 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.503104 30912 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:19.512524 31131 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.433012 30892 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.452346 31157 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.504745 31140 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.524114 31147 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.525190 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:19.510306 30638 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:19.456012 30992 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.517848 30635 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.562480 30992 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.564395 28332 raft_consensus.cc:2368] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:19.527077 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.516505 31121 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:19.528344 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:19.567598 30638 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:19.568010 31046 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:19.529109 27737 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:19.529778 28337 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:19.518644 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.507196 30986 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.570891 30986 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.532393 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.572364 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.573632 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.567622 31121 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:19.574779 31121 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.576804 30827 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.574805 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.530090 31201 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.538276 31140 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.460249 30995 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.553061 30892 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.556614 31149 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.560041 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:19.528139 31054 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:19.566784 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:19.568476 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:19.503257 30766 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:19:19.578624 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.534250 31145 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.585922 27658 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:19.579906 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:19.520689 30975 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.588061 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.588750 27731 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:19.589601 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.563597 31143 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.590202 27730 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:19.587545 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.593501 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.594049 27731 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:19.594767 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.594790 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:19.596567 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.599550 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:19.599606 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.601328 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.598043 31149 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:19.589851 31201 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.602972 31149 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:19.603531 31149 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:19.600466 31207 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:19.604702 31207 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:19.605669 31207 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.584059 31072 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:19.598311 31018 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.602278 31004 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:19:19.607833 31018 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.608826 31004 consensus_queue.cc:1243] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:19.610466 31018 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.592229 31054 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:19.612407 30987 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:19.612847 30987 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:19.613752 30987 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.614755 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 459) took 1730 ms. Trace:
I20260709 12:19:19.618108 30995 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.611912 31054 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.618726 30995 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.585573 31205 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:19.620009 31205 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.620433 31205 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:19.615309 27733 rpcz_store.cc:276] 0709 12:19:17.884667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.087409 (+202742us) service_pool.cc:224] Handling call
0709 12:19:18.099555 (+ 12146us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:19.237107 (+1137552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.614460 (+377353us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:19.614680 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2544910720}
I20260709 12:19:19.589844 31140 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.523701 31174 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.619220 30995 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.601007 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.614805 30987 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.639014 30638 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:19.639544 30638 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.639967 30638 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.636597 31075 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:19.641288 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1744) took 1313 ms. Trace:
I20260709 12:19:19.641541 31075 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.642316 27729 rpcz_store.cc:276] 0709 12:19:18.327671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.591122 (+1263451us) service_pool.cc:224] Handling call
0709 12:19:19.641221 (+ 50099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:19.644217 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.595067 30827 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:19.646198 30827 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.646608 30827 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:19.648777 27729 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:19.650362 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.652482 31050 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:19.653081 31050 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:19.654073 31050 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.626305 31143 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.657968 30995 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.658650 30975 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.602629 31171 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:19.628701 31174 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.659768 31171 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.660218 31171 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:19.636516 30534 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:19.675707 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:19.601892 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:19.676506 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.676942 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.654608 27729 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:19.661484 27589 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:19.661944 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:19.662298 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 160487us
I20260709 12:19:19.663511 27588 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:19.678750 27587 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:19.679615 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:19.680246 27584 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:19.681090 31147 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.675473 31059 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:19.682674 31059 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.614687 30635 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.622918 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.688524 27658 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:19.643035 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.689427 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1920) took 1100 ms. Trace:
I20260709 12:19:19.689814 27738 rpcz_store.cc:276] 0709 12:19:18.588749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.642761 (+1054012us) service_pool.cc:224] Handling call
0709 12:19:19.689371 (+ 46610us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:19.678526 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1741) took 1366 ms. Trace:
I20260709 12:19:19.690948 27658 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:19:19.690842 27729 rpcz_store.cc:276] 0709 12:19:18.312766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.652381 (+1339615us) service_pool.cc:224] Handling call
0709 12:19:19.654295 (+  1914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:19.678264 (+ 23969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.678461 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:19.684062 27586 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
W20260709 12:19:19.597319 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72195.4us
I20260709 12:19:19.693639 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:19.696422 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.698112 30995 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:19.675558 30892 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:19.701997 27658 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:19.702744 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.703666 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:19.703704 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.707020 27658 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:19.708626 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.709662 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.714277 31059 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.691895 31050 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.706411 31168 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:19.715610 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.716516 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:19.709317 31172 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:19.715915 31121 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.715678 31168 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.688670 31169 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:19.730104 27657 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:19.722363 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.732583 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.731231 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.735124 27661 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:19.706612 30892 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:19.738570 30892 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.741652 31121 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.684585 30534 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.731312 31169 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.745604 30534 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.717784 31174 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:19.707911 31143 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:19.632581 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.762784 30635 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.763969 31157 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:19.764155 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:19.764816 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132324us
I20260709 12:19:19.765745 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.766464 31140 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:19.766847 31140 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.766937 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.767273 31140 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:19.767834 31018 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:19.768218 31018 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.768548 31018 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.774573 30534 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.774946 31140 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.775113 30534 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.777165 30534 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.777261 31140 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.780766 31018 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.791458 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:19.791342 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.794375 27738 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:19.707710 30975 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:19:19.738862 31205 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:19.803048 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.803956 27732 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:19.796664 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2029) took 2861 ms. Trace:
I20260709 12:19:19.729137 31172 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:19.804952 27663 rpcz_store.cc:276] 0709 12:19:16.935039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.785964 (+850925us) service_pool.cc:224] Handling call
0709 12:19:17.919383 (+133419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.978298 (+ 58915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.796459 (+1818161us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54000e6fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.796610 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305724288}
I20260709 12:19:19.790603 30635 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.759416 30995 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:19.811236 27733 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11.
I20260709 12:19:19.808811 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.813120 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.813719 27738 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:19.796650 31174 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:19.829413 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.813218 30995 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.783638 31018 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.839375 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.847281 27655 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:19.848763 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.850070 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.850771 27655 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
W20260709 12:19:19.841140 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 628235us
I20260709 12:19:19.812463 30635 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.855445 31205 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.856411 31168 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.859429 31110 raft_consensus.cc:993] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:19:19.895699 31110 raft_consensus.cc:1081] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:19:19.896169 31016 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.862396 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:19.897732 27588 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:19.902292 31071 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:19.902827 31071 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:19.903820 31071 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.904897 31071 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:19.905066 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.863677 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:19.906032 27581 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:19.907636 31169 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.909451 31174 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.865100 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.912665 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:19.913776 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:19.914407 27655 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:19.868616 31162 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:19.870954 31147 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.870863 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:19.836777 31052 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:19.875164 27738 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
W20260709 12:19:19.892745 28333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 553) took 1990 ms (client timeout 1945 ms). Trace:
I20260709 12:19:19.895109 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:19.915293 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.915470 27581 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:19.915477 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:19.837350 30995 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:19.926877 27663 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
W20260709 12:19:19.926416 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55629.9us
W20260709 12:19:19.930162 28333 rpcz_store.cc:269] 0709 12:19:17.901881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.941522 (+ 39641us) service_pool.cc:224] Handling call
0709 12:19:19.892415 (+1950893us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b5400491bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.892640 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2203364096}
I20260709 12:19:19.937071 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:19.938934 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:19.940032 31097 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.942353 31097 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.947666 31147 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:19.948148 31147 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.948449 30635 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.948544 31147 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:19.950011 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 550) took 2351 ms (client timeout 2098 ms). Trace:
W20260709 12:19:19.950484 28331 rpcz_store.cc:269] 0709 12:19:17.598994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.599198 (+   204us) service_pool.cc:224] Handling call
0709 12:19:19.949722 (+2350524us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b5400962654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:19.949905 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4840920320}
I20260709 12:19:19.935634 31162 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.952286 30827 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.853765 30912 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.954582 31162 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.954713 30912 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:19.955514 27582 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:19.927944 31178 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:19.924937 31168 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:19:19.957285 30986 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.959851 27737 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:19.926715 31172 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.963059 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:19.976222 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.977209 27732 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:19.979417 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.954734 30827 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:19.990813 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:19.991452 27737 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:20.001539 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.001876 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:20.002869 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:20.004150 27732 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:20.003466 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:20.005995 27582 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:20.035820 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:20.005997 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:20.006677 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69833us
I20260709 12:19:20.005460 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:20.037303 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:20.015933 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.021025 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:20.038579 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:20.015851 30983 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:20.012038 30827 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.007879 27587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:56296
I20260709 12:19:20.056447 27587 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:20.056831 27587 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:20.024724 30986 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.022964 30892 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:19.964644 30638 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.005715 30995 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.033165 27655 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:19.971258 31149 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.034222 31168 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:20.032349 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 458) took 2155 ms (client timeout 1507 ms). Trace:
W20260709 12:19:20.068658 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65270us
I20260709 12:19:20.055548 30983 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:20.069742 30827 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.070631 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:20.059960 30987 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:20.060084 30635 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:19:20.056008 31162 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:20.107643 27661 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:20.076720 31157 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:19:20.057648 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.807s (SENT) [suppressed 18 similar messages]
I20260709 12:19:20.109177 27660 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:20.096894 31046 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.516s	user 0.004s	sys 0.006s
I20260709 12:19:20.095650 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:20.095806 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2026) took 1362 ms. Trace:
I20260709 12:19:20.133064 27580 rpcz_store.cc:276] 0709 12:19:18.733686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.735338 (+  1652us) service_pool.cc:224] Handling call
0709 12:19:18.739115 (+  3777us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:18.758984 (+ 19869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.095533 (+1336549us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:20.095748 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44163840}
I20260709 12:19:20.092098 27587 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.099030 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:20.096793 27736 rpcz_store.cc:269] 0709 12:19:17.877338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.942917 (+ 65579us) service_pool.cc:224] Handling call
0709 12:19:20.032103 (+2089186us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.032242 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2565857152}
I20260709 12:19:20.104350 27585 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:20.135535 27660 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:20.100093 30983 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:20.112720 30892 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.114359 28331 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:20.144538 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:20.145187 28333 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:20.115402 30995 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:20.116112 30987 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:20.118105 27582 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:20.120586 31155 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:20.122383 31149 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.093729 31110 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:20.125097 31064 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:20.125794 31097 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:20.125660 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.126423 31075 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:20.127381 27588 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:20.127967 27584 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:19:20.128424 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.130761 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.093149 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:20.107010 31159 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:20.060223 31178 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:19:20.142844 27655 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:20.114751 30827 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:20.170531 31207 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.172559 27738 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:20.106302 31168 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:20.312007 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.174955 28333 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:20.212631 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.245229 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:20.307520 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.318889 27732 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:20.320463 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 454) took 2462 ms (client timeout 1637 ms). Trace:
I20260709 12:19:20.319732 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1698) took 1278 ms. Trace:
I20260709 12:19:20.174396 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2039) took 1623 ms. Trace:
I20260709 12:19:20.322160 27655 rpcz_store.cc:276] 0709 12:19:19.041063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.142289 (+1101226us) service_pool.cc:224] Handling call
0709 12:19:20.142612 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.319451 (+176839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.319661 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:20.323056 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1752) took 1462 ms (client timeout 1819 ms). Trace:
W20260709 12:19:20.323567 27730 rpcz_store.cc:269] 0709 12:19:18.860186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.207896 (+347710us) service_pool.cc:224] Handling call
0709 12:19:20.322888 (+1114992us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54000a41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.322989 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810978176}
W20260709 12:19:20.247313 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 175774us
I20260709 12:19:20.324355 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:20.326226 27664 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 12:19:20.326895 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2054) took 1142 ms. Trace:
I20260709 12:19:20.327286 27664 rpcz_store.cc:276] 0709 12:19:19.184139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.325654 (+1141515us) service_pool.cc:224] Handling call
0709 12:19:20.325965 (+   311us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:20.326643 (+   678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.326840 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.329170 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1695) took 1289 ms. Trace:
I20260709 12:19:20.329563 27659 rpcz_store.cc:276] 0709 12:19:19.039985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.190599 (+150614us) service_pool.cc:224] Handling call
0709 12:19:20.328957 (+1138358us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.329105 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478992768}
W20260709 12:19:20.330483 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.321126 27735 rpcz_store.cc:269] 0709 12:19:17.857657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.918104 (+ 60447us) service_pool.cc:224] Handling call
0709 12:19:20.320225 (+2402121us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400316d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.320372 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2786535680}
I20260709 12:19:20.336086 27588 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:20.322629 27662 rpcz_store.cc:276] 0709 12:19:18.550679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.550844 (+   165us) service_pool.cc:224] Handling call
0709 12:19:18.572416 (+ 21572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:18.606131 (+ 33715us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:18.642719 (+ 36588us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.873408 (+230689us) log.cc:844] Serialized 14 byte log entry
0709 12:19:18.888355 (+ 14947us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:18.888428 (+    73us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:18.888486 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.437161 (+548675us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:19.540039 (+102878us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b180041c740. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:19.656245 (+116206us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b180041c740. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:20.173995 (+517750us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:20.174050 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.174329 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15255,"prepare.queue_time_us":39985,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":106,"raft.queue_time_us":31249,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":13834,"spinlock_wait_cycles":165314560,"thread_start_us":228772,"threads_started":1,"wal-append.queue_time_us":413793}
I20260709 12:19:20.325184 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:20.339872 27587 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:20.341076 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1738) took 2061 ms (client timeout 1807 ms). Trace:
W20260709 12:19:20.341552 27734 rpcz_store.cc:269] 0709 12:19:18.279871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.370129 (+ 90258us) service_pool.cc:224] Handling call
0709 12:19:20.331985 (+1961856us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.340980 (+  8995us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1837639552}
I20260709 12:19:20.294229 28335 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:20.344863 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.345985 27582 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:20.109414 31162 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:20.366505 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:20.301477 31050 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.586s	user 0.003s	sys 0.006s
I20260709 12:19:20.348296 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:42232 (request call id 1) took 2707 ms. Trace:
I20260709 12:19:20.348757 31207 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.343900 31075 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.366971 31012 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:19:20.302721 30635 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:20.441039 27729 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:20.144037 31191 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:20.148510 31046 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:20.392104 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.358429 31097 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.367058 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2056) took 1017 ms. Trace:
I20260709 12:19:20.394933 27732 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:20.367106 31162 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:20.367668 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:20.449906 27734 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:20.380395 31149 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:20.377574 28331 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:20.371770 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.368616 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.456388 27588 raft_consensus.cc:2368] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:20.370285 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:20.457186 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.377915 30900 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:19:20.405051 27702 leader_election.cc:400] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:20.405737 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2057) took 1055 ms. Trace:
I20260709 12:19:20.460932 27702 leader_election.cc:403] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:19:20.461208 27657 rpcz_store.cc:276] 0709 12:19:19.350469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.478398 (+127929us) service_pool.cc:224] Handling call
0709 12:19:20.405506 (+927108us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54000e60d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.405676 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":509362560}
I20260709 12:19:20.408041 27660 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:20.462780 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2062) took 1043 ms. Trace:
I20260709 12:19:20.462929 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.463160 27660 rpcz_store.cc:276] 0709 12:19:19.419909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.407400 (+987491us) service_pool.cc:224] Handling call
0709 12:19:20.407817 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.462518 (+ 54701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.462728 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.408721 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.457113 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:20.459774 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.420549 27737 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:20.465408 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.385389 31113 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:19:20.470475 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.380677 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:20.305756 31171 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.645s	user 0.002s	sys 0.008s
I20260709 12:19:20.367792 27736 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:20.379530 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 464) took 1927 ms. Trace:
I20260709 12:19:20.428799 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1694) took 1415 ms. Trace:
I20260709 12:19:20.379010 31157 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260709 12:19:20.429833 27583 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:20.435465 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.479158 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.480333 27660 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:20.435633 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.322113 31155 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:20.443425 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:20.482257 27730 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:20.482355 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:20.320820 30638 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.450611 27732 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:20.377264 27733 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:20.448818 27659 rpcz_store.cc:276] 0709 12:19:19.349666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.330485 (+980819us) service_pool.cc:224] Handling call
0709 12:19:20.366458 (+ 35973us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.366680 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.366715 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.366760 (+    45us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:20.366797 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.366869 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.366989 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.367168 30992 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:20.454144 27731 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:20.454517 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:20.489715 28329 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:20.384207 31110 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:20.289805 31147 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.405913 27656 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:20.461517 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56643.3us
I20260709 12:19:20.464847 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.409652 30983 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:20.504271 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.505419 27729 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:19:20.506422 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1922) took 1882 ms. Trace:
I20260709 12:19:20.412775 31168 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.465538 28337 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:20.465835 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 465) took 1932 ms. Trace:
I20260709 12:19:20.509356 27737 rpcz_store.cc:276] 0709 12:19:18.533168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.419874 (+1886706us) service_pool.cc:224] Handling call
0709 12:19:20.420279 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.465565 (+ 45286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.465778 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.422801 28330 rpcz_store.cc:276] 0709 12:19:17.614782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.900329 (+285547us) service_pool.cc:224] Handling call
0709 12:19:20.322098 (+2421769us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.322179 (+    81us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2024952192}
I20260709 12:19:20.380400 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:20.467195 27738 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:20.506759 27729 rpcz_store.cc:276] 0709 12:19:18.624291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.504849 (+1880558us) service_pool.cc:224] Handling call
0709 12:19:20.505196 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.505862 (+   666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.506360 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.339010 31159 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.387159 30766 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 12:19:20.388475 30987 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.470715 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2064) took 1042 ms. Trace:
I20260709 12:19:20.442327 30635 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:20.548182 30635 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.549551 30635 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:20.551972 28334 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:20.386572 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.449400 31225 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:20.555171 31225 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:20.555401 27587 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:20.556375 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.556087 31225 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.557260 31225 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:20.557577 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.472154 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:20.476116 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 418455us
I20260709 12:19:20.386340 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.476379 27661 rpcz_store.cc:276] 0709 12:19:19.013297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.736298 (+723001us) service_pool.cc:224] Handling call
0709 12:19:19.746606 (+ 10308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.108267 (+361661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.428407 (+320140us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b540022b354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:20.428683 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":829801216}
I20260709 12:19:20.476267 27735 rpcz_store.cc:276] 0709 12:19:18.452402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.378646 (+1926244us) service_pool.cc:224] Handling call
0709 12:19:20.378963 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.379185 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.379214 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.379257 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:20.379295 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.379372 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.379476 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.476893 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:20.484117 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.569063 28332 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:20.569675 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.485888 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1746) took 1879 ms. Trace:
I20260709 12:19:20.486315 27734 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:20.487020 27730 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:20.458974 31012 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.571493 27732 rpcz_store.cc:276] 0709 12:19:18.606768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.449979 (+1843211us) service_pool.cc:224] Handling call
0709 12:19:20.450363 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.485622 (+ 35259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.485830 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.572468 31012 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.573244 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:20.573933 27732 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:20.574847 31012 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:20.575412 30766 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:20.487946 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.465984 31046 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:20.576730 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.576895 31097 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.577924 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.578774 31207 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:20.497279 27581 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:20.580416 27581 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:20.580904 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1747) took 1973 ms. Trace:
I20260709 12:19:20.580987 31178 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:20.581432 31178 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:20.581346 27730 rpcz_store.cc:276] 0709 12:19:18.607156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.483025 (+1875869us) service_pool.cc:224] Handling call
0709 12:19:20.486840 (+  3815us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.580622 (+ 93782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.580843 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.501772 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:20.582594 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:20.582360 31178 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.510891 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1745) took 1904 ms. Trace:
I20260709 12:19:20.571152 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.583587 27738 rpcz_store.cc:276] 0709 12:19:18.606377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.449327 (+1842950us) service_pool.cc:224] Handling call
0709 12:19:20.466849 (+ 17522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.510648 (+ 43799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.510842 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.584868 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:20.584982 27588 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
W20260709 12:19:20.586652 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84997.1us
I20260709 12:19:20.338734 31064 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.587852 31205 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:20.485323 31155 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:20.588663 31205 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:20.460206 31149 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.520233 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1704) took 1374 ms. Trace:
I20260709 12:19:20.521220 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 1936) took 1023 ms. Trace:
I20260709 12:19:20.524240 27657 rpcz_store.cc:276] 0709 12:19:19.428346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.462207 (+1033861us) service_pool.cc:224] Handling call
0709 12:19:20.470173 (+  7966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.470380 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.470407 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.470453 (+    46us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:20.470489 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.470560 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.470663 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.526142 31110 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.528398 31147 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.478708 31171 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.613655 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:20.530043 31075 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.531773 27580 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:20.532109 31159 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:20.525234 31054 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 12:19:20.538674 28329 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:20.540468 27736 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:20.571916 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1748) took 1964 ms. Trace:
I20260709 12:19:20.632596 27734 rpcz_store.cc:276] 0709 12:19:18.607550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.483028 (+1875478us) service_pool.cc:224] Handling call
0709 12:19:20.483319 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.571649 (+ 88330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.571862 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.575923 30766 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:20.491367 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1743) took 2165 ms. Trace:
I20260709 12:19:20.634148 27733 rpcz_store.cc:276] 0709 12:19:18.325738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.376517 (+2050779us) service_pool.cc:224] Handling call
0709 12:19:20.376867 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.491002 (+114135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.491276 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.634446 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:20.496078 30992 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:20.590075 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:20.592362 27582 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:20.593164 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.595774 27656 rpcz_store.cc:276] 0709 12:19:19.145752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.321021 (+1175269us) service_pool.cc:224] Handling call
0709 12:19:20.405702 (+ 84681us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.494453 (+ 88751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.520026 (+ 25573us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b54000e60d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.520168 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193916288}
I20260709 12:19:20.595971 27731 rpcz_store.cc:276] 0709 12:19:19.498206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.601050 (+102844us) service_pool.cc:224] Handling call
0709 12:19:20.521055 (+920005us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b540017edd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:20.521161 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959682944}
I20260709 12:19:20.598210 27588 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:20.594353 27730 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:20.601308 27663 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:20.602047 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.606287 27585 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 5. (term mismatch)
I20260709 12:19:20.596051 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.610901 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:20.595497 30827 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:20.521603 31050 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.108s
I20260709 12:19:20.616932 27735 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 12:19:20.618163 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:20.619251 27485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 230) took 35943 ms (client timeout 15000 ms). Trace:
I20260709 12:19:20.619565 31149 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.620503 31064 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.623018 30983 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:20.626510 27581 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:20.744006 27585 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:20.744242 27582 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:20.744585 27585 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20260709 12:19:20.745008 27585 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:20.624138 31174 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.019s	sys 0.040s
I20260709 12:19:20.635890 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20260709 12:19:20.635893 27584 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:19:20.616212 27658 raft_consensus.cc:1217] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2]   Dedup: 3.2->[]
I20260709 12:19:20.748852 27658 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 12:19:20.749225 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.009s	sys 0.014s
W20260709 12:19:20.749585 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.009s	sys 0.014s
W20260709 12:19:20.750217 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 712921us
I20260709 12:19:20.750526 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1927) took 1254 ms. Trace:
I20260709 12:19:20.751446 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:20.751148 27731 rpcz_store.cc:276] 0709 12:19:19.496250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.748566 (+1252316us) service_pool.cc:224] Handling call
0709 12:19:20.748973 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.749621 (+   648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.749664 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.749724 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:20.750219 (+   495us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.750304 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.750443 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.755225 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:20.746580 27583 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:20.637445 31046 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:20.747743 27737 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:20.762156 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1926) took 1266 ms. Trace:
I20260709 12:19:20.752712 27587 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:20.762408 27730 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:20.644372 27729 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
W20260709 12:19:20.746891 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 156931us
I20260709 12:19:20.763391 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1929) took 1266 ms. Trace:
I20260709 12:19:20.762667 27733 rpcz_store.cc:276] 0709 12:19:19.495979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.635221 (+1139242us) service_pool.cc:224] Handling call
0709 12:19:20.635621 (+   400us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:20.761811 (+126190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.762093 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.614251 31171 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:20.764236 27704 leader_election.cc:400] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:20.765200 27733 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:20.745337 27582 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.764089 27730 rpcz_store.cc:276] 0709 12:19:19.496716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.749887 (+1253171us) service_pool.cc:224] Handling call
0709 12:19:20.761984 (+ 12097us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.763089 (+  1105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.763306 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.628904 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:20.766450 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1930) took 1269 ms. Trace:
I20260709 12:19:20.769088 27733 rpcz_store.cc:276] 0709 12:19:19.496959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.764490 (+1267531us) service_pool.cc:224] Handling call
0709 12:19:20.764810 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.766180 (+  1370us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.766383 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.767383 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:20.771181 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1923) took 2108 ms. Trace:
I20260709 12:19:20.776561 27735 rpcz_store.cc:276] 0709 12:19:18.645555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.614928 (+1969373us) service_pool.cc:224] Handling call
0709 12:19:20.615314 (+   386us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:20.743800 (+128486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.754237 (+ 10437us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.782994 27731 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:19:20.782823 27582 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:20.783669 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1924) took 2110 ms. Trace:
I20260709 12:19:20.783783 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1762) took 1093 ms. Trace:
I20260709 12:19:20.785061 27729 rpcz_store.cc:276] 0709 12:19:18.657919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.627565 (+1969646us) service_pool.cc:224] Handling call
0709 12:19:20.644124 (+ 16559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.763412 (+119288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.768542 (+  5130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.785301 27731 rpcz_store.cc:276] 0709 12:19:19.690277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.775692 (+1085415us) service_pool.cc:224] Handling call
0709 12:19:20.776074 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.783517 (+  7443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.783732 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:20.788530 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.789740 27732 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:20.795493 27730 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:19:20.795531 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1939) took 1285 ms. Trace:
I20260709 12:19:20.796185 27732 rpcz_store.cc:276] 0709 12:19:19.509768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.767764 (+1257996us) service_pool.cc:224] Handling call
0709 12:19:20.768085 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.794343 (+ 26258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.795474 (+  1131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.797092 27730 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:19:20.784593 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:20.803221 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:20.803700 31149 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:20.804009 31225 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:20.806636 31225 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:20.807777 31225 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:20.748210 31174 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.020s	sys 0.040s
I20260709 12:19:20.650250 31178 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:20.733668 31075 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:20.745527 30827 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:20.826336 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1942) took 1039 ms. Trace:
I20260709 12:19:20.670805 31205 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:20.768376 31228 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:20.747644 31054 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:20.804363 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:20.842759 27587 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:20.845544 27582 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:19:20.831310 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:20.847193 27731 rpcz_store.cc:276] 0709 12:19:19.787259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.825295 (+1038036us) service_pool.cc:224] Handling call
0709 12:19:20.825664 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.825894 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.825931 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.825988 (+    57us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:20.826031 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.826111 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.826266 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:20.861212 27582 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:20.861526 27582 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:19:20.861809 27582 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:20.855628 27737 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:20.862680 27732 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
W20260709 12:19:20.745684 27485 rpcz_store.cc:269] 0709 12:18:44.680714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.692469 (+ 11755us) service_pool.cc:224] Handling call
0709 12:19:20.619151 (+35926682us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:13.809647 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:13.810243 (+   596us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:13.810293 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:19:13.810311 (+    18us) tablet.cc:662] Decoding operations
0709 12:19:13.811157 (+   846us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:13.811237 (+    80us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:13.811262 (+    25us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:19:13.886407 (+ 75145us) tablet.cc:701] Row locks acquired
0709 12:19:13.886457 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:19:13.886651 (+   194us) write_op.cc:270] Start()
0709 12:19:13.886851 (+   200us) write_op.cc:276] Timestamp: P: 1783599553886581 usec, L: 0
0709 12:19:13.886875 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:19:13.974469 (+ 87594us) log.cc:844] Serialized 3224 byte log entry
0709 12:19:15.082539 (+1108070us) op_driver.cc:464] REPLICATION: finished
0709 12:19:15.895831 (+813292us) write_op.cc:301] APPLY: starting
0709 12:19:15.913438 (+ 17607us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:15.923622 (+ 10184us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:15.923813 (+   191us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:16.020354 (+ 96541us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:16.020782 (+   428us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0709 12:19:16.020856 (+    74us) write_op.cc:312] APPLY: finished
0709 12:19:16.021261 (+   405us) log.cc:844] Serialized 65 byte log entry
0709 12:19:16.147807 (+126546us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:16.169821 (+ 22014us) write_op.cc:454] Released schema lock
0709 12:19:16.226878 (+ 57057us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:16.372392 (+145514us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:16.372432 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36532,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":720696,"apply.run_cpu_time_us":19230,"apply.run_wall_time_us":455642,"num_ops":8,"prepare.queue_time_us":357924,"prepare.run_cpu_time_us":24178,"prepare.run_wall_time_us":315027,"raft.queue_time_us":129024,"raft.run_cpu_time_us":13124,"raft.run_wall_time_us":368891,"replication_time_us":1157389,"thread_start_us":758114,"threads_started":5,"wal-append.queue_time_us":314295,"wal-append.run_cpu_time_us":35373,"wal-append.run_wall_time_us":3103103}]]}
W20260709 12:19:20.865077 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60760.4us
I20260709 12:19:20.866214 27627 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11.
I20260709 12:19:20.866565 27627 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:20.819661 30983 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:20.829542 30995 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:20.846967 31149 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.883394 31226 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:20.883638 30979 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:20.884286 31226 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:20.885402 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:20.856078 31228 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:20.829483 27482 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:20.895444 31228 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:20.895678 31149 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:20.896019 27482 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 7, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 7 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:20.898031 31149 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:20.897939 27482 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 3 to 7, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 7 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:20.898573 31149 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:20.900453 27729 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:20.870258 30983 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:20.905743 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.009s	sys 0.017s
W20260709 12:19:20.906380 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.009s	sys 0.017s
W20260709 12:19:20.907050 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.005s	sys 0.046s
W20260709 12:19:20.907471 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.005s	sys 0.046s
W20260709 12:19:20.908013 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 552971us
I20260709 12:19:20.870832 30995 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:20.857285 31075 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:20.910019 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.914361 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:20.694523 30766 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:20.913182 27664 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.011s	sys 0.015s
W20260709 12:19:20.916572 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.924294 27735 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
W20260709 12:19:20.924950 27664 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.011s	sys 0.015s
I20260709 12:19:20.891588 31226 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:20.924947 27482 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:20.907390 30983 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:20.915853 31169 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.006s	sys 0.022s
I20260709 12:19:21.002429 27582 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:21.002655 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:20.903105 31162 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:20.899279 31149 raft_consensus.cc:2764] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:19:21.007752 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:21.013069 28329 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:21.013535 28329 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:21.014441 28329 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.016961 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:21.018251 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:20.903210 30992 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.019613 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.023742 30979 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11.
I20260709 12:19:21.024958 30979 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:21.024487 27482 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 7 to 9, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 9 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:19:21.004856 31169 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.830s	user 0.006s	sys 0.022s
I20260709 12:19:21.025298 30979 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:21.027036 27482 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:21.028076 30979 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.029487 27482 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 5, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 5 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:21.030541 30979 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:20.988255 30975 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:19:21.035136 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.037902 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1684) took 2114 ms. Trace:
I20260709 12:19:21.038370 27658 rpcz_store.cc:276] 0709 12:19:18.923455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.710456 (+787001us) service_pool.cc:224] Handling call
0709 12:19:19.710780 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.749238 (+1038458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.037637 (+288399us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54001e5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:21.037837 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2081950592}
I20260709 12:19:21.040444 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 474) took 1135 ms. Trace:
I20260709 12:19:21.040956 27732 rpcz_store.cc:276] 0709 12:19:19.905087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.801915 (+896828us) service_pool.cc:224] Handling call
0709 12:19:20.862441 (+ 60526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.863158 (+   717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.040184 (+177026us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b540017e3d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:21.040372 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":138346752}
I20260709 12:19:21.042771 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1702) took 1965 ms. Trace:
I20260709 12:19:21.043347 27663 rpcz_store.cc:276] 0709 12:19:19.077635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.171660 (+1094025us) service_pool.cc:224] Handling call
0709 12:19:20.171977 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.743824 (+571847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.042494 (+298670us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:21.042680 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":986892544}
I20260709 12:19:21.036094 30975 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:21.063925 30975 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:21.067566 31016 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:21.068881 31075 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:21.069893 27662 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:21.035724 30766 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:21.084398 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.008s	sys 0.019s
W20260709 12:19:21.084826 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.008s	sys 0.019s
W20260709 12:19:21.085538 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.086287 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.092350 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.098335 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.000s	sys 0.028s
W20260709 12:19:21.098796 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.000s	sys 0.028s
W20260709 12:19:21.099153 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 594249us
W20260709 12:19:21.100183 27659 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.007s	sys 0.022s
I20260709 12:19:21.076634 27482 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 2, term changed from 3 to 5, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added, no longer has a pending config: VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1), VOTER 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3), NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:21.085192 31159 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:19:21.098886 30635 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.000s
W20260709 12:19:21.100646 27659 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.007s	sys 0.022s
I20260709 12:19:21.076696 31149 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:19:21.095781 31064 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.163237 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.182305 27661 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:21.167037 31012 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:21.135172 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:21.138721 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 153417us
I20260709 12:19:21.171314 31207 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:21.142683 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.190930 27663 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:21.143005 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.191893 27658 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:21.172473 27482 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 6, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 6 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:21.173763 30635 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:21.173614 31174 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:19:21.174307 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 74293.6us
W20260709 12:19:21.106467 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.145797 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:21.204478 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.206383 30987 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:21.206699 30983 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.208612 31191 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:21.271533 31075 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:21.211418 31162 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:21.213194 31155 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.624s	user 0.005s	sys 0.009s
I20260709 12:19:21.217013 28329 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:21.219581 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2049) took 1091 ms. Trace:
I20260709 12:19:21.278164 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:21.219735 31253 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.214949 30995 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.220460 31178 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:21.221040 30827 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.151470 31159 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:21.202874 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2059) took 1823 ms. Trace:
I20260709 12:19:21.231403 27661 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:21.232390 31168 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:21.233264 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98198.8us
I20260709 12:19:21.234625 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.296278 28328 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:21.235543 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.276172 27732 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:21.297214 27663 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:21.285365 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:21.277758 27586 rpcz_store.cc:276] 0709 12:19:20.128376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.129048 (+   672us) service_pool.cc:224] Handling call
0709 12:19:20.129308 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.453546 (+324238us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:20.453807 (+   261us) log.cc:844] Serialized 14 byte log entry
0709 12:19:20.988741 (+534934us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.988812 (+    71us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:20.988876 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.211288 (+222412us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:21.219351 (+  8063us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.219498 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1542,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":113,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":14929,"spinlock_wait_cycles":745099008,"thread_start_us":491118,"threads_started":1,"wal-append.queue_time_us":698008}
W20260709 12:19:21.286641 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 519351us
I20260709 12:19:21.289247 27656 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: [7.7-9.8]
I20260709 12:19:21.299182 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:21.190351 31207 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:21.292157 27662 rpcz_store.cc:276] 0709 12:19:19.379055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.338298 (+959243us) service_pool.cc:224] Handling call
0709 12:19:20.387730 (+ 49432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:21.074022 (+686292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.202498 (+128476us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:21.202785 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1568327040}
I20260709 12:19:21.300240 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:21.300379 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:21.290027 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 293382us
I20260709 12:19:21.296667 27658 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:21.315199 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:21.316509 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.190050 31064 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.325215 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:21.329887 27661 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:21.283790 31075 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:21.289290 31157 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:21.285346 31162 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.270507 30635 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.308153 31147 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:21.364236 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:21.361861 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.369041 27662 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:21.370397 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:21.373818 27732 raft_consensus.cc:606] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:21.374269 27732 raft_consensus.cc:614] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:19:21.371004 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.371016 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.320436 30995 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:21.289750 31232 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:19:21.338495 31050 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.382512 31113 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:21.371258 31226 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.330767 31191 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.382514 31149 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:21.401059 31191 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.401686 31149 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:21.402731 31235 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:19:21.371336 31228 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.383571 31174 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:21.404584 31174 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:21.405052 31174 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:21.405299 31228 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:21.387591 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.407275 27658 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:21.325155 30827 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:21.323643 31016 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:21.331526 31239 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:21.330608 31054 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:21.331148 31237 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.395917 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.425184 31226 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:21.426855 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:21.284861 31155 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.398018 31050 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.427487 27586 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:21.428908 31155 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:21.429239 31050 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:21.430017 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:21.430624 27735 raft_consensus.cc:606] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:21.430969 27735 raft_consensus.cc:614] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:21.402562 31149 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:21.431387 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.400403 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.432130 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1769) took 1464 ms. Trace:
I20260709 12:19:21.320066 31168 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.388867 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.393303 31147 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:21.294492 31253 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.408867 31046 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:21.420493 31174 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.420794 30975 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.430843 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:21.447001 27663 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:21.432255 27662 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:21.432533 27731 rpcz_store.cc:276] 0709 12:19:19.968023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.859936 (+891913us) service_pool.cc:224] Handling call
0709 12:19:20.860259 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.860562 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.860589 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:20.865540 (+  4951us) log.cc:844] Serialized 26 byte log entry
0709 12:19:21.154122 (+288582us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:21.154184 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.154284 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:21.431920 (+277636us) raft_consensus.cc:1685] finished
0709 12:19:21.431941 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.432074 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":63436,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":80207,"thread_start_us":249909,"threads_started":1,"wal-append.queue_time_us":546116}
I20260709 12:19:21.434090 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1925) took 2772 ms. Trace:
I20260709 12:19:21.448256 27736 rpcz_store.cc:276] 0709 12:19:18.661590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.631033 (+1969443us) service_pool.cc:224] Handling call
0709 12:19:20.631370 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.631695 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:19:20.631733 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:20.638355 (+  6622us) log.cc:844] Serialized 26 byte log entry
0709 12:19:21.098056 (+459701us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:19:21.098129 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.098223 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:21.433885 (+335662us) raft_consensus.cc:1685] finished
0709 12:19:21.433907 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.434034 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":410905,"threads_started":1,"wal-append.queue_time_us":658060}
I20260709 12:19:21.407902 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.371354 27482 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 1 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:21.453276 27482 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 7 to 8, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 8 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:21.434927 27586 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:21.407826 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1749) took 2557 ms. Trace:
I20260709 12:19:21.456945 27734 rpcz_store.cc:276] 0709 12:19:18.850131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.637428 (+1787297us) service_pool.cc:224] Handling call
0709 12:19:20.637773 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.637990 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:19:21.084021 (+446031us) log.cc:844] Serialized 14 byte log entry
0709 12:19:21.406814 (+322793us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:21.407140 (+   326us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:21.407270 (+   130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.407654 (+   384us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.407765 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":762819,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":29,"raft.queue_time_us":15363,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":734439,"threads_started":2}
I20260709 12:19:21.437463 27657 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:21.411001 27587 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:21.459738 27587 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:21.384402 31162 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.417227 31016 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:21.437889 27656 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:21.444159 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.475872 27661 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:21.476756 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.477442 27661 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:21.432636 30638 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:21.478101 30638 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:21.478273 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.460565 27587 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.468262 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.433377 31191 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:21.479300 27587 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:21.479182 30638 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.480177 31147 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.444893 31239 raft_consensus.cc:2764] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 12:19:21.441394 31205 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.011s	sys 0.041s
I20260709 12:19:21.483016 28336 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:21.483665 28336 raft_consensus.cc:606] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:21.484982 31159 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.486896 31159 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:21.422964 31237 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.488787 27662 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:21.489740 31237 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:21.490334 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.490675 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.491262 27657 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:21.492595 31168 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:21.494438 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.494596 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.494282 27662 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:21.495244 27658 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:21.495723 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.496237 27662 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: [9.9-10.10]
I20260709 12:19:21.496376 27658 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:21.497887 27662 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:19:21.498220 27662 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:19:21.498435 27662 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:19:21.498678 27662 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:21.500551 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:21.501044 31253 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:21.483165 31205 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.011s	sys 0.042s
I20260709 12:19:21.508281 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:19:21.508509 31205 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:21.509533 31205 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.510264 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:21.511395 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:21.478966 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:21.512746 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.024s	sys 0.032s
W20260709 12:19:21.513090 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.024s	sys 0.032s
I20260709 12:19:21.470263 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:21.513370 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 608119us
W20260709 12:19:21.513868 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43693.2us
I20260709 12:19:21.416373 28329 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:21.516307 27482 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 3 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 4 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:21.471930 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.517906 27658 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:19:21.526713 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.531113 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1705) took 2320 ms. Trace:
I20260709 12:19:21.533365 27664 rpcz_store.cc:276] 0709 12:19:19.210297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.328085 (+1117788us) service_pool.cc:224] Handling call
0709 12:19:20.328376 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.984800 (+656424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.984847 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:21.005744 (+ 20897us) log.cc:844] Serialized 26 byte log entry
0709 12:19:21.280266 (+274522us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:21.280325 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.280422 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:21.530840 (+250418us) raft_consensus.cc:1685] finished
0709 12:19:21.530870 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.531039 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58294,"raft.queue_time_us":76371,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":18260,"thread_start_us":805746,"threads_started":2,"wal-append.queue_time_us":497601}
I20260709 12:19:21.536784 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2073) took 1667 ms. Trace:
I20260709 12:19:21.538021 27659 rpcz_store.cc:276] 0709 12:19:19.869688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.485992 (+616304us) service_pool.cc:224] Handling call
0709 12:19:20.490506 (+  4514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.490715 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:21.159727 (+669012us) log.cc:844] Serialized 14 byte log entry
0709 12:19:21.532926 (+373199us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:21.532979 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:21.533024 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.536280 (+  3256us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.536716 (+   436us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23467,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"thread_start_us":1028129,"threads_started":2}
W20260709 12:19:21.543681 28292 leader_election.cc:336] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.637s (SENT)
I20260709 12:19:21.544368 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:21.521801 31240 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
W20260709 12:19:21.565258 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.009s	sys 0.017s
W20260709 12:19:21.568776 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.009s	sys 0.017s
I20260709 12:19:21.519558 31178 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:21.517493 31253 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:21.405815 31207 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:21.531811 31225 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.017s	sys 0.017s
W20260709 12:19:21.656031 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 356917us
W20260709 12:19:21.728137 31225 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.019s	sys 0.026s
I20260709 12:19:21.628084 31016 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.729959 31016 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:21.722725 27734 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:19:21.730684 30986 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:21.731572 31178 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:21.732544 31178 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.733327 30986 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:19:21.724005 31149 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:21.734323 31241 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:21.649344 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2070) took 1859 ms. Trace:
I20260709 12:19:21.605311 31162 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:21.618045 30766 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.530s	user 0.010s	sys 0.000s
I20260709 12:19:21.602766 31157 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:21.736153 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:21.736356 31207 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:21.649791 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 370622us
W20260709 12:19:21.759145 27701 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.737s (SENT)
I20260709 12:19:21.759604 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:21.760773 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.761755 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.604432 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2053) took 2420 ms. Trace:
I20260709 12:19:21.531862 31075 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.762434 27655 rpcz_store.cc:276] 0709 12:19:19.183845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.323102 (+1139257us) service_pool.cc:224] Handling call
0709 12:19:20.323469 (+   367us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:20.324017 (+   548us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:19:20.421100 (+ 97083us) log.cc:844] Serialized 14 byte log entry
0709 12:19:21.150100 (+729000us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:21.164431 (+ 14331us) log.cc:844] Serialized 52 byte log entry
0709 12:19:21.164612 (+   181us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:21.164832 (+   220us) log.cc:844] Serialized 14 byte log entry
0709 12:19:21.164993 (+   161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.165069 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:21.604149 (+439080us) raft_consensus.cc:1685] finished
0709 12:19:21.604178 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.604348 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2754,"prepare.queue_time_us":302729,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":4671,"raft.queue_time_us":340445,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":78393,"thread_start_us":725414,"threads_started":2,"wal-append.queue_time_us":1042817}
I20260709 12:19:21.764124 28336 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:19:21.764442 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:21.765110 28328 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:21.769454 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.769866 27585 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:21.770709 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.737434 30638 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:21.673036 30992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.037s	sys 0.051s
I20260709 12:19:21.606940 31064 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:21.565368 31237 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:21.603765 30635 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:21.605832 30983 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:21.720273 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 241355us
I20260709 12:19:21.801518 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:21.720443 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206371us
W20260709 12:19:21.670676 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 345512us
W20260709 12:19:21.742420 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 737955us
I20260709 12:19:21.752517 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.749086 31241 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:21.749256 27660 rpcz_store.cc:276] 0709 12:19:19.790131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.481221 (+691090us) service_pool.cc:224] Handling call
0709 12:19:20.494582 (+ 13361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:20.494974 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:20.495017 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:20.512644 (+ 17627us) log.cc:844] Serialized 52 byte log entry
0709 12:19:20.989163 (+476519us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:20.989220 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.989303 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:21.649030 (+659727us) raft_consensus.cc:1685] finished
0709 12:19:21.649060 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.649270 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":535,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":15840,"thread_start_us":414891,"threads_started":1,"wal-append.queue_time_us":765882}
W20260709 12:19:21.775419 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.790133 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.780246 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.771152 27582 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260709 12:19:21.791468 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.771050 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.822211 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:21.827580 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:21.829459 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.781772 31046 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:21.797792 31174 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:21.837306 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:21.838131 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:21.849828 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 92331.5us
I20260709 12:19:21.858374 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.859627 27736 raft_consensus.cc:2368] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12.
I20260709 12:19:21.861018 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.862260 27731 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:21.783614 31249 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:21.845553 30635 raft_consensus.cc:2764] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:21.881536 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:21.874373 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.874769 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.884658 27734 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:21.886716 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:21.888543 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50496.1us
W20260709 12:19:21.895663 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.896747 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.897369 28336 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:21.897513 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:21.898384 28332 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:21.899696 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:21.906813 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:21.908725 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.909417 27734 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:21.910199 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.910717 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.911536 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.912120 27737 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:21.912197 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:21.913887 31046 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:21.914546 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.915088 27729 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:21.915146 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.915868 27735 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:21.915560 30986 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:19:21.874199 31254 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:19:21.867373 31012 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.918752 31012 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.920496 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.920980 31012 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:21.921166 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.921693 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.923120 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:21.803658 31169 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:19:21.924443 31169 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:21.924815 31169 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:21.925835 31169 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.926717 27729 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:21.927477 31252 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:21.927749 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.852285 31234 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:19:21.878898 31237 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.929380 31237 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.930145 31110 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:21.867597 31267 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:21.932358 27733 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:21.933554 27736 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:21.935899 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:21.936492 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.874840 30975 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:21.947249 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.947649 27655 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:19:21.936853 27736 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:21.939690 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:21.949157 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.931586 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:21.941231 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.943012 27735 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:21.950299 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.950516 27663 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:21.951587 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:21.951356 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.945119 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.945633 27588 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:19:21.954078 27659 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:19:21.952127 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.952776 27731 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:21.945647 27664 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
W20260709 12:19:21.957435 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75976.6us
I20260709 12:19:21.958197 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:21.959082 27659 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:19:21.959631 31168 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:21.960036 31168 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:21.960529 31168 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:21.961064 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.961464 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:21.963536 31075 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:21.967442 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:21.969640 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:21.970646 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:21.972728 27737 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:21.977048 27734 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:21.943476 27627 leader_election.cc:400] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12.
I20260709 12:19:21.982125 27627 leader_election.cc:403] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260709 12:19:21.982268 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 154774us
I20260709 12:19:21.982290 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:21.877312 31249 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:22.011967 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:21.950222 30987 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:21.938668 31174 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:21.952525 31169 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:19:22.046202 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 78110.9us
I20260709 12:19:21.955132 31178 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:21.876533 31064 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
W20260709 12:19:22.038800 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71506.8us
W20260709 12:19:22.073735 30995 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.680s	user 0.008s	sys 0.003s
I20260709 12:19:22.074249 31237 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:22.075459 31254 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:19:21.977789 31272 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
W20260709 12:19:22.080653 31263 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 12:19:22.081972 31012 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:22.082640 31012 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.082340 31178 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.083137 31012 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:22.084743 31178 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:21.874841 31270 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.182s
W20260709 12:19:22.050257 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107764us
I20260709 12:19:22.090207 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:22.049397 31155 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:22.052559 31234 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12.
W20260709 12:19:22.020063 31255 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:19:22.041992 31241 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:21.950430 31239 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:21.951467 30900 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 12:19:22.095137 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:22.091382 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 79497.3us
W20260709 12:19:22.121378 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 133942us
I20260709 12:19:22.123649 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:22.130004 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170027us
I20260709 12:19:22.309106 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:22.309835 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:22.020709 31050 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.103s
I20260709 12:19:22.103901 30995 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:19:22.141908 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 183769us
I20260709 12:19:22.142509 31075 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.123121 31155 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:22.198195 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.204761 31178 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.291972 30986 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:22.336694 31169 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:22.343528 31169 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:22.343389 30986 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.344813 31169 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:21.955024 31232 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.133s
I20260709 12:19:21.968508 30766 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.137s
I20260709 12:19:22.076889 31254 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:22.046257 31054 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.090s
W20260709 12:19:22.296674 31159 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.809s	user 0.015s	sys 0.002s
W20260709 12:19:22.297590 31207 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.538s	user 0.014s	sys 0.001s
W20260709 12:19:22.298331 31149 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.004s
I20260709 12:19:22.298772 30983 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:22.045718 30827 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 12:19:22.300323 31205 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.301370 31284 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.203555 31174 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.326493 30987 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:22.317464 31239 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:22.353132 31254 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:22.347793 31050 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:22.347201 30995 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:22.323711 31249 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:19:22.329125 31252 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:22.349604 31178 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.363044 31267 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:22.376143 31168 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.378253 31191 raft_consensus.cc:993] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:19:22.380656 31232 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:22.384449 31246 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:22.386829 31155 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.370831 31239 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:22.404462 31232 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:22.405824 31168 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.406080 31232 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.373896 31075 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.407616 31232 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:22.398646 31267 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:22.404611 31246 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:22.412379 31246 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.413798 31246 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:22.414484 31241 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:22.400713 31178 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.367769 31205 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:22.416792 31262 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.501s	user 0.004s	sys 0.000s
I20260709 12:19:22.418294 31205 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:22.403379 31254 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.368078 31284 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.423753 31254 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:22.425427 31284 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:22.279536 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.462164 27584 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:22.323205 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:22.469040 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 145826us
W20260709 12:19:22.291105 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.480502 27588 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:22.470829 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:22.318785 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 223704us
W20260709 12:19:22.475073 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.493057 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:22.494125 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.495988 27583 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:22.496665 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:22.499262 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:22.499763 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:22.500753 27583 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:22.504330 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:22.505426 27586 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:22.513844 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:22.311616 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:22.519002 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.529512 27583 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:22.519199 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.100049 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:22.533768 27585 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:22.525859 27661 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:19:22.334533 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:22.527981 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:22.520735 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.535470 27662 raft_consensus.cc:2376] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8. Re-sending same reply.
I20260709 12:19:22.536382 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1723) took 1059 ms. Trace:
I20260709 12:19:22.537125 27661 rpcz_store.cc:276] 0709 12:19:21.476998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.478060 (+  1062us) service_pool.cc:224] Handling call
0709 12:19:22.536021 (+1057961us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:22.536277 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2405775616}
I20260709 12:19:22.538407 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.536535 27581 raft_consensus.cc:2376] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12. Re-sending same reply.
I20260709 12:19:22.540122 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.527153 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.543812 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:22.545756 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:22.313524 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:22.349442 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 493716us
I20260709 12:19:22.304049 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 477) took 2341 ms. Trace:
I20260709 12:19:22.379228 30900 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:22.618892 30900 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:22.619397 30638 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.819s	user 0.011s	sys 0.000s
I20260709 12:19:22.621088 30900 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:22.622570 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.377671 30992 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:19:22.623292 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:22.624737 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.380620 31064 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:22.626730 31064 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.543998 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.401845 31169 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:22.581835 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.629318 30900 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:22.582813 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 275299us
I20260709 12:19:22.583940 31286 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.369233 30987 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:22.631600 31286 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.361876 30986 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:22.577222 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74288.9us
I20260709 12:19:22.638046 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:22.581840 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.596535 31225 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:22.640692 31225 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:22.641304 31225 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:22.642244 31162 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:22.643492 31110 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.644702 31147 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:22.645289 30638 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.645469 31147 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.646106 30638 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.648625 31155 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:22.649483 30983 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:22.597775 31285 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:22.598851 31289 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 12:19:22.651579 31285 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:22.652464 31265 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:22.653137 31265 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.604545 31251 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:22.654858 31251 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:22.655431 31251 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:22.656174 31171 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.656967 31171 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.615592 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:22.420127 31113 raft_consensus.cc:1081] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:19:22.397251 31050 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:22.618906 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.660826 31253 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:22.660127 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45099.7us
I20260709 12:19:22.662380 31050 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:22.662703 31265 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.644330 31110 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.652427 31285 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:22.617919 31178 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:22.670065 31178 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.670832 31110 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.672685 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:22.673918 31054 raft_consensus.cc:2764] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260709 12:19:22.674144 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:22.675312 31054 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:22.675756 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:22.678122 27735 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:19:22.678725 31252 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:22.679450 31149 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:22.679870 30638 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.684361 30995 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.684657 31251 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.692824 31178 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.599833 31157 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:19:22.697177 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66407.1us
I20260709 12:19:22.618669 27730 rpcz_store.cc:276] 0709 12:19:19.962658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.806505 (+843847us) service_pool.cc:224] Handling call
0709 12:19:20.806891 (+   386us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:21.281319 (+474428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:21.281759 (+   440us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:21.282334 (+   575us) log.cc:844] Serialized 52 byte log entry
0709 12:19:21.701494 (+419160us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:21.701560 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.701656 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:22.303789 (+602133us) raft_consensus.cc:1685] finished
0709 12:19:22.303813 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:22.303983 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15577,"raft.queue_time_us":80885,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":2809,"thread_start_us":874166,"threads_started":2,"wal-append.queue_time_us":841155}
I20260709 12:19:22.705897 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:22.708685 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.663144 31050 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:22.710008 31012 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.626s	user 0.014s	sys 0.000s
W20260709 12:19:22.703565 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 212249us
I20260709 12:19:22.698076 31252 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.598826 31288 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.116s
I20260709 12:19:22.728372 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.726800 31290 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.732331 31290 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.734333 28335 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:22.717907 31110 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:22.739687 31178 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:22.740592 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:22.741470 28328 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:22.711596 31286 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.746311 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.747421 27585 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:22.749747 31290 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.754798 30992 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:22.720306 31171 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.696810 31285 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:22.753052 31016 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.022s	user 0.010s	sys 0.011s
I20260709 12:19:22.758180 31285 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.759234 27583 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:22.755121 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:22.765654 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91737.9us
I20260709 12:19:22.767925 31169 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.768877 31169 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.722922 31252 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:22.771806 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96261.3us
I20260709 12:19:22.772914 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:22.774684 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:22.722223 31012 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.647003 31147 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.779587 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:22.780402 28328 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:22.781008 31251 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:22.777037 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:22.740633 31178 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:22.795379 27588 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:22.784219 31012 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:22.785068 31168 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:22.823735 31168 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.825209 31168 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:22.802234 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47316.2us
I20260709 12:19:22.827307 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:22.811693 31205 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:22.828949 30992 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:22.829473 30992 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.829772 31205 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:22.829903 30992 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:22.830339 31205 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:22.832042 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.812673 31286 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:22.834484 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.835229 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:22.819890 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:22.836668 28335 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:22.808357 27585 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:19:22.841598 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:22.798058 31050 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.845980 31265 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:22.845790 31205 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.846755 31234 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:22.847411 31234 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.847842 31234 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:22.849756 31285 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:22.851662 31288 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:22.852216 31288 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.853140 27587 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:22.854959 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:22.856364 31075 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:22.857018 31075 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.856865 31064 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.858111 31075 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.800170 31246 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.859983 31246 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.861243 30987 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.862417 31246 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:22.824002 31169 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:22.858341 31306 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:22.864006 31306 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:22.864382 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:22.865115 27588 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:22.865847 27585 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:56296
I20260709 12:19:22.866618 27585 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:22.820089 31046 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:19:22.866986 27585 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:22.864387 31306 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:22.869478 31306 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.821058 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:22.854771 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 502) took 1049 ms. Trace:
I20260709 12:19:22.871290 31306 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:22.874146 30827 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:22.904858 30827 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.905280 30827 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:22.884053 31178 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.886884 27587 rpcz_store.cc:276] 0709 12:19:21.804962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.805125 (+   163us) service_pool.cc:224] Handling call
0709 12:19:21.805498 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:22.852764 (+1047266us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540041a8a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:19:22.853761 (+   997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.854389 (+   628us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b54001fe5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:22.854677 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2194624768}
I20260709 12:19:22.910702 31251 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:22.893412 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:22.913674 31157 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:22.914182 31157 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.914664 31157 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:22.916368 31046 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.891597 31288 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.917322 31046 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:22.872866 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 52036.3us
I20260709 12:19:22.921689 31251 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:22.923156 31012 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:22.923609 31012 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:22.924856 31012 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:22.926172 31012 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:22.925230 31284 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:22.927822 31284 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.942015 31288 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:22.949805 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:22.951772 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 123283us
I20260709 12:19:22.952929 27586 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:22.955315 28328 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
W20260709 12:19:22.968662 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194084us
W20260709 12:19:22.969971 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 158666us
W20260709 12:19:22.974867 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 80795.2us
W20260709 12:19:22.976666 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119800us
I20260709 12:19:22.964072 31064 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:22.964139 31205 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:22.985112 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:22.985726 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:22.992977 31016 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:22.994796 31046 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:22.995543 31286 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:22.996073 31286 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.996246 30635 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:22.996503 31286 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:22.996783 30635 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:22.997859 30635 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.000550 31046 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:23.001387 31046 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:23.004703 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.973757 31285 raft_consensus.cc:2764] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
W20260709 12:19:23.018217 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.020885 31046 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:23.022818 31239 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.013s
I20260709 12:19:23.042169 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:23.042742 27585 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:22.989704 30987 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:23.045766 27583 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
W20260709 12:19:23.056531 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63988.5us
I20260709 12:19:23.054924 31254 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.064399 27585 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.047899 31171 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.031400 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 508278us
W20260709 12:19:23.074442 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:23.079459 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.080600 31251 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.081295 31251 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:23.083792 31267 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.672s	user 0.016s	sys 0.001s
W20260709 12:19:23.084914 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.084440 31267 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.085368 31251 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.086894 31267 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.088289 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:23.089311 31232 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.681s	user 0.008s	sys 0.008s
I20260709 12:19:23.089529 28328 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:19:23.075254 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:23.096606 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.050033 31239 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.063934 31254 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.066072 31149 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.095733 31046 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.110809 30987 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.127570 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73288.5us
W20260709 12:19:23.128168 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65883us
I20260709 12:19:23.091722 31171 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.134065 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:23.136397 27585 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:23.111310 31050 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:23.069329 31147 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260709 12:19:23.112970 31253 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:23.141475 27582 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:23.141719 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.142933 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:23.142505 27588 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:23.162341 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:23.151353 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:23.150516 31225 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.509s	user 0.012s	sys 0.008s
I20260709 12:19:23.126559 31252 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.147580 31284 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:23.174948 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:23.151414 31308 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:19:23.126828 30992 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.144999 30987 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.124521 31149 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.152774 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:23.178431 31110 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
W20260709 12:19:23.181334 31241 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.008s
I20260709 12:19:23.189778 31253 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:23.209735 31253 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:23.210196 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.210990 27584 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:23.211943 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.212559 27584 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:23.213534 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.214903 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.165138 31046 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.254669 27584 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:23.174770 31225 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.265972 31046 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.210268 31253 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:23.268352 31315 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:19:23.270963 31239 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.280243 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.281443 31290 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.281939 31157 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.282114 31290 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:23.282764 31314 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:19:23.291936 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 1961) took 1196 ms. Trace:
W20260709 12:19:23.296413 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 148900us
I20260709 12:19:23.272817 30638 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.312887 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.313982 27585 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:23.322609 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:19:23.343269 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.354645 27586 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:23.354586 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:23.356736 30766 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.357328 31311 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:19:23.357637 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.357415 30766 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:23.360359 31300 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:19:23.362892 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.363729 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:23.364411 31290 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:23.363096 31316 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:19:23.362483 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.371837 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:23.363103 27589 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:23.373721 28335 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:23.368747 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.374096 28335 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:23.377866 27583 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:23.344369 31232 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:23.296231 27736 rpcz_store.cc:276] 0709 12:19:21.941861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.969208 (+ 27347us) service_pool.cc:224] Handling call
0709 12:19:21.969496 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:21.969835 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:21.969862 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:21.970310 (+   448us) log.cc:844] Serialized 26 byte log entry
0709 12:19:22.622591 (+652281us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b1800265ae0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:22.756787 (+134196us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:23.071626 (+314839us) log.cc:844] Serialized 14 byte log entry
0709 12:19:23.093582 (+ 21956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:23.110292 (+ 16710us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:23.110333 (+    41us) raft_consensus.cc:1685] finished
0709 12:19:23.115829 (+  5496us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:23.137947 (+ 22118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60063,"prepare.queue_time_us":403435,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":24032,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":37563,"spinlock_wait_cycles":3151744,"thread_start_us":408120,"threads_started":2,"wal-append.queue_time_us":755105}
I20260709 12:19:23.365813 31046 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.386618 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:23.382418 27733 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:23.400624 31286 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.404518 31149 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.406278 31157 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.406889 30987 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.407441 30635 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:23.408031 30635 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:23.409104 30635 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.411969 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.412834 31232 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.414505 27735 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:23.416065 27732 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 12:19:23.416467 27732 pending_rounds.cc:85] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 4
I20260709 12:19:23.416706 27732 pending_rounds.cc:107] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:19:23.416968 27732 raft_consensus.cc:2889] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:23.421942 30766 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:23.430503 31323 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260709 12:19:23.430860 31317 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.252s	user 0.005s	sys 0.001s
I20260709 12:19:23.431149 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:23.431468 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77287.4us
I20260709 12:19:23.432309 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:23.433248 31157 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:23.394887 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.436108 31252 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.442338 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.445415 27659 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:19:23.450197 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.451086 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:23.453714 31171 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:23.454533 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90193.6us
I20260709 12:19:23.412387 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:23.438730 27704 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
W20260709 12:19:23.438511 30827 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.533s	user 0.007s	sys 0.011s
I20260709 12:19:23.460640 27664 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:23.397835 28335 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.460026 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.415105 31232 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:23.464676 28335 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:23.467401 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:23.468246 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:23.468268 31286 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.468636 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.469573 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:23.467260 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.379097 30638 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.473165 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.393280 31225 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:23.488577 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.501969 27733 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:23.505240 27738 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:23.397320 31046 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:23.513288 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60106.6us
I20260709 12:19:23.477913 30995 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:23.503616 31205 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.511s	user 0.008s	sys 0.003s
I20260709 12:19:23.514339 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:23.513960 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:23.463340 30827 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:23.533385 31322 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.004s	sys 0.001s
W20260709 12:19:23.533977 31064 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.000s
I20260709 12:19:23.534904 27734 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:19:23.553611 30638 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.554637 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.555301 27734 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:23.556145 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.556322 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.556836 27734 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:23.556983 27656 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:23.535391 27581 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:19:23.536109 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.540588 31254 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.543843 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.544715 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.545609 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.547832 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.559016 27658 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:23.548624 31225 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:23.548934 31253 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.526937 30995 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.531786 31157 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:23.581885 30638 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.583081 31046 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.552748 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:23.585462 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:23.566746 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.586196 30995 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:23.586563 31284 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:23.586892 30995 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:23.586623 28336 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:23.578173 27737 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:23.581465 31157 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.582670 30638 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.591459 27732 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:23.592159 28335 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:23.579600 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:23.568286 31265 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.722s	user 0.011s	sys 0.003s
I20260709 12:19:23.563997 27664 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
W20260709 12:19:23.569065 31234 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.721s	user 0.001s	sys 0.015s
W20260709 12:19:23.568140 31306 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.663s	user 0.013s	sys 0.000s
I20260709 12:19:23.587365 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.569155 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.589388 27731 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:23.530653 31205 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.602730 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:23.603791 31016 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:19:23.604450 31205 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.599017 31234 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.607904 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.608623 27731 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:23.609683 27659 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:23.609760 31234 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.610541 31306 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:23.611182 31306 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:23.612078 31267 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:23.612287 31306 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.613219 31267 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:23.613638 31267 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:23.614151 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:23.615047 31286 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:23.607131 31205 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:23.615502 31286 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:23.614619 31267 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.616102 31286 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:23.617177 31267 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:23.619475 31252 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.624946 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.628011 31012 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:23.628723 31012 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.630908 30979 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.631569 31012 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.631520 30979 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.633482 27660 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:19:23.633956 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.636072 27655 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 6: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:23.637136 31309 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.460s	user 0.014s	sys 0.006s
I20260709 12:19:23.637486 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:23.638531 27655 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:23.633958 30979 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:23.640529 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:23.640756 27664 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:23.634907 27658 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:23.645823 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:23.647590 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57413.2us
I20260709 12:19:23.649513 27734 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:23.648501 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:23.646538 27733 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:23.599520 27729 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:23.652140 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:23.689028 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.690766 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2075) took 1178 ms. Trace:
I20260709 12:19:23.692791 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.693766 31046 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:23.693850 30995 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.694572 31046 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.694800 30995 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.695894 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.697310 31046 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:23.693804 27662 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:23.697815 30995 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.691247 27582 rpcz_store.cc:276] 0709 12:19:22.512429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:22.512680 (+   251us) service_pool.cc:224] Handling call
0709 12:19:22.513508 (+   828us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:23.144853 (+631345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:23.690500 (+545647us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b54001892d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:23.690701 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1328941824}
I20260709 12:19:23.699486 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 507) took 1813 ms. Trace:
I20260709 12:19:23.687889 27735 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:23.701881 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:23.687557 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:23.703819 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:23.705489 27587 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:23.705670 27586 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:23.706569 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:23.591238 31171 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.702756 27589 rpcz_store.cc:276] 0709 12:19:21.886152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.886420 (+   268us) service_pool.cc:224] Handling call
0709 12:19:23.699162 (+1812742us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:23.699394 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3143161216}
I20260709 12:19:23.714604 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:23.706537 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54506us
I20260709 12:19:23.715660 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.716888 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:23.697316 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.715677 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:23.716341 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
W20260709 12:19:23.686951 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0660970259998521s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260709 12:19:23.739818 27552 connection.cc:256] RPC call timeout handler was delayed by -1.8199958329998935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1579s for a response
W20260709 12:19:23.740368 27552 connection.cc:256] RPC call timeout handler was delayed by -1.7820803520000936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
W20260709 12:19:23.740806 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6133970260000297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
I20260709 12:19:23.739856 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:23.741223 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5170970259998739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22240000000000004s for a response
W20260709 12:19:23.742574 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5043481410000368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260709 12:19:23.743016 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4468566079999619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260709 12:19:23.743453 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4197001760001058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21190000000000003s for a response
W20260709 12:19:23.743872 27552 connection.cc:256] RPC call timeout handler was delayed by -1.36822366399997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
W20260709 12:19:23.744215 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2144488960000217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 12:19:23.744486 27552 connection.cc:256] RPC call timeout handler was delayed by -1.194495833000019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22740000000000002s for a response
I20260709 12:19:23.761559 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.767843 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.768740 27735 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:23.772251 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.773020 27729 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:23.777015 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.777835 27733 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:23.779666 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.780601 27729 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:23.787859 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.788782 27733 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:23.792412 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.591605 31225 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.703583 31012 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.715096 31234 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:23.806901 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.810286 27735 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:23.707129 31252 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.762176 30766 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:23.822415 27658 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:23.824278 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.719594 31171 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.823052 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:23.841064 27589 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:23.842878 27729 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
W20260709 12:19:23.847157 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.847113 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:23.838837 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.856483 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:23.847383 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.864027 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.866636 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.869176 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.870215 27660 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:23.808691 31012 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:23.873710 27733 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:23.893121 27735 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:19:23.816191 31046 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:19:23.893728 27735 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 4
W20260709 12:19:23.897517 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 300743us
I20260709 12:19:23.897917 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:23.901747 27586 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:23.903563 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.905202 27735 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:19:23.905679 27735 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:23.843365 31050 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:19:23.857156 30635 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.911743 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.908135 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.906868 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.865957 30766 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:23.913988 27660 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:23.918306 27733 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:23.909461 31205 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:23.925165 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.906157 31046 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:19:23.941777 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.942788 27735 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:23.945844 27660 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:23.919203 30827 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:23.952942 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:23.953850 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129545us
I20260709 12:19:23.920621 31284 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:23.961488 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.967669 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.968096 27730 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:23.968490 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:23.969552 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.970860 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:23.961707 31290 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:23.972442 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.973323 27580 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:23.974274 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.976058 27584 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:23.976564 27584 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:23.977799 27584 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:23.979007 27584 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:23.979755 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:23.980376 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:23.981128 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:23.981464 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:23.982553 27660 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20260709 12:19:23.982494 27730 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:23.982966 30992 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:23.983224 31334 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:19:23.984759 27733 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:19:23.989429 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:23.989729 27729 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:19:23.990425 28328 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:23.991091 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:23.992038 27731 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:19:23.992920 31328 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:19:23.979773 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:23.985829 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:23.984786 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73523.3us
I20260709 12:19:24.002305 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:23.993628 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 221851us
I20260709 12:19:23.993099 27735 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:24.003516 27658 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:24.010818 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.011783 27589 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:24.013440 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.017551 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:24.018460 28329 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:24.023241 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.029376 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.009626 31149 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.053790 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:24.054854 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.012095 30638 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:24.062462 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82773.7us
I20260709 12:19:24.063453 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.065780 28329 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:24.030036 31205 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:23.953317 30635 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.010658 30979 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:19:24.042888 31234 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:24.048631 31225 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.052270 31046 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.082512 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:24.094708 31168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.941s	user 0.007s	sys 0.019s
I20260709 12:19:24.079229 30995 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:19:24.135897 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72548.4us
I20260709 12:19:24.108976 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:24.108282 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76499.2us
W20260709 12:19:24.109341 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52136.7us
I20260709 12:19:24.079924 31342 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.108071 27660 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:24.092769 31252 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.101289 31343 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:24.108754 30827 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:24.127234 31168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.974s	user 0.008s	sys 0.019s
I20260709 12:19:24.144435 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.049770 30766 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:24.151060 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 136160us
I20260709 12:19:24.151454 31345 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:24.159369 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.148823 31342 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.109985 31225 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:24.146170 31178 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:24.099680 31149 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:24.175686 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.104904 31046 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.154740 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.182432 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.141733 31350 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:24.148717 30638 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:24.155726 31343 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.189754 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:24.193599 31337 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:19:24.160143 30995 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:24.223799 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:24.224246 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:24.193943 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57019.7us
I20260709 12:19:24.194141 31349 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.194618 31205 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:24.155730 31012 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:24.187721 27661 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:24.224752 27657 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:19:24.246028 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:24.231628 31352 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:19:24.184321 31178 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:24.235869 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81313.5us
I20260709 12:19:24.236393 31168 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:19:24.253896 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 199119us
I20260709 12:19:24.254468 27584 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:19:24.258056 27584 pending_rounds.cc:85] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 3
I20260709 12:19:24.255038 27658 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:24.256467 28329 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:24.195389 31350 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:24.257392 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 248566us
I20260709 12:19:24.181361 30987 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:19:24.260565 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.224134 31343 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:24.261587 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:24.258551 27584 pending_rounds.cc:107] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:19:24.262148 27661 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:19:24.262348 27584 raft_consensus.cc:2889] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:24.264904 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.264619 30995 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:24.267112 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48352.7us
I20260709 12:19:24.229252 31050 raft_consensus.cc:2764] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 12:19:24.259490 31333 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 12:19:24.228647 30638 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:24.280011 27655 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.560s	user 0.009s	sys 0.001s
I20260709 12:19:24.277025 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.302351 31178 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.303270 31046 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:24.303444 30635 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:24.303947 31178 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:24.304109 30635 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.304669 30635 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.305028 31149 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.282469 27664 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:24.306412 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:24.305986 27655 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:24.273474 31306 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:24.311998 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.310448 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.275401 31252 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:24.285621 27657 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:24.266691 31350 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:24.315709 31363 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:24.318070 31363 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:24.318439 31363 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:24.319767 31364 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:24.320223 31364 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:24.320597 31364 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:24.320845 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.260313 31349 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.324570 31351 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:24.330291 31342 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.330695 30635 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.330824 31351 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.281325 31347 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 12:19:24.331301 31351 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:24.328665 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.256841 31234 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.327857 31168 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:24.347954 31168 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:24.349136 31168 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.349748 31364 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:24.350212 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68861.3us
I20260709 12:19:24.351636 31306 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.352169 31306 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.354305 31306 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.303948 31046 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:24.328168 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.327584 30992 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:24.317274 31252 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.340155 31157 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:24.340862 31286 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.724s	user 0.002s	sys 0.010s
I20260709 12:19:24.356503 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.359023 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:24.363718 30992 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.367147 31110 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.367712 31252 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:24.409858 31351 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.411410 31359 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:24.412127 31359 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.412606 31359 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:24.415934 31253 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.366156 30635 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:24.424538 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.429749 27587 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:24.365327 31342 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:24.398164 31350 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.401059 31349 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.404119 30827 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:24.389825 30638 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.406594 31363 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.364967 31347 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:19:24.453953 31363 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.375736 31306 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:24.455582 31306 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:24.456107 31306 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:24.420650 31149 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.459735 31149 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:24.421180 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 195743us
I20260709 12:19:24.462203 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.421823 31178 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.422883 31364 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:24.426330 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.472055 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.472470 27588 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:24.472616 27587 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:24.427274 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.431344 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.431644 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:24.432188 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75896.2us
I20260709 12:19:24.439209 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.391073 30992 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:24.475647 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:24.474264 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:24.445719 31350 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.433060 31253 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.441449 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.481379 27659 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:24.481737 27582 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:19:24.451761 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:24.469482 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.430616 30635 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.479348 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.450829 31343 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.376096 31286 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.486577 31351 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.492552 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:24.497475 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 270385us
I20260709 12:19:24.498728 27584 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:24.469496 31342 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:24.500523 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.420658 31110 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.479269 30827 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:24.520068 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.503123 27659 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:24.488040 31284 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:24.489672 30635 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:24.523790 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:24.536705 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67395.8us
I20260709 12:19:24.538009 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:24.538877 27656 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:24.545743 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47271.5us
I20260709 12:19:24.547412 31050 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:24.547789 31050 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.548120 31050 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:24.548208 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.549860 31046 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:24.555428 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.556245 27584 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:19:24.536295 31064 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:24.558290 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.559087 27585 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:24.563213 31050 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.565536 31050 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:24.566143 31050 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:24.481848 31178 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.569289 31178 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.576253 31050 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:24.577600 27736 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:24.579809 31110 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.579905 31342 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.582480 31286 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.497748 31253 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.532166 31284 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:24.584434 31284 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:24.595285 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.596241 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:24.598104 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:24.598661 27729 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:24.599534 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.600713 27656 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:24.605965 31343 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.606285 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:24.607018 27732 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:24.607620 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.609470 31171 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.611184 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.611567 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:24.611832 27738 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:24.612113 27731 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:24.612764 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.613361 27589 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:24.614487 30827 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.615594 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.616182 27730 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:24.615036 30827 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.619961 31361 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:24.620548 31361 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:24.621281 30827 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:24.628477 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.632515 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:24.603466 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.634824 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:24.635037 31149 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.635263 27730 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:19:24.635464 27656 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:24.614732 31359 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.637336 31253 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.637701 31359 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.637939 31252 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.638387 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:24.632877 30992 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.640223 31252 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.644029 27584 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:24.644718 27656 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:24.645043 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.645900 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.622138 30827 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:19:24.648521 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 154063us
I20260709 12:19:24.654510 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.655185 27733 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:24.655372 27729 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:24.647464 27732 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:24.672526 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:24.651418 31252 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:24.679075 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.656580 31251 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:24.653687 30827 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:24.689625 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 141494us
I20260709 12:19:24.685173 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 494) took 1429 ms. Trace:
I20260709 12:19:24.686026 30638 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:24.657388 31359 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:24.690889 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.691561 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:24.692689 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.694505 27732 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:24.697896 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.691414 27737 rpcz_store.cc:276] 0709 12:19:23.273480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:23.279776 (+  6296us) service_pool.cc:224] Handling call
0709 12:19:24.681938 (+1402162us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:24.682071 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665438976}
I20260709 12:19:24.702131 27731 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:24.621749 31361 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.703986 28329 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:19:24.704167 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:24.708242 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.710570 27731 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:24.689247 30995 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:24.692274 31251 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.710996 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:24.713045 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 158045us
I20260709 12:19:24.714134 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:24.693406 30827 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:24.693586 30638 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.693835 31252 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:24.735144 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.703117 31369 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.684969 30635 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.721973 31361 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:24.720309 28335 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:24.717226 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:24.716019 31343 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:24.695494 31359 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:24.732920 28334 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:24.734328 27552 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:24.734974 30638 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:24.741813 31343 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:24.743116 31343 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:24.736073 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:24.748381 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.749841 30992 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:24.749701 31168 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:24.738639 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.717537 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:24.707683 31342 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:24.730942 31251 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:24.756482 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.762363 31267 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:24.762877 31267 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:24.763374 31267 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:24.763518 31064 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:19:24.764725 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.765411 27583 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:24.757845 31360 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.427s	user 0.003s	sys 0.000s
I20260709 12:19:24.758739 31241 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:24.758809 31306 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.770987 27587 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:24.771735 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.779641 27584 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:24.773387 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.773149 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.780030 27659 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:24.736786 31332 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:24.750835 31369 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.785225 27737 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:24.802289 31369 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:24.786813 31205 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:19:24.786906 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69768.8us
I20260709 12:19:24.747222 31252 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.740835 31348 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:24.787549 31378 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:24.787606 31016 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.781648 31241 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.751178 31359 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:24.793069 31306 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:24.803287 31372 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:19:24.807008 31253 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:19:24.809670 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.775249 31286 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:24.819933 31286 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:24.820825 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.820942 31286 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.821796 30995 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:24.718654 31232 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 12:19:24.810043 31348 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.823553 31232 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:24.823827 31348 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:24.823982 31232 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:24.825368 27589 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:24.799439 30992 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:24.826328 30992 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:24.827368 30992 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.827745 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.828476 27588 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:24.811985 31016 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.769348 31379 raft_consensus.cc:2764] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:24.832049 31016 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.832626 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.834329 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:24.822881 30995 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:24.824962 31232 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.834708 27583 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:24.794255 30638 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:24.813046 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:24.813442 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.798058 31284 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:24.814728 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.816064 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.753175 31342 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:24.846884 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.805918 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:24.847889 27656 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:24.812913 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.841738 27584 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
W20260709 12:19:24.852159 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 154332us
W20260709 12:19:24.854712 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48903.4us
I20260709 12:19:24.855764 27662 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:24.854226 27584 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:24.851840 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:24.858526 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.859400 27585 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:24.859817 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.860416 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:24.862591 27587 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:24.811672 31378 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.864094 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.808518 31205 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:24.813793 31241 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:24.841054 30995 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.930713 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.916077 27663 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:24.906597 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.910208 27664 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[5.2-11.5]   Dedup: 5.2->[7.3-11.5]
I20260709 12:19:24.937320 27664 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 2
I20260709 12:19:24.937700 27664 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:19:24.937345 27656 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:24.938074 27664 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:24.913379 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:24.939401 27658 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:24.910544 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 97570.5us
I20260709 12:19:24.909224 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:24.879485 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.943979 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.945887 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.850499 31342 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:24.948000 27661 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12.
I20260709 12:19:24.912341 27586 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:24.913830 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:24.867242 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.955590 27589 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:19:24.956044 27589 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 9
I20260709 12:19:24.956305 27589 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:19:24.936671 27657 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:24.956702 27589 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:24.849809 31379 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:24.957906 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:24.846983 31046 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:24.965441 27585 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:24.866283 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.966871 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:24.816887 31306 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260709 12:19:24.857220 31064 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:24.971534 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.865899 31351 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:24.975848 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.930929 31369 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.975360 31149 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:24.914119 31232 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:24.910696 31347 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:24.946682 30995 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:24.992901 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.995190 27584 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:25.002475 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.004011 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.000391 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.023409 27585 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.005610 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:24.865425 31178 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.127s
I20260709 12:19:25.006573 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:24.913667 31378 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:25.009465 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:25.010385 27737 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:25.082202 27587 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:24.905431 31363 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.098s
I20260709 12:19:25.003577 31379 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.022325 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:25.026329 27586 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:25.091096 27657 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:19:25.036250 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:25.052822 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 6 to 10, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 10 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:25.060945 30635 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:25.031880 31381 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:25.102875 27581 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:25.051766 31064 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:25.060037 30638 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:25.122972 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165126us
I20260709 12:19:25.021572 31351 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:25.011389 31369 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.012701 31149 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.010035 31326 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:25.064981 27729 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:25.056452 27732 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:19:25.135756 27587 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:25.012895 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:25.081362 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:24.999629 31267 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.080263 27588 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:25.078418 31332 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.049s
W20260709 12:19:25.107615 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144898us
I20260709 12:19:25.109179 27585 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:25.130170 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.102623 31381 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.089864 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:25.138787 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.153591 28334 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:25.153769 27737 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:19:25.144793 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63418us
I20260709 12:19:25.145947 27586 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:19:25.146613 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.155975 27732 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:25.157352 27731 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:25.103813 31241 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 12:19:25.161917 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.163667 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.135363 31149 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.077167 31205 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.134893 31326 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.178963 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.152022 31332 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:25.168437 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:25.177843 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.182873 27729 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:25.168323 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:25.153028 31168 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:25.102732 31379 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.178058 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.180944 27659 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:25.195060 27589 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:19:25.155026 31381 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:25.200469 27736 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:19:25.166169 31359 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:25.188063 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57991.1us
I20260709 12:19:25.137534 30827 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.134819 31365 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20260709 12:19:25.193876 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 169249us
I20260709 12:19:25.138615 31290 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:19:25.203194 27701 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12.
I20260709 12:19:25.209466 27701 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:19:25.178368 31377 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:25.181824 31326 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.197630 31241 raft_consensus.cc:2764] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:19:25.214452 31350 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:25.212970 31377 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:25.233127 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.199761 31348 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.220312 31369 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.221658 27589 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:25.226058 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.226326 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:25.177564 31284 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:25.193697 31168 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:25.242506 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.253319 27736 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:25.249162 27584 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:25.234592 27589 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:25.263885 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68991.5us
W20260709 12:19:25.249537 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65241.4us
I20260709 12:19:25.265671 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:25.230198 31205 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:25.270839 27662 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:19:25.242872 31377 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:25.288262 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.291973 27736 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:25.292209 27552 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:25.292518 27552 leader_election.cc:403] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:19:25.258766 31252 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:25.257180 31168 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:25.234583 31016 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
W20260709 12:19:25.314949 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49372.4us
I20260709 12:19:25.315953 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:25.203244 31342 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.303478 31252 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:25.303903 31267 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:25.304996 31251 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.545s	user 0.006s	sys 0.006s
I20260709 12:19:25.311491 31050 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:19:25.353669 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.288551 31389 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:19:25.233505 31351 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.388182 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:25.314343 31388 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:25.389038 31050 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:25.389485 31050 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:25.390336 31050 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.391595 31050 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:25.386144 31251 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.394029 31251 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:25.395047 27660 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
W20260709 12:19:25.398517 31361 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.646s	user 0.009s	sys 0.000s
I20260709 12:19:25.320581 27658 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:25.328215 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.401624 27729 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:25.345240 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.298754 27589 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.353132 27583 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:25.354879 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:25.358458 31284 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:25.290926 30992 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:19:25.366294 31378 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.328925 31241 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
I20260709 12:19:25.371299 31386 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.373648 27655 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:19:25.377614 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:25.378123 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50355.7us
I20260709 12:19:25.429616 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:25.378983 31252 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.381458 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.381588 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.382546 27738 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:19:25.432969 27730 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:25.350512 30995 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.385607 31342 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.386144 28336 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:25.386929 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.295750 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
W20260709 12:19:25.411966 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57146.8us
I20260709 12:19:25.440869 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:25.441774 31284 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:25.418608 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.442322 31284 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.412859 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.396801 31326 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:25.367386 31382 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 12:19:25.425891 27589 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.317158 31390 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:25.430399 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:25.431875 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:25.446121 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:25.441450 31306 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:25.441931 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:25.442631 27585 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:25.451508 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.443975 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.452586 27582 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:19:25.453015 27730 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:25.442811 31284 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:25.423414 27729 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:25.444530 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.454041 27734 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:25.433357 31290 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:25.429710 31389 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:19:25.449070 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 246022us
I20260709 12:19:25.461891 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:25.453822 31064 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:19:25.464828 30995 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.427621 31397 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:25.459611 31378 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.503749 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:25.506811 27658 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:25.481760 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.443941 31267 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:25.508605 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:25.468405 31384 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 12:19:25.509984 31064 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:25.475334 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:25.524403 27662 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:25.508760 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.515509 27738 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:25.464828 30827 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:25.470844 31343 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:25.473156 31365 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:25.536338 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.537742 27589 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:19:25.538302 27589 pending_rounds.cc:85] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 7
I20260709 12:19:25.538725 27589 pending_rounds.cc:107] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:19:25.538995 27589 raft_consensus.cc:2889] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:25.539314 27589 pending_rounds.cc:107] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:19:25.539616 27589 raft_consensus.cc:2889] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:25.542045 31359 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:25.530655 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:25.550441 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.551050 27738 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:25.523978 31064 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:25.443907 31386 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:25.543294 31401 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:19:25.471725 31241 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:25.472455 31290 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:25.507138 30992 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:19:25.551425 27658 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:25.548249 27662 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:25.471974 31342 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:25.599088 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.537436 31365 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.567830 27487 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 9, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 9 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:25.545976 31348 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:25.639535 27738 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
W20260709 12:19:25.640973 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 194926us
I20260709 12:19:25.601487 31343 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:25.580155 31399 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:25.579763 31405 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:25.649345 31405 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:25.649135 31399 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.650404 31405 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:25.645955 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.540812 31381 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.630770 31290 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:25.633270 31241 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:25.661350 27738 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:19:25.647749 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:25.652854 31369 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:25.657084 27580 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: [7.5-12.7]
I20260709 12:19:25.660322 31406 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.554906 31168 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.606302 31389 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:19:25.556945 31377 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.663797 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:25.657269 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.668262 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.642953 31379 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.672361 27732 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:25.671089 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.648459 31365 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.674582 27583 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:25.676669 31168 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:25.679301 31241 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.691654 31343 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:25.693768 31398 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:19:25.695713 31326 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:25.696189 30827 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:25.696625 30827 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:25.697587 30827 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.681772 27589 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:25.698912 30827 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:25.671680 31064 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.701028 31064 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:25.704778 31381 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.685918 27580 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:25.706995 27487 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 11 to 13, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 13 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:25.659626 31342 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:25.687944 30995 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:25.688410 31306 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:25.689352 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:25.689834 31412 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:25.684257 31406 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.687521 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:25.657990 31267 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:19:25.665814 31369 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:25.719870 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.725132 31389 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:25.726048 31389 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:25.726361 31389 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:25.726445 31406 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:25.730537 31149 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.549s	user 0.009s	sys 0.001s
I20260709 12:19:25.730966 31149 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.733911 27656 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:25.714637 31306 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.742689 31149 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:25.745210 31267 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:25.746928 31386 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:25.747620 28336 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:25.748283 28336 raft_consensus.cc:606] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.748618 28336 raft_consensus.cc:614] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:25.748852 31397 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:25.749495 31388 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12.
I20260709 12:19:25.755870 31168 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:25.756287 31168 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:25.757296 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:25.757159 31168 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.758037 28332 raft_consensus.cc:2368] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:25.759560 27702 leader_election.cc:400] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:25.759904 27702 leader_election.cc:403] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:19:25.762737 27735 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:25.716836 31399 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.724012 30995 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:25.767310 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:25.781338 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:25.794786 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:25.807814 31342 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:25.793996 31284 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.814218 31284 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.831312 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:25.831986 27730 raft_consensus.cc:606] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.832430 27730 raft_consensus.cc:614] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:25.727205 31389 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.768800 31388 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.833503 31389 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:25.759438 31306 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:25.835377 31205 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.001s
I20260709 12:19:25.836280 27733 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:25.840260 31377 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:25.840868 31388 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
W20260709 12:19:25.842809 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 426100us
I20260709 12:19:25.842970 31395 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:25.844208 31381 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:25.844607 31381 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:25.845590 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:25.864928 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 193141us
I20260709 12:19:25.845674 31381 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:25.846776 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 434043us
I20260709 12:19:25.848134 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.848845 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.882067 27589 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.850697 31364 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:25.851693 30995 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:19:25.855000 31369 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.885241 27737 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:25.888509 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.889490 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.890928 27738 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:25.892287 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.893812 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:25.895517 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:25.896265 27737 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:25.861605 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:25.881531 27730 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:19:25.883550 31364 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.898279 31364 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:25.906913 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61414us
I20260709 12:19:25.907754 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:25.910277 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48825.4us
I20260709 12:19:25.910545 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:25.864109 31359 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:25.913938 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.915144 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:25.915907 27655 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:25.918529 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:25.919109 27662 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:25.919206 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:25.919751 27736 raft_consensus.cc:606] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:25.920089 27736 raft_consensus.cc:614] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:19:25.922879 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 198471us
I20260709 12:19:25.929113 27731 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:25.931834 27585 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:25.934137 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.930409 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:25.935307 27589 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:25.938925 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:25.933323 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:25.940239 27585 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:25.941665 27661 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:19:25.942112 27661 pending_rounds.cc:85] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:19:25.942371 27661 pending_rounds.cc:107] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:19:25.942653 27661 raft_consensus.cc:2889] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:25.936018 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:25.946199 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:25.946983 27657 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260709 12:19:25.948066 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:25.948660 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.948000 27662 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:25.913972 31363 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:25.913081 31253 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:25.936285 31381 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:25.960973 27662 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:19:25.925642 31391 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:25.970465 31415 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:25.928750 31359 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:25.959846 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:25.972124 27586 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
W20260709 12:19:25.975800 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.944587 31251 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:26.003240 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:25.918051 31171 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 12:19:26.020095 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:26.020993 27661 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:26.023857 27663 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:26.019505 31016 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:19:25.943909 31050 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
W20260709 12:19:26.026774 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.010474 31415 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:26.026652 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:26.028466 31415 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.029672 31415 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:26.026628 31050 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:25.948915 30995 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:19:26.036211 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2111) took 1080 ms. Trace:
W20260709 12:19:26.047763 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.049758 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:26.051288 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:26.058909 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:26.058981 27657 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:26.046608 27656 rpcz_store.cc:276] 0709 12:19:24.955572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:24.986741 (+ 31169us) service_pool.cc:224] Handling call
0709 12:19:26.035910 (+1049169us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:26.036126 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1621046144}
I20260709 12:19:26.069229 27733 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:19:26.103214 27585 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:25.958591 31394 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 12:19:26.066277 27487 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:26.114615 31405 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.189761 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:26.056948 31253 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:26.033366 31050 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:26.209452 31399 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:26.055688 31251 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:26.115250 31016 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.114614 31205 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:26.242882 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2891176729999643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
W20260709 12:19:26.333081 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 394228us
W20260709 12:19:26.349694 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2801176729999497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
W20260709 12:19:26.350147 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2684176729999308s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 12:19:26.350517 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0533176729998104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17920000000000003s for a response
W20260709 12:19:26.350847 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0398176730000159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
W20260709 12:19:26.351186 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0353176730000087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
I20260709 12:19:26.256541 31253 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:26.152529 31434 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:19:26.358130 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 444213us
I20260709 12:19:26.354596 31016 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.359066 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:26.360486 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:26.361004 31016 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:26.354800 31205 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.363902 31251 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.364710 31205 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:26.361629 31016 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:26.352766 31399 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:26.367311 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.368206 27735 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:26.224638 30766 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.324215 31417 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.367789 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:26.372510 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.223995 31405 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.376159 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:26.251971 31379 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:26.378396 31379 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.379300 27585 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:26.148598 31290 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.380740 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:26.381918 31290 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:26.324486 31397 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:26.381685 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:26.385372 31417 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.387679 31417 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:26.388185 30766 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.390611 30766 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:26.391098 31205 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:26.396878 31394 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:26.397769 31394 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:26.398231 31394 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:26.400533 31399 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:26.401273 31399 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:26.401690 31064 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.700s	user 0.006s	sys 0.006s
I20260709 12:19:26.401834 31399 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.352744 31253 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:26.403667 31171 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.404423 31171 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.407534 31050 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.408643 31050 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:26.414814 31365 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.700s	user 0.007s	sys 0.000s
W20260709 12:19:26.416086 31425 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:19:26.416291 31359 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:26.421890 31306 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.000s
I20260709 12:19:26.422452 31306 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.427498 31397 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.366031 31016 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:26.430347 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78537.2us
W20260709 12:19:26.435124 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53521.1us
W20260709 12:19:26.436425 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.449154 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.449715 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:26.448402 28332 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
W20260709 12:19:26.450392 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.450806 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:26.209959 31388 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:26.437799 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66799.6us
I20260709 12:19:26.457370 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:26.467487 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.465200 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:26.433732 30995 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.454763 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:26.499445 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.453774 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.433782 31406 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:26.429425 31394 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:26.429329 31171 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:26.452144 31064 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.452500 31359 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:26.529227 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.519641 27587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.037s	sys 0.080s
W20260709 12:19:26.543923 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.544013 27587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.038s	sys 0.080s
W20260709 12:19:26.538926 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73836.2us
I20260709 12:19:26.429656 31365 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.452241 31326 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.119s
I20260709 12:19:26.471140 31417 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.139s
W20260709 12:19:26.433688 31205 consensus_queue.cc:1175] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 8 status { last_received { term: 7 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:19:26.718159 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.014s	sys 0.007s
I20260709 12:19:26.709156 27586 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:19:26.763936 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.709012 27583 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.020s	sys 0.033s
W20260709 12:19:26.836689 27583 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.893s	user 0.020s	sys 0.041s
W20260709 12:19:26.743845 28336 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.009s	sys 0.013s
W20260709 12:19:26.838244 28336 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.884s	user 0.010s	sys 0.018s
W20260709 12:19:26.545545 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.835355 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 378031us
W20260709 12:19:26.835831 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.014s	sys 0.014s
W20260709 12:19:26.519654 30638 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.040s	sys 0.009s
W20260709 12:19:26.526587 31149 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.004s	sys 0.010s
W20260709 12:19:26.850327 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 395628us
I20260709 12:19:26.862701 27627 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:26.864717 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.031s	sys 0.004s
W20260709 12:19:26.865976 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.031s	sys 0.004s
I20260709 12:19:26.511658 31386 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:26.606356 30766 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:19:26.857920 30638 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.042s	sys 0.020s
I20260709 12:19:26.837495 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:26.867628 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:26.471020 31412 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.391s
I20260709 12:19:26.494962 31388 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.736382 31397 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:26.624718 31178 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.251s
I20260709 12:19:26.571877 31359 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:26.720289 31432 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:26.883682 31417 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:26.833936 31431 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:26.611564 31171 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.267s
I20260709 12:19:26.868772 31405 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:26.599563 31064 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.866137 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:26.888818 31364 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.990s	user 0.006s	sys 0.006s
I20260709 12:19:26.977362 31433 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:26.977788 31415 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.948s	user 0.012s	sys 0.000s
I20260709 12:19:26.977922 31364 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.979684 31405 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.980185 31405 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:26.981086 27660 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:26.982303 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1742) took 1067 ms. Trace:
I20260709 12:19:26.735730 31423 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.870855 31241 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:26.624820 31429 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.281s
W20260709 12:19:26.953649 27661 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.932s	user 0.011s	sys 0.004s
W20260709 12:19:26.957032 27657 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.810s	user 0.008s	sys 0.003s
I20260709 12:19:26.991475 27661 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:26.991582 27657 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:26.958884 27586 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:19:26.992250 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1755) took 1029 ms. Trace:
I20260709 12:19:26.906153 31432 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:26.906755 31359 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:26.975975 31426 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:19:26.959653 31434 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.008s	sys 0.011s
I20260709 12:19:26.976950 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:26.599354 30995 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:26.982378 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.009s	sys 0.022s
W20260709 12:19:27.003346 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.009s	sys 0.023s
I20260709 12:19:26.889153 31427 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:26.992731 27661 rpcz_store.cc:276] 0709 12:19:25.963383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.984309 (+ 20926us) service_pool.cc:224] Handling call
0709 12:19:26.992179 (+1007870us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:26.982707 27660 rpcz_store.cc:276] 0709 12:19:25.915302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.915431 (+   129us) service_pool.cc:224] Handling call
0709 12:19:25.915704 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:26.981502 (+1065798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:26.982064 (+   562us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:26.982221 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2449654144}
W20260709 12:19:26.987519 27658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.004s	sys 0.019s
I20260709 12:19:26.956331 31306 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:27.006224 27658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.005s	sys 0.019s
I20260709 12:19:27.001355 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:26.951485 31399 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.002846 27589 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:19:26.982245 31405 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:26.984495 31386 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:27.008129 27580 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:26.994287 31171 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:27.022171 27552 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9.
W20260709 12:19:27.026368 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.029496 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.029991 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.964190 31431 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:26.994359 31432 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:27.000105 31434 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:27.000696 31433 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:27.041800 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:26.994671 31241 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:27.043970 31435 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:19:27.044687 31431 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.040421 31149 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:27.045365 31440 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:19:27.046482 31420 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:19:27.046967 31306 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:27.047369 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:27.047703 31444 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.048015 31364 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:27.048153 31430 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:26.996299 31423 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.049266 31412 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:27.049998 31388 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.050478 30992 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:27.009783 31016 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:27.006462 31379 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.052594 27586 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:19:27.054306 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.126698 27586 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:27.051534 31171 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:26.944979 31417 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.055734 28332 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:27.019044 31406 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:27.065343 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.136171 27580 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:27.079707 30766 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:27.030200 31168 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:27.065004 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 529) took 1135 ms. Trace:
I20260709 12:19:27.076892 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:27.078166 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.030066 31386 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.058462 31426 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:19:27.074177 31149 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:27.081947 31405 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.061228 31361 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:19:27.133960 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:27.065519 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 185369us
I20260709 12:19:27.119114 31415 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:27.119397 31365 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:27.153937 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97905.6us
I20260709 12:19:27.061391 31433 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:27.178298 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44422.8us
I20260709 12:19:27.179555 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1752) took 1238 ms. Trace:
W20260709 12:19:27.161634 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.011s	sys 0.041s
I20260709 12:19:27.151074 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:27.165560 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:27.183540 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.011s	sys 0.041s
I20260709 12:19:27.185446 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.188251 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.188949 27580 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:27.190922 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:27.193046 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 108128us
I20260709 12:19:27.194591 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:27.199553 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.200299 27588 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:27.212687 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.179956 27662 rpcz_store.cc:276] 0709 12:19:25.940989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.991413 (+ 50424us) service_pool.cc:224] Handling call
0709 12:19:25.991857 (+   444us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:26.017409 (+ 25552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:26.017451 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:26.018216 (+   765us) log.cc:844] Serialized 52 byte log entry
0709 12:19:26.191585 (+173369us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:26.886634 (+695049us) log.cc:844] Serialized 14 byte log entry
0709 12:19:26.901977 (+ 15343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:26.902110 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:27.179291 (+277181us) raft_consensus.cc:1685] finished
0709 12:19:27.179309 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:27.179486 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61292,"prepare.queue_time_us":766761,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":533429,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":20321,"thread_start_us":627323,"threads_started":3,"wal-append.queue_time_us":842629}
W20260709 12:19:27.215272 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.088232 31430 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:27.179695 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.542s (SENT) [suppressed 20 similar messages]
I20260709 12:19:27.219882 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.080544 31444 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.227988 27580 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:19:27.229393 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 363318us
I20260709 12:19:27.131486 30992 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.237178 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.237922 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.231267 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:27.232241 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:27.238709 28332 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:27.221771 31378 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.225471 31399 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:27.244930 31378 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.245291 31399 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.247498 31378 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:27.135138 31454 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.093s
I20260709 12:19:27.212405 31306 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:27.259742 31064 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.260421 31451 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:27.261029 31451 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:27.261840 31394 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.651s	user 0.004s	sys 0.005s
I20260709 12:19:27.262274 31394 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.263721 31428 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:27.263481 31406 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.264154 31428 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:27.264569 31428 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:27.251235 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2138) took 1324 ms. Trace:
I20260709 12:19:27.267655 27583 rpcz_store.cc:276] 0709 12:19:25.926764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.927504 (+   740us) service_pool.cc:224] Handling call
0709 12:19:25.927803 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:26.857228 (+929425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:26.857264 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:26.877955 (+ 20691us) log.cc:844] Serialized 26 byte log entry
0709 12:19:27.086997 (+209042us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:27.087044 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.089251 (+  2207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:27.251004 (+161753us) raft_consensus.cc:1685] finished
0709 12:19:27.251024 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:27.251164 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1141,"raft.queue_time_us":102947,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":95188,"thread_start_us":1034931,"threads_started":2,"wal-append.queue_time_us":268876}
I20260709 12:19:27.261854 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1754) took 1302 ms. Trace:
W20260709 12:19:27.288064 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55904.8us
W20260709 12:19:27.290772 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59575.8us
I20260709 12:19:27.291985 27627 leader_election.cc:400] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:19:27.292467 27627 leader_election.cc:403] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:19:27.322175 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:27.325349 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:27.271783 27658 rpcz_store.cc:276] 0709 12:19:25.959000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.007931 (+ 48931us) service_pool.cc:224] Handling call
0709 12:19:26.008630 (+   699us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:26.137666 (+129036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:26.137718 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:26.366180 (+228462us) log.cc:844] Serialized 52 byte log entry
0709 12:19:27.006636 (+640456us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:27.006671 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.006741 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:27.261619 (+254878us) raft_consensus.cc:1685] finished
0709 12:19:27.261640 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:27.261792 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126425,"raft.queue_time_us":384817,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":640011,"threads_started":1,"wal-append.queue_time_us":763486}
I20260709 12:19:27.232151 31412 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.232300 31016 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:27.173735 31405 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.342128 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:27.342943 27583 raft_consensus.cc:2368] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11.
W20260709 12:19:27.343564 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 148993us
W20260709 12:19:27.335430 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 250499us
I20260709 12:19:27.336086 31064 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:19:27.161010 31434 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.336438 31459 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:19:27.337088 31437 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:27.337524 31447 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:27.374840 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.337667 31149 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.338913 31394 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:27.372786 27630 leader_election.cc:400] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11.
I20260709 12:19:27.381305 27630 leader_election.cc:403] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:19:27.256969 31440 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:27.319768 31451 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.328922 31178 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9.
I20260709 12:19:27.340539 31412 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.377533 31378 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.381764 31365 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.352792 30995 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:27.393959 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.364713 31399 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:27.378898 31426 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
W20260709 12:19:27.371400 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 157195us
I20260709 12:19:27.369273 31016 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:27.404104 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.407464 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.409127 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.411412 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:27.411876 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.414486 27658 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
W20260709 12:19:27.417588 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 43493.2us
I20260709 12:19:27.420497 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.423447 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:27.424049 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:27.424935 27731 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:27.426461 27730 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
W20260709 12:19:27.428050 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.411309 31378 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.430347 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:27.430521 31440 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.411418 31365 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.433486 31365 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:27.434099 31458 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:27.434760 31423 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.372876 31290 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:19:27.436606 27735 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
W20260709 12:19:27.395581 30827 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.685s	user 0.012s	sys 0.001s
I20260709 12:19:27.267247 31171 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.403839 31451 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.414263 31064 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:19:27.443053 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.391955 31241 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:27.438792 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:27.450147 27736 raft_consensus.cc:606] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:27.453323 27736 raft_consensus.cc:614] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:27.318579 30766 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.440641 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:27.417186 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84664.2us
I20260709 12:19:27.456933 27580 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:27.429121 31394 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:27.455588 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:27.444450 31447 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:27.434916 31456 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.106s
I20260709 12:19:27.460132 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:27.532882 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:19:27.520426 31378 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.447561 31267 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:19:27.453951 31326 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:19:27.510607 27583 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:27.453323 31361 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:19:27.462420 27731 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: [8.5-11.6]
I20260709 12:19:27.457002 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:27.457736 31477 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:27.434792 31458 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.493515 31471 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:27.492120 31442 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:27.494151 31457 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:19:27.520648 31306 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.016s
W20260709 12:19:27.462919 31253 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.056s	user 0.011s	sys 0.001s
I20260709 12:19:27.455037 31451 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.539847 31016 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:27.540871 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1751) took 1602 ms. Trace:
I20260709 12:19:27.541471 31476 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:27.520653 31426 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:27.654808 31389 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.821s	user 0.016s	sys 0.000s
W20260709 12:19:27.548753 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91836.3us
I20260709 12:19:27.552717 27729 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:27.434120 31386 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.687616 27734 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
W20260709 12:19:27.546602 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 366887us
I20260709 12:19:27.598613 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:27.695228 27738 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
W20260709 12:19:27.548259 31381 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.545s	user 0.016s	sys 0.000s
I20260709 12:19:27.699679 27661 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:19:27.538499 31050 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.129s	user 0.015s	sys 0.000s
I20260709 12:19:27.592659 31442 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:27.550069 31477 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:27.652880 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:27.627450 27659 rpcz_store.cc:276] 0709 12:19:25.938326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.957150 (+ 18824us) service_pool.cc:224] Handling call
0709 12:19:25.957611 (+   461us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:19:26.327572 (+369961us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:19:26.947549 (+619977us) log.cc:844] Serialized 14 byte log entry
0709 12:19:27.133926 (+186377us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:19:27.176093 (+ 42167us) log.cc:844] Serialized 78 byte log entry
0709 12:19:27.176941 (+   848us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:27.177022 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.179221 (+  2199us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:27.538161 (+358940us) raft_consensus.cc:1685] finished
0709 12:19:27.538194 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:27.540768 (+  2574us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":138115,"prepare.queue_time_us":679296,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":42,"raft.queue_time_us":179578,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":63536,"thread_start_us":906780,"threads_started":3,"wal-append.queue_time_us":469856}
W20260709 12:19:27.670856 31463 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:19:27.444274 31437 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:19:27.679729 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:27.712906 27737 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:27.683598 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:27.681037 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:27.684168 31388 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.624840 31433 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.703526 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:27.672734 31476 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:27.704257 31417 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:27.705022 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 518482us
I20260709 12:19:27.656201 31430 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:27.676156 31168 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.003s
I20260709 12:19:27.707854 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:27.656991 31444 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:27.673301 31394 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:27.710331 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:27.710723 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:27.710875 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58059.3us
I20260709 12:19:27.592868 31471 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.629140 31253 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:27.703962 31455 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 12:19:27.901297 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:27.714293 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.772140 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.114573 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:27.739790 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:28.116847 27583 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:19:27.918104 27581 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:19:27.872469 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.730686 27657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1753) took 1784 ms (client timeout 1711 ms). Trace:
I20260709 12:19:28.097699 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:28.118844 27582 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
W20260709 12:19:27.716511 31466 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:19:27.901541 31050 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.126483 31050 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.129240 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:28.130101 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:27.752100 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:28.113958 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:28.134528 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:28.118597 27657 rpcz_store.cc:269] 0709 12:19:25.946356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.959226 (+ 12870us) service_pool.cc:224] Handling call
0709 12:19:27.730482 (+1771256us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b5400344254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:27.730611 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88410368}
I20260709 12:19:28.109161 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:28.134593 27580 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:28.135486 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.767539 31363 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.709s	user 0.033s	sys 0.076s
I20260709 12:19:27.703521 31482 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:28.129509 31432 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.085s	user 0.006s	sys 0.005s
I20260709 12:19:28.153796 27585 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:19:27.967370 31359 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.962s	user 0.010s	sys 0.002s
I20260709 12:19:27.769605 31377 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:27.713564 30992 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:27.706314 31386 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:27.705641 31379 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.714988 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:28.155709 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.717347 31465 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:19:27.712044 31477 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:28.174718 31482 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:28.179744 31470 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.649s	user 0.000s	sys 0.001s
W20260709 12:19:28.155846 31363 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.125s	user 0.040s	sys 0.115s
I20260709 12:19:28.158131 31326 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:27.710707 31483 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:27.983157 31251 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.715584 31475 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.186259 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:28.173442 31476 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:27.767441 31405 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:27.767643 31427 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:27.714277 31442 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:27.687513 31426 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:28.155486 31447 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.616s	user 0.011s	sys 0.000s
I20260709 12:19:28.199182 27662 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:28.151675 31444 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:28.206425 31378 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:28.207892 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.277544 31483 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.204135 31432 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:28.219569 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 504660us
I20260709 12:19:28.192394 31423 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.223862 31326 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:28.209046 31377 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:28.154987 31439 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20260709 12:19:28.216753 31381 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.282939 31475 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.158113 31394 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:28.222054 31437 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:28.223600 31363 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:28.278388 31428 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.013s	user 0.012s	sys 0.002s
I20260709 12:19:28.208496 31359 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.217792 31379 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.284224 31427 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:28.320189 31485 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:28.320698 31418 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.890s	user 0.011s	sys 0.019s
I20260709 12:19:28.309619 31476 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:28.309504 31405 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.389326 31326 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:28.456596 31464 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:28.457973 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2144) took 1404 ms. Trace:
I20260709 12:19:28.459120 31359 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:28.459607 31171 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.321218 31447 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11.
I20260709 12:19:28.389386 31050 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.464465 27660 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:28.466131 31253 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:28.471678 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:27.794639 31388 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.479138 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 558) took 1683 ms. Trace:
I20260709 12:19:28.467528 31363 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.398553 31377 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:28.284354 31386 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.540769 27589 rpcz_store.cc:276] 0709 12:19:26.796065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.796215 (+   150us) service_pool.cc:224] Handling call
0709 12:19:26.796599 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:27.008652 (+212053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.475282 (+1466630us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:28.479015 (+  3733us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473539840}
I20260709 12:19:28.399739 31444 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.486709 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 559) took 1689 ms. Trace:
I20260709 12:19:28.494166 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:28.475728 31483 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:28.532655 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.277877 31251 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:28.525177 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53585.9us
I20260709 12:19:28.439222 31451 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.488306 27588 rpcz_store.cc:276] 0709 12:19:27.053508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.202963 (+149455us) service_pool.cc:224] Handling call
0709 12:19:27.203226 (+   263us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:27.210663 (+  7437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:27.210692 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:27.211191 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:19:27.586351 (+375160us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:27.586390 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.586458 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:28.457763 (+871305us) raft_consensus.cc:1685] finished
0709 12:19:28.457782 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.457916 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12293,"raft.queue_time_us":214014,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":19330,"thread_start_us":380991,"threads_started":2,"wal-append.queue_time_us":1014039}
I20260709 12:19:28.449120 31064 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:28.437641 31493 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 12:19:28.487587 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 524) took 2112 ms. Trace:
W20260709 12:19:28.536074 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:28.487362 31490 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.335s	user 0.001s	sys 0.000s
I20260709 12:19:28.542016 27584 rpcz_store.cc:276] 0709 12:19:26.797156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.797313 (+   157us) service_pool.cc:224] Handling call
0709 12:19:26.836012 (+ 38699us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:27.327654 (+491642us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:27.327704 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:27.328331 (+   627us) log.cc:844] Serialized 26 byte log entry
0709 12:19:27.425091 (+ 96760us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:27.425165 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.463265 (+1038100us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b540011c954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:28.478457 (+ 15192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:28.478530 (+    73us) raft_consensus.cc:1685] finished
0709 12:19:28.478567 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.486621 (+  8054us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33113,"raft.queue_time_us":201232,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":28670,"spinlock_wait_cycles":683517696,"thread_start_us":243758,"threads_started":2,"wal-append.queue_time_us":322837}
I20260709 12:19:28.400501 31423 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.549467 27733 rpcz_store.cc:276] 0709 12:19:26.369256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.369420 (+   164us) service_pool.cc:224] Handling call
0709 12:19:26.369683 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:26.466963 (+ 97280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:26.466999 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:26.467046 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:26.467080 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.472439 (+2005359us) spinlock_profiling.cc:243] Waited 97.1 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:28.481384 (+  8945us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.481506 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223241600}
I20260709 12:19:28.449625 31381 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.463915 31428 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.488021 31439 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.543270 31430 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:28.543859 31050 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.578694 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:28.580314 27733 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:28.580996 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1817) took 1035 ms. Trace:
I20260709 12:19:28.581393 27733 rpcz_store.cc:276] 0709 12:19:27.545817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.570268 (+1024451us) service_pool.cc:224] Handling call
0709 12:19:28.580912 (+ 10644us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.481241 31378 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.591594 27663 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12.
I20260709 12:19:28.591663 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:28.494714 31405 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.579450 31475 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.489148 31485 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:28.600850 27737 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.864s	user 0.009s	sys 0.000s
I20260709 12:19:28.607554 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:28.608253 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1809) took 1185 ms. Trace:
I20260709 12:19:28.608645 27737 rpcz_store.cc:276] 0709 12:19:27.422570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.443862 (+ 21292us) service_pool.cc:224] Handling call
0709 12:19:28.608175 (+1164313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.528693 31171 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.610036 27733 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:28.525535 31464 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.551501 31444 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.571521 31447 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:19:28.633312 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:28.553658 31064 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.637498 27664 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:28.639225 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:28.639531 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:28.639906 27664 raft_consensus.cc:2368] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:28.640748 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:28.642514 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:28.643347 27664 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:28.631625 31494 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.421s	user 0.001s	sys 0.000s
W20260709 12:19:28.653090 31498 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:19:28.587359 31050 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.658890 31378 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.648975 31253 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:28.723812 31253 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.645820 31503 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:28.659814 31499 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
W20260709 12:19:28.663851 31495 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260709 12:19:28.669159 31405 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.677862 31427 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.760433 27659 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:27.138610 28336 rpcz_store.cc:276] 0709 12:19:25.924904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.925154 (+   250us) service_pool.cc:224] Handling call
0709 12:19:25.925541 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:25.925844 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:19:26.847865 (+922021us) log.cc:844] Serialized 14 byte log entry
0709 12:19:27.059300 (+211435us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:27.059379 (+    79us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:27.059822 (+   443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.059879 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:27.059987 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2970,"prepare.queue_time_us":644123,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":61509,"raft.queue_time_us":79273,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":8446,"thread_start_us":1091552,"threads_started":2}
I20260709 12:19:28.684429 31428 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.686174 31426 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:28.692840 31464 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.583601 31439 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.605078 31441 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.695837 27657 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:28.700624 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:28.708179 27664 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 12:19:28.717238 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.741289 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 17) took 2715 ms. Trace:
I20260709 12:19:28.648758 31471 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.645140 31149 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:28.589627 31394 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:28.769805 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.782756 27736 rpcz_store.cc:276] 0709 12:19:26.026317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.026499 (+   182us) service_pool.cc:224] Handling call
0709 12:19:28.740988 (+2714489us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:28.741164 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3267463040}
I20260709 12:19:28.742090 31482 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:28.805009 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 220061us
W20260709 12:19:28.799939 31494 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:19:28.843340 31504 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.844120 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:28.897636 27736 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:28.832847 31405 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.818369 31451 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.818579 31471 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.917925 31471 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:28.926005 31416 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 2.955s	user 0.023s	sys 0.025s
I20260709 12:19:28.927488 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2130) took 3270 ms. Trace:
I20260709 12:19:28.927855 27587 rpcz_store.cc:276] 0709 12:19:25.656550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.656693 (+   143us) service_pool.cc:224] Handling call
0709 12:19:25.664591 (+  7898us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:25.671654 (+  7063us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:25.671684 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:25.672226 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:19:25.867512 (+195286us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:26.544463 (+676951us) log.cc:844] Serialized 14 byte log entry
0709 12:19:26.544714 (+   251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:26.544812 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:28.925287 (+2380475us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:28.927111 (+  1824us) raft_consensus.cc:1685] finished
0709 12:19:28.927224 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.927394 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6651,"prepare.queue_time_us":261604,"prepare.run_cpu_time_us":1714,"prepare.run_wall_time_us":30428,"raft.queue_time_us":88352,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":100534,"spinlock_wait_cycles":138580736,"thread_start_us":781986,"threads_started":2,"wal-append.queue_time_us":263838}
W20260709 12:19:28.929085 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1806) took 1507 ms (client timeout 1850 ms). Trace:
W20260709 12:19:28.929611 27735 rpcz_store.cc:269] 0709 12:19:27.421585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.423137 (+  1552us) service_pool.cc:224] Handling call
0709 12:19:28.928874 (+1505737us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:28.929011 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28529408}
I20260709 12:19:28.935307 27735 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:19:28.936529 31386 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:28.943977 31381 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.819509 31149 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.764962 31503 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.849663 27586 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:28.851563 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1801) took 2964 ms (client timeout 1939 ms). Trace:
I20260709 12:19:28.862212 31444 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.862392 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:28.864887 31369 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:28.869421 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 586) took 1827 ms. Trace:
I20260709 12:19:28.871049 27731 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:28.871562 31475 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:28.959499 27729 rpcz_store.cc:269] 0709 12:19:25.886923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.888270 (+  1347us) service_pool.cc:224] Handling call
0709 12:19:28.851262 (+2962992us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:28.851425 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3598418688}
I20260709 12:19:28.872308 31050 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.873716 31171 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.798487 31441 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.913836 31504 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.915889 31405 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:28.987565 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 527) took 2101 ms. Trace:
I20260709 12:19:28.916858 31451 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:28.953595 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 528) took 1542 ms (client timeout 1633 ms). Trace:
I20260709 12:19:28.959834 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1813) took 1498 ms. Trace:
I20260709 12:19:28.959928 28335 rpcz_store.cc:276] 0709 12:19:27.041584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.042927 (+  1343us) service_pool.cc:224] Handling call
0709 12:19:27.059692 (+ 16765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:27.329749 (+270057us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:19:27.367880 (+ 38131us) log.cc:844] Serialized 14 byte log entry
0709 12:19:27.552268 (+184388us) log.cc:844] Serialized 14 byte log entry
0709 12:19:27.552439 (+   171us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:27.552495 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:27.552567 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.869202 (+1316635us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:28.869245 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.869363 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":67304,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":14792,"raft.queue_time_us":223256,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"spinlock_wait_cycles":620559616,"thread_start_us":183741,"threads_started":1,"wal-append.queue_time_us":712446}
I20260709 12:19:28.799808 31426 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:28.990595 27658 rpcz_store.cc:269] 0709 12:19:27.410920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.411121 (+   201us) service_pool.cc:224] Handling call
0709 12:19:28.953423 (+1542302us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:28.953531 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4428160}
I20260709 12:19:28.992038 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:28.962810 27735 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:19:28.947196 31149 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:28.967706 31437 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.000s
I20260709 12:19:28.971657 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:28.972551 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.972625 31364 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.949287 31503 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.004180 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:28.972726 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:28.973443 31439 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.947985 31379 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.968225 31485 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:28.974088 31475 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:28.949927 31458 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:28.950788 31412 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:28.978116 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:28.979384 27729 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:19:28.988118 27655 rpcz_store.cc:276] 0709 12:19:26.874709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.874817 (+   108us) service_pool.cc:224] Handling call
0709 12:19:26.875512 (+   695us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:19:26.893312 (+ 17800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:26.893343 (+    31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:19:26.952655 (+ 59312us) log.cc:844] Serialized 78 byte log entry
0709 12:19:27.130494 (+177839us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:27.130569 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.130755 (+   186us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:28.923617 (+1792862us) spinlock_profiling.cc:243] Waited 6.86 ms on lock 0x7b24002ac098. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:19:28.961789 (+ 38172us) raft_consensus.cc:1685] finished
0709 12:19:28.961809 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.976161 (+ 14352us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9647,"raft.queue_time_us":120812,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":30137,"spinlock_wait_cycles":15767936,"thread_start_us":124930,"threads_started":1,"wal-append.queue_time_us":474791}
W20260709 12:19:28.967787 31442 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.660s	user 0.006s	sys 0.003s
I20260709 12:19:28.990688 27731 rpcz_store.cc:276] 0709 12:19:27.461671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.870477 (+1408806us) service_pool.cc:224] Handling call
0709 12:19:28.870822 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.959561 (+ 88739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.959777 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.000996 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
W20260709 12:19:29.002537 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.986904 31440 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:29.016209 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 532) took 1600 ms. Trace:
I20260709 12:19:29.018818 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:29.017017 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.016957 27735 raft_consensus.cc:606] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:29.021445 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.021342 27735 raft_consensus.cc:614] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:19:29.021961 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.005542 31437 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:29.026266 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:29.026834 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.028357 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.029116 27655 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:28.992571 31050 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:19:29.035174 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.019728 27656 rpcz_store.cc:276] 0709 12:19:27.416040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.434032 (+ 17992us) service_pool.cc:224] Handling call
0709 12:19:27.434409 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:27.434825 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:27.435173 (+   348us) log.cc:844] Serialized 14 byte log entry
0709 12:19:27.661875 (+226702us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:27.662514 (+   639us) log.cc:844] Serialized 26 byte log entry
0709 12:19:27.662702 (+   188us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:27.662752 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.662884 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:29.015998 (+1353114us) raft_consensus.cc:1685] finished
0709 12:19:29.016016 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.016150 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40394,"prepare.queue_time_us":7736,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":1597,"raft.queue_time_us":105794,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":37216,"thread_start_us":225942,"threads_started":1,"wal-append.queue_time_us":557089}
I20260709 12:19:29.040622 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:28.998574 31426 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.043026 27583 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:19:29.042538 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.011188 31471 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:29.046545 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.046864 27655 raft_consensus.cc:2368] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:29.048465 27581 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:29.048632 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.051771 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1814) took 1561 ms. Trace:
I20260709 12:19:29.052150 27736 rpcz_store.cc:276] 0709 12:19:27.490130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.898372 (+1408242us) service_pool.cc:224] Handling call
0709 12:19:28.915275 (+ 16903us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.954984 (+ 39709us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.955030 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:28.976672 (+ 21642us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.017535 (+ 40863us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:29.017574 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.017636 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:29.051583 (+ 33947us) raft_consensus.cc:1685] finished
0709 12:19:29.051601 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.051718 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":52847,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":3557,"thread_start_us":8298,"threads_started":1,"wal-append.queue_time_us":52563}
I20260709 12:19:29.035737 31440 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.055009 31426 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.055380 31440 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.056419 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.056921 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.057536 27589 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:29.057948 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.058470 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.059031 27589 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:29.057538 31427 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.044970 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:19:29.057018 27587 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:29.034829 31458 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:29.008365 31364 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.049265 27583 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:29.070837 27656 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:19:29.071434 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.073370 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.077075 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.043519 31064 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.079452 31451 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.077656 27583 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:29.090330 27587 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:29.092095 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.096194 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.100903 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.101956 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.103638 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.104277 27657 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:29.105131 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.105640 27657 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:29.101907 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.107163 27655 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:29.108279 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.109472 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:29.110450 31414 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 2.746s	user 0.011s	sys 0.033s
I20260709 12:19:29.110960 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:29.112319 31421 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 3.088s	user 0.028s	sys 0.060s
I20260709 12:19:29.018386 31442 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.154738 31442 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.074826 31458 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:29.156025 31454 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:29.156595 31454 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.158648 31454 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:29.078145 31377 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.000s
I20260709 12:19:29.159456 31064 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.159708 31377 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.159952 31064 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.161825 31377 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.161891 31064 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.162472 31064 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:29.163771 31364 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.165372 31475 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:29.165649 31505 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.350s	user 0.000s	sys 0.002s
I20260709 12:19:29.169144 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.169904 31284 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:29.170305 31284 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:29.170753 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:29.171136 31284 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.172173 31284 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:29.172457 31423 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.173749 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.174510 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.174850 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:29.175873 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1759) took 1496 ms (client timeout 1850 ms). Trace:
W20260709 12:19:29.176285 27660 rpcz_store.cc:269] 0709 12:19:27.682906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.683232 (+   326us) service_pool.cc:224] Handling call
0709 12:19:29.175706 (+1492474us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b540022b354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.175819 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1725674368}
I20260709 12:19:29.176786 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.177346 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.177956 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.178335 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.179111 27660 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:19:29.179888 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.180828 27661 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:29.181586 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.182094 27661 raft_consensus.cc:2376] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11. Re-sending same reply.
I20260709 12:19:29.182339 31251 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.182897 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.115948 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.184901 27658 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:29.186177 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.186844 27658 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:29.124066 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2144) took 2676 ms (client timeout 2349 ms). Trace:
W20260709 12:19:29.187539 27585 rpcz_store.cc:269] 0709 12:19:26.447440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.450550 (+  3110us) service_pool.cc:224] Handling call
0709 12:19:29.123847 (+2673297us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b540011bf54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.123981 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3885429120}
W20260709 12:19:29.187877 31462 consensus_queue.cc:1175] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 10 status { last_received { term: 9 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:19:29.175239 28329 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:29.156086 31458 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:29.190059 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.191183 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:29.127873 31509 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:19:29.128551 31253 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:29.109385 31486 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.996s	user 0.001s	sys 0.012s
I20260709 12:19:29.195293 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.135912 31405 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.136977 31441 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.195758 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.139773 31171 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.196329 27580 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:29.196403 31171 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.140655 31437 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.199719 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.199604 31437 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.141368 31427 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:29.142513 31432 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.216970 27729 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:29.218003 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.218688 27729 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 6: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:29.219666 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:29.145375 31388 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:29.148288 27586 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:29.217444 27487 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:29.222604 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:29.077788 31477 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.795s	user 0.009s	sys 0.000s
I20260709 12:19:29.217089 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:29.224313 31477 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.225442 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:29.149885 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 527) took 1035 ms. Trace:
I20260709 12:19:29.226663 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.226653 27733 rpcz_store.cc:276] 0709 12:19:28.114774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.584719 (+469945us) service_pool.cc:224] Handling call
0709 12:19:29.149725 (+565006us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.149829 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40541440}
I20260709 12:19:29.227566 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.227931 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.228194 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:29.226193 27487 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 5, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 5 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:29.151293 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.230113 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
W20260709 12:19:29.127002 31476 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.630s	user 0.006s	sys 0.000s
W20260709 12:19:29.179935 27663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 533) took 1763 ms (client timeout 1944 ms). Trace:
I20260709 12:19:29.245164 27487 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:19:29.246399 27663 rpcz_store.cc:269] 0709 12:19:27.416346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.416957 (+   611us) service_pool.cc:224] Handling call
0709 12:19:29.179767 (+1762810us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.179885 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1099972864}
I20260709 12:19:29.247591 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.248884 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.247170 27487 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 10, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 10 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:29.249571 27663 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:29.250341 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:29.250622 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 221) took 26997 ms (client timeout 15000 ms). Trace:
I20260709 12:19:29.193218 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1758) took 1514 ms. Trace:
I20260709 12:19:29.194152 31462 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:29.251456 27662 rpcz_store.cc:276] 0709 12:19:27.678900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.687746 (+  8846us) service_pool.cc:224] Handling call
0709 12:19:27.711143 (+ 23397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.308787 (+597644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.193004 (+884217us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b5400210554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:29.193164 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1030148992}
I20260709 12:19:29.252579 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.253623 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.254601 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.132565 31504 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:29.203938 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.184898 27589 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:29.204632 27656 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
W20260709 12:19:29.123247 31326 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.597s	user 0.009s	sys 0.000s
I20260709 12:19:29.205351 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.268218 27735 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:29.206933 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.209438 27701 leader_election.cc:400] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:29.270084 27657 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:29.211413 27655 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:29.203294 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:29.201851 31437 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.220309 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.159125 31454 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.199210 31171 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.163830 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:29.231210 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.236392 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.237789 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.244341 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.244832 31477 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.245939 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.256021 27729 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:19:29.250648 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 5, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 5 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:29.255278 27662 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:29.162931 31064 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.259724 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.199652 31343 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 12:19:29.216506 31427 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.222097 31456 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.137s
I20260709 12:19:29.470106 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.284480 27586 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:19:29.627851 27586 pending_rounds.cc:85] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 9
I20260709 12:19:29.628180 27586 pending_rounds.cc:107] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:19:29.316089 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.628727 27586 raft_consensus.cc:2889] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:29.347829 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.290391 27658 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:29.630944 27658 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:19:29.626293 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 416932us
I20260709 12:19:29.570845 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.632309 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:29.438408 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:29.626739 27660 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:29.633446 28336 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:19:29.426146 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.633579 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:29.424057 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:29.634382 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 431126us
I20260709 12:19:29.634883 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:29.346969 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.636315 27729 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 5
I20260709 12:19:29.636499 27581 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:29.636757 27729 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:19:29.318524 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.637274 27729 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:29.637907 27582 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:19:29.407011 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.635982 27738 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:29.281328 27657 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:29.635651 27659 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:29.660463 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.661091 27659 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:29.562341 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:29.387156 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.663573 27589 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:29.654086 27656 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:29.664731 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2131) took 1021 ms. Trace:
I20260709 12:19:29.665045 27656 rpcz_store.cc:276] 0709 12:19:28.643661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.288037 (+644376us) service_pool.cc:224] Handling call
0709 12:19:29.627176 (+339139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.664496 (+ 37320us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.664669 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.660585 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2119) took 1020 ms. Trace:
W20260709 12:19:29.667614 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.315384 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:29.666574 27659 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:29.667023 27657 rpcz_store.cc:276] 0709 12:19:28.640242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.280553 (+640311us) service_pool.cc:224] Handling call
0709 12:19:29.281019 (+   466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.660289 (+379270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.660524 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.639536 28334 raft_consensus.cc:2410] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:19:29.638724 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.689940 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2117) took 1050 ms. Trace:
I20260709 12:19:29.690472 27655 rpcz_store.cc:276] 0709 12:19:28.639777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.110744 (+470967us) service_pool.cc:224] Handling call
0709 12:19:29.689876 (+579132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.691809 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.692070 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.692427 27663 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:29.693473 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 567) took 2642 ms. Trace:
I20260709 12:19:29.699707 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.700029 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.226207 31489 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:29.703869 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:29.704566 27658 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:29.693851 27586 rpcz_store.cc:276] 0709 12:19:27.050902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.052195 (+  1293us) service_pool.cc:224] Handling call
0709 12:19:27.052486 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.849212 (+1796726us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540041a8a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c606f 00007f0e9a33ef60 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62
0709 12:19:29.147958 (+298746us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b58008c0fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:19:29.693235 (+545277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.693421 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":849577088}
I20260709 12:19:29.707502 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.708148 27586 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:29.708810 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.711930 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.712551 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:29.713532 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.714103 27587 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 5.
I20260709 12:19:29.719102 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.721339 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.722272 27586 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:19:29.723834 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.724193 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.724889 27662 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:29.452656 31430 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.860s	user 0.011s	sys 0.000s
W20260709 12:19:29.726414 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91120.7us
I20260709 12:19:29.739362 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:29.739825 27661 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:19:29.742249 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:29.743212 27583 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:29.287164 31429 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.732381 27582 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:29.473582 31464 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.745754 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.272469 31388 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.724462 27738 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
W20260709 12:19:29.734138 31284 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.005s
I20260709 12:19:29.733474 27487 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 7, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 7 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:29.735042 27656 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:29.314405 31476 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:29.748019 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.611987 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.716768 31489 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:29.761679 27735 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:29.717512 31369 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:29.436787 31445 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.313s
I20260709 12:19:29.749411 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.690074 31504 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:29.651012 31377 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:29.746366 27582 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:29.751137 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.697086 31397 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.229661 31241 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.494s
I20260709 12:19:29.698978 31267 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.757619 27589 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:19:29.759203 31251 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.795137 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.748342 27587 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:19:29.759675 27583 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:19:29.759706 27585 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.008s	sys 0.015s
W20260709 12:19:29.798466 27585 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.008s	sys 0.016s
I20260709 12:19:29.748829 27729 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:19:29.705281 31110 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 12:19:29.744254 31430 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:29.287019 31326 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.805730 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.806656 31326 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:29.760977 31394 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.003s	sys 0.020s
W20260709 12:19:29.807824 31394 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.004s	sys 0.021s
I20260709 12:19:29.785156 27487 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:29.811661 28330 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:29.812119 27487 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 8, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:29.813494 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 611) took 1676 ms. Trace:
I20260709 12:19:29.813894 28330 rpcz_store.cc:276] 0709 12:19:28.136904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.206902 (+ 69998us) service_pool.cc:224] Handling call
0709 12:19:29.813270 (+1606368us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005a2654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.813409 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2918498048}
I20260709 12:19:29.815929 31241 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:29.816476 31241 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:29.817613 31241 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.706151 31454 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.786836 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.758450 27588 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:29.770779 31016 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.022s	sys 0.014s
I20260709 12:19:29.762719 31511 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:29.762945 31171 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:29.780179 31232 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.008s	sys 0.017s
I20260709 12:19:29.804215 28337 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:29.804378 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.768586 31489 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:29.800808 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.772001 31284 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:29.759522 31388 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.852569 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.855527 27729 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:29.793751 31397 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.861975 31397 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:29.826606 27587 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:29.835346 31326 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:19:29.826386 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:29.814399 27487 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 4, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:29.774629 31423 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:19:29.764066 31464 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:29.250988 27482 rpcz_store.cc:269] 0709 12:19:02.252901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.253243 (+   342us) service_pool.cc:224] Handling call
0709 12:19:29.250416 (+26997173us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:22.072896 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:22.073660 (+   764us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:22.073713 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:19:22.073740 (+    27us) tablet.cc:662] Decoding operations
0709 12:19:22.316114 (+242374us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:22.316210 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:22.322403 (+  6193us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:19:22.344324 (+ 21921us) tablet.cc:701] Row locks acquired
0709 12:19:22.344349 (+    25us) write_op.cc:260] PREPARE: finished
0709 12:19:22.344522 (+   173us) write_op.cc:270] Start()
0709 12:19:22.344657 (+   135us) write_op.cc:276] Timestamp: P: 1783599562344465 usec, L: 0
0709 12:19:22.344680 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:19:22.345428 (+   748us) log.cc:844] Serialized 4806 byte log entry
0709 12:19:22.778727 (+433299us) op_driver.cc:464] REPLICATION: finished
0709 12:19:23.108810 (+330083us) write_op.cc:301] APPLY: starting
0709 12:19:23.127737 (+ 18927us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:23.180620 (+ 52883us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:23.213599 (+ 32979us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:23.215580 (+  1981us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:23.216075 (+   495us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0709 12:19:23.216158 (+    83us) write_op.cc:312] APPLY: finished
0709 12:19:23.216812 (+   654us) log.cc:844] Serialized 89 byte log entry
0709 12:19:23.291343 (+ 74531us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:23.371677 (+ 80334us) write_op.cc:454] Released schema lock
0709 12:19:23.382162 (+ 10485us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:23.467414 (+ 85252us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:23.467445 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1046259,"reported_tablets":49,"spinlock_wait_cycles":15744,"child_traces":[["op",{"apply.queue_time_us":285252,"apply.run_cpu_time_us":6265,"apply.run_wall_time_us":392122,"num_ops":12,"prepare.queue_time_us":315592,"prepare.run_cpu_time_us":46709,"prepare.run_wall_time_us":325209,"raft.queue_time_us":367839,"raft.run_cpu_time_us":7699,"raft.run_wall_time_us":188980,"replication_time_us":433763,"thread_start_us":502298,"threads_started":4,"wal-append.queue_time_us":61367,"wal-append.run_cpu_time_us":31039,"wal-append.run_wall_time_us":1956064}]]}
I20260709 12:19:29.848419 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:29.844213 31232 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.009s	sys 0.017s
I20260709 12:19:29.820631 31267 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:29.862308 31462 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.611s	user 0.010s	sys 0.000s
I20260709 12:19:29.836392 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033381
I20260709 12:19:29.862882 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.866451 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.840164 31511 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:29.867786 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2129) took 1224 ms. Trace:
W20260709 12:19:29.868702 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1802) took 3981 ms (client timeout 2242 ms). Trace:
I20260709 12:19:29.869422 31427 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:29.870424 31064 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.820034 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:29.819276 31241 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:29.753556 31429 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:29.890630 27732 rpcz_store.cc:269] 0709 12:19:25.887234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.889197 (+  1963us) service_pool.cc:224] Handling call
0709 12:19:29.868553 (+3979356us) spinlock_profiling.cc:243] Waited 3.42 s on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.868641 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7876818688}
I20260709 12:19:29.913192 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.914384 27732 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 12:19:29.823248 31016 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.022s	sys 0.015s
I20260709 12:19:29.833083 31454 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:29.916610 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.917582 27734 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:29.889981 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:29.890465 27661 rpcz_store.cc:276] 0709 12:19:28.642982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.206732 (+563750us) service_pool.cc:224] Handling call
0709 12:19:29.867603 (+660871us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.867732 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1079681280}
I20260709 12:19:29.921025 27583 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:29.922024 27586 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:19:29.900101 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 525) took 1785 ms (client timeout 1633 ms). Trace:
I20260709 12:19:29.879257 28336 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:29.904663 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:29.925423 27589 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:29.915736 27729 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:29.901834 27588 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:29.927214 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:29.846066 31284 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:29.881187 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:29.855171 31388 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:29.872859 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:29.923830 27737 rpcz_store.cc:269] 0709 12:19:28.114234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.632949 (+518715us) service_pool.cc:224] Handling call
0709 12:19:29.899878 (+1266929us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b540029fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.900005 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1218151808}
I20260709 12:19:29.937355 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:29.940975 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:29.943444 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:29.951875 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:29.956007 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 83243.7us
I20260709 12:19:29.912061 31462 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:29.821463 31251 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.961378 27487 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:29.962692 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.037s	sys 0.037s
W20260709 12:19:29.964856 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.037s	sys 0.038s
I20260709 12:19:29.839603 31171 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:29.975414 27487 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 7, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:29.973773 27589 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:29.940398 31284 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:29.952883 31441 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:29.982801 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:29.990370 27735 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:29.889458 31430 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 12:19:29.986253 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2121) took 1345 ms. Trace:
I20260709 12:19:29.959029 31432 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:29.991593 27664 rpcz_store.cc:276] 0709 12:19:28.640696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.101738 (+461042us) service_pool.cc:224] Handling call
0709 12:19:29.985996 (+884258us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:29.986159 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1620734848}
I20260709 12:19:29.993983 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.999145 27734 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:30.004865 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:29.998167 27487 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 8, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 8 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:30.005752 27734 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:29.954996 31394 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:19:29.952271 31427 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:19:29.892582 31511 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:29.992429 27738 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:30.068279 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.069247 27734 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:30.074551 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 439685us
I20260709 12:19:30.075721 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.077335 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:30.078492 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 516164us
I20260709 12:19:30.005611 31441 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:30.010365 31432 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:30.105111 27487 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 9, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 9 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:30.079521 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:29.996243 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 726766us
I20260709 12:19:30.085342 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:30.112072 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.112687 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:30.111920 27487 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 9, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:30.113552 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.117666 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.120183 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:30.134716 27487 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 6, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 6 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:30.104262 31462 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:30.114353 31388 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:30.115784 31432 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:30.152359 27487 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:30.144021 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:30.161600 27581 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.525s	user 0.011s	sys 0.002s
I20260709 12:19:30.124969 31205 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:30.170641 27581 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:30.171942 27735 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:30.150265 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.169584 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:30.175472 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:30.145756 31483 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:30.158313 31397 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:30.158246 31462 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:30.178642 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:30.141397 31524 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:19:30.189010 31523 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:30.187357 31377 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:19:30.189296 31326 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:30.190389 31397 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:30.209785 31064 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:30.190706 27664 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:30.177354 31205 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:30.216506 31504 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:30.228899 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.221553 31462 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:30.263772 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:30.254900 27738 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
W20260709 12:19:30.257706 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.003s	sys 0.029s
I20260709 12:19:30.167373 31432 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:30.220420 31405 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:30.220661 31483 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:30.265319 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.004s	sys 0.029s
I20260709 12:19:30.260977 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 535) took 1056 ms. Trace:
I20260709 12:19:30.271806 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2156) took 1076 ms. Trace:
W20260709 12:19:30.268402 27730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.921s	user 0.018s	sys 0.018s
I20260709 12:19:30.223908 31523 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:30.273309 27730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.018s	sys 0.018s
I20260709 12:19:30.272305 27585 rpcz_store.cc:276] 0709 12:19:29.203845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.204021 (+   176us) service_pool.cc:224] Handling call
0709 12:19:29.204274 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.204442 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:29.798988 (+594546us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.271305 (+472317us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.271395 (+    90us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:30.271455 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.271542 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.271735 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":803109,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":1040074,"threads_started":2}
W20260709 12:19:30.274048 27733 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.014s	sys 0.002s
I20260709 12:19:30.270113 27731 rpcz_store.cc:276] 0709 12:19:29.204732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.204852 (+   120us) service_pool.cc:224] Handling call
0709 12:19:29.221796 (+ 16944us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.256354 (+ 34558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.256383 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.274431 (+ 18048us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.823861 (+549430us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:30.254134 (+430273us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.260528 (+  6394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.260635 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.260699 (+    64us) raft_consensus.cc:1685] finished
0709 12:19:30.260740 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.260914 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5152,"raft.queue_time_us":45891,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":20428,"thread_start_us":799543,"threads_started":2,"wal-append.queue_time_us":719440}
W20260709 12:19:30.274751 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.002s	sys 0.015s
I20260709 12:19:30.256417 31397 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:30.276252 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.002s	sys 0.015s
W20260709 12:19:30.276479 27733 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.014s	sys 0.003s
I20260709 12:19:30.255493 31064 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:30.250645 31326 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:30.294885 27735 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:30.353845 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 600935us
W20260709 12:19:30.346127 27660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.019s	sys 0.035s
W20260709 12:19:30.330256 27553 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.011s	sys 0.006s
W20260709 12:19:30.330536 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 373081us
I20260709 12:19:30.259593 31504 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:30.356071 27553 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.011s	sys 0.008s
I20260709 12:19:30.230701 31441 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.358937 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:30.363336 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 419997us
I20260709 12:19:30.364351 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:30.266801 31405 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:30.364707 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:30.379777 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.395714 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:30.355831 27660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.020s	sys 0.035s
I20260709 12:19:30.358018 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2016) took 1111 ms. Trace:
I20260709 12:19:30.267405 31464 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:30.257588 31205 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:30.360966 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 714836us
W20260709 12:19:30.323299 31522 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:19:30.355677 31326 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:30.357636 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.343040 31527 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:30.281924 31523 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:30.468374 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110798us
I20260709 12:19:30.470206 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.399837 27733 rpcz_store.cc:276] 0709 12:19:29.246687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.247429 (+   742us) service_pool.cc:224] Handling call
0709 12:19:29.247896 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.267180 (+ 19284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.267210 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.267654 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.652491 (+384837us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:30.071491 (+419000us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:19:30.207128 (+135637us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b24002c5b38. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:19:30.353924 (+146796us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.357594 (+  3670us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.357700 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.357761 (+    61us) raft_consensus.cc:1685] finished
0709 12:19:30.357782 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.357946 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18806,"raft.queue_time_us":24205,"raft.run_cpu_time_us":13053,"raft.run_wall_time_us":837015,"spinlock_wait_cycles":39368320,"thread_start_us":1004690,"threads_started":2,"wal-append.queue_time_us":559209}
I20260709 12:19:30.355186 27735 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:30.408928 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.471714 27664 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:30.412719 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.472931 27661 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:30.473742 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.476289 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.476668 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.479183 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.479617 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.479923 27735 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:30.476496 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.480960 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.482771 27735 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:19:30.484006 27735 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:30.480993 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.368515 31171 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:30.458007 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93719.1us
I20260709 12:19:30.362833 31504 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:30.461464 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.464627 31464 raft_consensus.cc:2764] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:19:30.516013 27661 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:30.460582 31386 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:30.470913 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.487643 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:30.522208 27659 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:30.490586 27664 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:30.522640 27661 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:30.286123 31251 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:30.442205 27552 connection.cc:256] RPC call timeout handler was delayed by -2.7638676189999387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
I20260709 12:19:30.454938 31405 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:30.363133 31454 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
W20260709 12:19:30.474957 31458 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.285s	user 0.013s	sys 0.000s
W20260709 12:19:30.474135 31521 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:19:30.503783 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.524026 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:30.528012 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:30.507793 31515 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.365s	user 0.001s	sys 0.000s
W20260709 12:19:30.548604 31489 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.681s	user 0.013s	sys 0.000s
I20260709 12:19:30.527833 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:30.549065 31489 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.551740 28334 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:30.508606 31171 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:30.548049 27663 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 12 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 12:19:30.534461 31430 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.005s	sys 0.018s
W20260709 12:19:30.564885 27552 connection.cc:256] RPC call timeout handler was delayed by -2.4911676189999525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2242s for a response
I20260709 12:19:30.513252 31523 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:30.569531 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 427933us
I20260709 12:19:30.374395 31483 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.544273 31251 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:30.559114 31394 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:30.593786 31523 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:30.576995 31430 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.005s	sys 0.018s
W20260709 12:19:30.631644 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.007s	sys 0.030s
W20260709 12:19:30.632366 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.007s	sys 0.030s
W20260709 12:19:30.639593 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 530051us
I20260709 12:19:30.640447 27735 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:19:30.641144 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.642683 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:30.656248 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:30.658203 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:30.555884 31458 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.665879 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:30.588122 31251 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:30.687573 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 419828us
I20260709 12:19:30.690701 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:30.638441 31290 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.008s	sys 0.033s
I20260709 12:19:30.696269 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1816) took 3164 ms. Trace:
I20260709 12:19:30.696842 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:30.700207 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44065.8us
I20260709 12:19:30.701241 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.702679 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.556258 31064 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:30.637359 31489 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:30.664291 31527 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:30.698685 27735 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:30.681332 31388 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:30.696753 27730 rpcz_store.cc:276] 0709 12:19:27.531520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.933567 (+1402047us) service_pool.cc:224] Handling call
0709 12:19:28.934007 (+   440us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:28.978768 (+ 44761us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.978817 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:28.991883 (+ 13066us) log.cc:844] Serialized 52 byte log entry
0709 12:19:29.263841 (+271958us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:30.274999 (+1011158us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.695828 (+420829us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.695956 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.696015 (+    59us) raft_consensus.cc:1685] finished
0709 12:19:30.696032 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.696201 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37537,"prepare.queue_time_us":1260416,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2344,"raft.queue_time_us":120081,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":17712,"thread_start_us":1608107,"threads_started":3,"wal-append.queue_time_us":846451}
W20260709 12:19:30.714205 31290 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.008s	sys 0.033s
I20260709 12:19:30.693840 31251 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:30.665866 31363 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
W20260709 12:19:30.753149 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50567.1us
I20260709 12:19:30.665221 31326 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.680619 31110 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:19:30.724030 31369 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:30.754078 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 280342us
I20260709 12:19:30.778578 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:30.785482 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:30.787827 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.005s	sys 0.009s
W20260709 12:19:30.788664 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.006s	sys 0.009s
I20260709 12:19:30.790304 31369 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:30.791236 31369 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:30.792346 31369 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:30.793315 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:30.799266 31386 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:30.800420 31386 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:30.735590 31440 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:30.710122 31454 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:19:30.740294 31388 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:30.904276 31388 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:30.906847 31432 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.625s	user 0.006s	sys 0.002s
I20260709 12:19:30.739979 31377 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:30.831367 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 582) took 1007 ms. Trace:
I20260709 12:19:30.834632 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.908367 31532 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.909076 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.908942 31532 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:30.910260 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:30.907286 31432 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.826318 31427 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:19:30.876647 31397 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.001s
I20260709 12:19:30.768006 31064 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:30.911801 31427 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:30.938589 31427 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:30.913281 31397 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:30.939833 31427 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:30.941444 31397 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:30.943109 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2022) took 1384 ms. Trace:
I20260709 12:19:30.943527 27729 rpcz_store.cc:276] 0709 12:19:29.558427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.219611 (+661184us) service_pool.cc:224] Handling call
0709 12:19:30.219957 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:30.220324 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.220351 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:30.220830 (+   479us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.665650 (+444820us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:30.694630 (+ 28980us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.695711 (+  1081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.695784 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.942891 (+247107us) raft_consensus.cc:1685] finished
0709 12:19:30.942911 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.943042 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4851,"prepare.queue_time_us":140124,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":1127,"raft.run_cpu_time_us":497,"raft.run_wall_time_us":36308,"thread_start_us":397199,"threads_started":1,"wal-append.queue_time_us":681707}
I20260709 12:19:30.945691 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.851449 31440 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:30.870772 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84837.8us
I20260709 12:19:30.871066 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 536) took 1775 ms. Trace:
I20260709 12:19:30.884974 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.756022 31472 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.106s
I20260709 12:19:30.902751 31267 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:30.910778 27735 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:30.835748 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51734.9us
I20260709 12:19:30.908007 27586 rpcz_store.cc:276] 0709 12:19:29.824128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.957278 (+133150us) service_pool.cc:224] Handling call
0709 12:19:29.976047 (+ 18769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.976314 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:30.361970 (+385656us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.830988 (+469018us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.831046 (+    58us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:30.831092 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.831155 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.831286 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55375,"prepare.queue_time_us":385461,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"thread_start_us":777422,"threads_started":2}
I20260709 12:19:30.973233 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.974637 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:30.945894 27733 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:30.946434 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2032) took 1214 ms. Trace:
I20260709 12:19:30.975984 27737 rpcz_store.cc:276] 0709 12:19:29.731916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.232709 (+500793us) service_pool.cc:224] Handling call
0709 12:19:30.233143 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:30.253743 (+ 20600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.253771 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:30.257377 (+  3606us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.741542 (+484165us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:30.768876 (+ 27334us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.769508 (+   632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.769698 (+   190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.946233 (+176535us) raft_consensus.cc:1685] finished
0709 12:19:30.946252 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.946380 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69341,"prepare.queue_time_us":121304,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":49641,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":20469,"spinlock_wait_cycles":70912,"thread_start_us":402700,"threads_started":1,"wal-append.queue_time_us":659768}
I20260709 12:19:30.948671 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:30.887807 31437 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:30.947705 31241 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.024s	user 0.010s	sys 0.003s
I20260709 12:19:30.989570 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.990242 27737 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:30.991760 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.993474 27737 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:30.995399 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.814217 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.980377 28334 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:30.951074 27660 rpcz_store.cc:276] 0709 12:19:29.095552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.634684 (+539132us) service_pool.cc:224] Handling call
0709 12:19:29.634936 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.635129 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:19:30.403008 (+767879us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.870668 (+467660us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.870723 (+    55us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:30.870770 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.870839 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.870990 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106772,"prepare.queue_time_us":566720,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":131083,"thread_start_us":1127987,"threads_started":2,"wal-append.queue_time_us":540481}
I20260709 12:19:30.978359 27729 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:30.982448 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:30.996763 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 182937us
I20260709 12:19:30.983162 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:30.997702 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.998919 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:30.911288 31532 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.004279 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:31.004789 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:31.024427 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.025058 27586 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:31.021773 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.985486 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:31.010591 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 278710us
W20260709 12:19:30.981218 31462 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.699s	user 0.011s	sys 0.000s
I20260709 12:19:31.017954 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:31.036898 27589 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:30.907528 31253 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.096s
I20260709 12:19:31.019207 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:31.042306 27735 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [7.5-10.7]
I20260709 12:19:31.020001 27737 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 12:19:31.055160 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:31.020509 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:31.005210 31377 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:31.029498 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:30.995844 31397 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:31.033773 31545 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:31.053710 31429 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.050985 27729 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:31.069713 27586 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:19:31.072499 31565 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:19:31.072297 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:31.072825 31445 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:19:31.073997 31564 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:19:31.159338 27587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.000s	sys 0.009s
W20260709 12:19:31.080674 31405 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.533s	user 0.005s	sys 0.005s
I20260709 12:19:31.016597 31171 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:19:31.083257 27737 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [5.5-9.6]
W20260709 12:19:31.085148 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 99171.8us
W20260709 12:19:31.162207 27587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.000s	sys 0.009s
I20260709 12:19:31.162808 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.163070 27737 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [11.9-12.10]
I20260709 12:19:31.163851 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.040129 31417 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:31.130611 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:31.135581 27730 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:31.026494 31542 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.096s
I20260709 12:19:31.048071 31462 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.113937 31369 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:31.061154 31458 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.145983 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 581) took 1323 ms. Trace:
W20260709 12:19:31.142812 31549 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.373s	user 0.001s	sys 0.000s
I20260709 12:19:31.099068 31559 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:31.079905 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 570) took 1335 ms. Trace:
I20260709 12:19:31.186404 27583 rpcz_store.cc:276] 0709 12:19:29.822163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.953624 (+131461us) service_pool.cc:224] Handling call
0709 12:19:29.954003 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.954268 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:30.369327 (+415059us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.145519 (+776192us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:31.145592 (+    73us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:31.145654 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.145754 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.145912 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":136781,"prepare.queue_time_us":354410,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":1350,"raft.queue_time_us":234212,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":6923,"thread_start_us":1063999,"threads_started":3,"wal-append.queue_time_us":542277}
I20260709 12:19:31.098505 31377 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:31.100050 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2047) took 1188 ms. Trace:
W20260709 12:19:31.181699 27734 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.009s	sys 0.005s
W20260709 12:19:31.198625 27734 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.010s	sys 0.005s
I20260709 12:19:31.146242 31476 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:31.184939 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.007s	sys 0.007s
W20260709 12:19:31.148195 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 483328us
W20260709 12:19:31.203492 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.008s	sys 0.007s
I20260709 12:19:31.137204 31397 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:31.206288 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2020) took 1756 ms. Trace:
I20260709 12:19:31.190157 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.207794 27586 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:31.143397 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:31.191844 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 161128us
I20260709 12:19:31.163030 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 572) took 1386 ms. Trace:
I20260709 12:19:31.191110 27588 rpcz_store.cc:276] 0709 12:19:29.744964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.823087 (+ 78123us) service_pool.cc:224] Handling call
0709 12:19:29.832934 (+  9847us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:29.879769 (+ 46835us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:30.002254 (+122485us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.799613 (+797359us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:30.800305 (+   692us) log.cc:844] Serialized 52 byte log entry
0709 12:19:30.800457 (+   152us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:30.800492 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.800569 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.079700 (+279131us) raft_consensus.cc:1685] finished
0709 12:19:31.079718 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.079849 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":319614,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":31368,"thread_start_us":665851,"threads_started":2,"wal-append.queue_time_us":1008645}
I20260709 12:19:31.060452 31556 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.133s
W20260709 12:19:31.199525 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 173122us
W20260709 12:19:31.165449 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:31.178464 31504 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.658s	user 0.011s	sys 0.000s
W20260709 12:19:31.201545 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59587.3us
I20260709 12:19:31.198173 27729 rpcz_store.cc:276] 0709 12:19:29.911662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.029972 (+1118310us) service_pool.cc:224] Handling call
0709 12:19:31.050607 (+ 20635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.099720 (+ 49113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.099982 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:31.208734 27660 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:31.209084 27587 rpcz_store.cc:276] 0709 12:19:29.776694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.881559 (+104865us) service_pool.cc:224] Handling call
0709 12:19:29.881930 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.882164 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:19:30.080851 (+198687us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.162621 (+1081770us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:31.162702 (+    81us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:31.162761 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.162835 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.162965 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18347,"prepare.queue_time_us":375749,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":2740,"raft.queue_time_us":419180,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":128,"thread_start_us":1261240,"threads_started":3,"wal-append.queue_time_us":642923}
I20260709 12:19:31.135409 31464 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:19:31.216317 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.189666 31394 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.181624 31417 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:31.217108 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:31.199765 31538 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.447s	user 0.001s	sys 0.001s
I20260709 12:19:31.185658 31542 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:31.060976 31440 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.227599 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.202632 31458 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:31.150830 31588 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 12:19:31.177450 31405 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.239032 31483 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:31.238899 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.201224 31432 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.219165 31559 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:19:31.240142 31572 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:19:31.232235 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.259020 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:31.222165 31504 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:31.323542 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5977372550000837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:19:31.206301 31397 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.239915 31417 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.228974 31394 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.266505 31559 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.318540 27730 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:31.228889 31568 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:31.324132 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5950372550000793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260709 12:19:31.327940 27738 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:31.336256 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 534) took 2131 ms. Trace:
I20260709 12:19:31.233860 31593 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:19:31.341535 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 102746us
W20260709 12:19:31.342339 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4852372549999018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
W20260709 12:19:31.349735 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2953372550000495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
I20260709 12:19:31.346565 27736 rpcz_store.cc:276] 0709 12:19:29.204357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.204467 (+   110us) service_pool.cc:224] Handling call
0709 12:19:29.229068 (+ 24601us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.271944 (+ 42876us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.271989 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.272490 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.864419 (+591929us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:19:30.272696 (+408277us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:30.272762 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.272862 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.319466 (+1046604us) raft_consensus.cc:1685] finished
0709 12:19:31.319486 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.336163 (+ 16677us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27440,"raft.queue_time_us":27595,"raft.run_cpu_time_us":11795,"raft.run_wall_time_us":597990,"spinlock_wait_cycles":94781312,"thread_start_us":849325,"threads_started":1,"wal-append.queue_time_us":1271188}
W20260709 12:19:31.358562 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 126400us
I20260709 12:19:31.359738 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:31.364815 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105906us
I20260709 12:19:31.365823 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:31.367302 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:31.327023 31171 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.326962 31601 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:31.339526 31397 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:31.252194 31253 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.101s
W20260709 12:19:31.379805 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 142460us
I20260709 12:19:31.370998 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.341327 31456 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:31.341770 31559 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:31.381155 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.354439 31429 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:31.342916 31241 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:31.382197 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:31.341924 31585 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:31.386761 27660 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:31.380851 31593 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:31.374931 31573 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:19:31.365621 31417 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:31.398237 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:31.553225 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 171116us
I20260709 12:19:31.373296 31427 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:31.386011 31551 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260709 12:19:31.381026 31558 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:31.378779 31464 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:31.375393 31432 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:31.389495 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:31.561122 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 171703us
I20260709 12:19:31.487634 31504 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:31.386747 31394 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.395450 31369 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:19:31.553887 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 155758us
I20260709 12:19:31.568274 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.554250 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.388196 31594 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:31.395452 31511 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:19:31.571671 31511 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:31.572129 31511 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:31.572345 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2173) took 2068 ms. Trace:
I20260709 12:19:31.572729 27580 rpcz_store.cc:276] 0709 12:19:29.503617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.741427 (+237810us) service_pool.cc:224] Handling call
0709 12:19:29.741774 (+   347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:29.763549 (+ 21775us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:19:30.255809 (+492260us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.006743 (+750934us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:31.059458 (+ 52715us) log.cc:844] Serialized 52 byte log entry
0709 12:19:31.059656 (+   198us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:31.059705 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.059773 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.130688 (+ 70915us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:19:31.572037 (+441349us) raft_consensus.cc:1685] finished
0709 12:19:31.572143 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.572289 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":199181,"prepare.queue_time_us":391963,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":184248,"raft.run_cpu_time_us":1257,"raft.run_wall_time_us":202558,"spinlock_wait_cycles":15505664,"thread_start_us":1049721,"threads_started":3,"wal-append.queue_time_us":1064663}
W20260709 12:19:31.461975 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 392230us
I20260709 12:19:31.575573 31580 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:19:31.575809 31578 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:19:31.399881 31429 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.487493 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.563606 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.381089 31171 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.400306 31587 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:31.464682 31601 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:31.569736 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:31.577517 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:31.382536 31591 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.183s
I20260709 12:19:31.548898 31542 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:31.588135 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.622394 27580 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:31.559288 31458 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:31.565989 31595 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:19:31.559130 31253 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:31.568969 31570 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:31.584810 31441 raft_consensus.cc:2764] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:19:31.547511 31241 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.564680 31462 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:31.627247 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57549.8us
W20260709 12:19:31.606346 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:31.628183 31430 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:31.628759 31253 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.625766 31557 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 12:19:31.652810 31557 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:31.653177 31557 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:31.654407 31388 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.750s	user 0.005s	sys 0.004s
I20260709 12:19:31.655066 31253 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:31.655874 31541 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260709 12:19:31.635802 31284 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.630s	user 0.014s	sys 0.001s
I20260709 12:19:31.629503 31587 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.632251 31558 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:31.632757 31543 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.062s	user 0.002s	sys 0.000s
I20260709 12:19:31.364573 31568 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:31.640856 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77335.6us
I20260709 12:19:31.624596 31542 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:31.635154 31251 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.872s	user 0.003s	sys 0.004s
I20260709 12:19:31.654913 31388 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.650378 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.401811 31456 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.697083 31456 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.654314 31430 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:31.698616 31241 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.635047 31171 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:31.638321 31205 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.151s	user 0.010s	sys 0.000s
W20260709 12:19:31.666247 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:31.700050 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50275.6us
I20260709 12:19:31.671797 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.700286 31205 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.701370 27737 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:31.701910 31241 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:31.702224 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:31.702425 31241 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:31.702555 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:31.591858 31429 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:31.703171 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1828) took 1190 ms. Trace:
I20260709 12:19:31.703737 31606 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:31.704200 31606 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:31.704608 31606 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:31.704798 31604 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:19:31.704981 31441 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:31.705881 31441 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:31.706331 31441 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:31.587082 31571 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:19:31.707737 31571 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:31.708056 31571 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:31.708892 31571 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.710242 31571 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:31.603219 31504 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:19:31.642125 31570 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:31.650334 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74920us
I20260709 12:19:31.649290 31599 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:31.728009 31610 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:19:31.728082 31599 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:31.728691 31599 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:31.558854 31585 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.732708 31568 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.693948 31251 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.698288 31430 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:31.631019 31590 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:19:31.701300 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.706956 31326 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.626781 31458 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.703732 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2043) took 1856 ms. Trace:
I20260709 12:19:31.703565 27737 rpcz_store.cc:276] 0709 12:19:30.512300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.164028 (+651728us) service_pool.cc:224] Handling call
0709 12:19:31.164284 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.701805 (+537521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.702959 (+  1154us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:31.703117 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1234704128}
I20260709 12:19:31.718492 31570 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:31.676307 31284 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.720814 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.720010 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.737708 27731 rpcz_store.cc:276] 0709 12:19:29.847055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.276232 (+429177us) service_pool.cc:224] Handling call
0709 12:19:30.340293 (+ 64061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.702957 (+1362664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.703518 (+   561us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002e90d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:31.703670 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3132419328}
W20260709 12:19:31.723546 31603 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.165s	user 0.004s	sys 0.000s
I20260709 12:19:31.724037 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:31.724399 31545 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.000s
W20260709 12:19:31.655273 31064 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.705s	user 0.009s	sys 0.000s
I20260709 12:19:31.730201 27733 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:31.715752 31405 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:31.736512 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.718852 31394 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:19:31.738422 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.740514 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.722638 31587 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.741356 27586 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:31.721310 31432 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:31.712847 31253 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.747618 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1821) took 1342 ms. Trace:
I20260709 12:19:31.737275 31596 raft_consensus.cc:2764] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:31.755875 27733 rpcz_store.cc:276] 0709 12:19:30.405665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.074530 (+668865us) service_pool.cc:224] Handling call
0709 12:19:31.729760 (+655230us) spinlock_profiling.cc:243] Waited 7.37 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:31.729964 (+   204us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:31.747207 (+ 17243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.747566 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16948992}
I20260709 12:19:31.720404 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.752516 27737 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:31.753397 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.754042 27729 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:31.688736 31558 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.758983 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.738963 31458 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:31.757467 27731 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:31.757580 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.744765 31568 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:31.707198 31441 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.746443 31171 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:31.746980 31545 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:31.764653 27586 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:31.716450 31241 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.753261 31064 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:31.766970 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2025) took 2065 ms. Trace:
I20260709 12:19:31.817591 27732 rpcz_store.cc:276] 0709 12:19:29.701442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.224692 (+523250us) service_pool.cc:224] Handling call
0709 12:19:30.225172 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:30.253684 (+ 28512us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.253715 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:30.259209 (+  5494us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.740455 (+481246us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b1800146380. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:31.008263 (+267808us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:31.008301 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.008363 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.763824 (+755461us) raft_consensus.cc:1685] finished
0709 12:19:31.763841 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.766908 (+  3067us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52855,"raft.queue_time_us":34684,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":19392,"spinlock_wait_cycles":7913728,"thread_start_us":267586,"threads_started":1,"wal-append.queue_time_us":1081948}
I20260709 12:19:31.768399 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.767697 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:31.819314 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50953.5us
I20260709 12:19:31.820077 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:31.748039 31464 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 12:19:31.771966 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.733443 31593 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.775442 31456 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:31.769976 31504 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:31.767076 31618 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:31.776479 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2179) took 1111 ms. Trace:
I20260709 12:19:31.776769 31364 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:31.777156 31171 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.780153 27731 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:31.781517 31559 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.783744 31253 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.785455 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:31.755730 31405 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:31.789047 27660 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:19:31.790596 31437 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:31.792260 31602 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:19:31.792771 31284 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.795226 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:31.206815 27734 rpcz_store.cc:276] 0709 12:19:29.450003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.142333 (+692330us) service_pool.cc:224] Handling call
0709 12:19:30.165591 (+ 23258us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:30.165982 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.166010 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:30.178178 (+ 12168us) log.cc:844] Serialized 52 byte log entry
0709 12:19:30.533509 (+355331us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:31.199042 (+665533us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.205651 (+  6609us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.205874 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.205981 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.206039 (+    58us) raft_consensus.cc:1685] finished
0709 12:19:31.206058 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.206219 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2277,"prepare.queue_time_us":210939,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":2342,"raft.queue_time_us":2960,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":89,"thread_start_us":973291,"threads_started":2,"wal-append.queue_time_us":751991}
W20260709 12:19:31.843930 31603 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:19:31.839970 31364 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:31.796314 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:31.797405 31585 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.755716 31612 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:31.800931 31606 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.763453 31568 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.807130 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:31.817518 31429 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.811295 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:31.823807 27737 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
W20260709 12:19:31.877039 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65845.9us
I20260709 12:19:31.877504 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.826807 27655 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:31.877754 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:31.826714 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:31.833973 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62125.1us
W20260709 12:19:31.834702 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67075.5us
I20260709 12:19:31.835659 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:31.864960 31405 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.832281 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:31.886091 27733 raft_consensus.cc:2376] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10. Re-sending same reply.
W20260709 12:19:31.886343 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59760us
I20260709 12:19:31.887187 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:31.887878 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:31.888012 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.888806 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.890084 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.841058 31580 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:31.891690 31580 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.892131 31580 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:31.892170 31171 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.891845 31364 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.893057 31364 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:31.893205 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.893827 27734 raft_consensus.cc:2376] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10. Re-sending same reply.
I20260709 12:19:31.894630 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.894860 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:31.861526 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:31.775130 31511 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.895710 27731 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:31.896448 31559 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:31.832677 31596 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:31.897259 31432 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:31.897468 31596 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:31.897765 31432 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.898062 31596 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:31.831893 31456 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:31.832337 31426 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:31.899627 31456 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:31.897275 31511 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:31.836949 31064 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:31.770560 31575 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:19:31.816658 27658 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:31.900759 31205 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:31.862468 31528 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:31.745222 31542 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.901479 31528 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:31.901124 31064 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.902064 31542 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:31.903160 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1819) took 1503 ms. Trace:
I20260709 12:19:31.878198 27737 raft_consensus.cc:2376] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11. Re-sending same reply.
I20260709 12:19:31.886123 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.904572 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:31.905344 27737 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:31.880695 27732 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:31.825752 27582 rpcz_store.cc:276] 0709 12:19:30.664678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.664947 (+   269us) service_pool.cc:224] Handling call
0709 12:19:30.665294 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.013727 (+348433us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:31.013756 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:31.013796 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:31.013825 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.776273 (+762448us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:31.776313 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.776426 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25875,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":8508,"spinlock_wait_cycles":800567552}
I20260709 12:19:31.905530 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.882107 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.906399 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:31.902833 27733 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:31.906711 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.907508 27731 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:31.909587 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:31.898167 31432 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:31.859231 31437 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:31.903535 27735 rpcz_store.cc:276] 0709 12:19:30.400129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.043312 (+643183us) service_pool.cc:224] Handling call
0709 12:19:31.045795 (+  2483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:31.104432 (+ 58637us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.104461 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:31.104876 (+   415us) log.cc:844] Serialized 26 byte log entry
0709 12:19:31.322623 (+217747us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:31.322983 (+   360us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.371890 (+ 48907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.684851 (+312961us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:31.726809 (+ 41958us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b18002fc680. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:31.779175 (+ 52366us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b18002fc680. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:31.779864 (+   689us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:31.779904 (+    40us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.902969 (+123065us) raft_consensus.cc:1685] finished
0709 12:19:31.902990 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.903104 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46117,"prepare.queue_time_us":268,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":154689,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":13071,"spinlock_wait_cycles":156952832,"thread_start_us":144434,"threads_started":1,"wal-append.queue_time_us":236321}
I20260709 12:19:31.915452 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:31.908524 27734 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:31.882664 27580 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:31.877748 31568 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:31.912673 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.902383 31528 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.917681 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:31.972589 31611 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260709 12:19:31.959865 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2042) took 2128 ms. Trace:
I20260709 12:19:31.932190 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:31.885038 31524 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:31.973555 27738 rpcz_store.cc:276] 0709 12:19:29.831596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.265327 (+433731us) service_pool.cc:224] Handling call
0709 12:19:30.265700 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.346239 (+1080539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.959642 (+613403us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:31.959806 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2442271488}
I20260709 12:19:31.954314 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
W20260709 12:19:31.960482 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53833.2us
I20260709 12:19:31.975514 27733 raft_consensus.cc:606] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:31.976348 27733 raft_consensus.cc:614] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:31.918220 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:31.979930 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62311.2us
I20260709 12:19:31.983922 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.911450 31602 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:31.878006 31612 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:31.916467 31205 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:31.922149 31614 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.157s	user 0.002s	sys 0.000s
W20260709 12:19:31.920480 31617 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:19:31.971716 31571 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:31.911715 31599 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.918524 27662 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:19:31.974087 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61511.8us
I20260709 12:19:31.918833 31587 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:31.995589 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:31.920889 31464 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:31.918133 31618 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:31.995867 27701 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:31.998565 27655 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:31.911530 31377 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 12:19:32.002928 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:31.973917 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
W20260709 12:19:31.991757 31577 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:19:31.912313 31405 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:31.921468 31558 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:31.980532 31016 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:32.006086 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88108.7us
I20260709 12:19:31.980530 31284 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:31.902822 31594 consensus_queue.cc:1175] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 6 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:19:31.916294 31476 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:31.989885 31575 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:19:31.917164 31559 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:32.028821 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:32.069852 31616 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:19:32.009244 27657 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:32.118140 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.052553 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.125021 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.024905 27656 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:31.998911 31576 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:32.101397 31426 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:32.141891 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.009778 31602 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:32.025806 31397 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.630s	user 0.006s	sys 0.005s
I20260709 12:19:32.127102 31613 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:19:32.132195 31599 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:31.998020 31524 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:32.133664 31571 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:31.998849 31585 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.136s
I20260709 12:19:32.114972 31587 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:32.140177 27655 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
W20260709 12:19:32.073669 31621 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:19:32.142390 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:32.125097 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129301us
I20260709 12:19:32.053248 31574 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.051023 31511 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:32.029469 31417 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.030862 31545 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:32.125428 31476 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.105878 31454 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:32.158872 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 106434us
I20260709 12:19:32.137753 31558 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:32.176674 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.268718 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.176806 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.177217 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.148700 31429 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.142004 31464 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:32.273902 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.274508 27655 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:32.149061 31426 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:32.201306 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.127439 31445 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:19:32.165833 31386 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:19:32.142859 31576 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:32.136096 31523 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:19:32.183609 27730 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:32.172791 31587 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:32.152007 31559 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.270804 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:32.165094 31568 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.166399 31397 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.236965 31618 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.287787 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.273104 31476 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:32.157631 31602 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.307096 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2049) took 1535 ms. Trace:
I20260709 12:19:32.184942 31574 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.323726 27736 rpcz_store.cc:276] 0709 12:19:30.771419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.353046 (+581627us) service_pool.cc:224] Handling call
0709 12:19:31.381534 (+ 28488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:31.393239 (+ 11705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.393268 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:31.399805 (+  6537us) log.cc:844] Serialized 26 byte log entry
0709 12:19:31.748350 (+348545us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:31.758171 (+  9821us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.758693 (+   522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.758761 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.306847 (+548086us) raft_consensus.cc:1685] finished
0709 12:19:32.306870 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.307027 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11541,"prepare.queue_time_us":11143,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":1439,"raft.queue_time_us":14167,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":64171,"spinlock_wait_cycles":4731648,"thread_start_us":128564,"threads_started":1,"wal-append.queue_time_us":279678}
I20260709 12:19:32.314307 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.281757 31545 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:32.238081 31511 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.327265 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.240744 31284 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:32.282438 31576 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:32.278033 31454 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.298895 31575 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:32.334793 31064 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:32.324877 31386 raft_consensus.cc:2764] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:32.371189 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.341842 27736 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:32.276300 31417 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.327093 27660 raft_consensus.cc:2368] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
W20260709 12:19:32.353757 31625 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:19:32.353181 31426 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.359879 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2050) took 1588 ms. Trace:
I20260709 12:19:32.361771 27655 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:32.333292 31523 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:32.374092 31574 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.377596 31437 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.332901 31612 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.378249 31456 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.377979 31596 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.291954 31524 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.411103 27661 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:32.279748 31571 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:32.356608 31624 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:19:32.390950 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:32.357494 31570 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.610s	user 0.007s	sys 0.005s
I20260709 12:19:32.419209 31437 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.419916 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.420789 27655 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:32.421640 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.422639 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.423290 27655 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:32.390359 27735 rpcz_store.cc:276] 0709 12:19:30.771703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.979051 (+1207348us) service_pool.cc:224] Handling call
0709 12:19:31.980351 (+  1300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:31.982492 (+  2141us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.982518 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:31.984358 (+  1840us) log.cc:844] Serialized 26 byte log entry
0709 12:19:32.178132 (+193774us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:32.178231 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.178399 (+   168us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.359668 (+181269us) raft_consensus.cc:1685] finished
0709 12:19:32.359687 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.359818 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3481,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":5179,"spinlock_wait_cycles":1742208,"thread_start_us":193319,"threads_started":1,"wal-append.queue_time_us":338454}
I20260709 12:19:32.368958 31585 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:32.367769 31575 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:32.381892 31458 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.614s	user 0.007s	sys 0.002s
I20260709 12:19:32.432267 31458 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:32.433111 31608 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.861s	user 0.000s	sys 0.005s
I20260709 12:19:32.434578 31458 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.435719 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.436285 27655 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:32.442986 31630 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:19:32.323848 31618 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.392534 31523 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:32.396485 31241 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:32.366708 31284 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.416960 27582 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:32.418809 31596 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:32.418946 31364 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.000s
I20260709 12:19:32.419409 31570 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.431800 31575 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:32.447749 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:32.474713 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.448364 31559 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:32.426275 31545 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.451519 31363 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:32.452365 31568 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.481974 31363 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:32.453352 31397 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:32.482940 31363 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.483721 31456 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.483954 31363 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:32.485699 31454 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.454459 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.487636 27659 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:32.395030 31426 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:32.455945 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.495294 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.496825 31570 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.404248 31064 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:19:32.458922 27737 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:32.459267 27735 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:19:32.503830 27587 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:32.504813 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.505362 27735 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:32.506114 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:32.459580 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.465917 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.467000 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.451057 31606 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.469573 27655 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
W20260709 12:19:32.472635 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80119.6us
I20260709 12:19:32.415676 31405 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:32.472226 31580 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.579s	user 0.005s	sys 0.006s
I20260709 12:19:32.470916 31511 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.523785 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:32.469823 31542 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.567s	user 0.000s	sys 0.013s
I20260709 12:19:32.458312 31417 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:32.512069 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46291.4us
I20260709 12:19:32.516562 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.501124 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:32.398279 31464 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.534899 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.542150 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.537041 31417 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:32.543195 31511 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:32.522244 31570 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:32.543221 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.524372 31454 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:19:32.548983 27738 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:19:32.525198 31594 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.552990 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:32.532395 31630 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:19:32.558120 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.558728 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:32.562076 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.565127 31618 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:32.565287 27659 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:32.569038 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.569847 27660 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:32.569900 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.523864 31638 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:32.577082 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.584741 31251 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.524547 31456 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:19:32.591768 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.589727 27660 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:19:32.547854 31570 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:32.593955 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.594986 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.553853 31417 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:32.594692 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:32.586229 31432 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.615s	user 0.009s	sys 0.002s
I20260709 12:19:32.595498 27731 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:32.526172 31580 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.595490 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.546599 31440 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 12:19:32.598625 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.607877 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.588726 31548 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.592600 31064 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:19:32.612214 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78314.4us
I20260709 12:19:32.590829 31638 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:32.611155 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.567907 31594 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.616395 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.614710 27659 raft_consensus.cc:2368] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:32.618039 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.618726 27733 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:32.615360 27660 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:19:32.620946 27660 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:19:32.621487 27738 raft_consensus.cc:2368] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:32.621538 27660 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:19:32.621977 27660 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:32.622428 27731 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:32.623716 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.624248 31284 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.614208 31548 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.625272 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:32.626863 31548 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.627739 31638 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.611866 31585 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:32.569628 31575 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.555694 31523 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.613121 31064 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:32.640710 27582 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:32.608042 31417 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.605734 31432 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.616106 31241 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:32.646461 27659 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:32.648954 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 541) took 1913 ms. Trace:
I20260709 12:19:32.651549 27730 rpcz_store.cc:276] 0709 12:19:30.735835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.344045 (+608210us) service_pool.cc:224] Handling call
0709 12:19:31.344824 (+   779us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:19:31.380289 (+ 35465us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:31.400508 (+ 20219us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.112213 (+711705us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b1800366e80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:32.162376 (+ 50163us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:19:32.183239 (+ 20863us) log.cc:844] Serialized 608 byte log entry
0709 12:19:32.183468 (+   229us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:32.320789 (+137321us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.327471 (+  6682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.327571 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.648730 (+321159us) raft_consensus.cc:1685] finished
0709 12:19:32.648753 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.648890 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37744,"prepare.queue_time_us":78526,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":3890,"raft.queue_time_us":52599,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":12739,"spinlock_wait_cycles":3073408,"thread_start_us":31791,"threads_started":1,"wal-append.queue_time_us":679618}
I20260709 12:19:32.639461 31585 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:32.656292 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 660726us
I20260709 12:19:32.668267 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.668905 27584 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:32.669104 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.670099 27586 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:19:32.671232 27588 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:19:32.676318 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.656906 31573 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:32.642746 31594 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.661703 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.681257 27658 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 11. Ops: []
I20260709 12:19:32.683827 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.685364 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.686558 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.688911 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.690032 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.690778 27580 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:32.687148 27583 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:32.695241 27587 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:32.696269 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:32.697835 27583 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:32.679195 27730 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:32.703012 27730 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:32.704241 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.705677 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:32.705852 27580 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:32.706664 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.707491 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.665256 31585 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:32.629281 31570 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.645385 31613 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:19:32.703892 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:32.692667 31618 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:32.703891 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:32.731124 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:32.731614 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.732228 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:32.715853 27730 raft_consensus.cc:606] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:32.732718 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.733181 27730 raft_consensus.cc:614] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:32.734459 27730 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:32.725217 31476 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:32.726379 31645 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:19:32.683064 31573 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:32.700242 31641 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:32.728960 27732 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:32.685432 31575 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.720821 31557 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.736640 27730 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [5.5-9.6]
W20260709 12:19:32.741315 31649 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:19:32.700788 31559 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:32.728184 31064 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.743124 31587 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.750459 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:32.760735 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.772361 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.778220 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 544) took 1041 ms. Trace:
I20260709 12:19:32.788249 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.788008 27733 rpcz_store.cc:276] 0709 12:19:31.736287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.984787 (+248500us) service_pool.cc:224] Handling call
0709 12:19:31.992090 (+  7303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:32.621204 (+629114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.777900 (+156696us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:32.778135 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1440684416}
I20260709 12:19:32.813711 27732 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:19:32.801292 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.765293 31618 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:32.774191 31064 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:32.817212 31618 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.766474 31557 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:32.764562 31527 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:32.764322 31397 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:32.786628 31593 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:32.781303 31634 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:32.822103 31476 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.824582 31599 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:32.773649 31432 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.766669 31615 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:32.764196 31284 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:32.774300 31573 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.831442 31397 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:32.782591 31558 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:32.836141 31573 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:32.781945 31587 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.833734 31634 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:32.833787 31599 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:32.773685 31559 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.769580 31576 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.837553 31558 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:32.833853 31593 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:32.841730 31634 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:32.841777 31599 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.776160 31641 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.847755 31634 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:32.850075 31576 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.850557 31585 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.847154 31586 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:32.854307 31586 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:32.855552 31586 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:32.856823 31586 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:32.846285 31593 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:32.858814 31580 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:32.854317 31641 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:32.853158 31599 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:32.856308 31628 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.499s	user 0.001s	sys 0.003s
I20260709 12:19:32.902154 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.908469 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:32.910933 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2178) took 1911 ms. Trace:
I20260709 12:19:32.919884 27581 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:32.919673 27585 rpcz_store.cc:276] 0709 12:19:30.999710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.024818 (+ 25108us) service_pool.cc:224] Handling call
0709 12:19:31.025171 (+   353us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:31.050890 (+ 25719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.050921 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:31.060580 (+  9659us) log.cc:844] Serialized 52 byte log entry
0709 12:19:32.024496 (+963916us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b1800039100. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:32.171631 (+147135us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:32.171666 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.171726 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.855793 (+684067us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:32.910267 (+ 54474us) raft_consensus.cc:1685] finished
0709 12:19:32.910352 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.910717 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48171,"raft.queue_time_us":26022,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":10334,"spinlock_wait_cycles":64600576,"thread_start_us":146950,"threads_started":1,"wal-append.queue_time_us":1227267}
W20260709 12:19:32.911839 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:32.923877 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.923969 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:32.925971 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:32.926370 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:32.924476 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.934948 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:32.936223 27586 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:32.937625 27589 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:32.938721 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.940281 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.946445 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:32.916220 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:32.949560 27584 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:32.957448 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 554) took 1171 ms. Trace:
W20260709 12:19:32.957808 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.759142 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:32.961190 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.962735 27655 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 12:19:32.976836 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 217611us
I20260709 12:19:32.979151 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:32.981771 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.962608 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.964003 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:32.965713 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:32.984637 27660 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:32.969079 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.969456 27733 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:19:32.986361 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:32.975943 27663 rpcz_store.cc:276] 0709 12:19:31.785681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.785795 (+   114us) service_pool.cc:224] Handling call
0709 12:19:31.786229 (+   434us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:31.786767 (+   538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.786807 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:31.787626 (+   819us) log.cc:844] Serialized 52 byte log entry
0709 12:19:31.988477 (+200851us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:31.998918 (+ 10441us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.999337 (+   419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.999411 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.493654 (+494243us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b1800010720. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:32.854935 (+361281us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b1800010720. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1
0709 12:19:32.860689 (+  5754us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b1800010720. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:32.956767 (+ 96078us) raft_consensus.cc:1685] finished
0709 12:19:32.956819 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.957198 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15,"prepare.queue_time_us":17204,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":2541,"raft.queue_time_us":22360,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":15856,"spinlock_wait_cycles":25912576,"thread_start_us":200446,"threads_started":1,"wal-append.queue_time_us":212234}
I20260709 12:19:32.973145 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2063) took 1078 ms. Trace:
I20260709 12:19:32.989288 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:32.993610 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:32.988286 27731 rpcz_store.cc:276] 0709 12:19:31.894183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:32.627248 (+733065us) service_pool.cc:224] Handling call
0709 12:19:32.627605 (+   357us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:32.628024 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:32.628631 (+   607us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.783214 (+154583us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:32.786434 (+  3220us) log.cc:844] Serialized 52 byte log entry
0709 12:19:32.786593 (+   159us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:32.786851 (+   258us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.787065 (+   214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.787148 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.972424 (+185276us) raft_consensus.cc:1685] finished
0709 12:19:32.972491 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.972840 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":261,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":1179,"raft.queue_time_us":2813,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":33606,"thread_start_us":135371,"threads_started":1,"wal-append.queue_time_us":169374}
I20260709 12:19:32.997088 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:32.997318 31363 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:32.997476 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:32.998993 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.018997 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:33.020891 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:33.021059 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:33.021729 27731 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:33.022327 27586 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:19:32.969022 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:32.998445 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:32.998783 31364 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:32.999636 28332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:32.837037 31397 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.001101 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.001423 31432 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:33.003028 28336 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:33.003989 31527 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:33.004148 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:32.844177 31558 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:33.005522 31205 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:32.786835 31504 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:19:32.995426 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:33.025417 27585 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.027917 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:33.029613 31585 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.030381 27587 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:19:33.030920 31574 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:33.032641 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:33.033459 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 556) took 1221 ms. Trace:
I20260709 12:19:33.035514 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:33.036108 27735 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:33.026178 31432 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:33.046640 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.051052 27736 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:33.051841 31364 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:33.082729 27662 rpcz_store.cc:276] 0709 12:19:31.812240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.812381 (+   141us) service_pool.cc:224] Handling call
0709 12:19:31.838635 (+ 26254us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:32.114176 (+275541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.032870 (+918694us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:33.033269 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183045376}
W20260709 12:19:33.053566 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84997.2us
I20260709 12:19:33.054081 31612 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:33.055716 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:32.989050 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:33.073487 28332 raft_consensus.cc:606] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:33.084177 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:33.084585 27733 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:33.098520 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:33.085973 27586 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:33.104890 28332 raft_consensus.cc:614] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:33.103746 27737 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:33.096781 31576 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:33.112119 31576 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.114354 31559 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.114727 31576 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.094843 31570 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:33.082863 31205 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:33.120743 31205 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.094363 31432 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:33.127038 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:33.130491 27733 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:33.135799 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:33.120692 27732 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:33.144135 27732 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:33.146653 31417 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.154709 27735 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:33.163981 27731 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:19:33.165347 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 176543us
I20260709 12:19:33.166419 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:33.167609 31504 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:33.168496 31504 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:33.169252 31504 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:33.170516 31504 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.190326 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:33.190212 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:33.200230 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 78173.4us
W20260709 12:19:33.206691 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.208603 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
W20260709 12:19:33.221516 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55320.3us
I20260709 12:19:33.217870 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:33.225111 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:33.224674 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.228219 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:33.232194 31638 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:33.236918 31612 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:33.237433 31612 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:33.237854 31612 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:33.246379 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:33.195962 31574 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.248394 27660 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:33.255082 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:33.257458 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 557) took 1444 ms. Trace:
I20260709 12:19:33.257874 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:33.264008 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:33.259729 27661 rpcz_store.cc:276] 0709 12:19:31.812539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.812653 (+   114us) service_pool.cc:224] Handling call
0709 12:19:31.812966 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:32.412234 (+599268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.256979 (+844745us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:33.257282 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1375123328}
I20260709 12:19:33.273171 31426 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:19:33.273295 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 163747us
W20260709 12:19:33.275068 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.279675 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.280076 27701 leader_election.cc:400] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
I20260709 12:19:33.280459 27701 leader_election.cc:403] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:33.280948 31527 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:33.281942 31363 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:33.283411 27660 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:33.284659 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:33.281576 31527 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:33.290784 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.291651 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [7.5-13.7]
I20260709 12:19:33.294900 31615 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.294994 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.295568 31615 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.296401 27586 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:19:33.297405 31559 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 8. Current term is 9.
W20260709 12:19:33.298653 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.299369 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.306005 27662 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:33.306058 27582 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:33.311566 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110156us
I20260709 12:19:33.275425 31613 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:19:33.313778 31171 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:19:33.328307 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.315130 31171 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:33.380944 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:33.415105 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.374402 31171 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:33.426743 27662 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
W20260709 12:19:33.427347 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46430.6us
I20260709 12:19:33.379236 31364 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:33.439420 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 159527us
I20260709 12:19:33.450670 27584 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:33.416085 31363 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:19:33.477447 31574 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.477775 31542 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.478242 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 537) took 2764 ms. Trace:
I20260709 12:19:33.478536 31542 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.479496 31241 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.480135 31241 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.481407 31542 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.481824 27589 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
W20260709 12:19:33.484421 31593 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.627s	user 0.008s	sys 0.012s
I20260709 12:19:33.485363 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:33.485320 31593 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.486544 31559 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:33.484342 31571 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.478749 27729 rpcz_store.cc:276] 0709 12:19:30.713771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.234427 (+520656us) service_pool.cc:224] Handling call
0709 12:19:31.339296 (+104869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.757551 (+418255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.477735 (+1720184us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:33.478043 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":953324800}
W20260709 12:19:33.491354 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43531.3us
I20260709 12:19:33.496498 27586 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
W20260709 12:19:33.497998 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.502025 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63226.3us
I20260709 12:19:33.502861 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:33.503880 31634 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.004s
I20260709 12:19:33.504186 31388 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:33.506402 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:33.504215 31571 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.491163 31364 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.507721 31527 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.508832 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:33.509094 31571 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.509428 31364 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.510195 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:33.513046 31394 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:33.515862 27580 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:33.521881 31574 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:19:33.522104 31558 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.523150 27580 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:33.523645 27734 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:19:33.524240 31570 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:33.524695 31570 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:33.524881 27580 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:33.525800 31570 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.527354 31570 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:33.527549 31661 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.294s	user 0.005s	sys 0.000s
W20260709 12:19:33.529253 31627 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.144s	user 0.002s	sys 0.000s
W20260709 12:19:33.529765 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.531466 27730 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:19:33.523703 31171 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.532840 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:33.533617 27730 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.535655 27729 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:19:33.538113 27586 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:33.541808 31171 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:33.543817 27583 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:19:33.544206 27583 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:33.544446 27583 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:19:33.544720 27583 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:33.546164 27583 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:33.538107 31523 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:33.546604 31504 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:33.547410 31574 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.547256 31504 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.547991 31574 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.548759 31253 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:33.549252 31253 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:33.550364 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:33.554625 31504 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:33.555244 31504 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:33.557057 31504 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:33.559351 31599 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.000s
I20260709 12:19:33.561712 31587 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:33.563364 31587 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:33.563797 31587 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:33.564311 31397 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.525s	user 0.014s	sys 0.002s
I20260709 12:19:33.564905 31397 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.565680 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1781) took 1569 ms. Trace:
I20260709 12:19:33.565745 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.566668 28332 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:33.489756 31615 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:33.567936 28330 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:33.568372 27581 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:19:33.568900 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:33.569326 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:33.562623 31290 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:33.570215 31290 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.571471 31613 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.571362 31290 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.571967 31613 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.572742 31290 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:33.573742 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 550) took 1834 ms. Trace:
I20260709 12:19:33.574245 31613 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.574149 27736 rpcz_store.cc:276] 0709 12:19:31.739111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:32.373101 (+633990us) service_pool.cc:224] Handling call
0709 12:19:33.050549 (+677448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:33.096059 (+ 45510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.573463 (+477404us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54002ea4d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:33.573638 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1506154368}
I20260709 12:19:33.577279 27736 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:33.577180 31634 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:19:33.577621 31472 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:33.579542 31638 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:33.580077 31655 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:33.580152 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.580683 31655 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:33.581019 31662 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:19:33.566236 27658 rpcz_store.cc:276] 0709 12:19:31.996363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.996529 (+   166us) service_pool.cc:224] Handling call
0709 12:19:31.996874 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:32.686091 (+689217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.565181 (+879090us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b5400291154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:33.565501 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1573408128}
I20260709 12:19:33.581830 31655 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.583935 31016 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:33.584410 31016 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.585632 31612 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.585549 31016 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.564234 31427 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:19:33.562950 31586 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.706s	user 0.010s	sys 0.009s
I20260709 12:19:33.567502 27737 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:33.494417 31241 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.569546 27738 raft_consensus.cc:606] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:33.614707 27585 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.532s	user 0.001s	sys 0.011s
I20260709 12:19:33.589140 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:33.591277 27733 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:19:33.591445 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:33.591620 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85354.1us
I20260709 12:19:33.572774 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:33.600066 27731 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:19:33.600960 27734 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:33.611786 28336 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:33.612030 27586 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:19:33.623472 27586 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:33.588256 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.608687 27738 raft_consensus.cc:614] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:33.550390 31253 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.567099 31397 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.595300 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:33.629861 27583 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:19:33.630590 27580 raft_consensus.cc:2410] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:19:33.621076 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:33.626864 27582 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:33.590961 28334 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:33.598829 27735 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
W20260709 12:19:33.602643 31659 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.366s	user 0.002s	sys 0.000s
I20260709 12:19:33.587600 31427 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:33.633810 31427 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.587311 31016 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:33.636788 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.604496 31666 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
I20260709 12:19:33.594774 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:33.633828 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:33.634408 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:33.637912 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48890.6us
I20260709 12:19:33.526811 31432 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.621379 31655 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:33.659063 27730 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:33.659078 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:33.605150 31558 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:33.594555 31593 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:33.667675 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164872us
I20260709 12:19:33.668825 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:33.671422 31647 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:33.674342 31326 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:33.676982 31432 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.677554 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:33.678098 31557 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:33.678571 31557 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.679726 31557 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.681663 31557 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:33.656270 31542 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:33.683944 31618 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:33.684721 31570 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:33.685253 31570 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.685694 31570 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.671757 31253 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:33.683785 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:33.688097 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:33.699745 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:19:33.702647 27587 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:19:33.703720 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 214404us
I20260709 12:19:33.705585 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:33.705888 31574 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.713208 27588 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:33.716672 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47991.5us
I20260709 12:19:33.716854 31429 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:33.717495 31429 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:33.718050 31429 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:33.719480 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 85910us
I20260709 12:19:33.720407 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:33.720610 27584 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:33.589469 31586 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260709 12:19:33.663460 31427 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:33.726258 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67357.6us
I20260709 12:19:33.715195 27588 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:19:33.730057 31574 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:33.731638 31586 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:33.732079 31586 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:33.732445 31483 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:33.733306 31586 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.734007 31483 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:33.734458 31483 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.734782 31586 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:33.735764 31483 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.737444 31483 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:33.738314 27552 leader_election.cc:400] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:33.738670 27552 leader_election.cc:403] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:19:33.739482 31417 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:33.735965 27583 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:33.666471 31504 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.743472 27580 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
W20260709 12:19:33.748201 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84878.6us
I20260709 12:19:33.748472 31574 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:33.749374 31602 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:19:33.749560 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.746556 31542 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:33.743505 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:33.751785 31464 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:33.752558 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47149.3us
I20260709 12:19:33.754375 28330 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:19:33.756424 31587 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.762255 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:33.772027 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.780925 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:33.750836 31364 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:19:33.781878 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:19:33.791520 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43533.8us
I20260709 12:19:33.741976 31559 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:19:33.805773 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:33.790678 31405 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:19:33.812168 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.789582 31612 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:33.773615 31417 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:19:33.778795 31397 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20260709 12:19:33.830770 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 111413us
I20260709 12:19:33.798038 31585 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:33.834620 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48467.7us
I20260709 12:19:33.832787 27658 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:33.815356 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:33.833917 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.816023 31405 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.785420 31464 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.819888 31599 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:33.843526 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68209.2us
I20260709 12:19:33.847877 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:33.819175 31364 consensus_queue.cc:1243] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:19:33.805001 31472 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:19:33.862216 27660 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:33.864248 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:33.870651 27658 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:33.847787 31405 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.845791 31171 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:33.817121 31504 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:19:33.844386 31602 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:33.864193 31363 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:33.860829 31205 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:33.885581 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:33.888064 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:33.890574 27663 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:19:33.866911 31527 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:33.895896 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.898936 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:33.876871 31618 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.887013 31432 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
W20260709 12:19:33.904882 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.905239 31671 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.905710 31671 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.906499 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:33.907785 31587 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.909623 31527 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.910564 31647 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:33.911394 31594 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.911576 31647 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:33.911908 31527 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:33.911965 31594 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.909693 31523 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:33.913228 31523 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.913625 31523 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.911937 31647 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.905741 31548 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 12:19:33.882910 31612 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:19:33.884019 31171 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.904737 31064 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:33.918267 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.937740 27662 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:33.970996 27662 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:33.958597 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.927798 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.947453 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.931716 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:33.976136 27589 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 7: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:33.977474 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 194066us
I20260709 12:19:33.916929 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:33.889997 31504 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:33.890388 31364 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:33.985737 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.923982 31669 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:19:33.851505 31585 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.939227 31638 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:19:33.922904 31523 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:34.025185 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.903733 31205 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:33.984680 31397 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:33.990391 27662 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:34.006295 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:34.019960 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132016us
I20260709 12:19:33.926477 31326 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:33.928205 31441 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:33.914506 31594 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:33.976426 31171 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:33.981616 31587 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:33.984882 31637 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:33.902444 31574 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.138s
I20260709 12:19:34.053927 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:33.941607 27483 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 8, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 8 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:33.923946 31570 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:33.994105 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:34.021790 31638 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.920127 31647 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.040321 31326 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:33.989094 31364 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.064807 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:34.055537 31441 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:33.985316 31064 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:33.992637 31618 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.062793 31599 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.078809 27662 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:34.074443 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:34.092650 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:34.097215 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.066569 31548 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:34.100406 31548 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.102516 31548 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:34.065773 31417 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.095674 27737 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:34.098305 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:34.104168 27729 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:19:34.071137 31671 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.119797 27662 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 12:19:34.136519 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.070744 31613 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:19:34.139272 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:34.145126 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.117350 31363 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:34.159255 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 306020us
I20260709 12:19:34.077498 31637 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.137797 31647 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:19:34.139814 31542 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:19:34.135726 31587 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:34.138181 31602 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:34.170219 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71830.8us
W20260709 12:19:34.169697 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57702us
I20260709 12:19:34.098675 31599 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.146504 31671 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.176340 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:34.180706 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45658.7us
I20260709 12:19:34.166837 31674 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.083s
I20260709 12:19:34.189898 31602 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.192183 31585 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.193001 31523 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.193070 31364 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:34.193392 31677 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:19:34.170830 31326 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:34.194983 31672 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260709 12:19:34.167182 31558 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.118764 31417 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.210703 27583 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:19:34.211092 27583 pending_rounds.cc:85] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:34.211302 27583 pending_rounds.cc:107] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:19:34.211546 27583 raft_consensus.cc:2889] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:34.145160 31429 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.195626 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2199) took 1074 ms. Trace:
I20260709 12:19:34.205205 27582 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:34.204183 27587 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:34.214319 27587 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:19:34.214779 27587 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:34.174455 31574 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:34.218683 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103944us
I20260709 12:19:34.183041 31593 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.138695 31205 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.217103 31388 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.210222 31523 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:19:34.226864 31637 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.195834 31504 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:34.228264 31405 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.217069 31364 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:19:34.212211 27581 rpcz_store.cc:276] 0709 12:19:33.121560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:33.613100 (+491540us) service_pool.cc:224] Handling call
0709 12:19:33.634010 (+ 20910us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:33.666979 (+ 32969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:33.667007 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:33.698729 (+ 31722us) log.cc:844] Serialized 26 byte log entry
0709 12:19:33.871538 (+172809us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:33.871575 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.871653 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:34.195382 (+323729us) raft_consensus.cc:1685] finished
0709 12:19:34.195408 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:34.195561 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3392,"raft.queue_time_us":27909,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":24763,"thread_start_us":172325,"threads_started":1,"wal-append.queue_time_us":364892}
I20260709 12:19:34.203722 31615 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:34.234570 31559 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.240612 31458 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:34.240509 31559 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.241873 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.246050 31559 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.248358 31405 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.221333 31587 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:34.250341 31586 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:34.249428 31615 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.197968 31671 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.233135 31527 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.213972 31558 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.264241 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.224401 31574 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.276737 27657 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:34.233915 31593 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.251407 31570 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.192247 31441 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.271550 31559 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:34.278064 31427 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.529s	user 0.012s	sys 0.000s
I20260709 12:19:34.238711 31388 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.304973 31671 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:34.306098 31678 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:19:34.306342 31388 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:34.261543 31458 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 12:19:34.288218 31585 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:34.299875 31545 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:19:34.300879 31683 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:34.301311 31685 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:19:34.302019 31690 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:19:34.304044 31593 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:34.308734 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47857.6us
I20260709 12:19:34.259485 31594 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:19:34.255916 31504 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.265578 31587 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:34.290191 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65801.6us
I20260709 12:19:34.315248 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:34.271710 31637 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
W20260709 12:19:34.328904 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87124.8us
I20260709 12:19:34.281818 31586 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:34.349906 31686 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:19:34.350644 31586 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.342542 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:34.351363 27730 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:34.356107 31673 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.305s	user 0.000s	sys 0.004s
I20260709 12:19:34.357286 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2209) took 1391 ms. Trace:
I20260709 12:19:34.357661 27586 rpcz_store.cc:276] 0709 12:19:32.965986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:33.537588 (+571602us) service_pool.cc:224] Handling call
0709 12:19:33.537933 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:33.633917 (+ 95984us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:33.633949 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:33.634401 (+   452us) log.cc:844] Serialized 26 byte log entry
0709 12:19:33.829236 (+194835us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:33.829285 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:33.829351 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:34.355355 (+526004us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:34.357000 (+  1645us) raft_consensus.cc:1685] finished
0709 12:19:34.357020 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:34.357226 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66087296,"thread_start_us":194482,"threads_started":1,"wal-append.queue_time_us":258623}
I20260709 12:19:34.361114 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.363948 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:34.365209 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:34.358852 31638 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.371800 31594 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:34.372228 31594 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:34.373111 31587 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.372258 31638 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:34.277516 31363 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.375469 27730 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
W20260709 12:19:34.380362 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.374440 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:34.327239 31558 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.307978 31559 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.374485 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:34.326536 31618 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:34.310127 31527 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.388526 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:34.392156 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:34.428122 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67126.3us
I20260709 12:19:34.326287 31655 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:19:34.381932 31253 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:34.429700 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55282.6us
W20260709 12:19:34.439157 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.430164 31016 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:34.445032 31417 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.373174 31594 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.448242 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.445087 31016 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:34.454459 31429 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:34.454327 31702 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:19:34.456183 31655 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:34.398144 31585 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:19:34.428542 31599 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:34.473773 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211612us
I20260709 12:19:34.456633 31655 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:34.479866 31574 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.480316 31655 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.481580 31064 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:34.481709 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.487864 31064 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.488431 31064 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.471724 31253 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.491788 31064 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.492677 31253 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.452661 31394 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:34.457399 31417 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:34.495806 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:34.503798 27663 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:34.452762 31548 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:19:34.507953 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.493013 27483 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 10, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 10 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:34.475265 31594 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:34.455006 31016 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:34.582831 31707 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
W20260709 12:19:34.584632 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45326.7us
I20260709 12:19:34.587085 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:34.565418 31397 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:34.594485 27656 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:34.602754 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.604112 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.604739 31706 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:34.624158 27664 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:34.605926 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.616250 27655 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:34.620656 27663 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:34.575359 31394 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:34.627744 27661 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:19:34.627697 31394 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.628754 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:34.630842 31171 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.556s	user 0.007s	sys 0.003s
I20260709 12:19:34.642722 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:34.649310 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.635202 31615 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.631253 31171 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.659184 31064 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.659389 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.664427 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:34.665223 27655 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:34.666007 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.667479 27655 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:19:34.673907 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.676844 27656 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:34.676924 31559 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.657160 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:34.643105 31417 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:34.677862 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51432.6us
I20260709 12:19:34.678639 27656 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:34.691234 27586 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:19:34.691474 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:34.692024 27584 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:34.697826 27583 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:19:34.694137 27586 pending_rounds.cc:85] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 6
I20260709 12:19:34.699277 27586 pending_rounds.cc:107] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:19:34.699890 27586 raft_consensus.cc:2889] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:34.698304 27583 pending_rounds.cc:85] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:34.700803 27583 pending_rounds.cc:107] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:19:34.700990 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.701164 27583 raft_consensus.cc:2889] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:34.702026 27584 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:34.703647 31464 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:34.704394 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.705579 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.706139 27582 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:34.708750 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.711750 27655 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:34.715022 31171 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.682586 31504 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.721721 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:34.722821 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54505.4us
W20260709 12:19:34.692655 31697 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:19:34.721146 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:34.725903 27580 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:34.682034 31704 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:34.728638 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.731916 27588 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:34.699136 31417 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:34.722433 31253 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.680154 31064 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.745113 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:34.746428 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:34.754304 27581 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:34.754719 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.770057 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:34.772081 31704 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:34.772514 31704 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:34.772855 31704 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:34.774693 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.775841 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:34.776443 31064 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:34.777159 31695 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.503s	user 0.007s	sys 0.005s
W20260709 12:19:34.783355 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:34.783968 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.785022 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.754423 31417 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:34.790804 27581 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:34.786549 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:34.792078 27586 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:19:34.795182 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:34.746172 31253 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.798231 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.794351 31417 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:34.799433 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:34.800204 31417 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.800678 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:34.801879 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:34.797590 31523 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:34.802953 31523 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:34.802991 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.808053 31523 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:34.807628 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:34.809417 27588 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:34.810665 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:34.796397 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:34.809791 27734 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:34.804221 27580 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:19:34.815827 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:34.818102 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:34.820449 27583 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 12:19:34.821038 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.842015 27730 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:19:34.824155 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:34.846273 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48178.6us
I20260709 12:19:34.821460 31441 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:34.842761 31171 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:34.886596 27552 leader_election.cc:400] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11.
I20260709 12:19:34.884618 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:34.887943 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.888875 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:34.888034 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.890214 27664 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:34.890581 27736 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:34.892366 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.893321 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.896121 27659 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:34.854496 31064 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:34.928921 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.885375 31441 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:34.932291 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:34.924350 27660 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:19:34.855993 31464 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 12:19:34.937902 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.855130 31548 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 12:19:34.943703 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.893131 31618 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:34.922935 31638 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.960911 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.961875 31638 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:34.962321 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:34.933910 31504 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:34.963269 31504 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.964623 31638 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:34.965454 31504 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:34.966068 31699 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:34.966780 31699 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:34.966889 31718 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:19:34.936964 31441 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:34.974606 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:34.975642 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:34.977500 31548 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:19:34.968135 31699 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:34.979489 31706 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:19:34.978919 31680 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.920s	user 0.005s	sys 0.000s
I20260709 12:19:34.981750 31699 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:34.986739 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 577) took 1111 ms. Trace:
I20260709 12:19:34.987285 27657 rpcz_store.cc:276] 0709 12:19:33.874962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:33.875099 (+   137us) service_pool.cc:224] Handling call
0709 12:19:33.891124 (+ 16025us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:34.281945 (+390821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:34.986403 (+704458us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:34.986658 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":784643456}
I20260709 12:19:34.992089 31655 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:35.006350 31472 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:19:35.008751 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:35.010924 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:35.011739 28329 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:35.016786 31527 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:35.029403 31405 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:35.029912 31405 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:35.030787 31405 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.032064 31405 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:35.053923 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:35.056629 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:35.066423 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:35.070617 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:35.071487 27585 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:35.071955 31326 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:35.072804 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:35.076130 31699 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:35.086901 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
W20260709 12:19:35.089053 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126815us
I20260709 12:19:35.090142 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:35.093072 27657 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:19:35.098599 27736 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:35.108708 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:35.115010 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:35.116706 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50391.4us
I20260709 12:19:35.119707 27589 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:35.058807 31290 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.121044 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:35.123592 31251 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:35.119843 31504 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:35.136348 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 204146us
I20260709 12:19:35.154562 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:35.091118 31599 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.109727 31585 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:35.173509 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:35.176726 27588 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:35.180526 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:35.181846 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45898.8us
I20260709 12:19:35.112624 31574 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:35.197619 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:35.181203 31677 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:19:35.109720 31618 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.194180 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.154454 31251 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:35.198280 28332 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:35.214421 31559 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:19:35.215935 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.225077 31427 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:35.225412 31615 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:35.226179 31615 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.227394 31585 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:35.231793 31690 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.232343 31690 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.234692 31690 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:35.236279 31698 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:35.236886 31698 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:35.237344 31698 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:35.240432 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 236) took 18903 ms (client timeout 15000 ms). Trace:
W20260709 12:19:35.240957 27487 rpcz_store.cc:269] 0709 12:19:16.367485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.367796 (+   311us) service_pool.cc:224] Handling call
0709 12:19:35.240203 (+18872407us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:31.030405 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:31.075148 (+ 44743us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:31.075221 (+    73us) write_op.cc:435] Acquired schema lock
0709 12:19:31.075234 (+    13us) tablet.cc:662] Decoding operations
0709 12:19:31.077052 (+  1818us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:31.077134 (+    82us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:31.077172 (+    38us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:19:31.078827 (+  1655us) tablet.cc:701] Row locks acquired
0709 12:19:31.078890 (+    63us) write_op.cc:260] PREPARE: finished
0709 12:19:31.079068 (+   178us) write_op.cc:270] Start()
0709 12:19:31.079216 (+   148us) write_op.cc:276] Timestamp: P: 1783599571079008 usec, L: 0
0709 12:19:31.079245 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 12:19:31.230530 (+151285us) log.cc:844] Serialized 7878 byte log entry
0709 12:19:32.096348 (+865818us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a343e28 00007f0e9dd7e581 00007f0e9dd7dd58 00007f0e9dd829f1 00007f0e9dd829a9 00007f0e9dd82939 00007f0e9dd82901 00007f0e9dd81b2d 0000000000388634 0000000000388479
0709 12:19:32.375958 (+279610us) op_driver.cc:464] REPLICATION: finished
0709 12:19:32.528354 (+152396us) write_op.cc:301] APPLY: starting
0709 12:19:32.532660 (+  4306us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:32.642689 (+110029us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:32.642712 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:32.823854 (+181142us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:32.824150 (+   296us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=40
0709 12:19:32.824202 (+    52us) write_op.cc:312] APPLY: finished
0709 12:19:32.824594 (+   392us) log.cc:844] Serialized 139 byte log entry
0709 12:19:32.824774 (+   180us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:32.824954 (+   180us) write_op.cc:454] Released schema lock
0709 12:19:32.825019 (+    65us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:32.825390 (+   371us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:32.825422 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6618418,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":99463,"apply.run_cpu_time_us":10844,"apply.run_wall_time_us":309868,"num_ops":20,"prepare.queue_time_us":492812,"prepare.run_cpu_time_us":25563,"prepare.run_wall_time_us":1079794,"raft.queue_time_us":220955,"raft.run_cpu_time_us":3821,"raft.run_wall_time_us":60821,"replication_time_us":1296536,"spinlock_wait_cycles":106233216,"thread_start_us":765615,"threads_started":4,"wal-append.queue_time_us":511570,"wal-append.run_cpu_time_us":43590,"wal-append.run_wall_time_us":2140018}]]}
I20260709 12:19:35.246420 27483 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 9 to 10, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 10 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:35.247897 31427 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:35.207208 27735 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:35.204807 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:35.257539 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 249831us
W20260709 12:19:35.202659 31724 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:19:35.193147 31253 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:35.260553 27735 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:35.156339 31290 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.180423 31599 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.212070 31251 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:35.182300 31064 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:35.211886 31504 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:35.279568 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:35.285106 27585 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:19:35.285477 27585 pending_rounds.cc:85] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:35.285779 27585 pending_rounds.cc:107] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:19:35.285883 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.286015 27585 raft_consensus.cc:2889] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:35.286718 27735 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:35.282747 27656 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:35.292539 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:35.294153 27657 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:35.291769 27483 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:35.275655 31613 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.326463 27483 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:35.321973 31427 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.345355 31427 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:35.347930 27483 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 2 to 5, term changed from 5 to 6, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to <none>, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:35.349597 27483 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:35.353041 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.283423 31064 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.360889 31599 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:35.361996 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:35.362378 31064 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:35.362634 28332 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:35.363982 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:35.363688 31504 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.365551 31064 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:35.366006 31064 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.366420 31064 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:35.367930 31618 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:35.278218 31690 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:19:35.372057 31690 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:35.372416 31690 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:35.373353 31690 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.374575 31690 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:35.376648 27483 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:19:35.343843 31523 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.000s
I20260709 12:19:35.379050 27483 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:35.379964 31523 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.380545 31290 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:35.381224 27483 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 7 to 9, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:35.382161 31615 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:35.383126 28334 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260709 12:19:35.384457 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.411706 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:35.412823 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:35.416119 27582 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:19:35.396919 27483 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 7, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 7 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:35.382635 31523 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:35.455762 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 301333us
I20260709 12:19:35.457329 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:35.458950 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:35.459726 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 275934us
I20260709 12:19:35.466691 27582 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:19:35.465684 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.352453 31613 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.456795 31729 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.483228 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:35.493244 27582 pending_rounds.cc:85] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:35.502041 27582 pending_rounds.cc:107] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:19:35.502774 27582 raft_consensus.cc:2889] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:35.504048 27585 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20260709 12:19:35.483259 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 393200us
I20260709 12:19:35.506052 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:35.541251 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:35.542778 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:35.507087 31599 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:35.549559 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 43630.1us
I20260709 12:19:35.550916 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:35.558167 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:35.562630 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:35.563712 27736 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:35.564811 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.546290 31734 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:19:35.565420 27735 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:35.547930 31326 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:35.547585 31016 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:35.507750 31729 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.581904 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:35.563297 31585 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:35.558212 31599 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.608012 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.608968 27735 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:35.582422 31405 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:35.563083 31574 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:19:35.616107 27731 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:35.621476 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.617910 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:35.590137 31733 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:35.625236 27735 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:35.624814 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165960us
I20260709 12:19:35.630934 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:35.624890 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 132044us
I20260709 12:19:35.582166 31527 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
W20260709 12:19:35.652905 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 241307us
I20260709 12:19:35.652423 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:35.659307 27732 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
W20260709 12:19:35.621014 31441 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.000s
I20260709 12:19:35.661417 31417 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:35.656256 31730 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
W20260709 12:19:35.669008 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 188113us
I20260709 12:19:35.621677 31615 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:35.652884 31739 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.674269 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:35.679740 31364 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:35.684298 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:35.621888 31585 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:19:35.619199 31735 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:19:35.658985 31363 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:35.700114 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:35.659562 31504 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:35.708940 31429 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:35.724478 31744 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:19:35.724377 31429 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.724838 31599 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:35.725572 31740 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:35.726076 31740 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.726444 31740 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:35.726764 31429 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:35.728406 31613 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:35.657953 31698 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.692211 31733 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.698558 31363 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.706413 31699 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.738814 31685 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:35.732257 31754 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:35.733482 31364 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:35.708690 31542 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.720778 31594 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.709129 31253 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.668251 31441 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.706735 31427 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.671866 31417 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.778893 31363 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:35.793792 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 212826us
W20260709 12:19:35.802754 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 256311us
I20260709 12:19:35.872121 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:35.784338 31704 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.011s	user 0.011s	sys 0.002s
I20260709 12:19:35.734870 31735 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:35.759506 31736 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:35.708212 31251 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.692090 31739 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.929837 27658 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:35.779136 31618 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:35.815315 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 141089us
I20260709 12:19:35.934538 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:35.819645 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 236046us
I20260709 12:19:35.788754 31574 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:35.793563 31429 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:19:35.802054 31742 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.110s	user 0.004s	sys 0.008s
I20260709 12:19:35.785774 31504 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.913417 27737 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:35.852583 31692 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.916440 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:35.908896 31726 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.618s	user 0.012s	sys 0.006s
I20260709 12:19:35.792896 31733 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.817507 31699 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.864780 31594 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.924207 31016 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:19:35.908058 31734 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:35.908677 31602 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:19:35.917220 31735 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:35.948594 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 255831us
I20260709 12:19:35.934887 31729 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:35.869462 31253 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.919534 31613 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:35.911072 31427 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:35.957713 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:35.939874 31429 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.864938 31542 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:35.923286 31704 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.023602 28334 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:19:36.025214 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:36.025583 27655 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:19:36.025871 27657 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:35.979841 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45381.2us
I20260709 12:19:35.989722 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:35.989037 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:35.939637 31574 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:36.031489 31747 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:19:36.031692 31542 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:36.032675 31574 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.032802 31542 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:19:36.033478 31741 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 12:19:36.034410 31542 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:35.953353 31692 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:36.035704 31750 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
I20260709 12:19:36.037207 31692 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.037321 31733 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:36.038623 31739 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:36.039167 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:36.039856 27729 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:36.041222 31698 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:36.002825 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:36.002842 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 318603us
I20260709 12:19:36.043223 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.043875 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:36.044719 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:36.045712 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.046257 31759 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:36.046737 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.047222 27660 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:36.003645 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:35.987146 31647 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:36.017711 28330 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:19:36.007274 31613 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:36.063383 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:36.030496 31749 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:19:35.958875 31364 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.048256 31704 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.048542 31594 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:36.049280 31752 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:19:36.050303 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.023018 31758 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:36.046814 31759 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.053471 31615 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:36.054783 31504 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:35.955334 31745 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:19:36.066344 31064 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.699s	user 0.005s	sys 0.009s
I20260709 12:19:36.008901 31429 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:36.064494 31647 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:36.060209 31733 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.060540 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:36.062427 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.070423 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.062741 31768 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:36.063509 31613 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:36.064210 27580 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:36.030591 31699 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.066418 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:36.067612 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.051867 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.068500 27655 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:36.073345 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:36.043893 31441 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.076480 27581 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:19:36.148572 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2206) took 1284 ms. Trace:
I20260709 12:19:36.148706 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.148938 27658 rpcz_store.cc:276] 0709 12:19:34.852926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:34.884877 (+ 31951us) service_pool.cc:224] Handling call
0709 12:19:36.136837 (+1251960us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:36.136966 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2376962304}
I20260709 12:19:36.058589 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:36.074205 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 122481us
I20260709 12:19:36.064920 31698 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.035095 31542 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:36.043119 31690 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.668s	user 0.013s	sys 0.000s
I20260709 12:19:36.060417 31574 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:36.121521 27588 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:36.106391 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:36.127441 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.136961 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:36.137063 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.147912 27662 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:36.077993 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:36.072956 31729 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.150595 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:36.147231 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 140487us
I20260709 12:19:36.076958 31757 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.155735 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.156332 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.162648 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:36.163672 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 422266us
I20260709 12:19:36.167253 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.133069 31647 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:36.136638 31671 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.198649 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:36.203800 31763 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:19:36.204092 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.204741 31417 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.205291 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.205845 27658 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:36.088779 31734 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.206699 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:36.207357 31692 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:36.207930 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.089870 31594 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:19:36.209327 27661 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:36.203649 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.209046 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.216857 27737 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:36.217751 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.207270 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:36.213595 27729 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:36.225488 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.226179 27729 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:36.213580 31671 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.227674 31740 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.229426 31671 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:36.208563 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:36.218854 27659 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:19:36.230510 31740 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:36.216276 31692 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:36.232445 31692 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.235733 27735 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:36.227141 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.167269 31698 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.249594 27657 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:36.219990 31690 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:36.088239 31758 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.209005 31729 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.205870 31417 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:36.132761 31064 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.202271 31464 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:19:36.243832 28336 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
W20260709 12:19:36.209084 31760 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:19:36.220118 31427 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.250801 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:36.542497 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 6.622s	user 0.170s	sys 0.056s Metrics: {"bytes_written":30615,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":23907,"dirs.run_cpu_time_us":420,"dirs.run_wall_time_us":973,"drs_written":1,"lbm_read_time_us":357,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520412,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":26,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":23525,"threads_started":1,"update_count":31}
I20260709 12:19:36.489882 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.525791 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.258546 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.405224 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.520952 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:36.223146 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2208) took 1369 ms. Trace:
I20260709 12:19:37.321612 27660 rpcz_store.cc:276] 0709 12:19:34.853590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:34.937378 (+ 83788us) service_pool.cc:224] Handling call
0709 12:19:36.222960 (+1285582us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400229554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:36.223087 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2535801216}
I20260709 12:19:36.774888 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.501601 27731 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:37.346773 31753 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:37.346962 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1869) took 1344 ms (client timeout 1689 ms). Trace:
I20260709 12:19:36.457803 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.347337 31753 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:37.347985 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.0972e+06us
I20260709 12:19:37.348240 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1882) took 1128 ms. Trace:
I20260709 12:19:37.348665 27732 rpcz_store.cc:276] 0709 12:19:36.220189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.457474 (+237285us) service_pool.cc:224] Handling call
0709 12:19:37.348183 (+890709us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.348860 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:37.349933 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.350680 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1884) took 1129 ms. Trace:
I20260709 12:19:36.541368 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:36.237187 27729 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:37.349581 31753 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:36.301879 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91889.3us
I20260709 12:19:36.852886 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2098) took 1526 ms. Trace:
I20260709 12:19:36.789542 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.384369 27735 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:37.385098 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1875) took 1247 ms. Trace:
I20260709 12:19:36.306917 28332 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:19:37.394799 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 624) took 1317 ms. Trace:
I20260709 12:19:36.238241 31016 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:37.404913 31016 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:37.405355 31016 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:36.382890 31064 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.409740 31064 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:36.209481 31251 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:36.416940 31363 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.278964 31417 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:37.414268 31417 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.414733 31363 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.415489 31016 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.418229 31016 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:36.430044 31429 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.396843 31542 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:36.216720 31574 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.428900 31613 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.432073 31574 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.432570 31574 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:37.432600 31613 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.433131 31574 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.427744 31647 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.352770 31698 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:36.217415 31704 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:36.395889 31729 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.146308 31733 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.088601 31735 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:36.354594 31736 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:36.209792 31757 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:36.382166 31758 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.368557 31759 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.444329 31780 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:36.724113 31781 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.335s	user 0.003s	sys 0.000s
I20260709 12:19:36.445174 31782 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.468627 31704 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:37.471411 31782 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.472182 31757 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.473176 31733 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.473546 31782 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:36.296654 31775 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.061s
I20260709 12:19:36.445089 31773 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.477144 31775 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:37.477547 31775 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:37.477372 31773 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.012768 31793 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.480234 31793 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.479439 27586 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:37.482430 31793 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.482477 27588 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:37.027441 31253 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:37.030299 27582 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.961s	user 0.015s	sys 0.056s
I20260709 12:19:37.053859 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.094674 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.483847 27580 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:37.484315 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 624) took 1058 ms. Trace:
I20260709 12:19:37.120251 31290 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:37.188683 27585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.993s	user 0.012s	sys 0.025s
I20260709 12:19:37.486282 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 620) took 1083 ms. Trace:
W20260709 12:19:37.486574 27585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.291s	user 0.013s	sys 0.025s
I20260709 12:19:37.202355 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:37.222163 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.438s	user 0.012s	sys 0.019s
I20260709 12:19:37.206260 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.240191 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:37.249301 31655 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.486689 27588 rpcz_store.cc:276] 0709 12:19:36.401840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.404784 (+  2944us) service_pool.cc:224] Handling call
0709 12:19:37.484620 (+1079836us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.484717 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511266560}
I20260709 12:19:37.259178 27583 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:37.269812 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2227) took 1222 ms. Trace:
I20260709 12:19:37.275468 31764 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.275931 31740 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:37.492290 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.708s	user 0.013s	sys 0.020s
I20260709 12:19:37.492010 31764 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.493230 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2105) took 1919 ms. Trace:
I20260709 12:19:37.493372 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.277494 31754 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:37.494336 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.494519 31755 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.493600 27736 rpcz_store.cc:276] 0709 12:19:35.574287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:35.575361 (+  1074us) service_pool.cc:224] Handling call
0709 12:19:35.599157 (+ 23796us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:35.655418 (+ 56261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:35.655464 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:35.675300 (+ 19836us) log.cc:844] Serialized 26 byte log entry
0709 12:19:37.492852 (+1817552us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:37.492895 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.492977 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:37.493016 (+    39us) raft_consensus.cc:1685] finished
0709 12:19:37.493026 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:37.493177 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36561,"raft.queue_time_us":58794,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":1708260,"threads_started":1,"wal-append.queue_time_us":585588,"wal-append.run_cpu_time_us":3544,"wal-append.run_wall_time_us":1040303}
I20260709 12:19:37.494259 31754 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.495210 31755 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.495992 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.496711 27736 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:37.278105 31615 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.279950 31739 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:37.281566 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43633.6us
I20260709 12:19:37.275254 31745 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.282765 31766 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:37.284111 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.275223 31559 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:19:37.501425 27655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 596) took 1327 ms (client timeout 1536 ms). Trace:
I20260709 12:19:37.302953 31441 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.508639 31441 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.506999 31745 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.510203 27737 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:37.304016 31464 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.304993 31594 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.498940 31615 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.299168 31432 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.313490 31796 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:36.735982 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 669578us
I20260709 12:19:37.335961 31795 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.337816 31685 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.337992 31602 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.338562 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1881) took 1118 ms. Trace:
W20260709 12:19:37.339078 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 124238us
I20260709 12:19:37.340492 31772 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:37.342351 31798 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.345592 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:37.346073 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 123) took 19850 ms (client timeout 15000 ms). Trace:
I20260709 12:19:37.350986 27732 rpcz_store.cc:276] 0709 12:19:36.220799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.349679 (+1128880us) service_pool.cc:224] Handling call
0709 12:19:37.350627 (+   948us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.351135 31427 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:37.351677 31364 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:19:37.347354 27731 rpcz_store.cc:269] 0709 12:19:36.002790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.162263 (+159473us) service_pool.cc:224] Handling call
0709 12:19:37.346883 (+1184620us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.353874 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:37.352546 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:37.361361 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1880) took 1141 ms. Trace:
I20260709 12:19:37.542160 27729 rpcz_store.cc:276] 0709 12:19:36.219639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.226856 (+  7217us) service_pool.cc:224] Handling call
0709 12:19:37.361300 (+1134444us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:36.466495 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.365746 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.373697 31797 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.385543 27735 rpcz_store.cc:276] 0709 12:19:36.137923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.789218 (+651295us) service_pool.cc:224] Handling call
0709 12:19:37.385015 (+595797us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.382827 27733 rpcz_store.cc:276] 0709 12:19:35.326867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:35.327443 (+   576us) service_pool.cc:224] Handling call
0709 12:19:35.332590 (+  5147us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:35.548298 (+215708us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:35.548328 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:35.561933 (+ 13605us) log.cc:844] Serialized 52 byte log entry
0709 12:19:36.072220 (+510287us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b1800066d60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:36.165636 (+ 93416us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:36.165673 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:36.165745 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:36.852416 (+686671us) raft_consensus.cc:1685] finished
0709 12:19:36.852437 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:36.852807 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19366,"raft.queue_time_us":232931,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":21199,"spinlock_wait_cycles":2383232,"thread_start_us":308495,"threads_started":2,"wal-append.queue_time_us":770237}
I20260709 12:19:37.395232 28332 rpcz_store.cc:276] 0709 12:19:36.077147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.077708 (+   561us) service_pool.cc:224] Handling call
0709 12:19:37.394512 (+1316804us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54003ed0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.394705 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307932544}
I20260709 12:19:37.397567 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:37.407822 28295 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.932s (SENT) [suppressed 20 similar messages]
I20260709 12:19:37.410426 31064 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.396309 27485 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 7 to 8, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:37.426893 28335 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:19:37.428064 27663 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
I20260709 12:19:37.430160 31429 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.427225 31799 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.620008 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.619910 31799 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.621451 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.622083 28332 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:37.622478 31799 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.469062 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.622890 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.469556 31780 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.478899 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:37.480619 31773 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.131281 31676 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.485147 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 622) took 1059 ms. Trace:
I20260709 12:19:37.632253 27586 rpcz_store.cc:276] 0709 12:19:36.425311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.489624 (+ 64313us) service_pool.cc:224] Handling call
0709 12:19:37.484981 (+995357us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54002b98d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.485089 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511751936}
I20260709 12:19:37.484609 27580 rpcz_store.cc:276] 0709 12:19:36.425928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.094377 (+668449us) service_pool.cc:224] Handling call
0709 12:19:37.484261 (+389884us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:37.486357 27582 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.417s	user 0.016s	sys 0.056s
I20260709 12:19:37.238126 31585 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.487507 27584 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:37.640537 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 619) took 1239 ms. Trace:
I20260709 12:19:37.487732 28336 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:37.257010 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 8051 bytes on disk
I20260709 12:19:37.492084 31740 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.653788 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.512051 31594 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.495528 31655 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.658893 28328 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:37.279488 31527 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.495220 28333 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:19:37.491601 27587 rpcz_store.cc:276] 0709 12:19:36.052449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.054876 (+  2427us) service_pool.cc:224] Handling call
0709 12:19:36.055227 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:37.074057 (+1018830us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:37.074094 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:37.074147 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:37.074182 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.269546 (+195364us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:37.269598 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:37.269751 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2342407040}
I20260709 12:19:37.657719 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2232) took 1603 ms. Trace:
I20260709 12:19:37.520610 31602 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.616627 31793 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.618984 31599 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:37.440140 31542 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.501469 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:37.301981 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.131s	user 0.024s	sys 0.019s
W20260709 12:19:37.660965 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.490s	user 0.025s	sys 0.020s
I20260709 12:19:37.660914 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:37.661490 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.55503e+06us
I20260709 12:19:37.499859 31739 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.486397 31290 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.463057 31417 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:37.507297 27655 rpcz_store.cc:269] 0709 12:19:36.173663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.215464 (+ 41801us) service_pool.cc:224] Handling call
0709 12:19:37.501369 (+1285905us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.694185 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 554) took 1625 ms. Trace:
I20260709 12:19:37.516664 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 591) took 1341 ms. Trace:
I20260709 12:19:37.521631 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:37.531683 31736 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:37.494397 31764 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:37.520750 27730 rpcz_store.cc:276] 0709 12:19:36.219915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.774567 (+554652us) service_pool.cc:224] Handling call
0709 12:19:37.338500 (+563933us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.520833 31559 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:19:37.534816 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:37.545262 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.545516 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.512658 31796 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.550545 27485 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 8 to 9, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:37.706961 27732 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:37.707645 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1885) took 1486 ms. Trace:
I20260709 12:19:37.708005 27732 rpcz_store.cc:276] 0709 12:19:36.221027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.544977 (+1323950us) service_pool.cc:224] Handling call
0709 12:19:37.707576 (+162599us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.707870 27485 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:37.708693 31796 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.682466 31527 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.709184 31559 consensus_queue.cc:1243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:37.709772 27485 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 10 to 11, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:37.553229 27729 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:19:37.545832 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:37.513635 31615 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.711037 31527 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:37.711138 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2230) took 1657 ms. Trace:
I20260709 12:19:37.711905 31810 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:37.712632 31740 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.712400 27485 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 4 to 5, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 5 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:37.713349 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.467624 31753 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:37.714526 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.714301 31753 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.715376 27485 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 8 to 9, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 9 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:37.716248 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 181558us
I20260709 12:19:37.716569 31753 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.717027 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:37.717190 31753 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:37.512847 31441 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.625056 28332 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:19:37.511313 31745 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.515295 31735 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:37.629302 27733 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:19:37.538240 31253 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260709 12:19:37.637535 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:37.478338 31775 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.472203 31704 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.647760 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:37.640985 27584 rpcz_store.cc:276] 0709 12:19:36.401518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.199315 (+797797us) service_pool.cc:224] Handling call
0709 12:19:37.640463 (+441148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.697381 31429 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:37.512346 31432 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.277021 31765 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:37.656662 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb500 after lost signal to thread 26411
I20260709 12:19:37.715405 27580 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:37.718858 27734 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:19:37.719300 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1883) took 1498 ms. Trace:
I20260709 12:19:37.511756 31464 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:37.658012 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 305523us
I20260709 12:19:37.657531 27589 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
W20260709 12:19:37.719676 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 173929us
I20260709 12:19:37.665663 31793 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.682493 31542 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.539171 31766 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20260709 12:19:37.660147 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:37.730876 27733 rpcz_store.cc:276] 0709 12:19:36.220480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.623492 (+1403012us) service_pool.cc:224] Handling call
0709 12:19:37.623842 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:37.719052 (+ 95210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.719251 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.464933 31574 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:37.661283 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 159872us
I20260709 12:19:37.497174 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1879) took 1277 ms. Trace:
I20260709 12:19:37.739882 27736 rpcz_store.cc:276] 0709 12:19:36.219336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.495754 (+1276418us) service_pool.cc:224] Handling call
0709 12:19:37.497121 (+  1367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:37.501528 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:37.628482 31755 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:37.614778 31363 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 12:19:37.659829 27582 rpcz_store.cc:276] 0709 12:19:36.054122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.055674 (+  1552us) service_pool.cc:224] Handling call
0709 12:19:36.055931 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:36.056165 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:37.634582 (+1578417us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.657391 (+ 22809us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:37.657438 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:37.657481 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.657553 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:37.657668 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6884,"prepare.queue_time_us":289499,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"thread_start_us":1581073,"threads_started":2,"wal-append.queue_time_us":21626}
I20260709 12:19:37.663157 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:37.627458 31733 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.695289 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.747848 27586 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:19:37.520363 31685 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.697554 27588 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:37.697381 27737 rpcz_store.cc:276] 0709 12:19:36.175010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.053519 (+878509us) service_pool.cc:224] Handling call
0709 12:19:37.516508 (+462989us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.516608 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60896128}
I20260709 12:19:37.703199 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1808) took 1726 ms. Trace:
I20260709 12:19:37.729934 31647 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:37.551827 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 154375us
I20260709 12:19:37.623368 28337 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:37.711182 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1868) took 1726 ms. Trace:
I20260709 12:19:37.535354 31772 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:37.664721 31698 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:37.655509 31585 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:37.539808 27483 rpcz_store.cc:269] 0709 12:19:17.495850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.496106 (+   256us) service_pool.cc:224] Handling call
0709 12:19:37.345969 (+19849863us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:36.197688 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:36.231671 (+ 33983us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:36.231723 (+    52us) write_op.cc:435] Acquired schema lock
0709 12:19:36.231738 (+    15us) tablet.cc:662] Decoding operations
0709 12:19:36.233004 (+  1266us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:36.233093 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:36.233128 (+    35us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:19:36.234226 (+  1098us) tablet.cc:701] Row locks acquired
0709 12:19:36.234276 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:19:36.234480 (+   204us) write_op.cc:270] Start()
0709 12:19:36.234629 (+   149us) write_op.cc:276] Timestamp: P: 1783599576234384 usec, L: 0
0709 12:19:36.234653 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:19:36.322841 (+ 88188us) log.cc:844] Serialized 4362 byte log entry
0709 12:19:36.435125 (+112284us) op_driver.cc:464] REPLICATION: finished
0709 12:19:36.440143 (+  5018us) write_op.cc:301] APPLY: starting
0709 12:19:36.440258 (+   115us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:36.441197 (+   939us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:36.441319 (+   122us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:36.442446 (+  1127us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:36.444749 (+  2303us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0709 12:19:36.444810 (+    61us) write_op.cc:312] APPLY: finished
0709 12:19:36.445209 (+   399us) log.cc:844] Serialized 83 byte log entry
0709 12:19:36.445414 (+   205us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:36.447513 (+  2099us) write_op.cc:454] Released schema lock
0709 12:19:36.447600 (+    87us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:36.449872 (+  2272us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:36.449908 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"dns_us":198,"mutex_wait_us":2958901,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":4714,"apply.run_cpu_time_us":4061,"apply.run_wall_time_us":9702,"num_ops":11,"prepare.queue_time_us":435939,"prepare.run_cpu_time_us":76269,"prepare.run_wall_time_us":284763,"raft.queue_time_us":997,"raft.run_cpu_time_us":2823,"raft.run_wall_time_us":9667,"replication_time_us":200297,"thread_start_us":392398,"threads_started":4,"wal-append.queue_time_us":89086,"wal-append.run_cpu_time_us":5002,"wal-append.run_wall_time_us":1101859}]]}
I20260709 12:19:37.520097 31795 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.629992 31797 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:37.730458 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08001b8a40 after lost signal to thread 27521
W20260709 12:19:37.774821 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08001e0920 after lost signal to thread 27613
I20260709 12:19:37.626729 31757 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:37.732172 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:37.547506 31754 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.543941 31798 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.721504 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.056s	user 0.002s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":498,"lbm_reads_lt_1ms":4}
W20260709 12:19:37.733242 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 379431us
W20260709 12:19:37.722239 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 84789.3us
W20260709 12:19:37.808166 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:37.726349 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 318528us
I20260709 12:19:37.717389 27485 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:37.718129 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:37.717612 31753 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:37.737914 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:37.683933 31599 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.741204 27587 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:37.725706 31504 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:37.765969 31772 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:37.655228 31676 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.766081 31698 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:37.748802 27663 rpcz_store.cc:276] 0709 12:19:35.976901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:35.977053 (+   152us) service_pool.cc:224] Handling call
0709 12:19:35.990200 (+ 13147us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:37.617787 (+1627587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.702881 (+ 85094us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400211454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:37.703123 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3306213120}
I20260709 12:19:37.829902 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.830741 27732 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:37.718349 31559 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:37.736487 31704 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:37.750653 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.697273 28330 rpcz_store.cc:276] 0709 12:19:36.068437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.073106 (+  4669us) service_pool.cc:224] Handling call
0709 12:19:37.671304 (+1598198us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400962654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.694121 (+ 22817us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3060643584}
I20260709 12:19:37.739539 31775 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:37.736629 31429 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:37.753925 27729 rpcz_store.cc:276] 0709 12:19:35.985268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.552360 (+1567092us) service_pool.cc:224] Handling call
0709 12:19:37.552792 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:37.710963 (+158171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.711136 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:37.851534 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119429us
I20260709 12:19:37.816244 31753 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:37.853616 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:37.854197 27582 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:19:37.762620 31809 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.763350 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.763586 31253 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:37.764503 31594 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:37.780547 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080018b340 after lost signal to thread 27688
I20260709 12:19:37.812539 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:37.863612 27662 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:37.814114 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:37.815534 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.866416 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:19:37.816959 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98900.5us
I20260709 12:19:37.711688 27585 rpcz_store.cc:276] 0709 12:19:36.053514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.055536 (+  2022us) service_pool.cc:224] Handling call
0709 12:19:36.067244 (+ 11708us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:36.138384 (+ 71140us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:36.195680 (+ 57296us) log.cc:844] Serialized 14 byte log entry
0709 12:19:36.723485 (+527805us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:37.626134 (+902649us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.626537 (+   403us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.654937 (+ 28400us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.655084 (+   147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:37.655123 (+    39us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:37.655173 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.710934 (+ 55761us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b54000893d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:37.710979 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:37.711088 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41550,"prepare.queue_time_us":349559,"prepare.run_cpu_time_us":151,"prepare.run_wall_time_us":1197,"raft.queue_time_us":42935,"raft.run_cpu_time_us":453,"raft.run_wall_time_us":28025,"spinlock_wait_cycles":246854016,"thread_start_us":1450354,"threads_started":3,"wal-append.queue_time_us":164401}
I20260709 12:19:37.828583 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:37.833127 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:37.867971 27587 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.867746 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:37.837312 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:37.868659 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:37.787181 31016 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:37.869675 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08001a5a80 after lost signal to thread 27689
I20260709 12:19:37.869879 27485 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:37.870800 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08003eb7c0 after lost signal to thread 27762
W20260709 12:19:37.871062 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800162b60 after lost signal to thread 27763
I20260709 12:19:37.736356 31765 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.752321 31733 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:37.848954 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:37.855530 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.855218 27581 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:37.820266 31504 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:37.825636 31754 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.859726 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.862749 31253 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:37.840975 31064 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:37.871522 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57486.2us
I20260709 12:19:37.875726 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:37.833719 31698 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.879629 27732 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.885751 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:37.879952 27584 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:37.891762 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:37.881011 27729 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:37.897082 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:37.901590 27582 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
W20260709 12:19:37.903191 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 626) took 1476 ms (client timeout 1542 ms). Trace:
I20260709 12:19:37.894101 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:37.905406 27729 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:37.906881 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:37.907720 27729 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:37.903604 27589 rpcz_store.cc:269] 0709 12:19:36.426529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.520635 (+ 94106us) service_pool.cc:224] Handling call
0709 12:19:37.903013 (+1382378us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.903132 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":911325952}
I20260709 12:19:37.912503 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:37.910399 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1866) took 1954 ms (client timeout 1966 ms). Trace:
I20260709 12:19:37.909953 27738 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:19:37.913357 27586 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
W20260709 12:19:37.917943 27734 rpcz_store.cc:269] 0709 12:19:35.955984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.466191 (+510207us) service_pool.cc:224] Handling call
0709 12:19:37.898000 (+1431809us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:37.910337 (+ 12337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394055424}
W20260709 12:19:37.920393 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67086.1us
I20260709 12:19:37.921562 31821 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:37.923055 31432 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.924290 31798 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.925343 31585 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.927729 31793 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.881112 31016 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:37.930243 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080010d460 after lost signal to thread 26411
W20260709 12:19:37.931288 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08001c16a0 after lost signal to thread 27521
I20260709 12:19:37.931763 31429 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:37.932477 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080016ad60 after lost signal to thread 27613
I20260709 12:19:37.880591 31504 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:37.934442 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080012ba40 after lost signal to thread 27660
W20260709 12:19:37.937470 27685 debug-util.cc:398] Leaking SignalData structure 0x7b0800146220 after lost signal to thread 27688
W20260709 12:19:37.937794 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08002cad80 after lost signal to thread 27689
W20260709 12:19:37.938128 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080035ad80 after lost signal to thread 27703
W20260709 12:19:37.938751 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003a52e0 after lost signal to thread 27729
I20260709 12:19:37.878170 31818 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:37.881379 31253 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.909353 31615 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:37.947275 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:37.949880 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 286746us
I20260709 12:19:37.952387 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:37.954902 31064 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:37.955473 31064 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:37.955825 31064 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:37.960665 31780 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:37.956768 31798 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:37.961656 31798 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:37.962025 31798 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:37.903617 31766 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:37.902985 31755 raft_consensus.cc:2764] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:37.939354 27729 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:37.883729 31729 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:37.941560 27685 debug-util.cc:398] Leaking SignalData structure 0x7b0800314620 after lost signal to thread 27762
W20260709 12:19:38.008222 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080027ba80 after lost signal to thread 27763
W20260709 12:19:38.008697 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080028a920 after lost signal to thread 28292
W20260709 12:19:38.009325 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08001eafe0 after lost signal to thread 28332
W20260709 12:19:38.010006 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08002097a0 after lost signal to thread 28361
W20260709 12:19:38.010290 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080033e760 after lost signal to thread 31016
W20260709 12:19:38.010517 27685 debug-util.cc:398] Leaking SignalData structure 0x7b0800336940 after lost signal to thread 31064
I20260709 12:19:37.863740 31809 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:37.985754 31740 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.992851 31733 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.993611 31759 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.995354 31574 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:37.996227 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:37.994678 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.053748 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:38.053952 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.053906 31759 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.055680 31464 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.056097 31759 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.882349 31685 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:38.057042 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60881.9us
I20260709 12:19:38.058295 31766 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:38.058686 31766 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:38.059039 31766 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:38.059739 31574 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:38.060084 31574 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:38.060881 31574 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.062058 31574 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.063970 31797 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:38.064726 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70130.7us
I20260709 12:19:38.065395 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:38.066385 31796 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:38.066864 31796 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:38.067801 31796 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.068405 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:38.068871 31793 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:38.069110 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 592) took 1893 ms. Trace:
I20260709 12:19:38.069303 31793 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:38.069744 31793 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.069518 27735 rpcz_store.cc:276] 0709 12:19:36.175398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.548002 (+1372604us) service_pool.cc:224] Handling call
0709 12:19:37.548308 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:37.693967 (+145659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:37.694002 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:37.705067 (+ 11065us) log.cc:844] Serialized 26 byte log entry
0709 12:19:37.825505 (+120438us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:37.825564 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.958229 (+132665us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54000a4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:37.958282 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:38.068843 (+110561us) raft_consensus.cc:1685] finished
0709 12:19:38.068862 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.069053 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":738,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5117,"spinlock_wait_cycles":334123904,"thread_start_us":86669,"threads_started":1,"wal-append.queue_time_us":190723}
I20260709 12:19:38.006899 27588 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:37.913983 31417 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.092584 27658 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:38.092866 27485 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 9, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 9 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:38.021041 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080037ba80 after lost signal to thread 31251
W20260709 12:19:38.096366 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080035c5a0 after lost signal to thread 31363
W20260709 12:19:38.096695 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080033b700 after lost signal to thread 31364
W20260709 12:19:38.097003 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003eaa60 after lost signal to thread 31417
W20260709 12:19:38.097285 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08002c0320 after lost signal to thread 31427
W20260709 12:19:38.097566 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08002c0ba0 after lost signal to thread 31429
W20260709 12:19:38.097848 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003cc8e0 after lost signal to thread 31432
W20260709 12:19:38.098120 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003c9360 after lost signal to thread 31441
W20260709 12:19:38.098457 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003d53e0 after lost signal to thread 31464
W20260709 12:19:38.098788 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08000fb480 after lost signal to thread 31542
W20260709 12:19:38.099056 27685 debug-util.cc:398] Leaking SignalData structure 0x7b0800376340 after lost signal to thread 31559
I20260709 12:19:38.023206 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:37.916420 31647 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:19:37.932850 31527 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:38.007861 31729 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.052302 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.054344 28335 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:38.003819 31818 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:38.099741 27584 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:38.100128 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003bdda0 after lost signal to thread 31574
W20260709 12:19:38.110845 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080035c800 after lost signal to thread 31594
I20260709 12:19:38.007148 31602 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:37.909803 31825 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:38.102656 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.115942 27588 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:38.104624 27732 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:38.116823 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.108196 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.117480 27588 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:38.117589 27735 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:38.117746 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.118361 27584 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:38.118914 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:38.111267 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003feec0 after lost signal to thread 31613
W20260709 12:19:38.119750 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003d5660 after lost signal to thread 31647
W20260709 12:19:38.120095 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08001b58c0 after lost signal to thread 31698
W20260709 12:19:38.120347 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003e1020 after lost signal to thread 31704
W20260709 12:19:38.101393 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:38.122509 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.127205 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:38.128074 31827 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:19:38.128916 31822 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:19:38.129243 31755 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.129896 31542 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.129735 31755 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.130330 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
I20260709 12:19:38.131029 31795 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.131410 31809 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.137338 27733 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:38.101182 31647 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:38.138330 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080038e9c0 after lost signal to thread 31729
W20260709 12:19:38.139943 27685 debug-util.cc:398] Leaking SignalData structure 0x7b0800345ca0 after lost signal to thread 31733
I20260709 12:19:38.140513 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.137458 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.141305 27584 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:38.141337 27588 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:38.142298 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.143028 27584 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:38.143790 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:38.144016 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:38.142320 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08002288e0 after lost signal to thread 31735
I20260709 12:19:38.144565 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:38.144886 27685 debug-util.cc:398] Leaking SignalData structure 0x7b0800256040 after lost signal to thread 31736
W20260709 12:19:38.145411 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc220 after lost signal to thread 31757
W20260709 12:19:38.145767 27685 debug-util.cc:398] Leaking SignalData structure 0x7b080037b020 after lost signal to thread 31758
I20260709 12:19:38.145465 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:38.146018 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003184a0 after lost signal to thread 31759
I20260709 12:19:38.147682 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.121349 27485 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 1 to 7, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:38.149992 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.107204 31729 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:38.158215 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:38.160593 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:38.162611 27588 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:38.165308 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 279595us
I20260709 12:19:38.051533 31740 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.168061 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.155511 27584 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:38.169265 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.157070 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:38.154765 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:38.146261 27701 connection.cc:256] RPC call timeout handler was delayed by -1.5052477920000911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2311s for a response
I20260709 12:19:38.156351 31795 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:38.112347 31527 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:38.189306 31527 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.170125 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.190040 27655 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
I20260709 12:19:38.167018 31739 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.191466 31758 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.191975 31758 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.192986 31740 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.193957 31758 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.193714 31795 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:38.196488 31795 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.173523 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.136857 31602 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:38.176054 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:38.155834 27685 debug-util.cc:398] Leaking SignalData structure 0x7b08003d3b40 after lost signal to thread 31767
I20260709 12:19:38.274988 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.176362 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:38.276814 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 100546us
I20260709 12:19:38.178269 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:38.277611 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.183005 27737 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:38.177661 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.184064 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.184805 27584 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:38.196905 31818 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.199807 31599 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.172390 31765 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.279122 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2246) took 1072 ms. Trace:
I20260709 12:19:38.201960 28335 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:38.203557 27485 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 8, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 8 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:38.279675 27584 rpcz_store.cc:276] 0709 12:19:37.207190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.169037 (+961847us) service_pool.cc:224] Handling call
0709 12:19:38.279028 (+109991us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:38.277447 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:38.208109 27553 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:38.281525 27553 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:19:38.281752 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97770.5us
W20260709 12:19:38.281998 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 74041us
I20260709 12:19:38.282502 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.259513 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 623) took 1833 ms. Trace:
I20260709 12:19:38.275728 27733 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:38.283156 27589 rpcz_store.cc:276] 0709 12:19:36.425615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.257611 (+1831996us) service_pool.cc:224] Handling call
0709 12:19:38.258856 (+  1245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.259072 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.259109 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.259161 (+    52us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:38.259205 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.259276 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.259439 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:38.284474 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.110039 31733 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260709 12:19:38.156737 31745 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:38.156576 31736 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:19:38.300209 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.152652 31647 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:38.201864 27660 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:19:38.208874 27586 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:38.287633 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.287681 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.381108 27737 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:38.378193 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.381983 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.289894 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.281181 27485 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 2 to 7, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 7 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:38.293761 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.291234 31613 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:38.292553 31599 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:38.292186 31759 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.295156 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.356432 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:38.405561 27659 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:38.275785 31764 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:38.378131 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:38.378803 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.378432 31733 consensus_queue.cc:1243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:19:38.379385 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:38.384011 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 225885us
I20260709 12:19:38.384876 27737 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:38.302001 31745 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:38.383947 27589 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:38.382668 27732 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:38.288363 31795 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:38.390952 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101132us
I20260709 12:19:38.418139 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.296356 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.385877 31613 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:38.399479 27587 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.529s	user 0.010s	sys 0.000s
I20260709 12:19:38.422219 27587 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
W20260709 12:19:38.423295 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1876) took 2265 ms (client timeout 1810 ms). Trace:
I20260709 12:19:38.421518 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:38.400103 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 390024us
I20260709 12:19:38.401361 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.424742 27585 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:38.176857 31831 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:38.404686 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 127309us
I20260709 12:19:38.423465 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:38.424000 27738 rpcz_store.cc:269] 0709 12:19:36.158289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.541006 (+382717us) service_pool.cc:224] Handling call
0709 12:19:38.416973 (+1875967us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:38.423217 (+  6244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1264172928}
I20260709 12:19:38.426436 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2245) took 1981 ms. Trace:
I20260709 12:19:38.426611 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:38.426816 27587 rpcz_store.cc:276] 0709 12:19:36.444771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.425698 (+1980927us) service_pool.cc:224] Handling call
0709 12:19:38.425964 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.426149 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.426172 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.426211 (+    39us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:38.426243 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.426298 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.426388 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:38.406872 27662 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:38.408056 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.428123 27586 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:38.428202 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.408533 27582 raft_consensus.cc:2376] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8. Re-sending same reply.
I20260709 12:19:38.269300 31802 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.409673 27733 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:38.269469 31602 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.407877 31764 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:38.418666 31775 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.295285 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.416523 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.413472 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.412420 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.418910 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.398061 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:38.419472 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.419925 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.481204 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.428813 27581 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:38.429406 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.409034 31835 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:19:38.438422 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 155773us
I20260709 12:19:38.481513 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:38.417850 31755 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.411469 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:38.488099 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76753.8us
I20260709 12:19:38.420418 31745 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:38.330240 31772 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.431245 31733 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:38.397331 31797 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.481873 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2253) took 1272 ms. Trace:
I20260709 12:19:38.482163 27580 raft_consensus.cc:2376] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11. Re-sending same reply.
I20260709 12:19:38.507808 27588 rpcz_store.cc:276] 0709 12:19:37.209147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.160346 (+951199us) service_pool.cc:224] Handling call
0709 12:19:38.481805 (+321459us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:38.481074 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.481488 28328 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
W20260709 12:19:38.485486 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56163.3us
I20260709 12:19:38.514829 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.516602 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.425874 31613 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:38.519857 31838 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:19:38.523955 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.524933 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:38.525138 31735 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.806s	user 0.011s	sys 0.000s
I20260709 12:19:38.428421 31831 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:38.525980 31831 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:38.526948 27660 raft_consensus.cc:1217] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Deduplicated request from leader. Original: 5.4->[7.5-14.8]   Dedup: 10.6->[12.7-14.8]
I20260709 12:19:38.527446 27660 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 6
I20260709 12:19:38.527657 27660 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:19:38.527879 27660 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:38.529209 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.405855 31064 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.428309 31759 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:38.531492 31759 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.531991 31759 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.532735 31064 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.534168 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.439149 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:38.535653 27732 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.489194 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.539410 31799 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.539978 31799 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.540314 31432 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:38.540870 31432 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.541313 31432 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:38.486939 31775 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.545316 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.546624 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.547091 27583 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 5.
I20260709 12:19:38.492774 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.548108 27581 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:38.491091 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:38.414871 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:38.549973 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.550271 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.551249 27662 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:38.551782 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.505896 31765 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:19:38.479643 31754 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.000s
I20260709 12:19:38.505261 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2247) took 1297 ms. Trace:
I20260709 12:19:38.559813 27585 rpcz_store.cc:276] 0709 12:19:37.207501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.485996 (+1278495us) service_pool.cc:224] Handling call
0709 12:19:38.504750 (+ 18754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.504934 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.504959 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.505002 (+    43us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:38.505031 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.505082 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.505202 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:38.400348 31766 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.480020 31764 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:38.516784 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.490983 31795 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:38.486878 31527 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.515924 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:38.582028 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66209.5us
I20260709 12:19:38.582979 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.515578 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
W20260709 12:19:38.584705 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 184861us
I20260709 12:19:38.585254 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.585506 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.558399 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.558916 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.493868 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:38.586899 31740 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:38.587486 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93688.7us
I20260709 12:19:38.562856 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.590101 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:38.560882 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.591251 31755 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:38.597469 31755 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.597736 31740 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.597966 31755 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:38.563410 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.477174 31802 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:38.599454 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.600057 27732 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:38.600234 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.591830 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.516454 28331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:38.508016 31253 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:38.599992 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:38.649513 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49606us
I20260709 12:19:38.649904 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.651028 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.641775 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.593384 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.642557 27736 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
W20260709 12:19:38.660317 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67020.7us
I20260709 12:19:38.660508 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.661136 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.661721 31837 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:38.662417 31837 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:38.645732 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45472.2us
I20260709 12:19:38.558218 31733 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:38.559628 31765 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:38.573288 31797 raft_consensus.cc:2764] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:19:38.603211 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:38.645155 31574 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.583s	user 0.010s	sys 0.002s
I20260709 12:19:38.661187 27736 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:19:38.573341 31754 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:38.647879 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:38.640744 31064 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:38.663669 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.598829 31804 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:38.670737 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.662357 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:38.602600 31427 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:38.674276 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.690276 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.668577 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.640651 31251 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:38.680148 27737 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:38.563405 31775 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.687752 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.663731 31765 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.666366 28331 raft_consensus.cc:606] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.667047 31735 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:38.668103 31733 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:38.667769 31780 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.692549 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.694283 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.640481 31527 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.706013 27733 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
W20260709 12:19:38.706782 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 280217us
I20260709 12:19:38.707407 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.714850 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:38.683176 31753 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.814s	user 0.009s	sys 0.001s
W20260709 12:19:38.687433 31793 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.617s	user 0.013s	sys 0.000s
I20260709 12:19:38.717983 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:38.720296 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.690689 31427 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.723963 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.721302 31754 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:38.703048 31574 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.724706 27736 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:38.703544 31251 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.725643 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:38.705495 31839 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:38.726303 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.708958 31797 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:38.706363 31804 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:38.708905 31735 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.798398 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.739873 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.800376 27733 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:38.768188 27659 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:38.778453 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.805595 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.757565 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:38.689419 31417 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260709 12:19:38.763130 31754 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:38.782377 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.739580 31795 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:38.764338 31780 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.760912 31839 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.727591 31427 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:38.716472 31599 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 12:19:38.760500 31251 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.760680 31290 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.705390 31837 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:19:38.780689 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.781672 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.833313 27738 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:38.782105 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 626) took 1438 ms. Trace:
I20260709 12:19:38.832993 27588 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:19:38.838587 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:38.839998 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.840904 27588 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:38.812363 31798 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.850s	user 0.009s	sys 0.004s
I20260709 12:19:38.740011 27732 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:38.779404 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:38.846916 27587 raft_consensus.cc:2376] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8. Re-sending same reply.
I20260709 12:19:38.848297 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:38.784118 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.849843 27657 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:38.851217 27657 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:19:38.788053 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.719587 31064 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:38.725375 31808 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:38.805533 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:38.727572 31739 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:38.820668 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.766368 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.763166 31846 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:38.827526 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:38.773128 31842 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:19:38.765609 31797 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.797588 31735 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:38.775695 31772 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.806926 27733 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:38.790756 31253 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.788527 31766 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:38.859658 31698 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.950s	user 0.011s	sys 0.000s
I20260709 12:19:38.876329 27658 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:19:38.726786 31753 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.860563 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:38.771974 31016 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.789881 31802 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.823805 31839 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:38.814028 31795 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:38.788874 31527 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.865384 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:38.727135 31793 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:38.865437 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 103119us
I20260709 12:19:38.894434 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.895579 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:38.869202 27659 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.827883 31829 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:38.870412 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:38.826455 31363 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:38.870854 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.873453 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.904532 27738 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:19:38.874142 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.825940 31754 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:38.876861 27658 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 8
I20260709 12:19:38.906987 27658 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:19:38.834249 28329 rpcz_store.cc:276] 0709 12:19:37.344169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.344399 (+   230us) service_pool.cc:224] Handling call
0709 12:19:37.344788 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:37.858876 (+514088us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:19:38.162019 (+303143us) log.cc:844] Serialized 14 byte log entry
0709 12:19:38.480678 (+318659us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.480738 (+    60us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:38.480779 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.781820 (+301041us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:38.781860 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.782005 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":303072,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":282113,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":1181647104,"thread_start_us":596811,"threads_started":2,"wal-append.queue_time_us":354582}
I20260709 12:19:38.855396 31835 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:38.884503 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:38.869064 31846 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.782343 31613 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:38.768658 31733 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.874521 31797 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:38.828737 31759 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.870128 31780 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:38.891301 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.919708 27584 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:38.897826 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:38.898346 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 173155us
I20260709 12:19:38.898891 27729 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:19:38.864671 31798 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.905351 27735 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:19:38.921164 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:38.905872 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2259) took 1171 ms. Trace:
I20260709 12:19:38.907280 27658 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:38.893035 31795 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:38.918041 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:38.924669 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:38.924866 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:38.919723 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:38.921883 27587 rpcz_store.cc:276] 0709 12:19:37.734458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.905114 (+1170656us) service_pool.cc:224] Handling call
0709 12:19:38.905409 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.905581 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.905606 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.905645 (+    39us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:38.905674 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.905724 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.905820 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:38.925261 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.926388 27738 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 7.
I20260709 12:19:38.898211 31574 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:38.885216 31698 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:38.929589 31850 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:19:38.930981 31698 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.910110 31064 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:38.935667 27731 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:38.938232 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1877) took 2736 ms (client timeout 2194 ms). Trace:
W20260709 12:19:38.938665 27731 rpcz_store.cc:269] 0709 12:19:36.201886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.545271 (+1343385us) service_pool.cc:224] Handling call
0709 12:19:38.937967 (+1392696us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:38.938152 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2825075456}
I20260709 12:19:38.940289 27734 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:38.942076 31655 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:38.947288 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86823.5us
I20260709 12:19:38.948136 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 627) took 2521 ms. Trace:
I20260709 12:19:38.948523 27589 rpcz_store.cc:276] 0709 12:19:36.426813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.423154 (+1996341us) service_pool.cc:224] Handling call
0709 12:19:38.423461 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.431759 (+  8298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.431789 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.431831 (+    42us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:38.431860 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.947906 (+516046us) spinlock_profiling.cc:243] Waited 8.09 ms on lock 0x7b540011ce54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:38.947948 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.948077 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18599680}
I20260709 12:19:38.949844 31772 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:38.950001 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:38.950440 31772 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:38.950673 27589 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:38.951485 31772 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.954567 31808 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:38.964339 31857 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:19:38.965698 31851 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:19:38.969986 31527 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:38.970463 31527 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:38.970981 31504 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.966s	user 0.009s	sys 0.000s
I20260709 12:19:38.972823 31753 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:38.975294 31602 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.977296 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 625) took 2551 ms. Trace:
I20260709 12:19:38.978117 31759 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.978922 27580 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:38.979423 31363 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:38.906509 31802 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:38.981120 31647 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.581s	user 0.006s	sys 0.003s
I20260709 12:19:38.981966 31829 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:38.983083 31831 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.983747 31793 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:38.985000 31432 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.970994 31527 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:38.986919 31736 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:38.988349 31729 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.808s	user 0.011s	sys 0.000s
I20260709 12:19:38.988848 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:38.990868 31754 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.995779 31835 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.000795 27729 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
W20260709 12:19:39.002063 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77499.7us
I20260709 12:19:39.002535 31837 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:19:39.004227 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 597) took 1194 ms. Trace:
I20260709 12:19:39.087103 27734 rpcz_store.cc:276] 0709 12:19:37.809968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.918895 (+108927us) service_pool.cc:224] Handling call
0709 12:19:38.940118 (+1021223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.940744 (+   626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.004009 (+ 63265us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400317c50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:39.004164 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2347836672}
W20260709 12:19:39.008021 27583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2258) took 1793 ms (client timeout 1557 ms). Trace:
W20260709 12:19:39.088805 27583 rpcz_store.cc:269] 0709 12:19:37.214444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.493085 (+278641us) service_pool.cc:224] Handling call
0709 12:19:39.007756 (+1514671us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000889d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:39.007935 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2421272320}
I20260709 12:19:39.090317 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:39.008353 27587 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260709 12:19:39.091336 27587 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:39.091697 27587 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260709 12:19:39.092048 27587 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:39.009532 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.011602 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.012305 28328 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:39.014101 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.015362 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.015683 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.016273 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.016615 27662 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:39.094772 27660 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:19:39.095619 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.017405 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.018023 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:39.097801 27738 raft_consensus.cc:606] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:39.098323 27738 raft_consensus.cc:614] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:39.009557 31613 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:38.955200 31808 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:39.024150 28337 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.026563 31798 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.027366 31676 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:39.028453 31615 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.029124 31740 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.885046 31804 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:39.030915 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 133159us
I20260709 12:19:39.031036 31655 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:39.034678 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.037535 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:39.043625 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.047891 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:39.051059 27584 rpcz_store.cc:276] 0709 12:19:36.426232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.291525 (+1865293us) service_pool.cc:224] Handling call
0709 12:19:38.291844 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.920196 (+628352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:38.977054 (+ 56858us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b5400089dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:38.977233 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434721664}
I20260709 12:19:39.056835 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 600) took 1246 ms. Trace:
I20260709 12:19:39.059890 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:38.916570 31846 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.065688 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.048554 31758 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:38.985877 31745 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:38.993160 31764 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.080277 27659 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:39.085671 27701 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:39.070962 31647 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.104321 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2251) took 1895 ms. Trace:
I20260709 12:19:39.105203 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2238) took 1682 ms. Trace:
I20260709 12:19:39.083873 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.155351 27588 rpcz_store.cc:276] 0709 12:19:37.423159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.875107 (+1451948us) service_pool.cc:224] Handling call
0709 12:19:38.875830 (+   723us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:39.050997 (+175167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.051029 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:39.051714 (+   685us) log.cc:844] Serialized 26 byte log entry
0709 12:19:39.103563 (+ 51849us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:39.103613 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.103711 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:39.105002 (+  1291us) raft_consensus.cc:1685] finished
0709 12:19:39.105025 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.105144 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11323,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":8380,"thread_start_us":219001,"threads_started":2,"wal-append.queue_time_us":43405}
I20260709 12:19:39.157393 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.157434 27729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:39.157953 27588 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:39.158344 27729 raft_consensus.cc:606] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:39.158843 27729 raft_consensus.cc:614] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:39.066042 31802 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:39.071107 31829 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:39.086802 31729 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.137133 31655 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.124356 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.125368 27661 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 8: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:39.126606 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:39.109380 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:39.131613 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:39.186007 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 76758.7us
W20260709 12:19:39.186641 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55120us
I20260709 12:19:39.129047 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.187804 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:39.186790 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:39.132977 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46422.7us
I20260709 12:19:39.086834 31685 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.134828 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.141480 31733 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:39.134971 27730 rpcz_store.cc:276] 0709 12:19:37.810629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:37.949179 (+138550us) service_pool.cc:224] Handling call
0709 12:19:37.949556 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.939668 (+990112us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.939698 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:38.939796 (+    98us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:38.939847 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.056599 (+116752us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b540029fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:39.056645 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.056778 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2276471168}
I20260709 12:19:39.082150 31736 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:39.202085 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:39.135649 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.203732 27737 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:39.204576 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.205150 27737 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:39.207058 31755 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.608s	user 0.006s	sys 0.004s
W20260709 12:19:39.145095 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59525.6us
I20260709 12:19:39.139102 31758 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:39.136128 31740 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:39.146351 31864 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:19:39.146919 31432 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.154055 27580 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:19:39.232589 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2243) took 1466 ms. Trace:
I20260709 12:19:39.054347 31504 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.233292 31740 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.232796 31758 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.234009 31758 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:39.234177 31733 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:39.235388 31504 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.155172 27585 rpcz_store.cc:276] 0709 12:19:37.208579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.562545 (+1353966us) service_pool.cc:224] Handling call
0709 12:19:38.642736 (+ 80191us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:38.643193 (+   457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.643234 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:38.691168 (+ 47934us) log.cc:844] Serialized 26 byte log entry
0709 12:19:39.103949 (+412781us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:39.104000 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.104091 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:39.104140 (+    49us) raft_consensus.cc:1685] finished
0709 12:19:39.104150 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.104270 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5028,"raft.queue_time_us":130,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":20206,"thread_start_us":377570,"threads_started":1,"wal-append.queue_time_us":388858}
I20260709 12:19:39.135270 31615 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.236938 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.237890 31615 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:39.238531 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:19:39.238183 31733 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.233094 27580 rpcz_store.cc:276] 0709 12:19:37.765863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.130802 (+1364939us) service_pool.cc:224] Handling call
0709 12:19:39.143916 (+ 13114us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:39.232279 (+ 88363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.232506 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.239919 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:39.240370 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:39.086725 31757 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:19:39.240640 27580 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:39.241705 31757 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:39.241925 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.124419 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2248) took 1916 ms. Trace:
I20260709 12:19:39.242587 31823 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:19:39.242660 27581 rpcz_store.cc:276] 0709 12:19:37.207760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.562491 (+1354731us) service_pool.cc:224] Handling call
0709 12:19:38.601763 (+ 39272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:38.642494 (+ 40731us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:19:38.643944 (+  1450us) log.cc:844] Serialized 14 byte log entry
0709 12:19:39.123915 (+479971us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.123968 (+    53us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:39.124015 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.124249 (+   234us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.124367 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1130,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":8947,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":4375,"spinlock_wait_cycles":4885760,"thread_start_us":478164,"threads_started":1,"wal-append.queue_time_us":497927}
I20260709 12:19:39.167151 31795 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.247784 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.247952 31736 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:39.172051 31866 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:19:39.174208 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:39.175153 31797 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.249500 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2240) took 1505 ms. Trace:
I20260709 12:19:39.250003 27587 rpcz_store.cc:276] 0709 12:19:37.744609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.926879 (+1182270us) service_pool.cc:224] Handling call
0709 12:19:39.007900 (+ 81021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.128516 (+120616us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.249263 (+120747us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b54000889d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:39.249428 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185288576}
I20260709 12:19:39.250532 31797 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.165293 31363 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.251564 27584 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:19:39.214861 31764 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.177686 31745 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:39.177573 31427 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.255630 31799 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.256783 31427 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.153909 31837 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:39.257452 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.257979 27655 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:39.258195 27734 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:19:39.179123 31831 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:39.180285 31729 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.181452 28334 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:39.182651 31647 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:39.133879 31861 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:19:39.189270 27731 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.262939 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:39.264189 31861 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:39.264554 31861 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.264896 31861 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:39.189880 31839 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:39.134153 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96684.2us
I20260709 12:19:39.188016 27583 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:39.268083 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:39.268751 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:39.194641 27589 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:39.197286 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:39.186254 31655 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.198202 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.118986 31772 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.196183 31775 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:39.164186 31802 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:39.145175 31765 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.212181 31685 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.196573 31860 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:39.295023 27662 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.196684 31854 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:39.214803 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.184538 31735 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:39.216243 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.216533 31808 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.217671 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2255) took 2005 ms. Trace:
I20260709 12:19:39.225232 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.225769 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:39.226600 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.228732 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:39.302678 27729 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:39.300884 27582 rpcz_store.cc:276] 0709 12:19:37.212536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.871350 (+1658814us) service_pool.cc:224] Handling call
0709 12:19:38.871773 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:38.872227 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:38.872272 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:38.872863 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 12:19:39.090370 (+217507us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:39.090409 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.090473 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:39.217450 (+126977us) raft_consensus.cc:1685] finished
0709 12:19:39.217472 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.217608 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2669,"raft.queue_time_us":910,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":7256,"thread_start_us":193279,"threads_started":1,"wal-append.queue_time_us":270513}
I20260709 12:19:39.304826 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.305972 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.306933 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.242120 31757 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:39.123811 27660 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:19:39.275305 31810 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:39.269158 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:39.314314 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85650.5us
W20260709 12:19:39.314683 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45583us
I20260709 12:19:39.316039 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.316176 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:39.317119 27729 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:39.270684 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 73448.9us
I20260709 12:19:39.272361 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:39.274003 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:39.321208 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48892.6us
I20260709 12:19:39.321847 27733 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:39.278096 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2262) took 1429 ms. Trace:
I20260709 12:19:39.280992 27580 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:19:39.280630 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.281772 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2250) took 1208 ms. Trace:
I20260709 12:19:39.169958 31251 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.281874 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.212147 27737 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:39.333689 27730 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:39.334869 27737 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:19:39.287609 27736 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:39.290697 31745 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:39.292155 31685 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.289880 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:39.274763 31427 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:39.299189 27731 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:39.231446 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:39.377987 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.378816 27733 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:39.379771 27731 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:39.248245 31795 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:39.309355 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:39.382730 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92925.6us
I20260709 12:19:39.323146 27584 rpcz_store.cc:276] 0709 12:19:37.848303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.134620 (+1286317us) service_pool.cc:224] Handling call
0709 12:19:39.134934 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:39.251943 (+117009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.270282 (+ 18339us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001fead4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:39.278025 (+  7743us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267695488}
I20260709 12:19:39.323562 27589 rpcz_store.cc:276] 0709 12:19:38.073437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.269797 (+1196360us) service_pool.cc:224] Handling call
0709 12:19:39.281325 (+ 11528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.281481 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.281503 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.281540 (+    37us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:39.281568 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.281615 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.281723 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.323629 27583 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:39.328233 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.329017 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.385777 27733 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:39.329636 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:39.387028 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57508us
I20260709 12:19:39.387712 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.385859 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.388744 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:39.388952 27589 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.331405 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.332654 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.289862 31797 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:19:39.318025 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.402685 27729 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:39.336588 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:39.403898 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:39.404639 27582 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:39.337970 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2116) took 1032 ms. Trace:
I20260709 12:19:39.404954 27737 rpcz_store.cc:276] 0709 12:19:38.306046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.334178 (+1028132us) service_pool.cc:224] Handling call
0709 12:19:39.334580 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:39.337641 (+  3061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.337900 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.268086 31755 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.359961 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2270) took 1078 ms. Trace:
I20260709 12:19:39.413196 27589 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:39.412055 27588 rpcz_store.cc:276] 0709 12:19:38.281809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.276588 (+994779us) service_pool.cc:224] Handling call
0709 12:19:39.280670 (+  4082us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.280903 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:39.356192 (+ 75289us) log.cc:844] Serialized 14 byte log entry
0709 12:19:39.359521 (+  3329us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.359589 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:39.359651 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.359738 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.359888 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18351,"raft.queue_time_us":1505,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"thread_start_us":25364,"threads_started":2}
I20260709 12:19:39.417774 27659 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:39.349413 31615 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:39.348227 31765 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:39.365051 31613 raft_consensus.cc:993] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:19:39.365696 27732 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: []
W20260709 12:19:39.308562 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82881us
I20260709 12:19:39.294971 31775 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:39.381634 27731 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:39.312388 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:39.412731 31797 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.384840 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.463001 27661 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:39.397634 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.304121 31854 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:39.401767 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.407002 27657 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:39.411496 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2271) took 1129 ms. Trace:
I20260709 12:19:39.333925 31602 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:39.295199 31837 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.089s
I20260709 12:19:39.285051 31772 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.348132 31794 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:39.413385 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.360474 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:39.349653 31798 raft_consensus.cc:2764] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:19:39.363000 27736 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:39.497722 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2118) took 1076 ms. Trace:
I20260709 12:19:39.498171 27736 rpcz_store.cc:276] 0709 12:19:38.421792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.362205 (+940413us) service_pool.cc:224] Handling call
0709 12:19:39.362551 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.497432 (+134881us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.497655 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.382529 31427 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.440079 27662 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:39.401265 31559 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:39.449400 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 634) took 1147 ms. Trace:
I20260709 12:19:39.412585 31759 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:39.452006 27730 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:39.454898 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.455531 27738 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:39.392865 27585 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:39.455606 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 143299us
I20260709 12:19:39.507802 27588 rpcz_store.cc:276] 0709 12:19:38.302501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.416597 (+1114096us) service_pool.cc:224] Handling call
0709 12:19:39.448774 (+ 32177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.448991 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.449027 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.449081 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:39.449135 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.449203 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.449332 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.456743 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.376426 31417 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:19:39.458889 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.458827 27580 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
W20260709 12:19:39.465233 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.421898 31780 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:39.438241 31872 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.320s	user 0.001s	sys 0.000s
I20260709 12:19:39.471472 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.432750 31795 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:39.433290 31615 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.476753 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.478351 27582 rpcz_store.cc:276] 0709 12:19:38.282050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.410726 (+1128676us) service_pool.cc:224] Handling call
0709 12:19:39.411017 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.411185 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.411208 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.411248 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:39.411276 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.411348 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.411445 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.484180 31733 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:39.486013 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88470.3us
I20260709 12:19:39.457893 31835 raft_consensus.cc:1081] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:19:39.492947 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:39.493988 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.561548 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:39.468740 31797 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:39.428246 31685 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:19:39.472862 31823 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:39.510578 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 279205us
I20260709 12:19:39.516885 27729 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:39.521116 27734 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
W20260709 12:19:39.527449 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.480631 31854 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:39.531816 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2266) took 1655 ms. Trace:
I20260709 12:19:39.571344 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.481163 31602 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.454438 31757 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.585216 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.586238 28329 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:39.586956 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:39.549786 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.551878 27737 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:19:39.495623 31765 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:39.559593 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82923.3us
I20260709 12:19:39.560988 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2265) took 1695 ms. Trace:
I20260709 12:19:39.454771 31764 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:39.564083 27730 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:39.505771 31427 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:39.589600 27581 rpcz_store.cc:276] 0709 12:19:37.865265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.245005 (+1379740us) service_pool.cc:224] Handling call
0709 12:19:39.245405 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.245661 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:39.452955 (+207294us) log.cc:844] Serialized 14 byte log entry
0709 12:19:39.560643 (+107688us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.560694 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:39.560752 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.560817 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.560935 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17221,"prepare.queue_time_us":222032,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":8076,"raft.queue_time_us":4583,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":9881,"thread_start_us":314084,"threads_started":2}
I20260709 12:19:39.560863 31615 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.551478 31559 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:39.575088 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:39.575479 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.607079 27661 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:39.608428 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.609776 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.579201 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:39.532100 31794 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:39.539011 27731 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:19:39.619560 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2276) took 1139 ms. Trace:
I20260709 12:19:39.619891 27731 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:39.589841 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.566092 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.537005 31846 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.597350 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.631729 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.600252 27588 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:39.633101 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2274) took 1200 ms. Trace:
I20260709 12:19:39.633585 27588 rpcz_store.cc:276] 0709 12:19:38.432839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.558868 (+1126029us) service_pool.cc:224] Handling call
0709 12:19:39.580670 (+ 21802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.632769 (+ 52099us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.633034 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.518333 31527 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.619973 27581 rpcz_store.cc:276] 0709 12:19:38.480311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.592964 (+1112653us) service_pool.cc:224] Handling call
0709 12:19:39.619059 (+ 26095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.619220 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.619242 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.619291 (+    49us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:39.619374 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.619424 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.619510 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.596098 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:39.621362 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59903.5us
W20260709 12:19:39.621814 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88384.7us
I20260709 12:19:39.635584 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.636529 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:39.581358 31602 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:39.572736 31795 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:39.659456 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.582453 31740 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:39.591056 31733 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:39.663035 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.663156 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.663570 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.673661 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:39.674677 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84912.3us
I20260709 12:19:39.674042 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:39.676632 31676 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:39.676657 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:39.677409 31676 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:19:39.677567 31811 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:39.677906 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.678246 31811 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:39.678809 31811 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:39.677865 31676 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:39.679749 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.681401 31676 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.683524 31891 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:39.684461 31891 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:39.664117 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:39.662623 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2254) took 1067 ms. Trace:
I20260709 12:19:39.666849 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2253) took 1176 ms. Trace:
I20260709 12:19:39.598263 31823 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:39.689459 27581 rpcz_store.cc:276] 0709 12:19:38.595750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.635073 (+1039323us) service_pool.cc:224] Handling call
0709 12:19:39.662109 (+ 27036us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.662313 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.662337 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.662381 (+    44us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:39.662415 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.662466 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.662571 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.690223 31823 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.691347 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2255) took 1095 ms. Trace:
I20260709 12:19:39.692152 27581 rpcz_store.cc:276] 0709 12:19:38.596108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.690553 (+1094445us) service_pool.cc:224] Handling call
0709 12:19:39.690855 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.691012 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.691045 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.691084 (+    39us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:39.691116 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.691167 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.691270 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.608291 31253 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:39.684393 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:39.671285 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1892) took 1562 ms. Trace:
I20260709 12:19:39.694327 27732 rpcz_store.cc:276] 0709 12:19:38.108787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.988678 (+879891us) service_pool.cc:224] Handling call
0709 12:19:39.365456 (+376778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.440875 (+ 75419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.671088 (+230213us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54002e7cd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:39.671228 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":865698688}
I20260709 12:19:39.604009 31736 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.693967 27581 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:39.694528 27588 rpcz_store.cc:276] 0709 12:19:38.490245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.634506 (+1144261us) service_pool.cc:224] Handling call
0709 12:19:39.666146 (+ 31640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.666399 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:39.666436 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.666492 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:39.666555 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.666629 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.666774 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.696316 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2256) took 1100 ms. Trace:
I20260709 12:19:39.696769 27581 rpcz_store.cc:276] 0709 12:19:38.596409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.693514 (+1097105us) service_pool.cc:224] Handling call
0709 12:19:39.693802 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.696046 (+  2244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.696252 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.697572 31253 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:39.698081 31253 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.698510 31253 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.703682 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:39.717031 27661 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:39.625589 31685 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:39.620123 31759 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:39.725404 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 129426us
I20260709 12:19:39.599385 31765 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.668430 31417 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:39.726241 27731 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:39.577453 31775 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.734468 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:39.767879 27588 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:39.801258 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2280) took 1029 ms. Trace:
I20260709 12:19:39.801708 27588 rpcz_store.cc:276] 0709 12:19:38.771896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.734812 (+962916us) service_pool.cc:224] Handling call
0709 12:19:39.767648 (+ 32836us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.800964 (+ 33316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.801191 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:39.736937 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.729913 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.674098 31808 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:39.724583 31767 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:39.674407 31835 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.142s
W20260709 12:19:39.815181 28330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 634) took 1396 ms (client timeout 1777 ms). Trace:
W20260709 12:19:39.816285 28330 rpcz_store.cc:269] 0709 12:19:38.418474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.419260 (+   786us) service_pool.cc:224] Handling call
0709 12:19:39.804063 (+1384803us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54002fab54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:39.815088 (+ 11025us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2059753728}
I20260709 12:19:39.803517 27664 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [9.7-10.8]
I20260709 12:19:39.818045 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:39.660948 31829 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:39.803025 31861 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.538s	user 0.007s	sys 0.004s
I20260709 12:19:39.730931 31810 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:39.730728 31685 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:39.730804 31799 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:39.737663 31251 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.725313 31740 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.730568 31887 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:39.839681 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.840462 27588 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:19:39.841698 31811 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.842574 27583 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
W20260709 12:19:39.859086 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.843681 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:39.862318 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 188316us
I20260709 12:19:39.864337 27583 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:39.739297 31759 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.866591 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2282) took 1022 ms. Trace:
I20260709 12:19:39.823221 31755 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:39.867004 27583 rpcz_store.cc:276] 0709 12:19:38.843943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.862929 (+1018986us) service_pool.cc:224] Handling call
0709 12:19:39.863238 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.866346 (+  3108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.866531 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:39.870271 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 141370us
I20260709 12:19:39.838511 31733 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:39.872484 27583 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:19:39.869990 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:39.874217 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.877467 28337 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:39.844709 31793 raft_consensus.cc:2764] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:19:39.884826 27588 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:39.887774 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.894243 27734 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:19:39.868662 31735 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.838184 31861 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.873155 31772 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.903834 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:39.907164 27588 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:39.834410 31795 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.892074 31810 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:39.907764 31808 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:19:39.823005 31794 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.834033 31559 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.892414 31835 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:19:39.937029 27583 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:39.931377 28334 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:39.902863 31894 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:19:39.942041 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 124084us
W20260709 12:19:39.942878 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.018s	sys 0.007s
W20260709 12:19:39.944587 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.018s	sys 0.007s
W20260709 12:19:39.945036 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 552503us
I20260709 12:19:39.898433 31676 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:19:39.948138 27662 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:39.949885 27662 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:36354
I20260709 12:19:39.950452 27662 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:39.950757 27662 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:39.956081 27662 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:39.958161 27662 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:39.873349 31685 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:39.963513 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45070.8us
I20260709 12:19:39.936240 31772 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:39.937976 31251 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:39.967890 31890 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:19:39.968232 31829 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:39.971427 31878 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.382s	user 0.000s	sys 0.001s
I20260709 12:19:39.972157 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:39.972529 31802 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.672s	user 0.011s	sys 0.000s
I20260709 12:19:39.946235 31253 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:39.946230 31754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.005s	sys 0.025s
I20260709 12:19:39.959299 31808 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:39.973016 31896 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
W20260709 12:19:39.977562 31899 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
W20260709 12:19:39.978050 31879 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.313s	user 0.001s	sys 0.000s
I20260709 12:19:39.983551 31894 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:39.986464 31908 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:39.930419 31888 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:19:39.946434 31885 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
I20260709 12:19:40.004897 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:40.006500 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:39.965945 31846 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.035588 31802 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.049672 31685 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.050494 31802 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.004217 31887 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:40.051632 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:40.051473 31887 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.054311 31887 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:40.055193 31902 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:19:40.056051 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:40.056694 27582 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:40.041836 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.042680 31837 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:19:40.043114 31886 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.330s	user 0.001s	sys 0.000s
W20260709 12:19:40.043475 31870 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:19:40.045205 31861 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:40.045949 31810 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.060472 31908 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:40.060814 31772 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:19:40.035807 31754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.005s	sys 0.025s
I20260709 12:19:40.058665 31829 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:19:40.061617 31527 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.064517 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.054335 31804 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:40.074267 31804 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:40.067624 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.075114 27582 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:40.075357 31804 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.076748 31804 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:40.077272 31891 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.078081 31891 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:40.086934 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:40.040447 31765 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:40.069301 31908 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:40.070185 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 396557us
W20260709 12:19:40.108136 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.108307 27552 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8.
I20260709 12:19:40.109599 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.072206 27588 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:19:40.090216 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 104826us
I20260709 12:19:40.062913 31797 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.094216 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.110767 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.106038 27582 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:40.096758 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:40.112537 28337 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:40.112967 28337 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:40.113150 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.113824 28337 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.117393 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.120435 27582 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:19:40.120745 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.087632 31891 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:40.122900 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.005s	sys 0.030s
W20260709 12:19:40.123307 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.006s	sys 0.030s
I20260709 12:19:40.073141 31811 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.123490 31901 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:40.124083 31901 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:40.123636 31891 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.124495 31901 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:40.120330 31895 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.113047 31759 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.127696 31908 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:40.128634 31427 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.000s
I20260709 12:19:40.129005 31908 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:40.129083 31427 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.131207 31733 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:40.132807 31427 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.133437 31427 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:40.089254 31740 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.135831 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.136701 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.137023 27589 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:19:40.137226 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.138443 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2273) took 1855 ms. Trace:
I20260709 12:19:40.139119 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.139714 27582 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:40.141393 31758 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.907s	user 0.009s	sys 0.005s
I20260709 12:19:40.141916 31908 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:40.142354 31908 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.070775 31837 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:40.138475 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.144057 31837 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.145354 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.057328 31894 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.147758 27586 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:40.147856 27582 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:40.148665 31797 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.150094 31894 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.150848 31894 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:40.150890 31759 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.151522 31799 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.152017 31795 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.152539 31772 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:40.159509 31772 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:40.161361 31655 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.872s	user 0.010s	sys 0.000s
I20260709 12:19:40.118073 31735 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:40.161868 31655 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.163348 28330 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:40.115271 31417 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:19:40.065376 31808 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 12:19:40.170863 31735 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.138860 27589 rpcz_store.cc:276] 0709 12:19:38.282535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.414303 (+1131768us) service_pool.cc:224] Handling call
0709 12:19:39.439683 (+ 25380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:40.137466 (+697783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:40.138213 (+   747us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b540006f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:40.138385 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1603038080}
I20260709 12:19:40.161970 31772 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.173028 31829 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:40.173619 31829 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.156874 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.175678 31895 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:40.177497 27584 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:40.124953 31891 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.185122 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:40.143216 31908 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:39.571712 27587 rpcz_store.cc:276] 0709 12:19:37.876397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.251073 (+1374676us) service_pool.cc:224] Handling call
0709 12:19:39.280058 (+ 28985us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.280271 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:19:39.350694 (+ 70423us) log.cc:844] Serialized 14 byte log entry
0709 12:19:39.531481 (+180787us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:39.531529 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:19:39.531574 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:39.531637 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:39.531756 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":250388,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":1428,"spinlock_wait_cycles":18344832,"thread_start_us":231557,"threads_started":2}
I20260709 12:19:40.190120 31655 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:40.183806 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 178987us
I20260709 12:19:40.175725 31504 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:40.196146 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:40.174741 31829 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.199136 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.231774 27589 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:40.231133 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:40.234874 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.179051 31894 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.232761 31895 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:40.234953 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 113353us
I20260709 12:19:40.255429 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:40.242074 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.252688 27587 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:19:40.253515 27582 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260709 12:19:40.236159 31615 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.616s	user 0.011s	sys 0.000s
W20260709 12:19:40.261555 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.261343 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:40.262013 27732 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:40.256438 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.264318 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.264758 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.264894 27729 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:40.265460 27736 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:40.266350 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.267098 27732 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:40.267343 27731 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:40.269086 31775 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:40.245850 31895 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.268810 31615 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.270134 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:40.271180 27585 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:19:40.271721 31895 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.272225 31615 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.248944 31676 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:19:40.276602 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.278767 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.279538 27731 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:40.281490 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.276221 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.282390 27659 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:40.282269 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.282609 27736 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:40.284318 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.285075 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.285652 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.286183 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.286731 27663 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:19:40.287896 27661 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:40.291059 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.291388 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.292145 27729 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:40.290449 27731 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:19:40.292927 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.293614 27730 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:40.228669 31504 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:40.254668 31854 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.648s	user 0.001s	sys 0.010s
I20260709 12:19:40.287874 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:40.249776 31602 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.002s
I20260709 12:19:40.296689 27729 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:40.298067 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.300211 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.251940 31795 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:40.296981 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.302485 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:40.295378 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:40.307880 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.308343 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.308689 28329 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:40.245189 31894 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.308753 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.303514 27733 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:40.295548 27730 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:40.304992 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.307572 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:40.310665 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.241349 31780 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:40.305822 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:40.275624 31251 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:40.312137 27663 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:40.312435 27661 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:40.281627 31735 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.314339 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:40.316222 27662 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:40.249955 31867 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.072s
I20260709 12:19:40.297614 31895 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.321817 27659 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:19:40.328140 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.325732 27664 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:40.326519 27661 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:40.324776 27658 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:19:40.294622 31775 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:40.331372 27662 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:40.331970 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.260926 31912 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:19:40.332571 27734 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.333433 27663 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:19:40.315490 31915 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:19:40.335242 31923 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:40.309355 31795 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:40.332405 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:40.294345 31913 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.343559 31835 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:40.305624 31918 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:19:40.337113 31775 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.345753 31735 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 12:19:40.358640 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.361161 31903 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:40.369912 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.306183 31759 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:19:40.385392 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.305389 31794 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:19:40.393222 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:40.402343 27661 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:40.404114 31894 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:40.404464 31798 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:40.404958 31798 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:40.404762 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.405602 31251 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:19:40.405982 31798 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.358181 31923 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:40.394990 31775 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.391881 31753 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:40.407114 31923 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:40.407382 31753 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:40.407601 31912 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.408444 31912 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:40.408475 31753 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.397663 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:40.309017 31602 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.372531 31795 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:40.306488 31854 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.404652 31894 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.428835 27734 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:40.432257 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.407133 31798 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:40.392333 31417 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:40.413523 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:40.439038 27656 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:40.384868 31780 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:40.421562 31775 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:19:40.434511 31753 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.436616 28332 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:40.417613 31913 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.437727 31417 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:40.358017 31901 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.444649 31839 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:40.447909 31923 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:40.456168 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44882.6us
I20260709 12:19:40.457022 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.430753 31733 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:19:40.458071 31417 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.464349 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 613) took 1144 ms. Trace:
I20260709 12:19:40.464542 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:40.465427 27734 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:19:40.464898 27655 rpcz_store.cc:276] 0709 12:19:39.320372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.320695 (+   323us) service_pool.cc:224] Handling call
0709 12:19:40.435033 (+1114338us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:40.435273 (+   240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:40.435517 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:19:40.435833 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 12:19:40.462945 (+ 27112us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:40.463381 (+   436us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:40.463428 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:40.463498 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:40.464282 (+   784us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4209,"prepare.queue_time_us":1837,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":4459,"spinlock_wait_cycles":6417152,"thread_start_us":22492,"threads_started":1}
I20260709 12:19:40.437302 31849 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:40.410195 31504 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.453634 31808 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:40.469496 31901 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:40.469705 31916 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:19:40.453930 31363 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:19:40.472126 31602 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.451289 31799 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:19:40.476243 31780 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.476872 31685 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:40.477250 31928 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:40.478050 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.478406 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.518573 28335 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:40.498821 31795 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.482443 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:40.468777 31914 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:19:40.520780 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:40.521865 31775 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:40.493880 27655 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
W20260709 12:19:40.496563 31917 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.302s	user 0.000s	sys 0.004s
I20260709 12:19:40.496714 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.519018 31602 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.537782 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.538550 27589 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:40.541651 27583 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [12.11-13.12]
I20260709 12:19:40.541857 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:40.505937 31504 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:40.544427 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.544970 27658 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:40.522493 31775 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.501101 31808 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:40.548480 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 178675us
I20260709 12:19:40.520841 31795 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.529194 31427 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
W20260709 12:19:40.559278 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 645) took 1364 ms (client timeout 1501 ms). Trace:
I20260709 12:19:40.559567 27588 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:19:40.559827 27586 rpcz_store.cc:269] 0709 12:19:39.194495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.214450 (+ 19955us) service_pool.cc:224] Handling call
0709 12:19:40.559059 (+1344609us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:40.559219 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1957279616}
I20260709 12:19:40.561295 31729 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Committing config change with OpId 11.9: config changed from index 4 to 9, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:40.541052 31794 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:19:40.563851 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1909) took 1485 ms. Trace:
I20260709 12:19:40.564287 27738 rpcz_store.cc:276] 0709 12:19:39.078430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.159655 (+ 81225us) service_pool.cc:224] Handling call
0709 12:19:40.563643 (+1403988us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54002e81d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:40.563794 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":612860800}
I20260709 12:19:40.564814 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.565613 27736 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:40.565555 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.566843 27736 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:40.567384 27735 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:40.547179 31854 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:40.556144 31602 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.571858 31775 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.577499 31930 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:40.578193 31930 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:40.578660 31930 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:40.560040 31892 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 12:19:40.602053 27658 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:40.608271 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:40.608661 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.611258 27661 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:40.612504 27583 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:40.615386 27586 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:40.623045 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.623270 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.623788 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.624513 27659 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:40.509990 31894 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.633993 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.563822 31794 consensus_queue.cc:1243] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:19:40.641321 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 648) took 1394 ms. Trace:
I20260709 12:19:40.647177 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:40.626215 31934 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:19:40.650928 31363 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:40.651409 31363 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.651839 31527 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.652295 31363 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.653645 31363 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:40.653678 31923 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.641968 28328 rpcz_store.cc:276] 0709 12:19:39.246977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.251446 (+  4469us) service_pool.cc:224] Handling call
0709 12:19:39.251773 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:40.052075 (+800302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:40.640972 (+588897us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:40.641235 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1838965504}
I20260709 12:19:40.553812 31849 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:19:40.657935 27738 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:40.627040 31895 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:40.678843 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.685525 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.689211 31799 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:40.690291 31915 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:40.690754 31915 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.692322 31772 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:40.692798 31772 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.693857 31772 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.697099 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 635) took 2315 ms. Trace:
I20260709 12:19:40.697615 27584 rpcz_store.cc:276] 0709 12:19:38.381230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.480754 (+1099524us) service_pool.cc:224] Handling call
0709 12:19:39.481055 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:40.178042 (+696987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:40.696833 (+518791us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:40.697034 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1600986880}
I20260709 12:19:40.657153 31602 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:19:40.701956 31602 consensus_queue.cc:1243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:19:40.700516 31918 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:19:40.703805 31891 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.000s
I20260709 12:19:40.701478 27583 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:40.705139 31915 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.643899 31615 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.616246 31417 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.706640 31915 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:40.714294 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.724654 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.728633 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.712826 31772 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:40.730240 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:40.721580 31883 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.984s	user 0.009s	sys 0.032s
I20260709 12:19:40.714731 31559 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.686802 31811 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:19:40.737130 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.742367 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1856) took 1410 ms. Trace:
I20260709 12:19:40.742839 27657 rpcz_store.cc:276] 0709 12:19:39.332282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.332406 (+   124us) service_pool.cc:224] Handling call
0709 12:19:40.741874 (+1409468us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:40.742292 (+   418us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37087616}
I20260709 12:19:40.728965 31615 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:40.737927 31883 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.009s	sys 0.032s
W20260709 12:19:40.765574 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.687405 31808 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.751667 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:40.769977 27585 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:19:40.772102 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.747087 31765 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.773870 31698 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:40.774355 31698 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:40.774703 31698 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:40.776460 31931 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.346s	user 0.005s	sys 0.000s
I20260709 12:19:40.783565 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2240) took 1335 ms. Trace:
I20260709 12:19:40.784026 27660 rpcz_store.cc:276] 0709 12:19:39.447954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.672997 (+225043us) service_pool.cc:224] Handling call
0709 12:19:40.783363 (+1110366us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b5400211454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:40.783502 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147494144}
I20260709 12:19:40.786850 27587 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:40.792965 27657 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:40.798684 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:40.799690 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:40.802291 31937 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:19:40.795418 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.754094 31837 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:19:40.750165 31909 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.798030 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:40.809131 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:40.809719 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.810105 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.811829 27660 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:40.812330 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.767216 31918 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:40.752519 31831 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 12:19:40.815934 27658 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Committing config change with OpId 11.9: config changed from index 4 to 9, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:40.752581 31888 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:19:40.805907 31615 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.830468 31908 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:40.834194 31846 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:19:40.834622 31846 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:40.842172 31930 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.815397 31754 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:40.818612 31795 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:19:40.846161 31888 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:40.846575 31888 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:40.823369 31837 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:40.847770 31837 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:40.848148 31837 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:40.823846 31895 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.850114 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.850715 27660 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.823809 31909 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:19:40.855680 31802 consensus_queue.cc:1175] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 13 status { last_received { term: 12 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:19:40.856894 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 638) took 1510 ms. Trace:
I20260709 12:19:40.857319 28334 rpcz_store.cc:276] 0709 12:19:39.347034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.347235 (+   201us) service_pool.cc:224] Handling call
0709 12:19:39.406077 (+ 58842us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:39.941305 (+535228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:40.856606 (+915301us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b5400611054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:40.856821 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1207479168}
I20260709 12:19:40.860802 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.855825 31417 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.873868 27660 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:40.876672 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.879941 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.880053 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:40.883440 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.885667 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.887112 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.891062 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 615) took 1563 ms. Trace:
I20260709 12:19:40.891516 27656 rpcz_store.cc:276] 0709 12:19:39.327777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.327924 (+   147us) service_pool.cc:224] Handling call
0709 12:19:40.890862 (+1562938us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:40.890994 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2421527296}
I20260709 12:19:40.893399 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.896513 31854 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:40.896584 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.898686 31854 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:40.898917 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.892545 31559 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
W20260709 12:19:40.900424 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.900787 31559 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:40.900938 31892 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:40.901182 31559 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:40.901615 31892 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.896739 31915 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:40.902063 31559 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.900193 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.904126 31810 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:40.904271 31915 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.904560 31810 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:40.905510 31810 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.906510 31810 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:40.908476 31835 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:40.909129 31835 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:40.910073 31835 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.910465 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.913057 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:40.915040 27731 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:19:40.916244 27731 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:19:40.871557 31615 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:40.876317 31793 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:40.920071 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:40.878626 31906 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:40.924366 27736 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 10: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:40.926710 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:40.927060 27736 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:19:40.928194 27736 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:19:40.899533 31939 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:40.930104 31647 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:40.919095 31915 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.931938 31920 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:19:40.932301 31837 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.933283 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:40.925487 31793 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.925712 31811 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:19:40.936316 31793 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:40.936899 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:40.923873 31615 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.925949 31946 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:19:40.939718 31906 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.940773 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:40.941493 27658 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:40.942909 27733 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:40.945762 31906 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:40.876891 31888 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.922394 31804 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:40.903242 31559 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:40.919586 31912 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:40.948462 31930 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.970474 31920 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:19:40.884164 31846 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:40.947539 31903 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:40.976778 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:40.978025 27733 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:40.899096 31698 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:40.939182 31615 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:40.991658 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 179366us
I20260709 12:19:40.993249 27729 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:41.005841 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:41.006260 31615 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:41.006891 31906 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:41.006989 31615 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.009935 31615 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:41.015707 31831 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.016357 31831 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.018537 31831 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:40.956051 31793 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:41.035446 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.955413 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32912 (request call id 0) took 2823 ms. Trace:
I20260709 12:19:40.982753 31846 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:41.040167 27732 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:40.955731 31647 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
W20260709 12:19:41.051663 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 251992us
I20260709 12:19:40.993834 31804 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:19:41.031356 31849 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:19:41.072067 27589 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:19:41.074121 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.075551 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.046478 31939 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:41.078418 31739 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:19:41.095747 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.044135 31793 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.078588 31839 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:19:41.103444 31733 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.118387 31733 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.118353 31939 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.120932 31733 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:41.121570 31953 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:19:41.123158 31733 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:41.123803 31733 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.127063 31733 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.078773 31831 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
W20260709 12:19:41.132203 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.082789 31903 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:19:41.136752 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.079043 31906 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:19:41.135802 31903 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:41.203944 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:41.214650 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.217072 31793 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:41.219568 31504 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:41.220027 31504 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:41.220906 31504 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.222182 31504 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:41.223886 31839 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.224427 31839 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.226547 31839 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:41.226608 31908 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.228221 31906 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:41.227224 31908 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.228691 31906 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:41.227207 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.229727 27729 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:41.229800 31906 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.205628 31903 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:41.230515 31908 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.207129 31837 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:41.232717 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.236199 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:41.237412 31772 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.002s
W20260709 12:19:41.237855 31363 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.005s
I20260709 12:19:41.236792 31906 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:41.237401 27729 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:41.247215 27662 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:19:41.259224 27662 pending_rounds.cc:85] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:19:41.259586 27662 pending_rounds.cc:107] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:19:41.259804 27662 raft_consensus.cc:2889] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:41.237684 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132927us
I20260709 12:19:41.240690 31698 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:41.263943 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.251114 31837 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.233214 31903 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.267935 31831 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:41.280797 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.280977 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:41.281471 27661 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:41.281780 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.283622 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.284291 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:41.286135 27663 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: [13.6-13.6]
I20260709 12:19:41.283047 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.287549 27656 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:41.292979 31829 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:41.301417 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 157211us
W20260709 12:19:41.299122 31964 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:19:41.304576 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:41.305368 27660 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:41.305047 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.291100 31903 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:41.310348 27661 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:19:41.303205 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.039300 31729 rpcz_store.cc:276] 0709 12:19:38.122441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:38.122597 (+   156us) service_pool.cc:224] Handling call
0709 12:19:39.006058 (+883461us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b54004fb3a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c5520 00007f0e9a34eb42 00007f0e9a34d800 00007f0e9a3469fa 00007f0e9f30e4c9 00007f0e9888ac54 00007f0e9888abe2 00007f0e9888ab11 00007f0e9888aa8c 00007f0e98889d42
0709 12:19:39.142849 (+136791us) log.cc:844] Serialized 494 byte log entry
0709 12:19:40.135162 (+992313us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x7b18002f8300. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:40.927181 (+792019us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55390,"raft.queue_time_us":55917,"raft.run_cpu_time_us":948,"raft.run_wall_time_us":97557,"spinlock_wait_cycles":46849664,"thread_start_us":129152,"threads_started":1,"wal-append.queue_time_us":293878,"wal-append.run_cpu_time_us":2060,"wal-append.run_wall_time_us":1111212}
I20260709 12:19:41.274667 31837 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.276994 27588 raft_consensus.cc:2955] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Committing config change with OpId 11.9: config changed from index 4 to 9, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:41.312753 27738 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 12:19:41.315938 27738 pending_rounds.cc:85] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:19:41.316318 27738 pending_rounds.cc:107] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 12:19:41.316586 27738 raft_consensus.cc:2889] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:41.304410 31893 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:19:41.313489 31759 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.003s	sys 0.025s
I20260709 12:19:41.321797 31893 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:41.324810 31893 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:41.326713 31932 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.892s	user 0.003s	sys 0.003s
W20260709 12:19:41.334473 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:41.335124 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.353286 27736 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260709 12:19:41.348287 31961 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:19:41.349915 31735 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:41.357260 31950 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.426s	user 0.000s	sys 0.000s
W20260709 12:19:41.359543 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:41.377118 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.377140 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:41.361522 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.378094 27581 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:41.378805 31735 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.381444 31735 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.376456 31867 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:41.382150 31867 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:41.382623 31967 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.151s	user 0.004s	sys 0.000s
I20260709 12:19:41.383903 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:41.385391 31867 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.387573 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2112) took 3184 ms. Trace:
I20260709 12:19:41.387955 27735 rpcz_store.cc:276] 0709 12:19:38.202874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.223526 (+1020652us) service_pool.cc:224] Handling call
0709 12:19:40.567178 (+1343652us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:40.567844 (+   666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:41.387363 (+819519us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54002e81d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:41.387512 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3089061760}
I20260709 12:19:41.394882 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.396042 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.396679 27581 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:41.397333 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:41.397951 27581 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:41.398602 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:41.399077 27581 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:41.400794 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.401137 27586 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:41.401250 27580 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:41.402060 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:41.402796 31974 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:19:41.403596 31757 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:41.364636 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:41.405393 27655 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:41.370275 31972 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:19:41.371094 27736 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:19:41.418807 27736 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 6
I20260709 12:19:41.419137 27736 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:19:41.419386 27736 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:41.373348 27587 pending_rounds.cc:85] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 3
I20260709 12:19:41.420258 27587 pending_rounds.cc:107] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:19:41.420581 27587 raft_consensus.cc:2889] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:41.374639 27582 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:41.425899 27657 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:19:41.322340 31759 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.003s	sys 0.025s
I20260709 12:19:41.377938 31910 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.450074 27581 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:41.428166 31363 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.467885 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:41.472621 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100504us
I20260709 12:19:41.475308 27581 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:19:41.476545 31976 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:41.476540 31829 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:41.477715 31829 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:41.477320 31976 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.478087 31829 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:41.479099 31829 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.480048 31976 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:41.475430 31685 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:41.480396 31829 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:41.480798 31685 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:41.488907 31685 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:41.464237 31910 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.493261 31837 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:41.494340 31910 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:41.496346 31253 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:41.471467 31363 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.498996 31363 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.482524 31427 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:41.483896 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:41.481140 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:41.500564 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:41.504117 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.513190 27730 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:41.505260 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:41.507869 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.512558 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.511050 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:41.524309 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.528640 27730 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:19:41.530052 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.534418 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.535948 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.541030 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:41.543716 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 136963us
I20260709 12:19:41.547259 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.494340 31754 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:41.548928 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.517839 31759 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:41.550923 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.550951 31758 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:41.560032 27586 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:41.550076 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:41.549551 31808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.018s	sys 0.022s
I20260709 12:19:41.544996 31835 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:41.566608 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.575347 27583 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:41.551245 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:41.575433 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.576148 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.551841 31753 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:41.518863 31903 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:19:41.525701 31867 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:19:41.557847 31754 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:41.594913 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 83955.6us
W20260709 12:19:41.598845 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.549086 31432 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
W20260709 12:19:41.577668 31808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.019s	sys 0.023s
I20260709 12:19:41.564707 31759 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:41.603957 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52785.9us
I20260709 12:19:41.604597 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:41.547917 31892 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:19:41.616250 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:41.684495 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68347.6us
I20260709 12:19:41.593433 31753 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:41.596289 31837 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:41.676751 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72265.3us
I20260709 12:19:41.699959 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.602483 31920 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:41.696636 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:41.683629 31944 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:19:41.618906 31797 raft_consensus.cc:2764] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 12:19:41.614147 31758 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:41.710677 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.613893 31839 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:41.713718 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:41.684312 31759 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:41.597689 31918 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 12:19:41.687970 31765 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:41.675896 31775 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.613090 31798 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.103s
I20260709 12:19:41.744241 27586 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:41.694815 31835 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:41.698575 31753 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:41.761112 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:41.774714 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.715541 31735 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:41.675629 31957 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.144s
I20260709 12:19:41.742540 31901 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:19:41.786974 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 78664.2us
I20260709 12:19:41.695194 31970 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:41.817195 28337 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:41.832732 31929 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:19:41.833098 31888 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:41.838037 31893 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:41.833223 31929 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:41.839988 31898 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.840528 31898 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.842602 31898 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.850297 27659 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:41.852413 31739 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.853379 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:41.853165 31739 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.855190 31793 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.855748 31793 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.857554 27659 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:41.857810 31793 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.859490 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:41.863494 31898 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:41.865855 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:41.867038 31898 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:41.863670 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:41.878993 31739 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:41.889807 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 1932) took 1029 ms. Trace:
I20260709 12:19:41.890439 27732 rpcz_store.cc:276] 0709 12:19:40.860288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:40.860563 (+   275us) service_pool.cc:224] Handling call
0709 12:19:41.889536 (+1028973us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54002a0654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:41.889722 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":410574208}
W20260709 12:19:41.891940 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:41.740391 31795 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:19:41.765884 31835 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:41.762826 31797 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:41.911314 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:41.912093 28334 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:41.915262 28330 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:19:41.918860 31758 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.904855 27472 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 81e6b2dabf8f4e6cb549ca250430e568 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:19:41.922593 31981 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.225s	user 0.000s	sys 0.000s
I20260709 12:19:41.786986 31906 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:41.753890 31839 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:41.914013 31797 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:41.934514 31797 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:41.908632 31901 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:41.943975 31980 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 12:19:41.790052 31962 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.145s
I20260709 12:19:41.929392 31957 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.902377 31615 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:41.955205 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:41.948915 31983 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 12:19:41.959702 31901 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.959915 31797 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.961779 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:41.962100 31797 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:41.912046 31765 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:41.930644 31941 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:19:41.903856 31831 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 12:19:41.929715 31936 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.779227 31775 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:41.978747 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 265112us
I20260709 12:19:41.925575 31920 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:41.977305 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.987950 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:41.966032 28337 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:41.996088 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:42.138466 28337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 643) took 2028 ms (client timeout 2058 ms). Trace:
W20260709 12:19:41.965040 31986 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
W20260709 12:19:42.138815 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 142824us
I20260709 12:19:42.170603 28329 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:42.184126 31808 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:42.168251 27733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.011s	sys 0.045s
W20260709 12:19:42.185261 27733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.011s	sys 0.046s
I20260709 12:19:42.184939 31808 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.951927 31615 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:41.975075 31985 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:19:42.170920 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:42.187433 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 210252us
W20260709 12:19:42.139250 28337 rpcz_store.cc:269] 0709 12:19:40.110041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:40.110507 (+   466us) service_pool.cc:224] Handling call
0709 12:19:40.850715 (+740208us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b18002c1360. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a383646 00007f0e9a2bd58e 00007f0e9a33ef9e 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c
0709 12:19:41.816700 (+965985us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b58006749d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c
0709 12:19:42.138408 (+321708us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3076440064}
I20260709 12:19:41.973237 31912 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.194842 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:41.944541 31835 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.169144 31797 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.135123 31559 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.134450 31860 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.186977 31504 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.195874 31867 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:42.189397 31793 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:19:42.171582 31939 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.169248 31733 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:42.170949 31910 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:42.195797 31795 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:42.206534 31987 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.282s	user 0.000s	sys 0.000s
I20260709 12:19:41.967573 31753 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:41.972852 31957 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.201910 31970 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:42.169694 31936 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:42.254678 31986 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:19:42.218617 31797 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.270135 31795 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.268812 31910 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.186964 31920 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.280398 31797 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.280839 31910 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.281883 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2170) took 1336 ms. Trace:
I20260709 12:19:42.282675 31795 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.250154 31860 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.200656 31912 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.282272 27734 rpcz_store.cc:276] 0709 12:19:40.945146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.266827 (+321681us) service_pool.cc:224] Handling call
0709 12:19:41.267259 (+   432us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:41.268329 (+  1070us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:41.808436 (+540107us) spinlock_profiling.cc:243] Waited 936 us on lock 0x7b1800031000. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:19:41.808838 (+   402us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.196490 (+387652us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.196681 (+   191us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:42.252440 (+ 55759us) log.cc:844] Serialized 52 byte log entry
0709 12:19:42.252668 (+   228us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:42.252940 (+   272us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.253130 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.261018 (+  7888us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:42.281674 (+ 20656us) raft_consensus.cc:1685] finished
0709 12:19:42.281694 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.281824 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33024,"prepare.queue_time_us":542786,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":63613,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":17177,"spinlock_wait_cycles":17964032,"thread_start_us":351782,"threads_started":1,"wal-append.queue_time_us":467155}
I20260709 12:19:42.287696 31795 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.288259 31795 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.290424 31795 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.290925 31795 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.291435 31795 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.292639 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:42.293385 31867 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:42.293633 31795 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.296746 31929 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.275241 31997 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:19:42.298317 31929 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:42.260676 31962 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:42.294673 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:42.300478 31997 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:42.300856 31417 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:42.300988 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:42.301667 28328 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:42.301963 31997 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:42.265868 31765 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.266347 31939 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.261861 31892 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.308355 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2171) took 1362 ms. Trace:
I20260709 12:19:42.256183 31504 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.297230 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 631) took 1309 ms. Trace:
I20260709 12:19:42.275885 31615 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.301290 31417 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:42.308732 27729 rpcz_store.cc:276] 0709 12:19:40.945458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.272365 (+326907us) service_pool.cc:224] Handling call
0709 12:19:41.273405 (+  1040us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:41.560265 (+286860us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:41.560309 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:41.560802 (+   493us) log.cc:844] Serialized 26 byte log entry
0709 12:19:41.781931 (+221129us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:41.781971 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.307984 (+526013us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b54002e77d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:42.308032 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:42.308080 (+    48us) raft_consensus.cc:1685] finished
0709 12:19:42.308093 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.308299 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":226001,"raft.queue_time_us":220050,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":23403,"spinlock_wait_cycles":154483456,"thread_start_us":186010,"threads_started":1,"wal-append.queue_time_us":619678}
I20260709 12:19:42.268774 31835 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:42.277563 31753 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:42.291193 31757 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.887s	user 0.007s	sys 0.001s
I20260709 12:19:42.313190 27733 rpcz_store.cc:276] 0709 12:19:40.988197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.302219 (+314022us) service_pool.cc:224] Handling call
0709 12:19:41.302610 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:41.559201 (+256591us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:41.559245 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:41.559810 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:19:42.185658 (+625848us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:19:42.185703 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.185770 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:42.297009 (+111239us) raft_consensus.cc:1685] finished
0709 12:19:42.297029 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.297170 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":268239,"raft.queue_time_us":257074,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":27815,"thread_start_us":606953,"threads_started":1,"wal-append.queue_time_us":735579}
W20260709 12:19:42.312182 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 117414us
I20260709 12:19:42.316502 31910 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:42.322474 31984 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
W20260709 12:19:42.323899 31990 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.339s	user 0.001s	sys 0.000s
I20260709 12:19:42.322104 31910 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.325994 31910 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:42.326893 31759 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.564s	user 0.002s	sys 0.005s
I20260709 12:19:42.333873 31775 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.327129 31901 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.327740 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:42.335433 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:42.328259 31829 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.819s	user 0.009s	sys 0.000s
I20260709 12:19:42.328631 31970 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.328765 31910 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:42.329602 31685 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.840s	user 0.007s	sys 0.003s
I20260709 12:19:42.347635 31685 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.307600 31892 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.349728 31685 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:42.329942 31754 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.654s	user 0.007s	sys 0.003s
I20260709 12:19:42.350559 31754 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.351733 31901 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.352715 31754 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.332736 31957 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.286906 31912 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.354056 31901 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.354244 31860 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.356298 31559 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:42.356916 31559 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:42.361095 31417 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.364104 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 656) took 1369 ms. Trace:
I20260709 12:19:42.364529 27588 rpcz_store.cc:276] 0709 12:19:40.994351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:40.994678 (+   327us) service_pool.cc:224] Handling call
0709 12:19:41.003649 (+  8971us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:41.139338 (+135689us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:41.139373 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:41.140663 (+  1290us) log.cc:844] Serialized 494 byte log entry
0709 12:19:41.276824 (+136161us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:41.340393 (+ 63569us) log.cc:844] Serialized 14 byte log entry
0709 12:19:41.340611 (+   218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.362514 (+1021903us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b18003bc6e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:42.363695 (+  1181us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:42.363753 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:42.363809 (+    56us) raft_consensus.cc:1685] finished
0709 12:19:42.363820 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.364025 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1774,"raft.queue_time_us":92,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":8627,"spinlock_wait_cycles":314461184,"thread_start_us":61307,"threads_started":1,"wal-append.queue_time_us":164756}
I20260709 12:19:42.367650 31772 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.368240 31772 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.368675 31772 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:42.370174 31991 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.581s	user 0.001s	sys 0.000s
I20260709 12:19:42.371790 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2174) took 1137 ms. Trace:
I20260709 12:19:42.372272 27736 rpcz_store.cc:276] 0709 12:19:41.234369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.354048 (+119679us) service_pool.cc:224] Handling call
0709 12:19:41.354246 (+   198us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:41.419963 (+ 65717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.371485 (+951522us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54000a55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:42.371717 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37810944}
I20260709 12:19:42.312242 31962 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.346746 31910 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.316327 31997 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:42.426362 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.441340 27584 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:42.374128 31253 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.458828 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.339526 31802 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.351156 31892 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.441051 31735 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.443022 31957 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.363178 31939 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.459395 31253 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.457435 31997 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.445709 31753 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.333750 31759 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.508128 27733 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:19:42.495944 31253 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:42.498492 31892 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.520210 31892 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.522486 31892 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.523408 32007 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.524008 32007 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.524681 31997 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.499449 31753 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.526239 32007 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.526757 32007 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.473553 31802 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.498761 31363 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.563596 31363 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.565892 31363 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.565982 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.566890 27730 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:42.511066 31613 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:19:42.568140 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.568912 27730 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:42.571267 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.571969 27730 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:42.573294 31892 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.573942 31892 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.576519 31892 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.580547 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:42.545459 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.031s	sys 0.058s
I20260709 12:19:42.464064 31970 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.548493 31504 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.527170 32007 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.480134 31957 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.549423 31759 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.465055 31962 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.549811 32006 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.510779 31427 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.550769 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.551852 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:42.663229 27734 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:42.553253 27737 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:42.554136 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.477842 31735 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.555133 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.666281 27735 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:42.527637 31753 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.555564 28334 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:42.560570 31910 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.581728 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:42.687765 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106123us
I20260709 12:19:42.689154 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:42.581755 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.032s	sys 0.058s
I20260709 12:19:42.601576 31901 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:42.579804 31906 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.610939 31504 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:42.637485 32007 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.654855 31957 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.656675 31970 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.664620 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
W20260709 12:19:42.696467 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:42.695894 27589 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:42.659265 31253 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.699828 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2326) took 1150 ms. Trace:
I20260709 12:19:42.693001 31363 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.695451 31997 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.694233 31753 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.651216 31559 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.693948 31901 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.719127 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:42.720582 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.737330 27661 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:42.667567 31427 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.723052 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.740625 27656 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
W20260709 12:19:42.740918 27738 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.010s	sys 0.019s
W20260709 12:19:42.741407 27738 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.010s	sys 0.019s
I20260709 12:19:42.695593 31759 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:42.722393 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.758289 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:42.759932 27664 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:42.739473 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.762753 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:42.743124 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:42.764380 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:42.738656 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:42.732609 27582 rpcz_store.cc:276] 0709 12:19:41.549274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.549405 (+   131us) service_pool.cc:224] Handling call
0709 12:19:41.549671 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:41.549855 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:41.980532 (+430677us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.699292 (+718760us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:42.699394 (+   102us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:42.699471 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.699563 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.699740 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21532,"prepare.queue_time_us":191381,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":67032,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":1127029,"threads_started":3,"wal-append.queue_time_us":302760}
I20260709 12:19:42.713052 32007 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.755913 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:42.666404 32006 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.771246 27730 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:42.734426 31970 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.774847 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:42.740141 31901 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:42.755913 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.786731 27658 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:42.724726 31810 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.772818 31775 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:42.792136 31810 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.794380 31810 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.711839 31772 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.795480 31945 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:42.796805 31772 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.797143 27657 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12.
I20260709 12:19:42.768646 28329 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:42.734067 31733 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.736119 31997 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:42.757397 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2172) took 1621 ms. Trace:
I20260709 12:19:42.738416 32009 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.779296 31253 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.780407 31735 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.781250 32006 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.782730 31559 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.707026 31504 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.785017 31962 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.705432 31906 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.789366 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:42.773765 32007 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.761461 31759 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.803232 27662 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:42.803602 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:42.821905 28329 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:42.800067 27738 rpcz_store.cc:276] 0709 12:19:41.135755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.323371 (+187616us) service_pool.cc:224] Handling call
0709 12:19:41.355534 (+ 32163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:41.376356 (+ 20822us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:41.812967 (+436611us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.742016 (+929049us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.746731 (+  4715us) log.cc:844] Serialized 14 byte log entry
0709 12:19:42.757058 (+ 10327us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:42.757113 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:42.757158 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:42.757222 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.757341 (+   119us) inbound_call.cc:177] Queueing success response
0709 12:19:42.783398 (+ 26057us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b18000552a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
Metrics: {"mutex_wait_us":580979,"prepare.queue_time_us":480972,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":4028,"raft.queue_time_us":138165,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":12247,"spinlock_wait_cycles":15505792,"thread_start_us":793377,"threads_started":2,"wal-append.queue_time_us":388427}
I20260709 12:19:42.738914 31363 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.739825 31753 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.830852 31363 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.813360 31735 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:42.832151 31753 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.832044 31735 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.834208 31735 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.834564 31936 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.837747 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.839756 27732 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:42.840636 31906 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.841290 31427 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.821857 31793 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:42.842284 31793 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.807080 32009 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:42.844055 31978 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.057s	user 0.004s	sys 0.000s
I20260709 12:19:42.844406 31793 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.845078 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.845682 32009 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.847724 31802 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.848665 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.849298 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:42.845712 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:42.850292 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.806023 31733 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.850883 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:42.851754 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:42.852368 31733 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.852870 31804 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.853261 27729 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:42.853372 31804 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.856307 31504 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.856851 31759 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:42.818166 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:42.861109 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:42.804113 31997 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:42.863471 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.865229 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:42.867408 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:42.797684 31970 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.876024 31798 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:42.876510 31798 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.876899 31798 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:42.877377 31970 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.859720 31253 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:42.877786 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.879721 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.880071 32007 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:42.881132 27658 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:19:42.881495 27658 pending_rounds.cc:85] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 11
I20260709 12:19:42.881331 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.881862 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:42.881888 27658 pending_rounds.cc:107] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:19:42.882715 27658 raft_consensus.cc:2889] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:42.896772 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:42.897676 27586 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:42.899418 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 403132us
I20260709 12:19:42.900789 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:42.883705 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:42.904460 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:42.903824 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:42.902310 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:42.907217 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:42.933007 31835 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:19:42.939981 31759 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.940361 31613 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.941612 31613 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.942032 31613 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:42.933079 31793 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.944356 31793 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:42.946532 31793 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.933275 31758 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 12:19:42.933133 31733 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.949048 31733 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.949448 31733 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:42.942409 31759 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:42.952824 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:42.955703 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49868.5us
I20260709 12:19:42.954309 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:42.957852 28330 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:42.956977 28329 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:42.903087 31363 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:42.967602 31363 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.899721 31253 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:42.896170 31797 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:19:42.976153 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:42.978354 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:42.902757 31802 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:42.988458 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:42.989342 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:42.989954 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:42.979411 27587 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:42.984568 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:42.998178 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94409.5us
I20260709 12:19:42.999859 27588 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 4.
I20260709 12:19:43.000393 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:42.956099 31860 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:42.999157 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.005716 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:43.001353 27552 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12.
I20260709 12:19:43.006608 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.006992 27552 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:19:42.997676 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.007174 28329 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:42.998162 27581 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:19:43.007694 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.001911 27580 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:19:43.008571 27584 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:43.002383 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.009863 28330 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:42.972617 31253 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:42.954332 31804 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:42.957837 31772 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:42.985980 31936 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.015630 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.011746 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.016106 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.010984 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:43.017481 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.017992 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:42.985733 31797 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:42.994163 31892 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.049324 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.020082 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 657) took 1070 ms. Trace:
I20260709 12:19:43.048710 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.060026 27588 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:43.060660 27581 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:43.059976 28328 rpcz_store.cc:276] 0709 12:19:41.949485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.949692 (+   207us) service_pool.cc:224] Handling call
0709 12:19:41.950119 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.302029 (+351910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.014737 (+712708us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54005eb854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:43.020009 (+  5272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":807650048}
I20260709 12:19:43.061393 27661 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:43.047845 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:43.062728 27662 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:43.066911 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.001744 31758 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.004848 31759 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:42.993852 32010 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.996492 31912 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.069669 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.059271 31793 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:43.059479 32005 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.059666 31929 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:43.075420 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 64558.5us
I20260709 12:19:43.005509 31860 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.084625 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.059685 31772 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.016072 31647 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:43.087602 27736 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:43.074009 27661 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:43.015923 31655 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.074153 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.058784 32007 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.210076 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.146807 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:43.141845 27628 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.4:39321 timed out after 1.640s (SENT) [suppressed 11 similar messages]
I20260709 12:19:43.211791 28335 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:43.112320 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.144548 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.218878 28337 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:19:43.020912 32011 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:43.082875 31802 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:43.083938 31759 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.084264 32006 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.248701 31939 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:19:43.250349 32019 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:43.250777 32019 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:43.251825 32019 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.084086 31559 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.084103 31753 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.214591 31647 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.079767 31970 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:43.070304 31797 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.083217 31901 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.071274 31936 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.235195 31655 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:43.241899 27663 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.010s	sys 0.009s
W20260709 12:19:43.242723 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.003s	sys 0.022s
I20260709 12:19:43.116633 31837 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:43.083837 31758 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.119868 31920 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.155510 31804 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.385419 31804 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.385986 31758 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:43.386919 31758 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.232266 31860 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:43.387522 31758 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:43.388335 32025 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:19:43.390056 31758 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.390175 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:43.390774 31758 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.391371 27733 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.392513 31804 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.239892 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.394286 27736 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:43.395143 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:43.396224 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.396729 27736 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:43.397467 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.397959 27736 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:43.398660 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.255821 31759 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:43.295847 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.003s	sys 0.022s
W20260709 12:19:43.296171 27663 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.010s	sys 0.010s
I20260709 12:19:43.092626 31775 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.180s
I20260709 12:19:43.072628 31892 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.301084 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:43.409200 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 108214us
W20260709 12:19:43.409715 27729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.008s	sys 0.014s
W20260709 12:19:43.410100 27729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.008s	sys 0.014s
I20260709 12:19:43.261416 32006 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.109545 32005 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:43.297362 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54761us
I20260709 12:19:43.246289 31802 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.102398 31912 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.419823 31912 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.229471 32007 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.230258 31929 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.231185 31906 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 12:19:43.274029 31647 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:43.371718 28329 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:43.260170 31908 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:43.365953 32024 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:43.366271 31989 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.442s	user 0.001s	sys 0.004s
I20260709 12:19:43.383317 31793 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12.
I20260709 12:19:43.399130 27736 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 4.
I20260709 12:19:43.089818 32010 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:43.404485 27660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.008s	sys 0.014s
I20260709 12:19:43.325924 32013 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:43.407456 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 321673us
W20260709 12:19:43.469017 27737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.013s	sys 0.011s
I20260709 12:19:43.421294 27664 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
W20260709 12:19:43.469470 27737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.013s	sys 0.011s
I20260709 12:19:43.431993 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:43.274298 31753 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.274425 31559 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.285858 31970 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.368157 31837 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:43.455615 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 548417us
W20260709 12:19:43.456161 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:43.456110 31976 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.999s	user 0.015s	sys 0.006s
I20260709 12:19:43.470573 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.426194 31802 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:43.477099 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.480517 27733 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:43.480839 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:43.481488 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:43.485224 32005 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.486239 32005 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:19:43.479224 31997 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.546s	user 0.008s	sys 0.003s
W20260709 12:19:43.487444 31976 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.016s	sys 0.007s
I20260709 12:19:43.493736 31793 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:43.441597 31758 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.457891 31908 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.504263 32023 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:19:43.494141 32017 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:19:43.477418 32013 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.457724 32024 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.501659 31997 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.532153 32023 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.565881 31908 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.577513 31892 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.577960 32019 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.579279 32013 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:43.580907 32012 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:19:43.568097 31936 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:43.587936 32016 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.477s	user 0.000s	sys 0.001s
W20260709 12:19:43.591210 31988 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.648s	user 0.005s	sys 0.000s
W20260709 12:19:43.591638 32018 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.473s	user 0.002s	sys 0.000s
W20260709 12:19:43.594965 32026 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
W20260709 12:19:43.601267 32042 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 12:19:43.602599 32021 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.485s	user 0.000s	sys 0.002s
W20260709 12:19:43.602851 32036 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 12:19:43.603729 32033 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:19:43.606981 32010 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.609045 31997 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:43.610286 31970 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:43.602607 32029 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:19:43.618023 32024 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:43.618921 32007 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.619665 31929 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.561650 32023 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:43.627947 32006 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.525445 32011 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:43.636474 32011 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.645833 32023 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.375998 27582 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:43.679107 27589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.015s	sys 0.012s
W20260709 12:19:43.812266 27589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.015s	sys 0.013s
W20260709 12:19:43.680015 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.000s	sys 0.029s
W20260709 12:19:43.681212 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 539161us
W20260709 12:19:43.814071 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.000s	sys 0.029s
W20260709 12:19:43.814587 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 822498us
I20260709 12:19:43.816512 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:43.456539 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.016s	sys 0.008s
W20260709 12:19:43.818172 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.011s	user 0.016s	sys 0.009s
I20260709 12:19:43.818670 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1955) took 1262 ms. Trace:
I20260709 12:19:43.820304 32046 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:43.821019 32046 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.821223 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.820333 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.822497 27659 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:43.823554 27655 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:43.825554 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:43.458415 27660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.008s	sys 0.014s
W20260709 12:19:43.685182 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.010s	sys 0.013s
I20260709 12:19:43.828244 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:43.828748 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.010s	sys 0.013s
I20260709 12:19:43.829274 31559 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.825326 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:43.685487 27585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.005s	sys 0.013s
W20260709 12:19:43.685583 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:19:43.687937 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 632) took 2382 ms. Trace:
I20260709 12:19:43.692688 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:43.809365 31613 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.867s	user 0.005s	sys 0.009s
I20260709 12:19:43.690737 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:43.695874 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:43.502343 31793 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:43.768678 31655 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.513985 31772 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.798488 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:43.806967 27735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.017s	sys 0.038s
I20260709 12:19:43.807221 32007 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:43.809445 31837 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.821602 32046 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.708563 31839 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:43.836366 27736 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:43.849409 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:43.852818 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.853859 27736 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:19:43.854251 31758 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.832268 32023 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:43.834220 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.013s	sys 0.000s
W20260709 12:19:43.834008 27585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.005s	sys 0.013s
I20260709 12:19:43.837001 27737 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:43.833751 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.852811 28328 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:43.829772 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:43.876307 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.018s	sys 0.038s
I20260709 12:19:43.855232 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:43.877427 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1894) took 1153 ms. Trace:
W20260709 12:19:43.836040 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 310795us
W20260709 12:19:43.875130 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.000s	sys 0.011s
I20260709 12:19:43.841439 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.800737 31753 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:43.884223 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.000s	sys 0.012s
W20260709 12:19:43.881875 27662 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.004s	sys 0.008s
I20260709 12:19:43.819211 27730 rpcz_store.cc:276] 0709 12:19:42.556076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.572644 (+ 16568us) service_pool.cc:224] Handling call
0709 12:19:42.573000 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.773983 (+200983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.818061 (+1044078us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b540044e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:43.818419 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455491968}
I20260709 12:19:43.834822 31908 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:43.899747 27662 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.004s	sys 0.008s
W20260709 12:19:43.889748 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95417.1us
I20260709 12:19:43.895303 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2183) took 1033 ms. Trace:
I20260709 12:19:43.903574 28335 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:19:43.905138 31970 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.906158 31970 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.935629 32050 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:43.936468 32050 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.938057 31970 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:43.939639 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.940778 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.942668 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.943602 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:43.945226 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:43.946386 28329 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:43.906925 31615 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:43.949522 31798 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.072s	user 0.010s	sys 0.010s
I20260709 12:19:43.950101 31798 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:43.908263 32055 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.066s	user 0.000s	sys 0.002s
W20260709 12:19:43.894371 31910 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.004s	sys 0.038s
I20260709 12:19:43.951740 27580 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:43.886477 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 379867us
I20260709 12:19:43.953378 31808 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.954466 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:43.955422 27733 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:43.955685 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 70963.9us
I20260709 12:19:43.956079 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2192) took 1091 ms. Trace:
I20260709 12:19:43.956544 27733 rpcz_store.cc:276] 0709 12:19:42.864734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.833157 (+968423us) service_pool.cc:224] Handling call
0709 12:19:43.955981 (+122824us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:43.956907 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:43.957019 31810 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:43.958325 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:43.835682 27731 rpcz_store.cc:276] 0709 12:19:41.306193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.360577 (+ 54384us) service_pool.cc:224] Handling call
0709 12:19:41.360930 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:41.526841 (+165911us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:41.526871 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:41.527343 (+   472us) log.cc:844] Serialized 26 byte log entry
0709 12:19:41.605816 (+ 78473us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:41.605857 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:41.605936 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:41.824359 (+218423us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b18001d3f20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b557 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:43.072119 (+1247760us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:43.687286 (+615167us) raft_consensus.cc:1685] finished
0709 12:19:43.687343 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.687683 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":165507,"raft.queue_time_us":182294,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":5791,"spinlock_wait_cycles":48070784,"thread_start_us":46970,"threads_started":1,"wal-append.queue_time_us":232867}
I20260709 12:19:43.913992 31363 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:43.891263 31929 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:43.918035 32060 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:43.968843 32060 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.969452 32060 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.968689 31802 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.891609 32007 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:43.921415 28334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 654) took 1975 ms (client timeout 1640 ms). Trace:
I20260709 12:19:43.964315 32050 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:43.971630 28334 rpcz_store.cc:269] 0709 12:19:41.945972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.961499 (+ 15527us) service_pool.cc:224] Handling call
0709 12:19:43.900489 (+1938990us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:43.921072 (+ 20583us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1363934592}
I20260709 12:19:43.954777 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:43.915110 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85573.8us
I20260709 12:19:43.973357 32007 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:43.973369 27734 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.974175 31837 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:43.974092 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:43.974731 31837 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.975157 31837 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.975682 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:43.977382 31808 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:43.977895 31808 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.978343 31808 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:43.980790 31929 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:43.978993 31910 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:43.982106 31910 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.981719 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:43.982447 31417 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:43.987627 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.988547 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:43.989667 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.990971 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.992818 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:43.993360 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2180) took 1132 ms. Trace:
I20260709 12:19:43.896951 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:43.960197 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:43.953230 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 640) took 1117 ms. Trace:
I20260709 12:19:43.902748 27735 rpcz_store.cc:276] 0709 12:19:42.861793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.899960 (+ 38167us) service_pool.cc:224] Handling call
0709 12:19:42.900476 (+   516us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:42.908845 (+  8369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:42.908890 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:42.953770 (+ 44880us) log.cc:844] Serialized 52 byte log entry
0709 12:19:43.889237 (+935467us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:43.890926 (+  1689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.891064 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:43.894897 (+  3833us) raft_consensus.cc:1685] finished
0709 12:19:43.894922 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.895202 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":186,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":30265,"thread_start_us":875298,"threads_started":1,"wal-append.queue_time_us":869410}
I20260709 12:19:43.967139 31901 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:43.968284 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2184) took 1106 ms. Trace:
W20260709 12:19:43.959945 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 700761us
I20260709 12:19:43.994663 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:43.990971 27734 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:43.996960 27738 rpcz_store.cc:276] 0709 12:19:42.835691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.839431 (+  3740us) service_pool.cc:224] Handling call
0709 12:19:42.839775 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.855461 (+ 15686us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:42.855498 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:42.855552 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:42.855585 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.952546 (+1096961us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b54000a4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:43.952623 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.952967 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35521408}
I20260709 12:19:43.993360 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2297) took 1131 ms. Trace:
I20260709 12:19:43.934423 31647 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.993880 27729 rpcz_store.cc:276] 0709 12:19:42.860924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.862551 (+  1627us) service_pool.cc:224] Handling call
0709 12:19:42.862871 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.863074 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:42.863306 (+   232us) log.cc:844] Serialized 14 byte log entry
0709 12:19:43.792398 (+929092us) log.cc:844] Serialized 14 byte log entry
0709 12:19:43.992339 (+199941us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:43.992411 (+    72us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:43.992528 (+   117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.992820 (+   292us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.993159 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":277753,"prepare.queue_time_us":240778,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":155,"raft.queue_time_us":235633,"raft.run_cpu_time_us":880,"raft.run_wall_time_us":435018,"thread_start_us":918972,"threads_started":3,"wal-append.queue_time_us":689874}
I20260709 12:19:44.004678 27731 rpcz_store.cc:276] 0709 12:19:42.862065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.966409 (+1104344us) service_pool.cc:224] Handling call
0709 12:19:43.967222 (+   813us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:43.967673 (+   451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:43.967702 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:43.967768 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:43.967806 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.967870 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.968016 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:43.959229 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:44.006708 27736 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:43.877950 27656 rpcz_store.cc:276] 0709 12:19:42.722972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.741547 (+ 18575us) service_pool.cc:224] Handling call
0709 12:19:42.741941 (+   394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:42.742893 (+   952us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:42.742933 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:42.768970 (+ 26037us) log.cc:844] Serialized 52 byte log entry
0709 12:19:43.819059 (+1050089us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:43.819153 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.819304 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:43.844435 (+ 25131us) raft_consensus.cc:1685] finished
0709 12:19:43.844455 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.875941 (+ 31486us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2249,"raft.queue_time_us":14311,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":5836,"thread_start_us":1012009,"threads_started":1,"wal-append.queue_time_us":825939}
I20260709 12:19:44.022156 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2196) took 1031 ms. Trace:
I20260709 12:19:43.957698 31810 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.026358 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.006705 27582 rpcz_store.cc:276] 0709 12:19:42.862741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.863191 (+   450us) service_pool.cc:224] Handling call
0709 12:19:43.992906 (+1129715us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b54002ba7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:43.993164 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1176628096}
I20260709 12:19:43.836424 31613 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:43.982496 31910 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.022872 27736 rpcz_store.cc:276] 0709 12:19:42.990361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.854759 (+864398us) service_pool.cc:224] Handling call
0709 12:19:44.022090 (+167331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:44.027294 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:44.034408 31797 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.667s	user 0.019s	sys 0.002s
I20260709 12:19:43.843459 31793 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.036661 27729 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:44.049319 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1960) took 1032 ms. Trace:
I20260709 12:19:44.049691 27729 rpcz_store.cc:276] 0709 12:19:43.017109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:44.031640 (+1014531us) service_pool.cc:224] Handling call
0709 12:19:44.036214 (+  4574us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:44.049072 (+ 12858us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.049265 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:44.038100 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2186) took 1175 ms. Trace:
I20260709 12:19:44.050562 27735 rpcz_store.cc:276] 0709 12:19:42.863094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:44.006579 (+1143485us) service_pool.cc:224] Handling call
0709 12:19:44.037252 (+ 30673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:44.037670 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:44.037703 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:44.037777 (+    74us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:44.037811 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.037882 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.038010 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:44.043594 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2185) took 1180 ms. Trace:
I20260709 12:19:44.034890 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:44.066120 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 107266us
I20260709 12:19:44.052723 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.075001 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 642) took 1238 ms. Trace:
I20260709 12:19:44.037297 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.076762 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1892) took 1354 ms. Trace:
I20260709 12:19:44.034982 31797 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.046883 31613 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.119818 31929 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.029412 31810 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:44.074257 28331 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:19:44.154788 27736 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
W20260709 12:19:44.025724 32002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.005s	sys 0.009s
I20260709 12:19:44.083387 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:44.047596 27730 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:44.068455 31735 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.051640 27738 rpcz_store.cc:276] 0709 12:19:42.862855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:44.019245 (+1156390us) service_pool.cc:224] Handling call
0709 12:19:44.042732 (+ 23487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:44.043145 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:44.043187 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:44.043277 (+    90us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:44.043312 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.043390 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.043514 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:44.145833 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60547.1us
I20260709 12:19:44.005227 31908 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:44.163748 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2191) took 1294 ms. Trace:
I20260709 12:19:44.164129 27730 rpcz_store.cc:276] 0709 12:19:42.864488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.895665 (+1031177us) service_pool.cc:224] Handling call
0709 12:19:44.158542 (+262877us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:44.151473 27658 rpcz_store.cc:276] 0709 12:19:42.836337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.836477 (+   140us) service_pool.cc:224] Handling call
0709 12:19:42.836875 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.883144 (+ 46269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.074390 (+1191246us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:44.074774 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58049920}
I20260709 12:19:44.162355 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:44.167629 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2188) took 1303 ms. Trace:
I20260709 12:19:44.167902 27737 rpcz_store.cc:276] 0709 12:19:42.863782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.886103 (+1022321us) service_pool.cc:224] Handling call
0709 12:19:44.167563 (+281460us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:44.172067 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 213868us
I20260709 12:19:44.173285 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.173981 31831 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:19:44.174402 31909 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:44.174988 31909 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.151752 27663 rpcz_store.cc:276] 0709 12:19:42.722335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.723004 (+   669us) service_pool.cc:224] Handling call
0709 12:19:42.723308 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.723756 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:43.403857 (+680101us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.005512 (+601655us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.029465 (+ 23953us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:44.033573 (+  4108us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:44.076214 (+ 42641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.076361 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.076588 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29447,"prepare.queue_time_us":263161,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":78,"raft.queue_time_us":36940,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":26252,"spinlock_wait_cycles":5670656,"thread_start_us":1066502,"threads_started":2,"wal-append.queue_time_us":437786}
I20260709 12:19:44.175418 31909 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.176294 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.177004 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
W20260709 12:19:44.177407 27657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 645) took 1312 ms (client timeout 1724 ms). Trace:
I20260709 12:19:44.173276 31735 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:44.177852 27657 rpcz_store.cc:269] 0709 12:19:42.864799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.864940 (+   141us) service_pool.cc:224] Handling call
0709 12:19:44.177054 (+1312114us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:44.177217 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2323362688}
I20260709 12:19:44.180480 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:44.182269 31767 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:44.182528 31793 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.182723 31767 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.183188 31767 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.184347 31908 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.187769 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.188731 27664 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:44.189127 32007 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:19:44.189364 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 182106us
I20260709 12:19:44.189625 32007 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.190040 32007 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.193732 31797 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.174499 31831 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.195042 31831 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:44.195945 31735 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.179276 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.200412 27657 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:44.174576 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.202378 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.202864 31906 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:44.203359 31906 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.203804 31906 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.214744 31901 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.177953 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.215402 31901 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:44.216245 31733 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.266s	user 0.011s	sys 0.015s
I20260709 12:19:44.216728 31733 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.218122 31559 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:44.218892 27586 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:44.218396 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1959) took 1201 ms. Trace:
I20260709 12:19:44.219122 31909 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.219878 27731 rpcz_store.cc:276] 0709 12:19:43.016882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:44.020347 (+1003465us) service_pool.cc:224] Handling call
0709 12:19:44.158508 (+138161us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:44.217965 (+ 59457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:44.218005 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:44.218083 (+    78us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:44.218121 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.218189 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.218326 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:44.220628 31759 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.791s	user 0.011s	sys 0.003s
I20260709 12:19:44.221333 31759 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.222697 31909 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.222932 28329 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:19:44.214179 31767 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.225487 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.226374 28332 raft_consensus.cc:2368] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:19:44.226814 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.226840 31831 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.228226 28335 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:44.225636 31767 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.239372 32050 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:44.253233 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:44.230242 32002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.005s	sys 0.010s
I20260709 12:19:44.261981 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:44.265570 28329 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:44.258453 27661 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:44.236497 31831 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:44.269214 28335 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:44.269447 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.271536 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.229935 31929 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.259281 31733 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.266170 31798 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.283599 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.249692 32050 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.275676 32005 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.286379 31647 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.272081 31906 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.282967 31735 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.308290 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:44.317893 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59514.7us
W20260709 12:19:44.322678 31860 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.867s	user 0.012s	sys 0.011s
I20260709 12:19:44.308501 31797 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.335152 31808 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.331862 27657 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:44.332332 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1903) took 1312 ms. Trace:
I20260709 12:19:44.335470 31735 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.336728 32050 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.313161 32007 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.315850 32041 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:44.339088 31559 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.339599 32046 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.340870 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.341552 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:44.343093 31759 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.343382 31906 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:44.344013 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.346601 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.323271 31860 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.349063 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.350554 27661 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:44.352277 31837 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.352726 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.353780 31918 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.353886 31910 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.354962 31797 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.355818 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.356158 31802 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:44.387583 27580 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:44.388986 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.390575 31253 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.391688 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.392355 31918 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.392985 31427 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.393687 31427 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.396200 31767 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:44.396878 31837 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:44.396809 31767 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.318128 31929 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.302397 31733 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.336779 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.361618 31793 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:44.365923 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.350744 31810 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.406131 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.360831 27662 rpcz_store.cc:276] 0709 12:19:43.020304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.074997 (+ 54693us) service_pool.cc:224] Handling call
0709 12:19:43.075498 (+   501us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:43.902834 (+827336us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:43.930227 (+ 27393us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.151703 (+221476us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:44.290780 (+139077us) log.cc:844] Serialized 52 byte log entry
0709 12:19:44.294165 (+  3385us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:44.294201 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.301915 (+  7714us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:44.332031 (+ 30116us) raft_consensus.cc:1685] finished
0709 12:19:44.332061 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.332270 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15505,"prepare.queue_time_us":957,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":169000,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":21749,"thread_start_us":1030536,"threads_started":2,"wal-append.queue_time_us":356602}
I20260709 12:19:44.368887 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:44.370618 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:44.373138 31559 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:44.411854 27661 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:44.372596 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.382408 27587 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:44.383184 27581 raft_consensus.cc:2410] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:44.384106 31910 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.416344 27659 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:44.385082 31860 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.386436 31797 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.386827 31735 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.387051 27738 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:44.355021 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.329268 32024 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:44.400328 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51365.1us
I20260709 12:19:44.464218 27730 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:44.368870 32041 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.371816 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.426337 27729 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:44.411413 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2329) took 2058 ms. Trace:
I20260709 12:19:44.467892 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.327224 32005 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.424223 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.433358 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.421918 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.433605 31918 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.434284 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.399158 31767 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.435348 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.436372 27589 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:44.437270 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:44.443717 31733 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.445387 32038 raft_consensus.cc:2764] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:19:44.391180 31253 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:44.448298 27555 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39872 dropped due to backpressure: service queue is full with 50 items [suppressed 5 similar messages]
I20260709 12:19:44.452303 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2303) took 1469 ms. Trace:
I20260709 12:19:44.453229 31793 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.460569 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2181) took 1599 ms. Trace:
I20260709 12:19:44.415932 31363 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:44.380636 31417 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:44.461700 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 120222us
I20260709 12:19:44.405536 31615 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:19:44.466729 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.467450 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:44.478900 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.417976 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:44.472781 27661 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:44.482354 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.487483 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.473312 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:44.470155 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.475137 27582 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:44.476823 27586 rpcz_store.cc:276] 0709 12:19:42.983074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.983281 (+   207us) service_pool.cc:224] Handling call
0709 12:19:44.452092 (+1468811us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:44.452237 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2785111808}
I20260709 12:19:44.484440 27735 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.491619 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.467960 27585 rpcz_store.cc:276] 0709 12:19:42.345016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.399136 (+ 54120us) service_pool.cc:224] Handling call
0709 12:19:42.399667 (+   531us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:42.400222 (+   555us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:42.814232 (+414010us) log.cc:844] Serialized 14 byte log entry
0709 12:19:43.960046 (+1145814us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:44.199730 (+239684us) log.cc:844] Serialized 52 byte log entry
0709 12:19:44.200087 (+   357us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:44.375700 (+175613us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.375935 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.380583 (+  4648us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:44.380668 (+    85us) raft_consensus.cc:1685] finished
0709 12:19:44.380679 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.403130 (+ 22451us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75521,"prepare.queue_time_us":99626,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":1527,"raft.queue_time_us":1660,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":8186,"thread_start_us":1429251,"threads_started":3,"wal-append.queue_time_us":692137}
W20260709 12:19:44.494047 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148571us
I20260709 12:19:44.472831 31797 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.470831 32041 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:44.476025 31906 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.507153 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.496963 27580 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:44.499027 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.470475 31759 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:44.500003 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.518815 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.478832 27732 rpcz_store.cc:276] 0709 12:19:42.861236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.864876 (+  3640us) service_pool.cc:224] Handling call
0709 12:19:42.865134 (+   258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.889134 (+ 24000us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:42.934259 (+ 45125us) log.cc:844] Serialized 14 byte log entry
0709 12:19:43.986285 (+1052026us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b18000a4c80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:44.217247 (+230962us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.217515 (+   268us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:44.217595 (+    80us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:44.217666 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.460062 (+242396us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b54000a64d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:44.460132 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.460359 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35232,"prepare.queue_time_us":73475,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":85,"raft.queue_time_us":35657,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":261,"spinlock_wait_cycles":93551744,"thread_start_us":41778,"threads_started":1,"wal-append.queue_time_us":1152345}
I20260709 12:19:44.500536 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.501672 27587 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.501868 31363 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.491583 31735 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.503531 32011 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.498548 31615 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:44.498987 31798 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:44.505221 32060 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.535s	user 0.009s	sys 0.003s
I20260709 12:19:44.507473 28337 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.472396 32024 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.512429 32007 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.512676 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.513630 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:44.515530 31901 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.519440 27658 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.502341 31417 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.519433 27729 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:19:44.522536 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.501736 27661 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:44.542172 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.542502 31735 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.543102 27661 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:44.543426 27584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2307) took 1532 ms (client timeout 1791 ms). Trace:
I20260709 12:19:44.534657 31798 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.544042 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.544337 32011 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.545320 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.546691 32011 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.547691 28329 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.548341 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.548962 27662 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:19:44.529124 31767 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.549810 31767 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.551867 31615 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.552487 31767 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.556510 32046 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.524679 27581 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:44.508617 32005 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:44.543848 27584 rpcz_store.cc:269] 0709 12:19:43.011216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:43.011417 (+   201us) service_pool.cc:224] Handling call
0709 12:19:44.543052 (+1531635us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000b1154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:44.543233 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3221011200}
I20260709 12:19:44.532150 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.478179 31837 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:19:44.535435 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:44.538018 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.546571 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.558980 27658 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:44.559125 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.516052 31427 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.489717 31810 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.539371 27733 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:44.564684 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.563751 27583 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:44.572161 27662 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:19:44.537361 28331 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.529592 32050 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.574397 27734 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:44.525882 31906 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.577400 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.577658 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.578359 27659 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:44.578860 27664 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:19:44.583950 28337 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:44.591410 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.592244 27731 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:44.593528 27658 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:44.576573 28329 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:44.599032 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.600239 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.600291 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.601284 32050 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.601794 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.602473 27730 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.603531 32050 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.604238 31860 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.604709 31860 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.606913 31860 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.610638 32007 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:44.611559 32007 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:44.612769 32007 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.615494 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1897) took 1658 ms. Trace:
I20260709 12:19:44.621097 27656 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:44.616554 27729 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:44.534893 31797 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.535082 32060 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.540913 31759 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.633282 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 1961) took 1616 ms. Trace:
I20260709 12:19:44.633736 27734 rpcz_store.cc:276] 0709 12:19:43.017355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:44.049442 (+1032087us) service_pool.cc:224] Handling call
0709 12:19:44.239019 (+189577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:44.584715 (+345696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.633050 (+ 48335us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:44.633223 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765382400}
I20260709 12:19:44.638679 27738 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:44.646373 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.655272 27663 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:44.658042 27658 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:44.477262 27555 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39872 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2345)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2344)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2336)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2323)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2334)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2349)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2327)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2326)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2353)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2337)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2322)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2350)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2343)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2335)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 677)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 678)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 682)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2321)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2324)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2325)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2332)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2333)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2338)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2341)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2355)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2362)
I20260709 12:19:44.675822 31920 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:19:44.668773 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.621318 27660 rpcz_store.cc:276] 0709 12:19:42.817472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.819968 (+  2496us) service_pool.cc:224] Handling call
0709 12:19:42.820295 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:42.820537 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:43.827995 (+1007458us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.317561 (+489566us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.320258 (+  2697us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.321212 (+   954us) log.cc:844] Serialized 14 byte log entry
0709 12:19:44.367645 (+ 46433us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:44.370128 (+  2483us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:44.381757 (+ 11629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:44.413310 (+ 31553us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:44.475796 (+ 62486us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":168588,"prepare.queue_time_us":280107,"prepare.run_cpu_time_us":157,"prepare.run_wall_time_us":165,"raft.queue_time_us":50917,"raft.run_cpu_time_us":724,"raft.run_wall_time_us":54259,"thread_start_us":1179053,"threads_started":3,"wal-append.queue_time_us":119474}
W20260709 12:19:44.681557 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 174915us
I20260709 12:19:44.685726 27628 leader_election.cc:400] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:19:44.686260 27628 leader_election.cc:403] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:44.687187 27658 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.689257 31808 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.690060 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.690768 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:44.684783 31733 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.702276 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.702904 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:44.701726 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72976.7us
I20260709 12:19:44.692502 31929 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.704804 31798 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:44.701584 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.697121 31793 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.708415 27584 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:44.716038 31735 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.722031 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.722301 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.722944 27661 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.724179 31759 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.724722 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.725442 27589 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:44.726408 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.727061 27589 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:44.728001 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.750695 27656 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.729132 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.728092 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.751509 27582 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:44.730340 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.729589 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.697342 31808 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:19:44.729174 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:44.732300 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.711575 31253 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.734582 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.736023 31906 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:44.736155 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.736824 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:44.737246 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.739061 31797 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.740584 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.752619 27729 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:44.752939 27659 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:44.754719 27735 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:44.764626 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.765692 27730 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:44.768318 27585 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:44.756644 27587 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:44.757618 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.769457 27588 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:19:44.770344 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.770592 27580 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:44.760265 27661 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:44.773149 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.774823 31363 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.774878 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.775418 31802 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.775908 27587 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.776575 27583 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:44.777226 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.777663 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.776262 31802 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:44.778359 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.779050 27581 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:44.780041 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.780143 31802 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:44.780701 27581 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:44.778390 27583 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:44.785384 27658 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:44.789532 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.791219 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.792666 32060 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.793651 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.770222 31793 raft_consensus.cc:2764] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:19:44.789284 27584 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:44.797909 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.798847 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.800828 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.784719 31906 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:44.801676 27586 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
I20260709 12:19:44.803167 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.794500 28334 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:44.810595 27587 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:44.810842 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:44.811000 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74280.1us
I20260709 12:19:44.818135 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.819631 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:44.820971 31765 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:44.821154 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:44.822046 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 122710us
I20260709 12:19:44.823582 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:44.824039 27583 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.786360 31837 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:19:44.810437 31906 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.821537 31765 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.835125 31765 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.828390 31831 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:44.829886 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:44.830008 31363 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:44.826438 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.841717 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:44.842427 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.843055 27587 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:44.827826 31995 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:44.836874 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.839679 31802 raft_consensus.cc:2764] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:44.836405 31906 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.840267 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:44.846776 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.825554 31860 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:44.847426 27585 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:19:44.845669 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.854660 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:44.835577 31808 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:44.852509 27581 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:44.858582 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.861743 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:44.862375 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.862573 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:44.863260 27735 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:44.863039 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:44.840833 31831 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:44.866029 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.844038 31995 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:44.867982 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.873039 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:44.868814 27581 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.851984 31860 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:44.870903 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53815.9us
I20260709 12:19:44.859117 31918 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:44.879729 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:44.880198 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.883662 31759 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:44.884126 31759 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:44.884506 31759 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:44.887022 32080 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:19:44.864063 31615 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:44.879247 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.888430 27730 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:44.867252 31802 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:44.867791 31808 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:44.868680 31831 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:44.847697 31559 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.901081 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.878584 31860 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.893184 32078 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:44.914471 27589 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:19:44.919345 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.919603 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:44.920370 27655 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:44.914251 32078 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:44.923710 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:44.923867 32078 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:44.927549 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:44.921099 31901 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.928817 31901 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.927807 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.935603 32078 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.936643 27586 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:44.937314 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.937912 31417 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:44.938886 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:44.939944 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:44.957779 27587 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:44.891402 31615 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:44.943877 27586 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:44.945477 32078 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:44.949476 27656 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:44.951043 27589 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:44.957008 27581 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:44.964799 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:44.969898 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.973043 27734 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
W20260709 12:19:44.965672 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 124027us
I20260709 12:19:44.974174 27730 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:44.975622 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:44.926955 31906 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:44.988118 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.972083 31615 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:45.082887 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 112520us
I20260709 12:19:45.083793 32007 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:19:45.084280 32007 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:45.084664 32007 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:19:44.993927 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:44.922885 28331 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:19:45.022823 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126183us
I20260709 12:19:45.010893 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:45.084686 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:44.987736 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:44.998172 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:45.097075 27583 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:45.097818 27735 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:19:45.026430 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:45.035195 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.019310 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.014745 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.099190 27730 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.113260 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:45.114590 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91663.4us
I20260709 12:19:45.115464 27588 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:45.117053 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 141565us
I20260709 12:19:45.117833 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.118867 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:45.125456 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 248286us
W20260709 12:19:45.034722 32041 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.501s	user 0.010s	sys 0.000s
I20260709 12:19:45.135118 27587 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:45.135967 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.136960 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:45.137631 27736 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:45.138144 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.143113 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.156483 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:45.112612 31363 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:45.157476 27729 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:45.158209 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.158663 27583 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:44.979729 31995 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.144929 31559 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.174464 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:45.174752 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 334530us
I20260709 12:19:45.175236 27587 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:45.182124 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:45.156548 31253 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:45.114166 31831 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.182574 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:45.170763 31363 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:45.193841 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:45.189309 27736 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:45.168918 32078 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:45.143708 32041 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.185510 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.195591 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 8.
W20260709 12:19:45.193413 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87654.6us
I20260709 12:19:45.196298 27730 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:45.029455 31808 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.197521 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.199718 27583 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:45.200094 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:45.200845 27729 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:45.172691 31918 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:45.202050 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.188943 31837 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:45.204597 27587 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:45.202994 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:45.206287 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.186800 31253 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:45.219877 27730 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:19:45.205407 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68529.8us
I20260709 12:19:45.221321 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:45.222514 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 242928us
I20260709 12:19:45.198712 31363 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:45.225154 27657 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14.
W20260709 12:19:45.225932 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51715.6us
I20260709 12:19:45.199338 31559 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.198557 32078 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.228233 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.228931 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.229635 27580 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:19:45.233264 27588 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:45.198275 32024 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.229976 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:45.234035 31849 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:19:45.217133 32081 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:19:45.235183 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:45.206892 31995 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:45.234329 32085 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:45.211410 32050 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:45.222890 31253 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:45.219400 31837 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.249475 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:45.251372 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.252856 27587 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:45.236707 32038 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:19:45.238680 32078 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:45.241211 31849 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.251813 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:45.259524 32038 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.262897 32038 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:45.261588 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.276659 28334 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:45.277647 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:45.252125 31837 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:45.278383 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.279207 27584 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 5.
I20260709 12:19:45.283919 27736 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:45.292906 27730 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:19:45.265763 31849 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:45.286770 31797 raft_consensus.cc:2764] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 12:19:45.295678 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60588.7us
I20260709 12:19:45.301771 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.302770 27584 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:45.300832 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.303270 27587 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
W20260709 12:19:45.308552 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 102190us
I20260709 12:19:45.280315 32050 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.314118 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:45.310796 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103127us
I20260709 12:19:45.319788 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.284796 31906 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.321836 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:45.322319 31765 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.323870 31802 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:45.324397 31802 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:45.325301 31802 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.325973 32007 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.323417 31918 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.332288 32041 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:45.332685 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:45.333854 32084 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:19:45.339596 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95073.7us
I20260709 12:19:45.340694 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:45.315578 32094 raft_consensus.cc:2764] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:19:45.343726 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:45.280612 32089 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:45.344323 27585 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:19:45.359454 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81155us
I20260709 12:19:45.361748 27733 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:45.343091 31908 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:19:45.376760 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 149438us
I20260709 12:19:45.359072 32024 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:45.355958 31417 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:45.378955 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:45.388756 28334 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:45.347461 31759 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:45.392957 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:45.394246 31759 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:45.394963 31765 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:45.396343 32091 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:19:45.406754 31831 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:45.411748 31901 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:45.358258 31793 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:19:45.418440 32038 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.421036 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:45.426352 31363 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.427632 31417 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:45.428996 31417 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.437299 32007 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:45.440846 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2321) took 1083 ms. Trace:
W20260709 12:19:45.441354 31860 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.000s
I20260709 12:19:45.441787 31860 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.442821 27580 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:45.443785 32101 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:45.444490 32101 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:45.445717 32101 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.446494 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.414937 31615 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.448091 32084 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:45.448367 32085 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.448763 32084 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:45.449190 31615 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.450047 32084 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.451766 27737 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:45.361405 32095 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.360316 32050 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 2, Time since last communication: 0.092s
I20260709 12:19:45.453133 32050 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:45.453464 32050 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:45.454340 32050 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:45.455516 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67650.7us
I20260709 12:19:45.457059 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:45.457762 31849 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.440973 31253 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.459545 32024 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:45.459698 31831 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:45.461679 31831 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:45.462045 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 677) took 1050 ms. Trace:
I20260709 12:19:45.462582 27585 rpcz_store.cc:276] 0709 12:19:44.341668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.358575 (+1016907us) service_pool.cc:224] Handling call
0709 12:19:45.376242 (+ 17667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.385544 (+  9302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.385579 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.385626 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:45.391516 (+  5890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.391582 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.391697 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.464326 27585 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:45.465397 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2325) took 1106 ms. Trace:
I20260709 12:19:45.465930 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2324) took 1107 ms. Trace:
I20260709 12:19:45.465830 27585 rpcz_store.cc:276] 0709 12:19:44.358685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.463758 (+1105073us) service_pool.cc:224] Handling call
0709 12:19:45.464108 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.465107 (+   999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.465330 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.466478 27580 rpcz_store.cc:276] 0709 12:19:44.358313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.442370 (+1084057us) service_pool.cc:224] Handling call
0709 12:19:45.442657 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.443203 (+   546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.465870 (+ 22667us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.468184 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2333) took 1031 ms. Trace:
I20260709 12:19:45.468524 27580 rpcz_store.cc:276] 0709 12:19:44.436742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.467383 (+1030641us) service_pool.cc:224] Handling call
0709 12:19:45.467682 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.467874 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.467902 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.467943 (+    41us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:45.467976 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.468033 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.468128 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.470036 27580 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:45.469980 32078 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.470842 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2338) took 1031 ms. Trace:
I20260709 12:19:45.376778 31908 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.471781 31908 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:45.441397 27583 rpcz_store.cc:276] 0709 12:19:44.357176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.439918 (+1082742us) service_pool.cc:224] Handling call
0709 12:19:45.440246 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.440465 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.440490 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.440553 (+    63us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:45.440582 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.440648 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.440768 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.473954 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2341) took 1033 ms. Trace:
I20260709 12:19:45.474318 27583 rpcz_store.cc:276] 0709 12:19:44.440674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.473121 (+1032447us) service_pool.cc:224] Handling call
0709 12:19:45.473424 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.473617 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.473647 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.473709 (+    62us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:45.473742 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.473796 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.473902 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.475406 28334 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:45.478366 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2346) took 1036 ms. Trace:
I20260709 12:19:45.448871 32085 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.464632 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 682) took 1121 ms. Trace:
I20260709 12:19:45.486909 27582 rpcz_store.cc:276] 0709 12:19:44.343125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.438283 (+1095158us) service_pool.cc:224] Handling call
0709 12:19:45.463034 (+ 24751us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.463230 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.463255 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.463293 (+    38us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:45.463338 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.464465 (+  1127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.464551 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.488047 31793 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.488658 31793 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.489065 31849 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:45.460994 32024 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.491297 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.478842 27586 rpcz_store.cc:276] 0709 12:19:44.442116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.477366 (+1035250us) service_pool.cc:224] Handling call
0709 12:19:45.477767 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.477993 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.478029 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.478082 (+    53us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:45.478120 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.478190 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.478304 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.494004 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.471207 27580 rpcz_store.cc:276] 0709 12:19:44.439708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.469552 (+1029844us) service_pool.cc:224] Handling call
0709 12:19:45.469858 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.470609 (+   751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.470792 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.505738 27588 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:19:45.509441 27585 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:45.510211 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2332) took 1075 ms. Trace:
I20260709 12:19:45.510618 27585 rpcz_store.cc:276] 0709 12:19:44.435568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.466983 (+1031415us) service_pool.cc:224] Handling call
0709 12:19:45.467257 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.509962 (+ 42705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.510155 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.378754 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.464666 31253 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:45.523224 32100 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:45.523836 32100 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:45.524797 32100 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.523255 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2347) took 1080 ms. Trace:
W20260709 12:19:45.527748 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 205956us
I20260709 12:19:45.527884 27581 rpcz_store.cc:276] 0709 12:19:44.442522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.493006 (+1050484us) service_pool.cc:224] Handling call
0709 12:19:45.507549 (+ 14543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.507737 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.507762 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.516181 (+  8419us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:19:45.523020 (+  6839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.523086 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.523202 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.528481 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.529326 27581 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:45.529901 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2360) took 1083 ms. Trace:
I20260709 12:19:45.530215 27581 rpcz_store.cc:276] 0709 12:19:44.446608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.528861 (+1082253us) service_pool.cc:224] Handling call
0709 12:19:45.529157 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.529677 (+   520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.529851 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.532344 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2362) took 1085 ms. Trace:
I20260709 12:19:45.532444 27580 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:45.533103 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2348) took 1090 ms. Trace:
I20260709 12:19:45.532756 27581 rpcz_store.cc:276] 0709 12:19:44.447170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.530983 (+1083813us) service_pool.cc:224] Handling call
0709 12:19:45.531247 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.532074 (+   827us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.532097 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.532135 (+    38us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:45.532161 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.532210 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.532299 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.533562 27580 rpcz_store.cc:276] 0709 12:19:44.442864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.516648 (+1073784us) service_pool.cc:224] Handling call
0709 12:19:45.532140 (+ 15492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.532882 (+   742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.533056 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.535468 32098 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.462081 31831 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.536360 31831 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:45.544353 27580 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 12:19:45.555841 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 294330us
I20260709 12:19:45.558136 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.559250 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.561334 27738 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
W20260709 12:19:45.566618 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:45.566612 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2355) took 1121 ms. Trace:
I20260709 12:19:45.566112 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:45.582559 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54153.3us
I20260709 12:19:45.582963 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.567474 27589 rpcz_store.cc:276] 0709 12:19:44.445253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.521160 (+1075907us) service_pool.cc:224] Handling call
0709 12:19:45.565986 (+ 44826us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.566200 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.566238 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.566294 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:45.566341 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.566411 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.566542 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:45.583580 27586 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.515970 32103 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:45.584728 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:45.589046 27736 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:45.585834 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.597224 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.581772 32007 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:45.588048 31918 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:45.607502 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.614955 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.588706 32095 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:45.590953 32098 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:45.612792 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.583671 32050 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:45.616469 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.591164 32103 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:45.564988 31906 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:19:45.632640 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.633240 27584 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.635541 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:45.644034 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.645097 27582 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:45.646188 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:45.654606 27586 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:45.657334 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.615370 31918 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.618546 31860 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:45.648142 31253 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:45.617941 32097 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:45.627146 31808 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:19:45.665006 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 322850us
I20260709 12:19:45.625994 32007 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:45.665795 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:45.676837 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:45.632510 32103 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:45.669651 32078 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:45.679562 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:45.637152 32098 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:45.679855 32094 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.682344 31253 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.711645 32038 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.711647 32102 raft_consensus.cc:2764] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
W20260709 12:19:45.711720 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46007.1us
I20260709 12:19:45.715138 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:45.664670 32085 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:45.720969 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 229773us
I20260709 12:19:45.722800 27584 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
W20260709 12:19:45.689510 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 310901us
I20260709 12:19:45.642318 31615 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.639427 32095 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:45.697522 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.693787 31808 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:45.679507 31918 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:45.695935 32086 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.134s
W20260709 12:19:45.725370 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 161506us
I20260709 12:19:45.712836 32060 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:45.715723 31253 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:45.740648 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:45.741500 27737 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.743578 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:45.715977 31363 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:45.746454 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110345us
I20260709 12:19:45.756917 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.769713 27580 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:45.731707 31808 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:45.771096 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:45.775619 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60620.1us
I20260709 12:19:45.777118 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:45.707973 32097 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.738780 32078 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:45.736353 32060 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:45.782748 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 218983us
I20260709 12:19:45.786245 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.757205 32102 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.790488 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.788271 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.796372 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:45.796784 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:45.797832 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:45.798393 31837 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.503s	user 0.007s	sys 0.000s
I20260709 12:19:45.799492 31797 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:45.799643 32121 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:45.798977 32111 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:45.800441 32085 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.800681 32111 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.796396 32097 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.785295 32078 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.801187 32111 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.800387 32121 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:45.789917 32060 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:45.798614 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.730974 31615 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.805234 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.746387 32089 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:19:45.798691 31831 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.815671 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.812335 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.816692 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:45.791564 31995 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.546s	user 0.006s	sys 0.004s
I20260709 12:19:45.818982 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.798929 31837 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.821478 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:45.801525 32078 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:45.825241 27586 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:45.810212 32085 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:45.791047 32109 raft_consensus.cc:2764] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 12:19:45.844161 32108 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:19:45.813771 32007 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.846313 32002 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:45.846902 32002 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.847637 32002 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:45.827785 32098 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:45.853039 32110 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:19:45.814052 31906 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:45.834091 32106 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:19:45.835484 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 678) took 1493 ms. Trace:
I20260709 12:19:45.837911 31798 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:45.839805 31615 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:45.846834 31793 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:45.813105 31363 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:45.817677 31417 raft_consensus.cc:2764] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:19:45.832091 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.790239 32086 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 12:19:45.911743 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:45.914453 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98883us
I20260709 12:19:45.919303 27587 rpcz_store.cc:276] 0709 12:19:44.341953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.360222 (+1018269us) service_pool.cc:224] Handling call
0709 12:19:45.385999 (+ 25777us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:45.604483 (+218484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:45.604522 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:45.650467 (+ 45945us) log.cc:844] Serialized 52 byte log entry
0709 12:19:45.763593 (+113126us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:45.763630 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.763703 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:45.835227 (+ 71524us) raft_consensus.cc:1685] finished
0709 12:19:45.835248 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.835419 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20110,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":274901,"threads_started":2,"wal-append.queue_time_us":157551}
I20260709 12:19:45.910416 32097 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:45.928915 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 151237us
I20260709 12:19:45.933560 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2342) took 1492 ms. Trace:
I20260709 12:19:45.923669 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:45.956796 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 124805us
I20260709 12:19:45.936444 27736 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:45.957384 27583 rpcz_store.cc:276] 0709 12:19:44.440988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.475193 (+1034205us) service_pool.cc:224] Handling call
0709 12:19:45.495124 (+ 19931us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:45.516363 (+ 21239us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:45.625315 (+108952us) log.cc:844] Serialized 14 byte log entry
0709 12:19:45.933069 (+307754us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:45.933139 (+    70us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:45.933203 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:45.933304 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:45.933475 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1980,"prepare.queue_time_us":220894,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":15178,"thread_start_us":364281,"threads_started":3}
I20260709 12:19:45.918583 31906 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:45.912995 32085 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:45.921861 31798 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:45.955705 31797 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:45.979951 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.978981 31798 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:45.978482 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:45.963955 32105 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:45.990200 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66669.4us
I20260709 12:19:45.991151 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.991408 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:45.967283 32101 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:45.995704 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.979023 31417 raft_consensus.cc:2764] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
I20260709 12:19:45.969046 32125 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:45.935168 32109 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
W20260709 12:19:45.990466 31908 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.517s	user 0.013s	sys 0.001s
I20260709 12:19:46.001802 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:45.913385 32095 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:46.010429 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46708.1us
W20260709 12:19:46.011945 32120 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:19:46.134958 27737 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:46.003629 32101 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:46.027501 31860 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:46.009495 32125 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.995661 32105 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.005262 31908 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.150254 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.150741 32105 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.151269 31908 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.151693 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.149263 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.152702 27664 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.152823 31797 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:46.153429 31797 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:46.154289 32125 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.155503 31860 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:46.156598 32101 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:45.961513 32097 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.156819 31860 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.024724 32132 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:46.006120 32098 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:46.126436 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.011055 32115 raft_consensus.cc:2764] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:46.002614 32119 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20260709 12:19:46.128942 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.032204 31906 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.129652 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.000664 32102 raft_consensus.cc:2764] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:19:46.132937 31253 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:46.013054 32114 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:19:46.140775 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.141633 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.149266 31837 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:46.215215 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66677.8us
I20260709 12:19:46.007647 31793 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:46.142566 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:46.160243 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4687792189999982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17090000000000002s for a response
W20260709 12:19:46.217578 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75122.6us
I20260709 12:19:46.158308 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:46.182396 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:46.194222 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.202541 32098 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.166411 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.165391 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:46.132829 31417 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:46.206311 31253 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.208357 31808 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:46.159819 32050 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.526s	user 0.015s	sys 0.002s
I20260709 12:19:46.159597 32060 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.147920 31918 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.211887 32002 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.213078 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.250944 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.166414 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.215188 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.155362 31798 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.237752 32126 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:19:46.149299 31831 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:46.259604 27664 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.243765 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.204594 32097 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:46.266461 27663 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:46.245784 31647 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:46.249960 27662 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:19:46.228051 32098 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:46.265388 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.294893 27656 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:46.295616 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.158776 32103 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.212790 32123 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.084s
I20260709 12:19:46.281739 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:46.283134 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.284108 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.284835 27589 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11.
W20260709 12:19:46.293821 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45433.4us
I20260709 12:19:46.301404 27733 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:46.281235 27659 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.302416 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:46.296438 27732 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:46.253885 32126 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:46.300236 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.305379 27734 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:46.305188 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.300531 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.215106 32085 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.300325 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.308092 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.308679 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.308616 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.309476 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.310487 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.280344 32132 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:46.329062 27661 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.329475 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.330475 27734 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.331740 27730 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:19:46.332242 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.332607 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:46.333427 31253 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:46.298061 31647 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:46.336897 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.339514 27664 raft_consensus.cc:2368] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13.
I20260709 12:19:46.340374 31253 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:46.340837 31253 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:46.341243 31253 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:46.347548 32138 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:46.348052 32138 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:46.349201 32130 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:19:46.349385 32138 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.350888 32138 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:46.311383 31559 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:19:46.351961 32131 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 12:19:46.352481 32002 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.354050 32060 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:46.310211 31793 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:46.357447 31793 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:46.360270 32078 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.521s	user 0.012s	sys 0.001s
I20260709 12:19:46.360980 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.360817 32078 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.361842 27662 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.311645 31831 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:46.326565 32101 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:19:46.348115 32128 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20260709 12:19:46.363682 27580 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:46.368197 27730 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [10.6-12.7]
W20260709 12:19:46.373315 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:46.382794 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:46.403471 27732 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:46.383886 27664 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:19:46.384836 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.386956 27661 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:19:46.391748 31253 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.390849 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:46.410858 27704 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11.
I20260709 12:19:46.411109 27733 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:46.411248 27704 leader_election.cc:403] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:19:46.412453 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:46.329298 32087 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:46.416594 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:46.419374 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 291846us
I20260709 12:19:46.421917 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.423098 31253 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.424258 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:46.424624 27737 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:19:46.418447 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:46.418843 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 224378us
I20260709 12:19:46.369029 32101 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:46.370797 32104 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:46.419354 32115 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:46.383054 31918 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:46.368320 31798 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.366668 31808 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:46.365645 32111 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.005s
I20260709 12:19:46.363288 32078 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.434689 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.421505 32002 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:46.420485 32089 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:46.418284 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 289666us
I20260709 12:19:46.419660 32087 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:46.369781 32133 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 12:19:46.420610 31417 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:46.368600 32136 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.098s
I20260709 12:19:46.427567 32103 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.507309 27583 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:19:46.500900 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70070.5us
I20260709 12:19:46.506127 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.376029 31647 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.504902 31808 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:19:46.452716 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:46.436997 32041 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 12:19:46.442831 32104 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:46.526145 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73718.9us
I20260709 12:19:46.502398 32111 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.531852 32112 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:46.532832 27587 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:46.364925 32126 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.499643 32002 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:46.472702 32115 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:46.512480 32129 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:19:46.549259 32121 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:46.549535 32087 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:45.742290 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:46.587695 31417 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.589953 32041 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:46.591228 31908 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:46.591794 31908 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:46.593297 32135 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.593922 32135 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.550398 32097 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:19:46.597234 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.599912 32153 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:19:46.553840 32146 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:46.488212 32060 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:19:46.601723 32060 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:46.602108 32060 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:46.563808 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2333) took 1589 ms. Trace:
I20260709 12:19:46.602567 32115 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.603073 32060 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.602872 27581 rpcz_store.cc:276] 0709 12:19:44.974955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:45.537563 (+562608us) service_pool.cc:224] Handling call
0709 12:19:45.927210 (+389647us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:45.927424 (+   214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:46.222756 (+295332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:46.222803 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:46.254510 (+ 31707us) log.cc:844] Serialized 26 byte log entry
0709 12:19:46.327691 (+ 73181us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:46.327736 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:46.327796 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:46.563394 (+235598us) raft_consensus.cc:1685] finished
0709 12:19:46.563415 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:46.563718 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35808,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":21937,"spinlock_wait_cycles":14095104,"thread_start_us":317824,"threads_started":2,"wal-append.queue_time_us":180159}
I20260709 12:19:46.605134 32135 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.560971 32002 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:46.471836 32132 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.472468 32101 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.609635 32101 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:46.614329 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.615935 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.616066 32098 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.616976 27661 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.618011 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.618784 27655 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:46.619230 32098 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.551309 32103 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:19:46.619897 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.620159 32103 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:46.620565 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:46.621166 32103 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.622066 27658 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:46.579931 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:46.623694 32147 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.267s	user 0.002s	sys 0.000s
I20260709 12:19:46.473325 31797 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:46.580428 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:46.601004 32104 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:46.573979 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67946us
I20260709 12:19:46.637414 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.685382 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.639086 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.686792 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:46.684185 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57894.7us
W20260709 12:19:46.675482 27701 leader_election.cc:336] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.774s (SENT)
I20260709 12:19:46.687759 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:46.682770 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55591.4us
W20260709 12:19:46.689625 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:46.694368 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.694723 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.695562 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.697019 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:46.697806 27701 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.789s (SENT)
I20260709 12:19:46.698257 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.721341 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.722275 27661 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 12:19:46.746213 27660 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:19:46.682621 31920 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:46.695202 27663 raft_consensus.cc:2376] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11. Re-sending same reply.
I20260709 12:19:46.753108 27657 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:46.753225 31559 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:46.758069 27589 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:46.760543 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.779814 32100 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:46.781939 32100 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.784658 31797 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:46.787531 31837 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:46.688390 32097 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:46.790578 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67244.4us
W20260709 12:19:46.794147 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96364.2us
I20260709 12:19:46.723757 32104 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:46.799249 27701 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.791s (SENT)
I20260709 12:19:46.799724 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:46.799959 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:46.797667 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.800859 27660 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:46.692329 32086 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.105s
I20260709 12:19:46.809079 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.782548 32100 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.763237 32111 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:46.817819 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 343248us
I20260709 12:19:46.699451 32153 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:19:46.823314 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.831342 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.831790 27738 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:19:46.832084 27661 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:46.833557 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.834013 27661 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:46.832126 27657 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11.
I20260709 12:19:46.835094 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:46.837380 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.837625 27663 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:46.840214 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.840685 31253 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:46.840919 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:46.846760 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.798580 32097 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.856709 27655 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:19:46.840431 31860 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.837174 31920 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:46.841132 32146 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:46.841338 31906 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:46.844815 31647 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:46.853159 31797 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:46.897011 27552 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14.
I20260709 12:19:46.897571 27552 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:19:46.905959 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:46.907119 27732 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:46.907156 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:46.907967 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:46.908186 27730 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:46.908564 27655 raft_consensus.cc:2376] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11. Re-sending same reply.
I20260709 12:19:46.921447 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:46.921579 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50635.4us
I20260709 12:19:46.924988 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.937759 27738 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:46.924070 27730 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:46.926352 27734 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:19:46.901918 31765 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:46.928534 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:46.925880 31797 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:46.933955 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.934687 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.872346 31647 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:46.936633 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.961163 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.902523 31417 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.926544 27732 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:46.952322 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:46.953740 27580 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:46.964332 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:46.926170 32146 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.986760 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.969914 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:47.022082 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52230.9us
I20260709 12:19:47.023114 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:46.987831 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.026135 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.988844 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:47.019486 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:46.987722 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:47.028494 27582 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:47.028430 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 233594us
I20260709 12:19:47.029764 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:46.871229 31860 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.048815 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:47.049889 27736 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:47.051765 27552 leader_election.cc:400] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13.
I20260709 12:19:47.051898 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:47.052798 27582 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:47.052983 27580 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:47.053603 27552 leader_election.cc:403] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:19:47.054185 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.037401 27584 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:47.031513 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:47.059790 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:47.060380 27588 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:46.916800 31920 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:46.903614 31906 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:47.065600 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.016s	sys 0.014s
I20260709 12:19:47.071935 31253 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.073206 31253 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:47.069046 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:47.029798 27701 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.826s (SENT)
I20260709 12:19:46.965940 32050 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:47.067996 27584 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 12:19:47.071408 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.017s	sys 0.014s
I20260709 12:19:46.965739 31793 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:47.091737 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:47.089432 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:47.125869 27655 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:47.096168 27663 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [10.6-12.7]
I20260709 12:19:47.127663 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:47.100072 27664 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:47.104738 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:47.106132 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76382.1us
I20260709 12:19:47.109279 31253 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:47.109620 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:47.064841 32135 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:47.133512 27582 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:47.110580 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.065184 32078 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:47.020248 32111 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:19:47.114811 28337 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:47.116854 28334 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:19:47.117105 28332 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:19:47.118402 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:47.118573 28328 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:19:47.118985 31920 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:47.119661 28329 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:47.120023 28330 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:19:47.132367 27701 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.897s (SENT)
I20260709 12:19:47.065411 32136 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13.
I20260709 12:19:47.134364 27657 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:47.139724 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.099807 32041 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:47.138809 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:47.140661 27588 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:47.055673 32002 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.146023 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.048352 31417 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.073319 32156 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 12:19:47.148077 27729 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:47.154028 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 244677us
I20260709 12:19:47.048206 32146 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.135805 32135 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:47.175021 27583 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:47.173677 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.185760 32115 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.147091 32167 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:47.184353 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52014.2us
I20260709 12:19:47.163272 32103 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.173599 32135 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:47.189257 32100 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.188709 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:47.188531 32165 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:19:47.164050 32041 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.177834 32098 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:19:47.208467 32138 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.824s	user 0.003s	sys 0.008s
I20260709 12:19:47.209457 27729 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:47.199446 32177 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.200598 32158 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 12:19:47.213546 32177 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.214159 32146 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.209743 32002 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.186525 32172 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:47.191612 32097 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.129125 31647 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:47.218854 32163 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:19:47.219084 28333 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:47.219233 27660 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:47.188668 32115 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.231513 32115 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:47.219934 32162 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
W20260709 12:19:47.220386 32155 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.299s	user 0.001s	sys 0.000s
I20260709 12:19:47.235306 31417 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.237082 32041 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.216244 32098 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:47.221323 32161 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:19:47.197326 32103 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.223894 31860 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.223881 32177 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.229143 27660 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:47.270792 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.010s	sys 0.010s
W20260709 12:19:47.271163 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.010s	sys 0.010s
W20260709 12:19:47.271864 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 837287us
I20260709 12:19:47.278482 27582 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:47.279011 27582 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:47.281590 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87218.2us
I20260709 12:19:47.282496 27581 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:19:47.284129 27701 leader_election.cc:336] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.956s (SENT)
I20260709 12:19:47.286787 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.204490 32112 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:19:47.287703 27583 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:47.269313 27660 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.279999 27582 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.285876 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.290560 27584 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:19:47.290874 27582 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:47.193136 32167 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.296464 27660 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:47.297482 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.298403 27581 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:19:47.294363 32089 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.791s	user 0.012s	sys 0.000s
W20260709 12:19:47.311904 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:47.321799 27582 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:19:47.330821 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.314785 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.329474 32095 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:47.287467 32098 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.294736 31906 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.299729 31860 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:47.344898 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.335882 32089 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11.
I20260709 12:19:47.349254 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1922) took 1214 ms. Trace:
I20260709 12:19:47.350297 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.351650 27662 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:19:47.352461 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.349730 27659 rpcz_store.cc:276] 0709 12:19:46.134677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.135200 (+   523us) service_pool.cc:224] Handling call
0709 12:19:47.349039 (+1213839us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:47.349192 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139600128}
I20260709 12:19:47.336591 32186 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.354903 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.342952 31918 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:19:47.348531 32101 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.738s	user 0.010s	sys 0.002s
I20260709 12:19:47.388796 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.392259 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.380614 32100 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 12:19:47.354387 32132 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:47.393494 32132 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:47.393872 32132 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:47.394044 32170 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:47.394910 31837 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:19:47.384275 32095 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.396824 32095 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:47.400024 32142 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.028s	user 0.010s	sys 0.012s
W20260709 12:19:47.400089 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:47.401283 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:47.401314 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.402397 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.380070 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.380700 32098 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:47.403795 28333 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:47.411270 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.354987 32172 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.375872 32146 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:47.399148 32095 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.428231 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.436420 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.427277 32190 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:47.380843 32177 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:19:47.416218 32136 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:47.448283 32132 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.450520 32132 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.422820 32181 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:19:47.418740 31797 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14.
I20260709 12:19:47.347009 31860 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.454835 31647 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:47.456182 32185 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:19:47.456261 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.457110 27734 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:47.451122 32078 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:47.460814 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.461534 27735 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 9: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:19:47.462384 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1920) took 1333 ms (client timeout 1571 ms). Trace:
W20260709 12:19:47.462778 27658 rpcz_store.cc:269] 0709 12:19:46.129304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.129431 (+   127us) service_pool.cc:224] Handling call
0709 12:19:47.462215 (+1332784us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:47.462331 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961117440}
I20260709 12:19:47.445294 28331 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:47.422348 32170 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:47.467520 32170 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.410274 31906 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
W20260709 12:19:47.468935 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:47.469705 32170 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.429662 32172 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.451753 32103 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.471987 27581 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:47.472751 32111 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.473268 32111 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.475488 31793 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.476190 31793 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.478720 31793 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:47.481168 32078 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:47.481873 27582 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:47.486521 27581 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:47.486809 32167 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:47.401970 31417 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:19:47.487951 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.489255 27581 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:47.490885 32098 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:47.491750 27587 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:47.492383 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.493196 27584 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:47.493849 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:19:47.494452 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.494369 32132 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:47.495139 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
W20260709 12:19:47.496436 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 140319us
W20260709 12:19:47.495374 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59054.7us
I20260709 12:19:47.497735 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.499598 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.501457 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.502156 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:47.504405 27738 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:47.505431 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.504614 31837 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:47.507493 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 127040us
I20260709 12:19:47.507766 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.502991 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.394059 32100 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.108s
I20260709 12:19:47.511129 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.511961 27658 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:19:47.503983 31797 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:47.513100 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.513350 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.513868 32190 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:19:47.514919 32182 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:19:47.514465 31908 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.511205 27587 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 9.
I20260709 12:19:47.516510 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.507535 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.523679 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.524035 27660 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:19:47.524459 27731 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:47.524883 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:47.520320 32078 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:47.526415 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:47.527597 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.528319 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:47.523022 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:47.529639 31797 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:47.530427 31906 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:47.526577 32181 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:19:47.531584 32184 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:47.511451 32170 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:47.532034 32007 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:47.532248 32112 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:19:47.533061 32193 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:19:47.532886 31918 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:19:47.533679 32089 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:47.542893 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.560325 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:47.519282 32100 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:19:47.562281 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.595889 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.564765 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.409154 32186 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.567421 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.579653 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:47.596626 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53799.8us
I20260709 12:19:47.602921 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.604110 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:47.607156 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.607880 27737 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:47.534317 32101 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:19:47.583155 32172 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:47.605509 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:47.638721 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 110475us
I20260709 12:19:47.552632 32103 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:19:47.583946 32104 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:47.583598 32095 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:47.582851 31798 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:19:47.618136 32150 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:19:47.618552 32183 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:47.619469 31918 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:47.619794 32194 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:19:47.656700 32190 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.657312 32190 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.659970 31860 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:47.667764 32135 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:47.668939 32186 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.670917 27486 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 9 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 11 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.673139 27486 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 9 to 10, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 10 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.646119 32095 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:47.676134 32095 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:47.628381 27737 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:47.603542 31797 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:47.690152 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.646732 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.591120 32170 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.645370 31837 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:47.670972 31849 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:19:47.690603 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.644342 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.694773 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:47.714305 27658 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:47.696564 27737 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:47.638208 32136 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:47.647495 32078 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:47.647687 31798 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:47.724291 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:47.728873 27737 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:47.723661 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:47.721334 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:47.735074 27589 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:47.736934 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.737881 32195 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.738230 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.695590 32102 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:47.739114 27658 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:47.714568 31797 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:47.725348 31417 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:19:47.719198 32078 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:47.721282 32190 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.717056 32149 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:19:47.738584 32195 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.749637 27731 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:19:47.744521 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:47.750560 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 190313us
W20260709 12:19:47.774433 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 209738us
I20260709 12:19:47.747665 27486 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:47.763923 32195 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.748320 32135 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:47.803392 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:47.804282 27658 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:47.760476 32078 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:47.805019 27586 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:47.811625 27583 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:47.812531 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:47.814551 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:47.816779 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.819811 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:47.743770 32102 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.802328 32104 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:47.822556 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:47.804780 31918 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:19:47.840083 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:47.992187 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 154459us
I20260709 12:19:47.995352 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.041208 27658 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:19:48.041700 27658 pending_rounds.cc:85] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 11
W20260709 12:19:48.039994 27485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 231) took 23894 ms (client timeout 15000 ms). Trace:
W20260709 12:19:48.049793 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 235317us
I20260709 12:19:48.050640 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:48.053074 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:47.847455 32206 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:47.836334 31918 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:47.815622 31798 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.041988 27658 pending_rounds.cc:107] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:19:48.079591 27658 raft_consensus.cc:2889] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:47.832333 32104 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:48.055186 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:47.832340 31797 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.056295 32089 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:48.068630 27486 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 6 to 9, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 9 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:19:48.116617 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63620.6us
I20260709 12:19:48.065259 32206 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:48.056902 31906 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:48.321617 27701 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11.
I20260709 12:19:48.068029 32085 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:48.042341 32202 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.264s
W20260709 12:19:48.411820 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 286287us
I20260709 12:19:48.322803 32170 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.119616 31417 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.260s
W20260709 12:19:48.376524 32146 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.879s	user 0.012s	sys 0.001s
I20260709 12:19:48.405486 27486 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:48.339627 31906 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:48.438505 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 443234us
I20260709 12:19:48.341013 32206 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:48.447124 32168 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
I20260709 12:19:48.447556 32209 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.408s
I20260709 12:19:48.449506 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.322446 32169 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.631s
I20260709 12:19:48.464931 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.462909 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:48.429229 32085 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:19:48.481901 32205 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
W20260709 12:19:48.484277 32212 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:19:48.486009 32214 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:19:48.449942 32146 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:48.487973 32146 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:48.487470 27486 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 7 to 8, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:48.488593 32146 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:48.473361 32085 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:48.490753 32085 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.491959 32085 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:48.046083 27485 rpcz_store.cc:269] 0709 12:19:23.924626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:23.962599 (+ 37973us) service_pool.cc:224] Handling call
0709 12:19:47.811038 (+23848439us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:38.884416 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:39.001171 (+116755us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:39.001214 (+    43us) write_op.cc:435] Acquired schema lock
0709 12:19:39.001232 (+    18us) tablet.cc:662] Decoding operations
0709 12:19:39.002389 (+  1157us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:39.002446 (+    57us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:39.002476 (+    30us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:19:39.135608 (+133132us) tablet.cc:701] Row locks acquired
0709 12:19:39.135704 (+    96us) write_op.cc:260] PREPARE: finished
0709 12:19:39.159932 (+ 24228us) write_op.cc:270] Start()
0709 12:19:39.165358 (+  5426us) write_op.cc:276] Timestamp: P: 1783599579144352 usec, L: 0
0709 12:19:39.165384 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:19:39.170930 (+  5546us) log.cc:844] Serialized 4764 byte log entry
0709 12:19:39.660480 (+489550us) op_driver.cc:464] REPLICATION: finished
0709 12:19:40.562542 (+902062us) spinlock_profiling.cc:243] Waited 56.5 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e9723e0ac 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e9dd7eef0
0709 12:19:40.779699 (+217157us) write_op.cc:301] APPLY: starting
0709 12:19:40.779822 (+   123us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:40.915535 (+135713us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:40.915583 (+    48us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:41.426903 (+511320us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:41.433214 (+  6311us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=24,mrs_lookups=0
0709 12:19:41.449853 (+ 16639us) write_op.cc:312] APPLY: finished
0709 12:19:41.491469 (+ 41616us) log.cc:844] Serialized 113 byte log entry
0709 12:19:41.586575 (+ 95106us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:41.605566 (+ 18991us) write_op.cc:454] Released schema lock
0709 12:19:41.652353 (+ 46787us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:41.759174 (+106821us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:41.762312 (+  3138us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":274552,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":1113338,"apply.run_cpu_time_us":34682,"apply.run_wall_time_us":923870,"cfile_cache_hit":46,"cfile_cache_hit_bytes":48070,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6217,"cfile_init":2,"lbm_read_time_us":25496,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":10,"num_ops":12,"prepare.queue_time_us":205127,"prepare.run_cpu_time_us":36657,"prepare.run_wall_time_us":463183,"raft.queue_time_us":237597,"raft.run_cpu_time_us":44679,"raft.run_wall_time_us":1199428,"replication_time_us":420729,"spinlock_wait_cycles":130055040,"thread_start_us":1482612,"threads_started":5,"wal-append.queue_time_us":407648,"wal-append.run_cpu_time_us":111352,"wal-append.run_wall_time_us":2620372}]]}
W20260709 12:19:48.495751 28335 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.754s	user 0.007s	sys 0.003s
I20260709 12:19:48.497246 28335 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
W20260709 12:19:48.500188 32200 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.618s	user 0.001s	sys 0.000s
W20260709 12:19:48.503922 32207 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.683s	user 0.001s	sys 0.000s
I20260709 12:19:48.504863 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2239) took 1113 ms. Trace:
I20260709 12:19:48.505821 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 677) took 1116 ms. Trace:
I20260709 12:19:48.506227 27731 rpcz_store.cc:276] 0709 12:19:47.389539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.762657 (+373118us) service_pool.cc:224] Handling call
0709 12:19:47.763182 (+   525us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:47.763743 (+   561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:47.763791 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:47.769487 (+  5696us) log.cc:844] Serialized 52 byte log entry
0709 12:19:47.941849 (+172362us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:47.941977 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:47.942209 (+   232us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:48.505617 (+563408us) raft_consensus.cc:1685] finished
0709 12:19:48.505635 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.505770 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":165,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":121041,"threads_started":1,"wal-append.queue_time_us":244030}
I20260709 12:19:48.508991 27731 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:48.509738 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2241) took 1118 ms. Trace:
I20260709 12:19:48.510159 27731 rpcz_store.cc:276] 0709 12:19:47.391769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.508351 (+1116582us) service_pool.cc:224] Handling call
0709 12:19:48.508761 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.509457 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.509674 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:48.448424 32203 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.171s
I20260709 12:19:48.450917 31906 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:48.513728 32218 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:48.514135 32218 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:48.514990 32218 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.516242 32218 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:48.505282 27730 rpcz_store.cc:276] 0709 12:19:47.391287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.503913 (+1112626us) service_pool.cc:224] Handling call
0709 12:19:48.504234 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.504448 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:48.504471 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:48.504526 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:48.504576 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.504649 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.504797 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:48.520895 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58057.3us
I20260709 12:19:48.522240 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:48.523523 27731 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:48.524593 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2242) took 1132 ms. Trace:
I20260709 12:19:48.525084 27731 rpcz_store.cc:276] 0709 12:19:47.391977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.511050 (+1119073us) service_pool.cc:224] Handling call
0709 12:19:48.523277 (+ 12227us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.524281 (+  1004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.524523 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:48.517359 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 673) took 1019 ms. Trace:
I20260709 12:19:48.411137 32138 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.532656 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:48.435590 32190 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:48.430377 32216 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:48.438753 32136 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.560325 27731 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:19:48.543633 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 679) took 1115 ms. Trace:
I20260709 12:19:48.451660 31417 consensus_queue.cc:1243] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 13.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:19:48.541460 32085 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:48.527055 27655 rpcz_store.cc:276] 0709 12:19:47.497906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.503874 (+  5968us) service_pool.cc:224] Handling call
0709 12:19:47.513208 (+  9334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:47.533013 (+ 19805us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:47.533066 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:47.533593 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:19:47.635032 (+101439us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:47.635078 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:47.635153 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:48.517065 (+881912us) raft_consensus.cc:1685] finished
0709 12:19:48.517093 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.517266 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42069,"raft.queue_time_us":37143,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8993,"thread_start_us":100987,"threads_started":1,"wal-append.queue_time_us":210606}
W20260709 12:19:48.562461 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.561717 28334 rpcz_store.cc:276] 0709 12:19:47.420048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.421007 (+   959us) service_pool.cc:224] Handling call
0709 12:19:47.421415 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:47.421805 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:47.421836 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:47.440041 (+ 18205us) log.cc:844] Serialized 26 byte log entry
0709 12:19:47.688475 (+248434us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x7b18003cbe00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:19:47.711502 (+ 23027us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:47.711630 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:47.711816 (+   186us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:48.518184 (+806368us) raft_consensus.cc:1685] finished
0709 12:19:48.518211 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.535106 (+ 16895us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":6619,"spinlock_wait_cycles":129563520,"thread_start_us":22647,"threads_started":1,"wal-append.queue_time_us":327344}
I20260709 12:19:48.559577 32190 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.567842 32220 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:19:48.569588 32104 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.571702 32136 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.604835 32060 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:48.605953 32136 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:48.571734 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2247) took 1160 ms. Trace:
I20260709 12:19:48.573571 31918 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.530254 32202 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
W20260709 12:19:48.582896 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.558769 32169 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:48.586510 32213 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:19:48.588231 27486 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 3 to 4, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.551537 32170 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
W20260709 12:19:48.595270 32204 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.472s	user 0.001s	sys 0.000s
W20260709 12:19:48.597630 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.597697 32206 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.599005 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2328) took 1903 ms. Trace:
W20260709 12:19:48.599660 32183 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.929s	user 0.008s	sys 0.003s
I20260709 12:19:48.616568 32183 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:48.616325 27662 rpcz_store.cc:276] 0709 12:19:46.695691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.698908 (+  3217us) service_pool.cc:224] Handling call
0709 12:19:47.351404 (+652496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:47.352037 (+   633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.598689 (+1246652us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b5400041d50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:48.598927 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1499856256}
W20260709 12:19:48.579282 32095 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.903s	user 0.009s	sys 0.003s
I20260709 12:19:48.582909 32085 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:48.621335 32085 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:48.626026 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 677) took 1508 ms (client timeout 1846 ms). Trace:
I20260709 12:19:48.602785 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:48.626533 32206 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.602989 32190 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.609055 27731 rpcz_store.cc:276] 0709 12:19:47.411455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.526144 (+1114689us) service_pool.cc:224] Handling call
0709 12:19:48.559863 (+ 33719us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.571470 (+ 11607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.571678 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:48.561409 32102 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.611315 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:48.613013 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.547773 32138 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.614398 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:48.600896 32078 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.789s	user 0.010s	sys 0.000s
I20260709 12:19:48.634088 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
W20260709 12:19:48.626475 28335 rpcz_store.cc:269] 0709 12:19:47.118000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.118132 (+   132us) service_pool.cc:224] Handling call
0709 12:19:48.625849 (+1507717us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b5400611054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:48.625962 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1372804864}
W20260709 12:19:48.738299 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 125366us
W20260709 12:19:48.703208 27627 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.672s (SENT) [suppressed 16 similar messages]
W20260709 12:19:48.632529 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2231) took 1700 ms (client timeout 1672 ms). Trace:
I20260709 12:19:48.694702 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1940) took 1598 ms. Trace:
W20260709 12:19:48.631409 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.000s	sys 0.010s
W20260709 12:19:48.741103 27738 rpcz_store.cc:269] 0709 12:19:46.931769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.970557 (+ 38788us) service_pool.cc:224] Handling call
0709 12:19:48.632290 (+1661733us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54001d3dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:48.632418 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":750732544}
W20260709 12:19:48.741910 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.001s	sys 0.014s
I20260709 12:19:48.743361 27738 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:48.745604 32105 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:19:48.745826 32101 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.746425 32101 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.747176 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.748376 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.749495 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.737176 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:48.738541 27587 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.631875 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2360) took 1161 ms. Trace:
I20260709 12:19:48.626755 27580 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:48.630527 27660 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:19:48.666136 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:48.736939 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 108378us
W20260709 12:19:48.707180 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:48.743021 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.670275 31908 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:48.750741 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2010) took 1062 ms. Trace:
I20260709 12:19:48.790124 27738 rpcz_store.cc:276] 0709 12:19:47.688386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.742504 (+1054118us) service_pool.cc:224] Handling call
0709 12:19:48.742849 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.745341 (+  2492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.750664 (+  5323us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:48.766250 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.741786 27664 rpcz_store.cc:276] 0709 12:19:47.096504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.135993 (+ 39489us) service_pool.cc:224] Handling call
0709 12:19:47.165867 (+ 29874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:47.237871 (+ 72004us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:47.286547 (+ 48676us) log.cc:844] Serialized 14 byte log entry
0709 12:19:47.551837 (+265290us) log.cc:844] Serialized 14 byte log entry
0709 12:19:47.553533 (+  1696us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:47.553580 (+    47us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:47.553624 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:47.678275 (+124651us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b18002c5480. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:48.693933 (+1015658us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b18002c5480. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:48.694390 (+   457us) spinlock_profiling.cc:243] Waited 71.8 ms on lock 0x7b54001e46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:48.694443 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.694623 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28924,"prepare.queue_time_us":142947,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":6135,"raft.queue_time_us":29692,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":1778,"spinlock_wait_cycles":239136384,"thread_start_us":284630,"threads_started":2,"wal-append.queue_time_us":302820,"wal-append.run_cpu_time_us":2289,"wal-append.run_wall_time_us":1077953}
I20260709 12:19:48.773906 32102 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:48.747982 32105 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:48.696997 32126 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:19:48.690840 32111 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:48.776288 27656 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 5.
I20260709 12:19:48.771756 32085 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.810029 32085 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.795506 31765 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:19:48.788190 32223 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:48.774511 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.807883 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 1951) took 1161 ms. Trace:
I20260709 12:19:48.776886 27587 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:19:48.666272 32104 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.691792 32098 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:48.665630 32169 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.665858 32203 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:48.666028 32220 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.666610 32225 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.691893 32181 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:48.815218 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.814698 27656 rpcz_store.cc:276] 0709 12:19:47.646022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.657285 (+ 11263us) service_pool.cc:224] Handling call
0709 12:19:48.665969 (+1008684us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:19:48.807793 (+141824us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76021504}
I20260709 12:19:48.666211 31918 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.776877 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1945) took 1135 ms. Trace:
I20260709 12:19:48.851744 27660 rpcz_store.cc:276] 0709 12:19:47.641698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.642746 (+  1048us) service_pool.cc:224] Handling call
0709 12:19:48.630058 (+987312us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:48.630208 (+   150us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.776600 (+146392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.776822 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42843776}
I20260709 12:19:48.813937 32060 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.774587 27582 rpcz_store.cc:276] 0709 12:19:47.470610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.470893 (+   283us) service_pool.cc:224] Handling call
0709 12:19:47.471219 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:47.482262 (+ 11043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.631547 (+1149285us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b54000893d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:48.631799 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24081152}
I20260709 12:19:48.818183 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:48.820566 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 355710us
I20260709 12:19:48.821756 27738 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:48.749673 32078 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.798297 32102 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:48.827006 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:48.796734 32121 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:19:48.829206 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:48.829463 32225 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.830207 28332 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:19:48.830845 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:48.833145 32203 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.842905 27659 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:48.826462 32169 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.845993 32138 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.814347 31765 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:48.827744 32157 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:19:48.812927 31906 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.880987 32225 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.789253 32115 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:19:48.817730 32098 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:48.887673 32098 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.888708 32098 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:48.889043 32203 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.890007 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:48.890615 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:48.798781 32119 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 12:19:48.854034 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2011) took 1165 ms. Trace:
I20260709 12:19:48.867700 32158 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:48.891808 27738 rpcz_store.cc:276] 0709 12:19:47.688709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.791424 (+1102715us) service_pool.cc:224] Handling call
0709 12:19:48.821562 (+ 30138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.853792 (+ 32230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.853981 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:48.892228 32158 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:48.893009 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:48.893821 32007 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:48.893347 32158 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.894181 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:48.894582 27737 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:19:48.894811 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:48.658521 32095 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.850615 32220 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.896193 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2238) took 1505 ms. Trace:
I20260709 12:19:48.896538 27737 rpcz_store.cc:276] 0709 12:19:47.391037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.765277 (+374240us) service_pool.cc:224] Handling call
0709 12:19:47.765646 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.894914 (+1129268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.895961 (+  1047us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:48.896130 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2595500800}
I20260709 12:19:48.820502 32105 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:48.825265 32202 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.821048 32104 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.897557 27733 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:48.897712 32220 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.872081 32103 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.898847 32103 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.897594 32202 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.899597 32104 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.849998 32085 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.900452 32085 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.901207 27737 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:48.901279 32202 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.879532 31765 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:48.882918 28334 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 7: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:48.886694 31906 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.891230 32169 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.854158 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.896935 32095 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:48.860913 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:48.912027 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57973.8us
W20260709 12:19:48.912426 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51635.9us
I20260709 12:19:48.912762 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.913069 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.913120 31417 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:48.914109 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:48.915212 32115 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:48.915661 31797 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:48.915772 32115 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:48.916112 31797 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:48.916501 31797 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:48.916764 32115 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.917222 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 591) took 2233 ms. Trace:
I20260709 12:19:48.918066 32138 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:48.917711 28331 rpcz_store.cc:276] 0709 12:19:46.684183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.688105 (+  3922us) service_pool.cc:224] Handling call
0709 12:19:46.753074 (+ 64969us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:47.445124 (+692050us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b54005378a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:19:47.464212 (+ 19088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.916860 (+1452648us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54005378a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34ba02 00007f0e9a347dd6 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:19:48.917123 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400971904}
I20260709 12:19:48.918782 32138 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:48.919792 32138 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.920171 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:48.914875 32218 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:48.921674 32078 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:48.922246 32227 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:48.923552 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:48.924180 32210 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.808s	user 0.001s	sys 0.004s
I20260709 12:19:48.925470 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 684) took 1290 ms. Trace:
I20260709 12:19:48.925833 28336 rpcz_store.cc:276] 0709 12:19:47.634915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.635167 (+   252us) service_pool.cc:224] Handling call
0709 12:19:47.643102 (+  7935us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:47.736295 (+ 93193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:47.736328 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:47.736903 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:19:47.834522 (+ 97619us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:47.834584 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:47.834687 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:48.923861 (+1089174us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b2400318008. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:19:48.925190 (+  1329us) raft_consensus.cc:1685] finished
0709 12:19:48.925211 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.925410 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80220,"raft.queue_time_us":94695,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"spinlock_wait_cycles":33439104,"thread_start_us":97191,"threads_started":1,"wal-append.queue_time_us":120037}
I20260709 12:19:48.924850 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:48.902822 32085 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:48.894217 32007 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:48.907519 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 681) took 1362 ms. Trace:
I20260709 12:19:48.939181 27732 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:48.909035 27737 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:48.938602 28333 rpcz_store.cc:276] 0709 12:19:47.544901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.610147 (+ 65246us) service_pool.cc:224] Handling call
0709 12:19:47.610530 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:47.638161 (+ 27631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:47.638195 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:47.646180 (+  7985us) log.cc:844] Serialized 26 byte log entry
0709 12:19:47.706633 (+ 60453us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:47.829103 (+122470us) log.cc:844] Serialized 14 byte log entry
0709 12:19:47.829353 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.907058 (+1077705us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b54005a2654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:48.907110 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:48.907166 (+    56us) raft_consensus.cc:1685] finished
0709 12:19:48.907264 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.907445 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70873,"prepare.queue_time_us":285929,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":3106,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":10637,"spinlock_wait_cycles":45182848,"thread_start_us":159242,"threads_started":2,"wal-append.queue_time_us":79421}
I20260709 12:19:48.948025 32218 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:48.947595 32007 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.948518 32218 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:48.948913 32218 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:48.949008 32007 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:48.949625 32135 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:48.862190 32060 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.859812 31918 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.953713 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2355) took 2324 ms. Trace:
I20260709 12:19:48.868199 32102 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.855247 32181 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:48.957994 27737 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:19:48.958482 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 255304us
W20260709 12:19:48.964033 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2229) took 2032 ms (client timeout 1783 ms). Trace:
I20260709 12:19:48.964275 27729 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 12:19:48.964478 27736 rpcz_store.cc:269] 0709 12:19:46.931193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.952989 (+ 21796us) service_pool.cc:224] Handling call
0709 12:19:48.963855 (+2010866us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:48.963968 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49851648}
I20260709 12:19:48.954130 27585 rpcz_store.cc:276] 0709 12:19:46.626478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.626924 (+   446us) service_pool.cc:224] Handling call
0709 12:19:46.627244 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:46.696725 (+ 69481us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:48.412286 (+1715561us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b18002c0d40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:19:48.544634 (+132348us) log.cc:844] Serialized 14 byte log entry
0709 12:19:48.854888 (+310254us) log.cc:844] Serialized 14 byte log entry
0709 12:19:48.855461 (+   573us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:48.870602 (+ 15141us) log.cc:844] Serialized 26 byte log entry
0709 12:19:48.870760 (+   158us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:48.870800 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.870883 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:48.948894 (+ 78011us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b18002c0d40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:48.950524 (+  1630us) raft_consensus.cc:1685] finished
0709 12:19:48.950549 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:48.950693 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3953,"prepare.queue_time_us":459384,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":24451,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":40306,"spinlock_wait_cycles":40135936,"thread_start_us":438208,"threads_started":2,"wal-append.queue_time_us":402577}
W20260709 12:19:48.965741 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.966878 27736 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:48.966796 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:48.969614 27732 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:19:48.969517 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:48.980201 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:48.978538 32105 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:48.984733 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:48.986372 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:49.004902 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:49.005142 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:49.015218 28337 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:49.003732 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:49.013553 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:48.986439 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.022526 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:49.024081 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:49.027802 32223 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:49.028080 31559 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:49.028272 32119 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:49.028493 32203 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.028941 32119 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.029208 32203 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.004638 32138 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:49.031689 32095 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:49.032187 32095 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:49.031886 32138 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.032930 31918 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:49.033125 32095 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.034273 32138 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:49.034827 32102 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:49.031915 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:49.035202 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:49.035574 32222 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:49.036391 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:49.037707 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57469.7us
I20260709 12:19:49.037811 32220 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:49.011952 32177 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:49.010069 32038 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.044735 32177 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.044942 32038 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:48.984236 32228 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.034394 32095 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:49.040593 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54268.9us
I20260709 12:19:48.950197 32135 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.041428 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:49.043895 27588 pending_rounds.cc:85] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 10
I20260709 12:19:49.070417 27588 pending_rounds.cc:107] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:19:49.070838 27588 raft_consensus.cc:2889] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:49.044086 32060 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.048880 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.087592 27586 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:19:49.088662 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:49.050489 27584 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:19:49.040073 32160 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:19:49.028128 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:49.052179 28337 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:49.047025 32177 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.053916 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:49.058105 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.096973 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:48.999847 32105 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:49.070487 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:49.106549 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:48.974000 32157 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.113672 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:49.097749 27585 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:49.115581 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.069211 32169 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:49.099058 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.121582 27587 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
W20260709 12:19:49.100818 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46977.2us
I20260709 12:19:48.984571 31908 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:19:49.102255 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:49.103559 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:48.967823 32183 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 12:19:48.984836 32149 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:49.090953 32227 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:49.097793 28336 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:19:49.093317 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.109525 32105 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:49.041050 32181 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.028679 31559 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:49.061285 32223 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:49.069372 32138 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:49.069031 31793 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:49.047762 32228 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.108131 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.125020 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.063469 32126 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:19:49.119442 31831 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:49.034952 32218 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.116843 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.154748 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.125094 32203 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:49.144127 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:49.134382 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:49.122709 32233 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:49.104250 32133 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:19:49.145735 27584 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:49.145291 27583 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 6.
I20260709 12:19:49.112332 32102 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:19:49.155763 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:49.107717 32086 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:49.127048 32149 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:19:49.161566 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.134346 31559 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:49.174685 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:49.119922 32157 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.133919 32219 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:49.176854 27585 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:49.162835 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.130697 32119 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:49.183140 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.184388 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.134076 31908 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:19:49.128845 31765 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:49.190821 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100326us
I20260709 12:19:49.132413 32227 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.134790 32190 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:19:49.163092 32138 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:49.153043 32181 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260709 12:19:49.259898 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:49.247136 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 186119us
I20260709 12:19:49.254266 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:49.239437 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76692us
I20260709 12:19:49.309626 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:49.180629 32203 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.310245 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.176674 32104 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:19:49.310616 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.180073 32086 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:49.180198 31831 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:49.311400 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:49.315346 32089 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:49.315616 32228 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.319933 27584 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:19:49.317734 27587 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:19:49.323047 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:49.333549 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.334252 27587 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:49.238471 31837 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.254027 32174 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:49.335716 27585 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:49.237375 31793 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 12:19:49.324808 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.238317 32234 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.338979 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.198810 32222 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:49.297327 32038 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:49.298699 32220 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.344374 27585 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:49.340745 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.345561 27587 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
W20260709 12:19:49.327006 32214 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:19:49.323362 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 188420us
I20260709 12:19:49.326452 32086 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:49.323405 32104 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:49.332108 32150 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:49.382292 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57533.4us
I20260709 12:19:49.362043 27585 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 4. Current term is 5. Ops: []
I20260709 12:19:49.336057 32167 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:49.383074 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:49.326136 32089 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:49.382951 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:49.340664 32102 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:19:49.503561 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 104036us
I20260709 12:19:49.504776 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.505424 27585 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:19:49.506543 27585 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:19:49.340432 32078 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:49.507597 27585 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:49.510154 27585 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:49.511384 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:49.512050 27585 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:19:49.317741 32229 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:19:49.514736 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:49.368938 31765 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:49.381994 32104 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:19:49.513137 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:49.521540 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:49.533991 27585 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:49.387137 32150 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:49.326628 32203 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.387220 32038 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:49.537869 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:49.333308 32223 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.537412 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:49.547348 32237 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.363s	user 0.000s	sys 0.001s
I20260709 12:19:49.537645 32202 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.549391 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2248) took 2123 ms. Trace:
I20260709 12:19:49.556224 27585 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
W20260709 12:19:49.543121 32242 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
W20260709 12:19:49.543354 32235 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.247s	user 0.000s	sys 0.000s
I20260709 12:19:49.346933 32228 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.170s
I20260709 12:19:49.389283 32135 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:49.386447 32167 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:49.340267 31837 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.513828 32126 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:19:49.364368 32250 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.193s
I20260709 12:19:49.502763 32102 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:49.556715 27585 pending_rounds.cc:85] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 4
I20260709 12:19:49.579882 27585 pending_rounds.cc:107] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:19:49.580257 27585 raft_consensus.cc:2889] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:49.342906 32234 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.522404 32078 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.556284 27731 rpcz_store.cc:276] 0709 12:19:47.426387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.750069 (+1323682us) service_pool.cc:224] Handling call
0709 12:19:48.750417 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:48.774657 (+ 24240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:48.774706 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:48.792401 (+ 17695us) log.cc:844] Serialized 26 byte log entry
0709 12:19:48.952202 (+159801us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:48.952272 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.952364 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:49.549162 (+596798us) raft_consensus.cc:1685] finished
0709 12:19:49.549185 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:49.549328 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37595,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7184,"thread_start_us":158180,"threads_started":1,"wal-append.queue_time_us":337760}
I20260709 12:19:49.513399 32169 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:49.563880 32116 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:19:49.379441 32220 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.544286 32150 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:49.537189 32085 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:49.364529 31831 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:49.346861 32222 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:49.572129 32098 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.681s	user 0.008s	sys 0.002s
I20260709 12:19:49.563508 32219 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:49.592777 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 680) took 2168 ms. Trace:
I20260709 12:19:49.572830 32167 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:49.691701 27734 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
W20260709 12:19:49.625823 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 240364us
I20260709 12:19:49.581449 32102 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:49.566316 32202 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.564862 32119 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:49.591454 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:49.591310 32243 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
W20260709 12:19:49.607080 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:49.708866 31797 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.792s	user 0.005s	sys 0.007s
I20260709 12:19:49.709338 31797 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.710230 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:49.710515 27586 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:49.710991 27661 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:49.711767 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:49.560212 32089 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.712405 27662 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:49.713747 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:49.714366 27656 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:49.714488 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:49.715161 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:49.715457 27658 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.716257 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:49.716816 27656 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:49.618692 31808 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:49.591598 32116 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:49.594354 32259 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:49.592646 32218 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:49.591935 32245 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 12:19:49.751807 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:49.752633 27662 raft_consensus.cc:2376] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11. Re-sending same reply.
W20260709 12:19:49.603507 32241 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.311s	user 0.000s	sys 0.000s
I20260709 12:19:49.698300 31253 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:49.760835 31253 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:49.700299 31798 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:49.702176 32103 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.704432 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2358) took 2234 ms. Trace:
I20260709 12:19:49.671537 27735 rpcz_store.cc:276] 0709 12:19:47.425967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.627511 (+1201544us) service_pool.cc:224] Handling call
0709 12:19:48.627867 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:48.670480 (+ 42613us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:48.670517 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:48.671015 (+   498us) log.cc:844] Serialized 26 byte log entry
0709 12:19:48.951912 (+280897us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:48.951969 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:48.952052 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:49.592551 (+640499us) raft_consensus.cc:1685] finished
0709 12:19:49.592573 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:49.592708 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65824,"raft.queue_time_us":45123,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":50269,"thread_start_us":280489,"threads_started":1,"wal-append.queue_time_us":398990}
I20260709 12:19:49.566073 32138 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:49.681810 32007 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.732s	user 0.008s	sys 0.004s
I20260709 12:19:49.712360 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:49.777416 27661 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:19:49.716374 27658 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:19:49.588583 27585 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []
I20260709 12:19:49.584239 32169 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:49.672911 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.002s	sys 0.005s
W20260709 12:19:49.780699 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.003s	sys 0.006s
W20260709 12:19:49.781097 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 725730us
I20260709 12:19:49.588583 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:49.592381 32227 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:49.720289 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.006s	sys 0.003s
W20260709 12:19:49.796999 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.006s	sys 0.004s
I20260709 12:19:49.707311 32098 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
W20260709 12:19:49.797610 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 775141us
I20260709 12:19:49.573206 32135 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:49.615201 32255 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.303s	user 0.000s	sys 0.002s
I20260709 12:19:49.597138 31837 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:49.599545 32220 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:49.797691 31253 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.810019 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:49.808260 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:49.718499 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 335641us
I20260709 12:19:49.811295 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.625334 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:49.619798 32203 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:49.744374 27660 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:49.571722 32126 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:49.717875 31417 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.804s	user 0.013s	sys 0.000s
W20260709 12:19:49.721683 32195 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.006s	sys 0.003s
I20260709 12:19:49.745348 32116 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:19:49.825652 32126 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:49.832785 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:49.769863 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:49.830943 32195 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.007s	sys 0.003s
I20260709 12:19:49.719465 31808 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:49.835528 32203 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:49.769863 27589 rpcz_store.cc:276] 0709 12:19:47.469977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.491506 (+ 21529us) service_pool.cc:224] Handling call
0709 12:19:47.491851 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:47.735634 (+243783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:49.673943 (+1938309us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:49.704347 (+ 30404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":558863744}
I20260709 12:19:49.786648 27656 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:49.745563 32157 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.718384 32119 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.693464 32219 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.820654 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:49.876791 27658 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:49.876142 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 593) took 1265 ms. Trace:
I20260709 12:19:49.877985 28334 rpcz_store.cc:276] 0709 12:19:48.610791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.610962 (+   171us) service_pool.cc:224] Handling call
0709 12:19:49.836648 (+1225686us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400520dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:49.875945 (+ 39297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":326297984}
I20260709 12:19:49.825800 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:49.765522 32259 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:49.831758 31253 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:49.830833 31417 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:49.844478 32267 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:49.804244 27664 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:49.858660 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:49.837874 32266 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 12:19:49.795485 32169 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:49.878932 27735 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:49.919680 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:49.881683 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:49.918874 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:49.921284 27661 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:49.924597 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 686) took 2253 ms. Trace:
I20260709 12:19:49.880754 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:49.932608 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97274.1us
I20260709 12:19:49.925099 28329 rpcz_store.cc:276] 0709 12:19:47.670802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.440943 (+770141us) service_pool.cc:224] Handling call
0709 12:19:48.448166 (+  7223us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:48.839443 (+391277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:48.839498 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:48.840192 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 12:19:49.082451 (+242259us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:49.082513 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:49.082634 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:49.924314 (+841680us) raft_consensus.cc:1685] finished
0709 12:19:49.924341 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:49.924514 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":217815,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"thread_start_us":619365,"threads_started":2,"wal-append.queue_time_us":294028}
I20260709 12:19:49.806494 32135 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:49.939946 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:49.943996 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:49.944873 27656 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:49.946630 27581 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:49.912995 32218 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.935531 32203 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:49.911409 32157 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:49.931614 31793 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:49.936730 32098 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:49.939222 32190 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:49.996023 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47051.8us
I20260709 12:19:49.939002 32183 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.016037 27658 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:49.932783 32085 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:19:49.936597 32261 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:49.947521 32078 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:49.951719 31417 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.023398 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.051869 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:50.052096 32105 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.907s	user 0.006s	sys 0.007s
I20260709 12:19:50.053020 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.006639 32222 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.008044 32007 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.017045 31793 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.052495 32105 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.049374 32041 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:19:49.995290 32218 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:50.069963 32265 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:19:50.060097 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.074754 27735 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:50.047243 27661 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
W20260709 12:19:50.073581 32263 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:19:50.040201 31797 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:50.125159 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:50.018854 32264 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260709 12:19:50.082614 32135 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.082839 31793 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.085171 32101 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.034422 32183 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.086503 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:50.131830 27583 raft_consensus.cc:2368] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10.
I20260709 12:19:50.132809 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:50.133220 32183 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.008813 32203 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.134634 27704 leader_election.cc:400] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10.
I20260709 12:19:50.134945 27704 leader_election.cc:403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:19:50.135735 32085 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:50.135828 32203 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.098516 32102 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.080096 32220 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.091965 32105 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.095391 32150 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.109933 32119 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.111111 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:49.990571 31808 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:50.094858 32149 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.858s	user 0.006s	sys 0.004s
I20260709 12:19:50.067876 31831 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.066363 32202 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:50.166795 32095 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.077s	user 0.012s	sys 0.006s
I20260709 12:19:50.055661 32181 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:19:50.073827 32266 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.065217 27656 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:50.182747 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.143841 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.090407 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.053319 32261 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.147229 32259 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.148262 32100 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:19:50.125187 31559 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.888s	user 0.005s	sys 0.009s
I20260709 12:19:50.150902 32126 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.151038 32233 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.077869 32041 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:50.061782 31417 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:50.157110 27657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 675) took 2638 ms (client timeout 1883 ms). Trace:
I20260709 12:19:50.210925 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:50.210796 27657 rpcz_store.cc:269] 0709 12:19:47.499953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.500172 (+   219us) service_pool.cc:224] Handling call
0709 12:19:50.138614 (+2638442us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:50.138754 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2588425088}
W20260709 12:19:50.158890 32104 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.608s	user 0.009s	sys 0.000s
I20260709 12:19:50.159361 32220 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.138161 32219 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.040516 32190 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.160977 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.072604 32222 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 12:19:50.116961 32267 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.165012 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 690) took 1161 ms. Trace:
I20260709 12:19:50.077535 32103 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 12:19:50.074385 32007 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:50.168629 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57610.8us
I20260709 12:19:50.169862 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2407) took 1056 ms. Trace:
I20260709 12:19:50.232198 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:50.170773 27658 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:50.133492 27583 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:50.165940 32218 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:50.173599 32086 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.785s	user 0.012s	sys 0.000s
I20260709 12:19:50.185325 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:50.169584 31808 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:50.173609 32203 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:50.239499 32203 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.240043 32203 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:50.239943 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.241665 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.242408 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:50.181263 31797 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.243440 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.243721 31797 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.244478 27661 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:50.217372 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:50.221760 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:50.247063 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.247010 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.247752 27661 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:50.248214 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.222563 28331 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 8. Ops: []
I20260709 12:19:50.175153 32149 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.255892 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.224613 27736 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:50.203125 32100 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:50.203469 32150 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.208840 32233 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:50.176070 31831 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.230574 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:50.261686 28329 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:50.226492 28328 rpcz_store.cc:276] 0709 12:19:49.003693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.037315 (+ 33622us) service_pool.cc:224] Handling call
0709 12:19:49.037646 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:49.052948 (+ 15302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:49.052979 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:49.053019 (+    40us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:49.053049 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.164734 (+1111685us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.164789 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.164942 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34628352}
I20260709 12:19:50.220662 32183 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:50.218950 32269 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.236169 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:50.237989 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.231938 27588 rpcz_store.cc:276] 0709 12:19:49.113035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.702537 (+589502us) service_pool.cc:224] Handling call
0709 12:19:49.703009 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:49.703687 (+   678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:49.703728 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:49.707915 (+  4187us) log.cc:844] Serialized 26 byte log entry
0709 12:19:49.977995 (+270080us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:49.978047 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:49.978178 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:50.169643 (+191465us) raft_consensus.cc:1685] finished
0709 12:19:50.169664 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.169794 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":341,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":2575,"thread_start_us":269637,"threads_started":1,"wal-append.queue_time_us":457902}
I20260709 12:19:50.248957 27656 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:50.208455 31559 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.272184 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:50.200276 32266 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.221212 32104 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.257532 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.265895 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:50.248662 32218 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:50.254062 32085 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10.
I20260709 12:19:50.297343 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:50.307790 27732 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:50.246771 32095 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:50.299559 27735 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:50.300055 27661 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:50.300258 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.303947 27736 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:50.308326 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:50.311254 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:50.331547 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.298039 32202 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:50.336987 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2014) took 1216 ms. Trace:
I20260709 12:19:50.338125 27737 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:50.337440 27736 rpcz_store.cc:276] 0709 12:19:49.120237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.160696 (+1040459us) service_pool.cc:224] Handling call
0709 12:19:50.336880 (+176184us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:50.331140 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66912.2us
I20260709 12:19:50.330054 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:50.339128 27588 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:50.329751 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:50.340198 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.340838 27583 raft_consensus.cc:2410] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:19:50.341143 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.341735 27656 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:19:50.342128 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2013) took 1212 ms. Trace:
I20260709 12:19:50.342487 27734 rpcz_store.cc:276] 0709 12:19:49.119953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.537151 (+417198us) service_pool.cc:224] Handling call
0709 12:19:50.332229 (+795078us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:50.332320 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289767680}
I20260709 12:19:50.342875 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.343602 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.343904 27736 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:50.343894 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:50.344197 27734 raft_consensus.cc:2376] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11. Re-sending same reply.
I20260709 12:19:50.345348 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.345525 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:50.231432 32267 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.350224 31253 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:50.355243 27661 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:50.297488 32103 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.370076 31253 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:50.351096 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:50.349404 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:50.338363 32116 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.508s	user 0.011s	sys 0.000s
I20260709 12:19:50.249207 32086 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.297948 32183 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:50.169696 32157 consensus_queue.cc:1175] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 9 status { last_received { term: 8 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:19:50.374385 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.312184 32095 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.330149 32097 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:19:50.385406 31253 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.375886 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.305586 32119 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:19:50.245992 32041 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.391778 32041 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:50.392880 32267 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.393748 32220 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:50.394093 32167 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.689s	user 0.004s	sys 0.004s
I20260709 12:19:50.394294 32220 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.298200 32269 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:50.395107 32276 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:19:50.396286 32086 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.396565 31417 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.346186 31860 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.397923 28334 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:50.355662 32085 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:50.401149 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50139.7us
W20260709 12:19:50.381212 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70060.7us
I20260709 12:19:50.382426 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2389) took 1090 ms. Trace:
I20260709 12:19:50.347997 32102 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.347800 32218 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:50.389074 27661 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:19:50.379158 32202 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:50.380276 32116 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:50.390381 32283 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:19:50.390682 31831 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:50.391207 32284 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:19:50.419719 32126 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.398058 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.380149 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:50.404033 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:50.497114 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93151.7us
I20260709 12:19:50.483259 32223 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.374023 31797 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.428000 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.500384 27738 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:50.430503 27661 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:50.431089 27655 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:19:50.501672 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:50.436259 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:50.502733 28332 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:50.390714 32104 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.378389 32103 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.439046 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2387) took 1281 ms. Trace:
I20260709 12:19:50.419523 32105 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:50.394800 32220 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.510983 32220 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.435456 32007 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.511605 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.512364 32216 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:50.512745 32216 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:50.512823 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.513559 27738 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:50.394518 32167 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.428016 32085 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:50.433395 32202 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:50.437415 32111 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:50.435650 32273 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 12:19:50.524497 27663 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:50.483948 32218 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.420113 31860 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.441432 32227 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:19:50.481386 32119 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.478011 32087 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:50.470613 32190 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:50.482539 32169 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.000s
I20260709 12:19:50.482817 32098 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.495193 28329 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:19:50.462392 27736 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:50.498147 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:50.367702 32291 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:50.487852 32248 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.398480 32095 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:50.469833 32183 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.504369 27661 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:19:50.552304 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2343) took 3350 ms. Trace:
I20260709 12:19:50.503444 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2393) took 1138 ms. Trace:
I20260709 12:19:50.553574 27583 rpcz_store.cc:276] 0709 12:19:49.365338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.351516 (+986178us) service_pool.cc:224] Handling call
0709 12:19:50.379975 (+ 28459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.499059 (+119084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.503389 (+  4330us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:50.505256 27585 rpcz_store.cc:276] 0709 12:19:49.157277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.795407 (+638130us) service_pool.cc:224] Handling call
0709 12:19:49.795718 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:49.795915 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:19:50.175695 (+379780us) log.cc:844] Serialized 14 byte log entry
0709 12:19:50.437932 (+262237us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:50.437986 (+    54us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:50.438043 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.438859 (+   816us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.438980 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":270487,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":89,"spinlock_wait_cycles":3586816,"thread_start_us":630478,"threads_started":2,"wal-append.queue_time_us":319710}
I20260709 12:19:50.500941 32097 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:50.519472 32202 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:50.523375 32103 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.523537 32111 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.372115 32100 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.524693 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.549419 32104 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.565400 27738 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:50.565654 32104 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.566298 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.517115 31797 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.567081 31797 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.567766 32104 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:50.569116 31798 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.801s	user 0.007s	sys 0.007s
I20260709 12:19:50.569296 31797 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:50.525563 32286 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:19:50.569895 27582 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:19:50.569839 32089 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:50.526141 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:50.570585 32259 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.571276 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2388) took 1385 ms. Trace:
I20260709 12:19:50.571049 32259 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.513603 32216 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.572281 32216 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:50.505564 32223 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.516623 31559 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.533713 31860 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.536403 32203 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.539963 32085 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.540551 32167 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:50.546725 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48714.9us
I20260709 12:19:50.591281 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:50.542526 32119 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.548521 28335 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:19:50.593887 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:50.552726 27659 rpcz_store.cc:276] 0709 12:19:47.196063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.357016 (+160953us) service_pool.cc:224] Handling call
0709 12:19:47.357350 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.870065 (+1512715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.542689 (+1672624us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.546298 (+  3609us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3415320960}
I20260709 12:19:50.541317 32227 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:50.557178 32190 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.564473 27657 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 7.
I20260709 12:19:50.570161 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 708) took 1517 ms. Trace:
I20260709 12:19:50.552901 32098 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.571637 27589 rpcz_store.cc:276] 0709 12:19:49.185573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.836441 (+650868us) service_pool.cc:224] Handling call
0709 12:19:49.860077 (+ 23636us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:49.941185 (+ 81108us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:50.339471 (+398286us) log.cc:844] Serialized 14 byte log entry
0709 12:19:50.434865 (+ 95394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:50.434980 (+   115us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:50.502218 (+ 67238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.571089 (+ 68871us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b54000be0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.571130 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.571227 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1742,"prepare.queue_time_us":170218,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":1917,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3253,"spinlock_wait_cycles":148259072,"thread_start_us":444653,"threads_started":2,"wal-append.queue_time_us":150780}
I20260709 12:19:50.573598 27661 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:50.600533 27580 rpcz_store.cc:276] 0709 12:19:49.053041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.053200 (+   159us) service_pool.cc:224] Handling call
0709 12:19:49.101365 (+ 48165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:49.328804 (+227439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:49.328855 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:49.328910 (+    55us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:49.328943 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.569938 (+1240995us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54001fe5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.569982 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.570106 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522441984}
I20260709 12:19:50.598630 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.545611 32291 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:50.580215 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:50.550973 32007 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:50.607681 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:50.551101 32220 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.547427 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:50.593966 27736 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:50.574545 32097 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:50.545466 32169 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.605497 32295 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:50.605415 32190 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.606321 32291 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:50.606839 27659 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
W20260709 12:19:50.614423 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69566.3us
I20260709 12:19:50.610385 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.616302 27736 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:50.586943 31797 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.615440 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.623648 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:50.632915 32285 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.182s	user 0.004s	sys 0.000s
I20260709 12:19:50.601109 32227 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:50.600440 31831 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:50.640240 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 710) took 1130 ms. Trace:
I20260709 12:19:50.645645 27587 rpcz_store.cc:276] 0709 12:19:49.510108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.538237 (+1028129us) service_pool.cc:224] Handling call
0709 12:19:50.538535 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.538725 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:19:50.538997 (+   272us) log.cc:844] Serialized 14 byte log entry
0709 12:19:50.639682 (+100685us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:50.639757 (+    75us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:50.639820 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.640002 (+   182us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.640156 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":535,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":104,"thread_start_us":99605,"threads_started":1}
I20260709 12:19:50.614163 32111 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.617410 31559 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:50.637997 32002 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:50.635524 32116 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
W20260709 12:19:50.659974 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 678) took 3259 ms (client timeout 3000 ms). Trace:
I20260709 12:19:50.553763 32248 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.645747 32103 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:19:50.653200 32299 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20260709 12:19:50.662177 27733 rpcz_store.cc:269] 0709 12:19:47.390156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.644032 (+253876us) service_pool.cc:224] Handling call
0709 12:19:50.649541 (+3005509us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:50.649707 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2720610048}
I20260709 12:19:50.613899 32220 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:50.649995 32292 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
W20260709 12:19:50.666765 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68244.9us
I20260709 12:19:50.643605 32041 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:50.665342 27736 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:50.673632 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:50.675191 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.669605 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.652184 31559 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:50.676707 27733 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:19:50.681028 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.639045 31797 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.684067 27733 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:50.703375 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.674396 32041 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:50.672396 32103 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:50.682226 31559 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:50.712363 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2244) took 3304 ms. Trace:
I20260709 12:19:50.667737 32233 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.716470 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:50.724436 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.733682 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.734282 27733 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:50.727371 32295 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:50.743667 32298 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.744278 32298 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.746387 32298 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.750226 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.751565 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.752235 27736 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:50.752344 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:50.753044 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:50.754033 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.754248 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.754889 27656 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:50.755223 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.755831 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.756861 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.756268 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:50.757351 32307 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:19:50.757464 27656 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:19:50.759555 32190 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.402782 27588 rpcz_store.cc:276] 0709 12:19:49.292258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.339807 (+1047549us) service_pool.cc:224] Handling call
0709 12:19:50.340088 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.340282 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:50.367403 (+ 27121us) log.cc:844] Serialized 14 byte log entry
0709 12:19:50.382087 (+ 14684us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:50.382147 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:50.382193 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.382257 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.382372 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31023,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":760,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":14494,"thread_start_us":14313,"threads_started":1}
I20260709 12:19:50.760389 32169 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.730427 32104 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:50.762506 32104 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.763399 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2376) took 1698 ms. Trace:
I20260709 12:19:50.763772 27581 rpcz_store.cc:276] 0709 12:19:49.064927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.065504 (+   577us) service_pool.cc:224] Handling call
0709 12:19:49.065805 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:49.986763 (+920958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.763168 (+776405us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.763341 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2013697408}
I20260709 12:19:50.764597 32104 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.766901 32269 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.770624 27738 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:50.770895 32203 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.772881 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.773474 27738 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:50.774277 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.778746 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.779397 27731 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:19:50.671159 32097 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.788684 31797 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.789103 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2001) took 3530 ms. Trace:
I20260709 12:19:50.732583 27730 rpcz_store.cc:276] 0709 12:19:47.408313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.519087 (+1110774us) service_pool.cc:224] Handling call
0709 12:19:48.898852 (+379765us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:48.899077 (+   225us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:48.904670 (+  5593us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:19:48.904701 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:48.905196 (+   495us) log.cc:844] Serialized 26 byte log entry
0709 12:19:49.101812 (+196616us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:19:49.101854 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.712039 (+1610185us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b54002a0b54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.712089 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:50.712135 (+    46us) raft_consensus.cc:1685] finished
0709 12:19:50.712152 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.712299 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":72,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":22628608,"thread_start_us":196256,"threads_started":1,"wal-append.queue_time_us":229295,"wal-append.run_cpu_time_us":3656,"wal-append.run_wall_time_us":1560328}
I20260709 12:19:50.729970 32041 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:50.789520 27729 rpcz_store.cc:276] 0709 12:19:47.259005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:47.696177 (+437172us) service_pool.cc:224] Handling call
0709 12:19:48.963963 (+1267786us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:48.964622 (+   659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.788919 (+1824297us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002a1a50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:50.789034 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2914865536}
I20260709 12:19:50.790869 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.798506 27730 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:19:50.798556 32248 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.799129 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2016) took 1678 ms. Trace:
I20260709 12:19:50.799518 27730 rpcz_store.cc:276] 0709 12:19:49.120757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.797919 (+1677162us) service_pool.cc:224] Handling call
0709 12:19:50.798202 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.798898 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.799065 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:50.756184 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 688) took 1661 ms. Trace:
I20260709 12:19:50.790892 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.801225 31797 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:50.800832 27732 rpcz_store.cc:276] 0709 12:19:49.094747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.094949 (+   202us) service_pool.cc:224] Handling call
0709 12:19:49.116526 (+ 21577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.312416 (+1195890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.755978 (+443562us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:50.756127 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2739145600}
I20260709 12:19:50.801867 31797 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:50.795674 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2356) took 1080 ms. Trace:
I20260709 12:19:50.802420 31797 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:50.801429 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.803424 27733 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:19:50.805131 27729 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:50.805701 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2015) took 1685 ms. Trace:
I20260709 12:19:50.730886 32119 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.791558 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.806208 32103 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.807648 32269 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:50.808053 32269 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.808436 32269 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:50.806008 27729 rpcz_store.cc:276] 0709 12:19:49.120497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.804666 (+1684169us) service_pool.cc:224] Handling call
0709 12:19:50.804942 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.805494 (+   552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.805651 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:50.810580 32227 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.756237 32233 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.811496 27729 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:19:50.811718 32220 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:50.813046 27729 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:19:50.814186 27729 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:50.802805 27662 rpcz_store.cc:276] 0709 12:19:49.715209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.753574 (+ 38365us) service_pool.cc:224] Handling call
0709 12:19:49.796030 (+ 42456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:49.809027 (+ 12997us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:50.178699 (+369672us) log.cc:844] Serialized 14 byte log entry
0709 12:19:50.335800 (+157101us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:50.761653 (+425853us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:19:50.762002 (+   349us) log.cc:844] Serialized 26 byte log entry
0709 12:19:50.762153 (+   151us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:50.762371 (+   218us) log.cc:844] Serialized 14 byte log entry
0709 12:19:50.762550 (+   179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.762635 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:50.795423 (+ 32788us) raft_consensus.cc:1685] finished
0709 12:19:50.795448 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:50.795603 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29393,"prepare.queue_time_us":344510,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":34699,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":25416,"spinlock_wait_cycles":2979200,"thread_start_us":460760,"threads_started":2,"wal-append.queue_time_us":231350}
I20260709 12:19:50.775203 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.782857 32308 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:50.819257 27736 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:50.820016 27730 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:50.820024 32291 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.827966 32291 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.822315 32227 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.833715 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.803524 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.834918 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.854336 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:50.852284 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.860242 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:50.822783 32202 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.873792 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:50.875222 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:50.880951 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:50.875973 32304 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:50.876119 32216 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 12:19:50.871169 32190 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 12:19:50.883335 32304 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:50.884267 32304 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:50.885110 31798 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:50.876083 32105 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:50.876016 31849 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:50.880921 32111 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:19:50.870782 32157 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:19:50.887014 32157 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:50.879658 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.896596 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:50.895747 27659 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:50.875845 31831 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:50.901412 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.857645 32115 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:50.861197 32291 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:19:50.903939 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 166326us
I20260709 12:19:50.898891 32157 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:50.908178 32115 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:50.912426 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.903668 32291 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:50.935349 27659 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:50.878891 32227 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:19:50.871537 32233 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:19:50.933604 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.937801 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.938657 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:50.945914 28337 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:50.938059 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:50.933444 32308 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 12:19:50.944736 32291 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:50.949975 27662 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:50.895486 32229 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:19:50.959501 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:50.960688 28330 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:19:50.961138 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.963219 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.966163 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.967383 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.970619 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.983078 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.984272 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:50.965209 32190 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:50.988131 27656 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:50.991673 32115 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:50.995687 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.990187 32104 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.996991 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:50.997537 32149 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:50.998205 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.000588 32223 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:50.998332 32220 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:50.997210 32104 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:50.999815 32277 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.750s	user 0.000s	sys 0.002s
I20260709 12:19:50.965580 31798 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:51.004031 32264 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:51.000456 32202 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.007644 27659 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:51.006796 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:50.966899 32227 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:19:51.012063 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.014472 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:50.986583 31802 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:50.965373 32216 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:19:50.980908 31849 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:51.021081 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.053171 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.007761 32190 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.042279 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:51.049360 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91434.6us
I20260709 12:19:51.050702 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.016345 32295 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.043131 31802 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:50.984681 32183 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:19:51.017974 32264 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:50.969938 31831 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.059638 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.060606 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.082870 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.019066 32223 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.048658 32259 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.121095 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2408) took 2007 ms. Trace:
I20260709 12:19:51.120954 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.117589 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.118312 28330 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:51.131264 27580 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:51.097791 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.121477 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.132668 27589 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:51.094980 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.096107 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.121992 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.122545 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.136209 27587 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:51.052680 32060 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.120391 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1961) took 1057 ms. Trace:
I20260709 12:19:51.100128 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 711) took 1393 ms. Trace:
I20260709 12:19:51.137171 27663 rpcz_store.cc:276] 0709 12:19:50.062794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.496000 (+433206us) service_pool.cc:224] Handling call
0709 12:19:50.496331 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.525973 (+ 29642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:51.120073 (+594100us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b5400343d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:51.120311 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53748480}
I20260709 12:19:51.137727 27583 rpcz_store.cc:276] 0709 12:19:49.707224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.554718 (+847494us) service_pool.cc:224] Handling call
0709 12:19:50.574785 (+ 20067us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.646377 (+ 71592us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:50.646409 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:50.646447 (+    38us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:50.646475 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:51.099912 (+453437us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:51.099957 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:51.100074 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164254080}
I20260709 12:19:51.138616 27663 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:51.140275 27656 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:51.089192 31802 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:51.148268 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 691) took 1849 ms (client timeout 1826 ms). Trace:
I20260709 12:19:51.117266 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:51.149641 27737 rpcz_store.cc:269] 0709 12:19:49.299101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.625004 (+325903us) service_pool.cc:224] Handling call
0709 12:19:51.148079 (+1523075us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54001d10d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:51.148203 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1207132416}
I20260709 12:19:51.130154 27586 rpcz_store.cc:276] 0709 12:19:49.113972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.712389 (+598417us) service_pool.cc:224] Handling call
0709 12:19:49.712742 (+   353us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:49.713184 (+   442us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:49.713214 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:49.713738 (+   524us) log.cc:844] Serialized 52 byte log entry
0709 12:19:49.988542 (+274804us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:49.988601 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:49.988698 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:50.999153 (+1010455us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:51.120790 (+121637us) raft_consensus.cc:1685] finished
0709 12:19:51.120812 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:51.121038 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3689,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":104,"spinlock_wait_cycles":10929920,"thread_start_us":274354,"threads_started":1,"wal-append.queue_time_us":496778}
I20260709 12:19:51.150938 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.018501 32220 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.059818 32190 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:51.096500 32287 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.697s	user 0.001s	sys 0.001s
W20260709 12:19:51.095515 32318 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:19:51.097185 32315 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:19:51.159909 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65027.3us
W20260709 12:19:51.160331 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:51.163256 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:51.159026 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:51.175287 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.177990 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.177139 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.178608 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:51.179520 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.180050 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:51.108804 32234 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.180421 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.206310 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:51.123814 32250 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:19:51.186822 32227 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:51.104200 32002 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:51.194038 32087 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.649s	user 0.008s	sys 0.003s
I20260709 12:19:51.154594 32304 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.195477 32104 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.155193 32060 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.154982 32041 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.055980 32095 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.170975 31798 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.201261 32098 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.098039 32324 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:51.211300 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.213470 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.214534 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.216902 32323 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:51.153687 32157 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:51.266312 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53067.7us
I20260709 12:19:51.222472 32227 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:51.056866 32202 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:51.158411 32216 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:51.240427 32298 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:51.231698 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:51.281772 27737 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:51.236799 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.117205 31559 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.239970 28333 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:51.203531 31849 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:51.244310 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 228748us
I20260709 12:19:51.283658 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.250236 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:51.088583 32295 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.149987 32223 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.209472 32250 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:51.231469 32002 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:51.262467 27583 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:51.263142 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.222195 32111 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.268218 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.228000 27589 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:51.267277 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.271204 27729 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:51.059240 32261 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.255479 32104 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:51.325533 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 692) took 1937 ms. Trace:
I20260709 12:19:51.299921 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.268922 32323 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:51.313010 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62842.6us
I20260709 12:19:51.331738 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.320183 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.325924 27731 rpcz_store.cc:276] 0709 12:19:49.387847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.807029 (+1419182us) service_pool.cc:224] Handling call
0709 12:19:50.807362 (+   333us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:50.808101 (+   739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:50.808131 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:50.816588 (+  8457us) log.cc:844] Serialized 52 byte log entry
0709 12:19:50.951092 (+134504us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:50.951128 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:50.951184 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:51.325320 (+374136us) raft_consensus.cc:1685] finished
0709 12:19:51.325340 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:51.325468 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":169,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":133570,"threads_started":1,"wal-append.queue_time_us":189877}
I20260709 12:19:51.334417 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.335170 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:51.337548 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2346) took 1630 ms. Trace:
I20260709 12:19:51.329217 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.338218 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.338833 27663 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:51.322871 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.337916 27664 rpcz_store.cc:276] 0709 12:19:49.709240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.711424 (+  2184us) service_pool.cc:224] Handling call
0709 12:19:49.711784 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:49.915535 (+203751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:51.337332 (+1421797us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:51.337493 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204802560}
I20260709 12:19:51.340118 28335 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:19:51.331894 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.340728 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:51.320256 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.342271 27589 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 9.
I20260709 12:19:51.342949 27583 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:51.290602 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:51.344501 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:51.355963 27589 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:19:51.343096 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.256896 31802 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.286275 32226 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.347199 27738 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:51.274590 32254 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:19:51.392695 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.394944 32259 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.277724 32227 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:51.408819 27732 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:51.348189 27733 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:51.410024 27737 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:51.273639 32202 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.351464 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:51.410945 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 243540us
I20260709 12:19:51.411113 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.281354 32298 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.244230 32304 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.350781 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:51.413282 27583 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:51.344594 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54058.4us
I20260709 12:19:51.216003 32264 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.414774 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.416333 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.417301 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:19:51.361922 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.373521 32104 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:19:51.430069 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68217.6us
I20260709 12:19:51.431151 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.375625 32060 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:51.376940 32279 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.992s	user 0.005s	sys 0.000s
I20260709 12:19:51.378218 32157 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.330039 32085 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:19:51.386720 32269 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.578s	user 0.010s	sys 0.000s
I20260709 12:19:51.299567 31849 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.350418 32203 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:51.329726 32097 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 12:19:51.341746 32323 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:51.398890 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:51.409175 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.476639 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:51.477768 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:51.411957 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:51.324980 32089 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:51.428153 28337 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:51.341228 32111 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.441344 32087 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:51.445580 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:51.416092 32298 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:51.395702 32259 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.412994 32202 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:51.447435 32085 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.348026 31797 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.483566 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.493677 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.485147 27737 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:51.447532 32119 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:51.442384 32326 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:51.496558 27583 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:51.489980 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.501739 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.481947 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.492197 28332 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:51.429071 32104 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.449219 32060 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.499531 27589 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:19:51.529405 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:51.451987 32264 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:51.504441 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.453795 32203 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.530902 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:51.487475 32223 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.533205 32041 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.476177 32269 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.495290 32087 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:51.549582 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:51.488147 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 241532us
W20260709 12:19:51.523932 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 102590us
I20260709 12:19:51.525658 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.408025 32226 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.533667 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.534950 27583 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:51.497871 32085 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:51.588574 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:51.537853 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.594561 27588 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:51.589660 27589 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:19:51.604451 27589 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:19:51.527701 32104 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:51.532577 31798 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:51.607383 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:51.610739 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:51.624125 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.536355 32324 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:51.538278 32119 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:51.543505 32203 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:51.619369 27737 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:51.564559 31849 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.587530 32220 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:51.556818 32259 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.593689 31860 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:51.539554 32060 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.640535 27663 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 6.
I20260709 12:19:51.641754 27731 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:51.673929 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.664660 32324 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:51.679965 27663 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:19:51.683468 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.691679 27737 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:51.670370 32327 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:51.671974 32220 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:51.693890 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.694234 31802 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.706393 32119 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.686674 32324 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:51.707544 32304 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.727627 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.673312 31860 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:51.741153 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 215628us
I20260709 12:19:51.743111 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.744091 28335 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:51.696516 32202 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.731869 32324 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:51.745828 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 135256us
I20260709 12:19:51.748504 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:51.749971 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:51.742441 32190 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.564s	user 0.013s	sys 0.000s
I20260709 12:19:51.758436 32216 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:51.761762 32202 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.763655 27737 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:51.764354 32002 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.778716 32323 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.774772 32227 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.692453 32087 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.726485 32327 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.768044 32269 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:51.768512 32291 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.810s	user 0.006s	sys 0.005s
I20260709 12:19:51.741272 32226 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.770083 32254 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:51.771067 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:51.785663 32216 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:51.774449 32085 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.775601 32060 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.776298 32203 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.827443 32203 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.777314 32298 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.829483 32298 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:51.830734 32302 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.164s	user 0.000s	sys 0.005s
I20260709 12:19:51.778560 32326 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:51.737459 32220 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:51.832403 32220 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.834364 32160 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:51.765214 31860 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:51.765919 32335 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:19:51.793838 27584 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260709 12:19:51.835798 27584 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 5
I20260709 12:19:51.836081 27584 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:19:51.836328 27584 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:51.836648 27584 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:19:51.836866 27584 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:51.834936 32160 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:51.759711 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:51.794185 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:51.741744 32111 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:51.800230 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2383) took 2677 ms (client timeout 3123 ms). Trace:
I20260709 12:19:51.819787 27588 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:51.795066 31918 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:51.844671 31918 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:51.845103 32326 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:19:51.845721 32326 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:51.845678 31918 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:51.844805 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50709.7us
W20260709 12:19:51.842341 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71360.3us
I20260709 12:19:51.847455 32326 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.843345 27661 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:19:51.799619 27589 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 13. Ops: []
I20260709 12:19:51.844003 32343 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:51.848845 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:51.843925 27582 rpcz_store.cc:269] 0709 12:19:49.118006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.310385 (+192379us) service_pool.cc:224] Handling call
0709 12:19:51.793320 (+2482935us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:51.795151 (+  1831us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2877969152}
I20260709 12:19:51.851643 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:51.852202 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93744.3us
I20260709 12:19:51.852665 32111 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:51.785867 32190 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.853173 32111 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:51.877615 27731 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:51.856308 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.856607 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:51.857717 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:51.852417 27582 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:51.823205 32216 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:51.887202 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.853173 32269 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:51.849663 32343 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:51.890004 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.887593 32344 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:19:51.948341 27657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 686) took 1833 ms (client timeout 1636 ms). Trace:
W20260709 12:19:51.948944 27657 rpcz_store.cc:269] 0709 12:19:50.114837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.227869 (+113032us) service_pool.cc:224] Handling call
0709 12:19:51.948076 (+1720207us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b5400041354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:51.948246 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":671683840}
I20260709 12:19:51.891225 32259 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:51.891719 32002 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.891500 32007 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:51.890846 32311 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:51.894152 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.956252 27662 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:51.896435 32202 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:51.899642 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.901728 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:51.904053 32227 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.905862 32323 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.912011 32250 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:51.913750 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2019) took 2792 ms (client timeout 2086 ms). Trace:
I20260709 12:19:51.917965 32126 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:19:51.966104 27735 rpcz_store.cc:269] 0709 12:19:49.121500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.089632 (+968132us) service_pool.cc:224] Handling call
0709 12:19:51.913489 (+1823857us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:51.913658 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266245888}
I20260709 12:19:51.921625 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2017) took 2800 ms. Trace:
I20260709 12:19:51.927600 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:51.967254 27730 rpcz_store.cc:276] 0709 12:19:49.121002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.800242 (+1679240us) service_pool.cc:224] Handling call
0709 12:19:50.800539 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.823723 (+ 23184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:51.921269 (+1097546us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:51.921534 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44374912}
I20260709 12:19:51.930660 32085 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:51.932430 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:51.936156 32160 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:51.968772 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:51.798655 32291 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:51.869601 32254 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:51.946615 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.883704 32111 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:51.952091 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62222.5us
I20260709 12:19:51.847136 31918 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:51.959426 28332 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:19:51.959286 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.930720 32266 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:51.945482 32269 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:51.962090 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62538.5us
I20260709 12:19:51.963639 31417 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:51.985055 32095 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:51.985457 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2027) took 1774 ms (client timeout 1597 ms). Trace:
W20260709 12:19:51.985881 27734 rpcz_store.cc:269] 0709 12:19:50.210556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.345076 (+134520us) service_pool.cc:224] Handling call
0709 12:19:51.985257 (+1640181us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b540044e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:51.985390 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1400566656}
I20260709 12:19:51.939996 32261 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:51.947487 32344 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:51.987635 32344 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:51.940546 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.986979 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:51.989205 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48811.4us
I20260709 12:19:51.990757 27701 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.991839 27662 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:51.961162 32089 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:19:51.954743 32169 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:51.975389 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:51.996279 27656 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:51.982726 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.875710 32183 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:19:51.983827 32002 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:51.979085 32126 consensus_queue.cc:1243] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:19:51.984472 32190 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:51.995533 32119 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:19:52.004019 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:52.004999 32291 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.005440 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:52.006856 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:51.984674 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:52.007930 27735 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:51.973256 32112 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:52.009900 32112 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:52.010818 32112 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.011904 32183 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:52.011960 32112 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:52.014050 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:51.977831 32254 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:52.017903 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 113983us
I20260709 12:19:52.019102 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:51.982954 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:52.028859 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46000us
W20260709 12:19:52.029382 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.017721 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:51.980749 32250 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:19:52.035388 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:51.989006 32269 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.042511 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.048604 27580 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:19:52.049477 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2361) took 2110 ms (client timeout 2187 ms). Trace:
W20260709 12:19:52.049916 27658 rpcz_store.cc:269] 0709 12:19:49.939072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.053989 (+114917us) service_pool.cc:224] Handling call
0709 12:19:52.049181 (+1995192us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.049393 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188641664}
I20260709 12:19:52.050480 27660 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:52.039975 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.041344 27589 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:19:52.041630 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.044595 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.053046 27583 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:52.053110 27582 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:52.055537 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.061493 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.062235 27662 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:51.994071 32089 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:52.047564 32190 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:19:52.020287 32183 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.034379 32291 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:19:52.027146 32119 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:52.035916 32254 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:52.037637 32098 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:52.056254 32007 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:52.047820 32326 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:52.084820 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:52.086059 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:52.087579 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.087957 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.035676 32311 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 12:19:52.092026 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:52.092788 27581 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:52.074252 32291 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:52.070652 32190 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.064193 32336 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:19:52.093763 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:52.101141 27735 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:52.102249 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.056740 32202 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:19:52.105185 27589 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 6. Current term is 7. Ops: []
I20260709 12:19:52.105096 27656 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:52.056106 31417 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:19:52.110337 27582 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:52.047139 32157 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:19:52.081872 32098 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:52.062831 32104 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.137200 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.137539 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.137779 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.132521 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:52.138635 27735 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:52.138118 27581 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:52.141117 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.145231 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.146255 27581 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:19:52.065891 32102 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:19:52.070236 32298 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260709 12:19:52.147601 32349 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:19:52.089736 32227 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:52.089943 32259 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:52.096211 32323 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:52.066975 32158 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:19:52.089879 32103 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:52.132867 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.183744 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.184708 32222 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:52.184958 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.185297 32222 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.186060 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:52.138644 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:52.188090 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.188301 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 694) took 2110 ms. Trace:
I20260709 12:19:52.183795 32102 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:52.188692 27729 rpcz_store.cc:276] 0709 12:19:50.078291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.814773 (+736482us) service_pool.cc:224] Handling call
0709 12:19:50.815044 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:51.320654 (+505610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.188051 (+867397us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:52.188244 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1048621824}
I20260709 12:19:52.189288 32102 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:52.151716 27583 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:52.190075 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:52.190820 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:52.191429 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.192058 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:52.192369 32354 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:19:52.149609 27735 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:19:52.193403 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 695) took 2035 ms. Trace:
I20260709 12:19:52.193957 32158 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:52.194212 27735 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:52.194384 32158 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:52.194957 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2269) took 2033 ms. Trace:
W20260709 12:19:52.196089 32348 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:19:52.195770 27735 rpcz_store.cc:276] 0709 12:19:50.161532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.193319 (+2031787us) service_pool.cc:224] Handling call
0709 12:19:52.193869 (+   550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.194740 (+   871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.194890 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.196316 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 682) took 2082 ms. Trace:
I20260709 12:19:52.196767 27660 rpcz_store.cc:276] 0709 12:19:50.113554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.571437 (+457883us) service_pool.cc:224] Handling call
0709 12:19:52.050237 (+1478800us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.051080 (+   843us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.196123 (+145043us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400292050. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.196257 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3400040320}
I20260709 12:19:52.197620 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2274) took 1992 ms. Trace:
I20260709 12:19:52.198016 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.197968 27735 rpcz_store.cc:276] 0709 12:19:50.205113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.196742 (+1991629us) service_pool.cc:224] Handling call
0709 12:19:52.197159 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.197328 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.197357 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:52.197401 (+    44us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:52.197432 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.197481 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.197576 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.198714 27660 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.187165 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.200305 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.200628 27731 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:52.201172 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2030) took 1973 ms. Trace:
I20260709 12:19:52.201360 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.201460 27731 rpcz_store.cc:276] 0709 12:19:50.227400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.200184 (+1972784us) service_pool.cc:224] Handling call
0709 12:19:52.200428 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.200985 (+   557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.201130 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.202260 27656 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:52.202900 27731 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:19:52.148681 32104 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:52.174693 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:52.133586 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.225392 27589 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:52.097551 32007 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:52.195060 27737 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:52.176052 32259 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.261924 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2271) took 2091 ms. Trace:
I20260709 12:19:52.204049 32183 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:52.205269 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:52.263721 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.136776 32089 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:52.207949 31798 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.573s	user 0.006s	sys 0.006s
I20260709 12:19:52.209509 27735 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:52.290680 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2275) took 2081 ms. Trace:
I20260709 12:19:52.291057 27735 rpcz_store.cc:276] 0709 12:19:50.209122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.198999 (+1989877us) service_pool.cc:224] Handling call
0709 12:19:52.203735 (+  4736us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.290437 (+ 86702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.290626 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.203367 27583 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:52.198024 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.211512 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:52.292384 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 124486us
I20260709 12:19:52.193993 27730 rpcz_store.cc:276] 0709 12:19:50.158165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.192563 (+2034398us) service_pool.cc:224] Handling call
0709 12:19:52.192892 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.193074 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.193100 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:52.193148 (+    48us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:52.193181 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.193237 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.193348 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.097728 32060 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:52.190845 27729 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:52.217103 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2279) took 1671 ms. Trace:
I20260709 12:19:52.296027 27731 rpcz_store.cc:276] 0709 12:19:50.544124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.202336 (+1658212us) service_pool.cc:224] Handling call
0709 12:19:52.202627 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.203222 (+   595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.215048 (+ 11826us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.110791 32291 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:52.137275 32098 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.229152 27662 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 5. Current term is 6. Ops: []
I20260709 12:19:52.131938 32150 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:52.038998 32343 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.233990 32111 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.238099 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.240065 27734 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.202847 32002 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:52.078522 32324 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:52.242874 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144674us
I20260709 12:19:52.224172 32218 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:52.226262 32304 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:52.214738 32323 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.255465 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:52.320647 27662 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:19:52.175777 32227 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:52.310683 32352 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:19:52.262347 27737 rpcz_store.cc:276] 0709 12:19:50.170519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.194572 (+2024053us) service_pool.cc:224] Handling call
0709 12:19:52.194841 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:52.261569 (+ 66728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.261851 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.264459 27580 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:52.347277 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.267935 27660 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:52.276836 31797 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.281733 27582 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.282235 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.350935 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:52.283339 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2021) took 2901 ms. Trace:
I20260709 12:19:52.209638 32158 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.284415 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 1968) took 1530 ms. Trace:
I20260709 12:19:52.352281 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:52.352470 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140394us
I20260709 12:19:52.353101 27586 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:52.285357 32269 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.285971 32216 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.287405 32254 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.287771 32089 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:19:52.276980 32259 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.288470 32183 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:52.211812 27581 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8.
W20260709 12:19:52.297925 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86496.4us
I20260709 12:19:52.298851 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:52.301060 27730 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:52.302117 27729 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 10. Ops: []
I20260709 12:19:52.367313 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2282) took 1809 ms. Trace:
I20260709 12:19:52.364835 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.360193 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.368039 27729 rpcz_store.cc:276] 0709 12:19:50.558338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.301537 (+1743199us) service_pool.cc:224] Handling call
0709 12:19:52.301904 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.366997 (+ 65093us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.367245 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.369025 27586 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:52.304836 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.304482 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 697) took 1728 ms. Trace:
I20260709 12:19:52.370862 27731 rpcz_store.cc:276] 0709 12:19:50.576501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.302799 (+1726298us) service_pool.cc:224] Handling call
0709 12:19:52.303288 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.304035 (+   747us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.304082 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:52.304141 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:52.304187 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.304265 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.304411 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:52.371642 32264 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.829s	user 0.006s	sys 0.006s
I20260709 12:19:52.372303 27731 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:52.372593 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:52.372885 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 700) took 1523 ms. Trace:
I20260709 12:19:52.373406 32254 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.373270 27731 rpcz_store.cc:276] 0709 12:19:50.849126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.371870 (+1522744us) service_pool.cc:224] Handling call
0709 12:19:52.372141 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.372683 (+   542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.372837 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.374344 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.308905 28335 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:52.375027 28330 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:52.376961 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:52.377797 28334 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:52.318492 32218 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.379505 32007 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.380088 32218 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.381124 32218 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:52.311065 32150 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.295146 31798 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:52.383767 31798 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:52.254884 32190 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.384253 31798 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.384722 32216 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.384918 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:52.385599 27587 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:19:52.304420 31417 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:52.260344 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.265192 32295 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:52.351809 27733 rpcz_store.cc:276] 0709 12:19:49.381467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.804303 (+1422836us) service_pool.cc:224] Handling call
0709 12:19:50.804646 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:51.410057 (+605411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.283000 (+872943us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:52.283246 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1249532672}
I20260709 12:19:52.397236 27735 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:52.397697 31798 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.355559 32304 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:52.384761 32150 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.353185 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:52.398535 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:52.399037 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45930us
I20260709 12:19:52.399451 31417 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.400240 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:52.400329 31417 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.401094 32150 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:52.401490 31417 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:52.401772 31908 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:52.402205 31908 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:52.402609 31908 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.398435 32304 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.408923 32269 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.409610 27589 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13.
I20260709 12:19:52.410446 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.412240 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2018) took 3291 ms. Trace:
I20260709 12:19:52.412619 27732 rpcz_store.cc:276] 0709 12:19:49.121264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.802185 (+1680921us) service_pool.cc:224] Handling call
0709 12:19:50.802465 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:51.409495 (+607030us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.412024 (+1002529us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:52.412186 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393763712}
I20260709 12:19:52.413177 32227 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.413663 32327 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.415313 32324 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.415582 32323 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:52.416699 32358 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:19:52.417696 32111 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.315351 32226 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:19:52.418450 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:52.418893 32226 consensus_queue.cc:1243] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:19:52.357939 28333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 689) took 3361 ms (client timeout 2852 ms). Trace:
I20260709 12:19:52.352311 27661 rpcz_store.cc:276] 0709 12:19:50.754205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.757758 (+  3553us) service_pool.cc:224] Handling call
0709 12:19:50.758085 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:51.848638 (+1090553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.284166 (+435528us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:52.284359 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2494690560}
W20260709 12:19:52.423192 28333 rpcz_store.cc:269] 0709 12:19:48.984181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.012897 (+ 28716us) service_pool.cc:224] Handling call
0709 12:19:52.344967 (+3332070us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.345098 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5111603456}
I20260709 12:19:52.424062 27661 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:19:52.359313 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2284) took 1510 ms. Trace:
I20260709 12:19:52.424518 27737 rpcz_store.cc:276] 0709 12:19:50.848576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.358535 (+1509959us) service_pool.cc:224] Handling call
0709 12:19:52.358830 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.358990 (+   160us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.359016 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:52.359074 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:52.359104 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.359156 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.359259 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.361590 27662 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 13. Ops: []
I20260709 12:19:52.354770 27734 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:52.317524 27656 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:19:52.383133 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.397974 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2286) took 1408 ms. Trace:
I20260709 12:19:52.427713 27735 rpcz_store.cc:276] 0709 12:19:50.989702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.396747 (+1407045us) service_pool.cc:224] Handling call
0709 12:19:52.397050 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.397705 (+   655us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.397919 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.359429 32360 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:52.397791 27733 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:52.434800 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2287) took 1285 ms. Trace:
I20260709 12:19:52.435124 27733 rpcz_store.cc:276] 0709 12:19:51.149556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.397365 (+1247809us) service_pool.cc:224] Handling call
0709 12:19:52.397622 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.434572 (+ 36950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.434750 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.353185 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:52.395651 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:52.423630 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:52.392738 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2281) took 1842 ms. Trace:
I20260709 12:19:52.436868 27730 rpcz_store.cc:276] 0709 12:19:50.550030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.300407 (+1750377us) service_pool.cc:224] Handling call
0709 12:19:52.300816 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.366029 (+ 65213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.392653 (+ 26624us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.438640 27730 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:19:52.428661 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.430017 27661 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:52.445123 27661 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
W20260709 12:19:52.430382 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.430907 27660 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:19:52.362196 32158 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:52.370211 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:52.452334 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 99241.4us
I20260709 12:19:52.329569 32219 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 12:19:52.453197 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:52.458630 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.458949 27586 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:52.460801 27583 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:52.453912 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2047) took 1184 ms. Trace:
I20260709 12:19:52.461342 27730 rpcz_store.cc:276] 0709 12:19:51.269450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.437913 (+1168463us) service_pool.cc:224] Handling call
0709 12:19:52.438323 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:52.439108 (+   785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.453835 (+ 14727us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.463032 27730 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
W20260709 12:19:52.456671 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.463805 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 703) took 1051 ms. Trace:
I20260709 12:19:52.464363 27730 rpcz_store.cc:276] 0709 12:19:51.412297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.462360 (+1050063us) service_pool.cc:224] Handling call
0709 12:19:52.462798 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.463499 (+   701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.463739 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:52.468217 27662 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:52.401510 32150 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:52.415591 32137 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:52.472708 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.473512 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.479748 27586 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:52.429742 32295 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:52.475461 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:52.477635 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.473925 27657 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:52.480522 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.484285 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.484818 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:52.485430 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:52.485518 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.458170 32111 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:52.481752 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.488902 31417 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:52.483196 27589 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:52.489974 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170813us
I20260709 12:19:52.451704 31831 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:52.466881 32104 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:52.490298 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:52.488126 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.491653 27583 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:52.491878 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:52.478421 32150 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:52.491533 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:52.478703 32137 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:52.489163 32111 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:52.515563 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:52.518419 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.459874 32226 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
W20260709 12:19:52.520368 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.512008 27589 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:52.496070 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:52.535032 31417 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:19:52.535689 31417 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.562239 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:52.571744 32333 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.586s	user 0.004s	sys 0.000s
I20260709 12:19:52.573967 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2376) took 1607 ms. Trace:
I20260709 12:19:52.574369 27659 rpcz_store.cc:276] 0709 12:19:50.967052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:51.132896 (+165844us) service_pool.cc:224] Handling call
0709 12:19:51.133227 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:51.272076 (+138849us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:51.328497 (+ 56421us) log.cc:844] Serialized 14 byte log entry
0709 12:19:51.637126 (+308629us) log.cc:844] Serialized 14 byte log entry
0709 12:19:51.637639 (+   513us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:51.662423 (+ 24784us) log.cc:844] Serialized 26 byte log entry
0709 12:19:51.667586 (+  5163us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:51.668022 (+   436us) log.cc:844] Serialized 14 byte log entry
0709 12:19:51.668313 (+   291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:51.923100 (+254787us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b1800426580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:51.924406 (+  1306us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b1800426580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:51.934898 (+ 10492us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b1800426580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:51.942837 (+  7939us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:51.942905 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:51.984911 (+ 42006us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b1800426580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1
0709 12:19:51.985393 (+   482us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b1800426580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:52.571159 (+585766us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:52.573779 (+  2620us) raft_consensus.cc:1685] finished
0709 12:19:52.573798 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.573911 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145,"prepare.queue_time_us":64912,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":81,"raft.queue_time_us":8362,"raft.run_cpu_time_us":573,"raft.run_wall_time_us":27119,"spinlock_wait_cycles":408087424,"thread_start_us":255866,"threads_started":1,"wal-append.queue_time_us":288206}
I20260709 12:19:52.577075 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:52.577761 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.578612 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.523929 32233 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:52.618578 31417 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.621119 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:19:52.621806 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:52.523396 32104 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:52.530292 32111 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:52.625425 27589 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:19:52.622790 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:52.524744 32336 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:52.496646 31798 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:52.528043 31831 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:52.626619 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:52.495098 32295 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.580219 32254 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:19:52.636283 32364 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:19:52.639570 32137 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.640940 32137 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:52.647962 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:52.624991 32233 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:52.649439 31559 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:52.654800 32344 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.667s	user 0.009s	sys 0.000s
I20260709 12:19:52.647380 32264 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:52.668519 32264 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.670017 32360 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:52.670774 27581 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:52.671205 32264 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.658244 31831 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.671905 32098 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.671859 32104 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.673143 27658 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13.
I20260709 12:19:52.660641 27586 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:52.665382 31908 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.666595 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 699) took 1817 ms. Trace:
I20260709 12:19:52.656677 32233 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:52.707181 27729 rpcz_store.cc:276] 0709 12:19:50.848861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.369127 (+1520266us) service_pool.cc:224] Handling call
0709 12:19:52.369527 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:52.369955 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.369998 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:52.370554 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 12:19:52.527924 (+157370us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:52.527968 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.528063 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:52.666308 (+138245us) raft_consensus.cc:1685] finished
0709 12:19:52.666333 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.666521 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":58419,"threads_started":1,"wal-append.queue_time_us":196236}
I20260709 12:19:52.642355 31798 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.731964 27729 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:52.737545 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2048) took 1123 ms. Trace:
I20260709 12:19:52.737944 27729 rpcz_store.cc:276] 0709 12:19:51.613789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.722040 (+1108251us) service_pool.cc:224] Handling call
0709 12:19:52.722449 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.737219 (+ 14770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.737464 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:52.740131 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.720036 27657 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:52.635514 32336 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.750161 27729 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:19:52.752282 27661 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:52.753275 27729 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:52.755360 31908 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.759403 32291 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.761834 32291 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.762537 32098 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.763746 32111 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.763850 32150 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.766255 32111 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.766423 32150 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.776934 27588 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 10: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:52.721233 32226 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:52.724115 32344 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.787621 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 688) took 3810 ms. Trace:
I20260709 12:19:52.788146 28331 rpcz_store.cc:276] 0709 12:19:48.977549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:48.979802 (+  2253us) service_pool.cc:224] Handling call
0709 12:19:49.914822 (+935020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.249998 (+335176us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.006547 (+1756549us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b5400520dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:52.787404 (+780857us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b5400520dd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.787558 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2645438976}
I20260709 12:19:52.815436 32344 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:52.824385 32233 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.812125 32295 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.831524 32233 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:52.831835 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.832647 32336 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.833451 31798 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.783595 32226 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.849275 32304 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.855540 32291 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:52.856129 32291 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.858489 32291 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:52.859138 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.856372 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:52.860663 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:52.861294 32291 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:19:52.863633 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:52.864123 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:52.864704 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:52.865119 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:52.865479 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.864075 32291 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:52.867496 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:52.867682 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:52.868697 32360 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:52.869210 32360 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.869557 32233 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:19:52.870263 32233 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:19:52.870141 32360 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.871495 32218 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:52.872284 32218 consensus_queue.cc:1243] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:19:52.872179 32233 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:19:52.873198 32233 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:52.873536 32233 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:52.874327 32233 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.879562 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.879791 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:19:52.881817 32371 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:19:52.892241 32181 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:52.892699 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:52.891249 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 683) took 2777 ms. Trace:
I20260709 12:19:52.903496 31417 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:19:52.906374 31417 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:52.908226 31417 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.920624 32226 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:52.924042 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2400) took 1085 ms. Trace:
I20260709 12:19:52.924566 27657 rpcz_store.cc:276] 0709 12:19:51.838300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:51.950473 (+112173us) service_pool.cc:224] Handling call
0709 12:19:52.923847 (+973374us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54001e4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.923967 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1138373376}
W20260709 12:19:52.925747 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.929963 32102 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.934831 32103 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:52.935370 32103 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:52.936421 32103 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:52.937758 32103 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:52.908638 32291 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:19:52.908119 32183 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:52.943231 32183 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:52.944003 32183 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.944968 32183 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:52.949934 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2391) took 1526 ms (client timeout 2028 ms). Trace:
W20260709 12:19:52.950508 27664 rpcz_store.cc:269] 0709 12:19:51.423496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:51.482513 (+ 59017us) service_pool.cc:224] Handling call
0709 12:19:52.949677 (+1467164us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.949844 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019527936}
I20260709 12:19:52.952345 32152 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:52.952852 32152 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:52.953976 32152 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:52.955554 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.956240 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:52.956746 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.957383 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:52.957475 27588 raft_consensus.cc:2376] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 8. Re-sending same reply.
I20260709 12:19:52.958647 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:52.959013 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:53.003003 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2288) took 1795 ms. Trace:
I20260709 12:19:53.003788 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2043) took 1761 ms. Trace:
W20260709 12:19:53.026612 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.962899 32291 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:52.897090 27655 rpcz_store.cc:276] 0709 12:19:50.113884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.572764 (+458880us) service_pool.cc:224] Handling call
0709 12:19:50.573055 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:50.573293 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:19:50.588143 (+ 14850us) log.cc:844] Serialized 14 byte log entry
0709 12:19:52.622478 (+2034335us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b1800420d00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a
0709 12:19:52.624824 (+  2346us) log.cc:844] Serialized 14 byte log entry
0709 12:19:52.890785 (+265961us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:52.890866 (+    81us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:52.890932 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.891065 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:52.891188 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14795,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4829,"raft.queue_time_us":698,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":4539,"spinlock_wait_cycles":12794624,"thread_start_us":287293,"threads_started":2,"wal-append.queue_time_us":78321,"wal-append.run_cpu_time_us":1760,"wal-append.run_wall_time_us":1089522}
W20260709 12:19:52.979863 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.028723 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:52.973145 31417 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:53.029704 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.030365 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:53.014793 28292 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13.
I20260709 12:19:52.978789 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:53.026854 27735 rpcz_store.cc:276] 0709 12:19:51.242424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.428425 (+1186001us) service_pool.cc:224] Handling call
0709 12:19:52.428736 (+   311us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:52.435816 (+  7080us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.435860 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:52.451990 (+ 16130us) log.cc:844] Serialized 52 byte log entry
0709 12:19:52.634458 (+182468us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:52.656901 (+ 22443us) log.cc:844] Serialized 14 byte log entry
0709 12:19:52.962160 (+305259us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:19:53.003387 (+ 41227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.003461 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:53.003501 (+    40us) raft_consensus.cc:1685] finished
0709 12:19:53.003603 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.003735 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":402123,"prepare.queue_time_us":15499,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":369461,"raft.run_cpu_time_us":1415,"raft.run_wall_time_us":503703,"spinlock_wait_cycles":32242432,"thread_start_us":140137,"threads_started":1,"wal-append.queue_time_us":333017}
I20260709 12:19:52.961479 32169 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:19:53.033797 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 74842.2us
I20260709 12:19:53.033980 32169 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:53.026227 27732 rpcz_store.cc:276] 0709 12:19:51.207255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.413575 (+1206320us) service_pool.cc:224] Handling call
0709 12:19:52.424267 (+ 10692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.424455 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:52.531119 (+106664us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.002640 (+471521us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:53.002708 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:53.002758 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.002822 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.002940 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":383942,"prepare.queue_time_us":69428,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":4462,"raft.queue_time_us":379452,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":147756,"threads_started":2,"wal-append.queue_time_us":104274}
I20260709 12:19:53.017776 28334 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:53.036240 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:53.003181 32119 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:53.040587 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:53.041220 32119 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:53.042564 32119 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:52.978302 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:53.043499 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2285) took 2153 ms. Trace:
I20260709 12:19:53.044018 28328 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:52.974556 32222 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:52.960525 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:53.046352 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:53.047041 32370 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.261s	user 0.001s	sys 0.001s
I20260709 12:19:53.047192 27581 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:53.012110 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2392) took 1588 ms. Trace:
I20260709 12:19:53.048346 27663 rpcz_store.cc:276] 0709 12:19:51.423886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.229716 (+805830us) service_pool.cc:224] Handling call
0709 12:19:52.230051 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.345473 (+115422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.345502 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:52.345545 (+    43us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:52.345595 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.011814 (+666219us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.011866 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.012026 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264925952}
I20260709 12:19:52.963515 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2397) took 3854 ms. Trace:
I20260709 12:19:53.003561 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:53.002709 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 379998us
I20260709 12:19:53.050148 27587 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:53.018843 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:53.051272 27586 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:53.000698 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2394) took 1302 ms. Trace:
I20260709 12:19:53.052307 27656 rpcz_store.cc:276] 0709 12:19:51.698755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.259977 (+561222us) service_pool.cc:224] Handling call
0709 12:19:52.260391 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.426429 (+166038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.000358 (+573929us) spinlock_profiling.cc:243] Waited 56.6 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.000620 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":130249472}
I20260709 12:19:52.987214 28333 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:53.048177 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:53.054314 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:52.959254 27588 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:53.030967 27585 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:52.979699 28330 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
W20260709 12:19:52.997671 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.055867 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.017594 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:53.057730 28335 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:53.057413 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.031495 28292 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:19:53.059975 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:53.060612 27581 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:53.060840 27656 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:53.061446 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:53.004076 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2042) took 1761 ms. Trace:
I20260709 12:19:53.062872 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.049356 27584 rpcz_store.cc:276] 0709 12:19:49.109060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:49.346371 (+237311us) service_pool.cc:224] Handling call
0709 12:19:51.793468 (+2447097us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:51.837302 (+ 43834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:52.963248 (+1125946us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b540006fc50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:52.963442 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5626986368}
I20260709 12:19:53.066880 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.068003 27729 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:53.070498 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.072561 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:53.073781 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:53.076218 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61634us
I20260709 12:19:53.062290 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:53.077477 28330 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 8: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:53.062781 27737 rpcz_store.cc:276] 0709 12:19:51.242193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.425446 (+1183253us) service_pool.cc:224] Handling call
0709 12:19:52.425741 (+   295us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:52.430792 (+  5051us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:52.430821 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:52.471268 (+ 40447us) log.cc:844] Serialized 52 byte log entry
0709 12:19:52.734474 (+263206us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:52.786091 (+ 51617us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.003767 (+217676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.003838 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:53.003874 (+    36us) raft_consensus.cc:1685] finished
0709 12:19:53.003889 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.004008 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":266070,"prepare.queue_time_us":51377,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":222629,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":25572,"thread_start_us":238694,"threads_started":1,"wal-append.queue_time_us":362296}
I20260709 12:19:53.079917 32060 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:53.080363 32060 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:53.080262 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:53.080892 32060 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.084101 32138 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:52.959546 32261 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:53.084612 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.084863 32138 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:53.085235 27732 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:53.085520 31831 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:53.088632 32202 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:53.089221 32202 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:53.090147 32202 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.091400 32183 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:53.043848 27731 rpcz_store.cc:276] 0709 12:19:50.889887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.374116 (+1484229us) service_pool.cc:224] Handling call
0709 12:19:52.374378 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.374564 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:52.425673 (+ 51109us) log.cc:844] Serialized 14 byte log entry
0709 12:19:52.561317 (+135644us) log.cc:844] Serialized 14 byte log entry
0709 12:19:52.871638 (+310321us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b1800255ac0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:52.993017 (+121379us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b1800255ac0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:19:52.993603 (+   586us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.042761 (+ 49158us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.043169 (+   408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:53.043218 (+    49us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:53.043258 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.043337 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.043447 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":176142,"prepare.queue_time_us":259033,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":11845,"raft.queue_time_us":440898,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":18820,"spinlock_wait_cycles":66053376,"thread_start_us":115206,"threads_started":1,"wal-append.queue_time_us":306014}
I20260709 12:19:53.091977 32183 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:53.092566 32316 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:53.093183 32160 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:53.092868 32183 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.093430 32316 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:53.093814 32316 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:53.010584 32152 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:53.091668 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.075754 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:53.096835 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:53.097582 32248 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:53.097972 32248 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:53.098302 32248 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:53.100883 32181 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:53.101275 32181 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:53.101619 32181 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:53.034986 32169 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.106472 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:53.107256 27656 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:53.107651 27656 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.107757 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:53.110098 27581 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 14. Ops: []
I20260709 12:19:53.110674 27735 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:19:53.110777 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2425) took 1015 ms. Trace:
I20260709 12:19:53.111141 27581 rpcz_store.cc:276] 0709 12:19:52.095304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.109662 (+1014358us) service_pool.cc:224] Handling call
0709 12:19:53.109937 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.110538 (+   601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.110726 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.112341 27581 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
W20260709 12:19:53.116400 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49058.6us
I20260709 12:19:53.115738 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:53.118253 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2429) took 1022 ms. Trace:
I20260709 12:19:53.117389 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.118775 27581 rpcz_store.cc:276] 0709 12:19:52.095980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.111954 (+1015974us) service_pool.cc:224] Handling call
0709 12:19:53.112197 (+   243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.118049 (+  5852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.118204 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.120525 27581 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 13. Ops: []
I20260709 12:19:53.121153 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2430) took 1022 ms. Trace:
I20260709 12:19:53.120952 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:53.122180 27663 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 10: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:53.122526 27581 rpcz_store.cc:276] 0709 12:19:52.098986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.120108 (+1021122us) service_pool.cc:224] Handling call
0709 12:19:53.120374 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.120944 (+   570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.121106 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.123287 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:53.114574 32262 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:53.114828 32089 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.126550 32089 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.130738 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 704) took 1591 ms. Trace:
I20260709 12:19:53.127666 32115 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:53.131800 32115 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:53.132225 32115 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.131278 27730 rpcz_store.cc:276] 0709 12:19:51.539783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.465503 (+925720us) service_pool.cc:224] Handling call
0709 12:19:52.465872 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:52.678120 (+212248us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:19:52.747943 (+ 69823us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.041986 (+294043us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.042203 (+   217us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:53.042264 (+    61us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:19:53.042304 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.130381 (+ 88077us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.130462 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.130646 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":204174,"prepare.queue_time_us":128298,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":79,"raft.queue_time_us":129107,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":174,"spinlock_wait_cycles":487337216,"thread_start_us":134328,"threads_started":1,"wal-append.queue_time_us":200260}
I20260709 12:19:53.129221 31559 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:53.134495 31559 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:53.128341 32138 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:53.135008 31559 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:53.135743 32327 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:53.133340 32115 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.136978 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.021489 32100 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:53.010878 32344 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.158746 31253 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:19:53.161331 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:53.162052 27663 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.108417 27656 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.150909 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:52.997164 32102 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:53.153228 27588 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:53.124119 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.143184 27581 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 10. Ops: []
I20260709 12:19:53.153841 31417 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:53.158667 32105 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:53.227574 27584 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:19:53.150993 32262 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
W20260709 12:19:53.165791 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2283) took 2317 ms (client timeout 1692 ms). Trace:
I20260709 12:19:53.168819 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:53.151530 32169 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.136168 32327 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.144387 31831 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.230446 27588 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:19:53.190078 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:53.230540 27738 rpcz_store.cc:269] 0709 12:19:50.848211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:51.006534 (+158323us) service_pool.cc:224] Handling call
0709 12:19:53.165586 (+2159052us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54000a55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:53.165711 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":751941376}
I20260709 12:19:53.190624 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.231210 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2435) took 1094 ms. Trace:
W20260709 12:19:53.143488 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68009.2us
I20260709 12:19:53.197058 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:53.231741 27588 rpcz_store.cc:276] 0709 12:19:52.137326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.226636 (+1089310us) service_pool.cc:224] Handling call
0709 12:19:53.226933 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.230931 (+  3998us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.231142 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.197309 31253 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:53.200806 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.232795 27735 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.182878 32111 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:19:53.166040 32316 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.167549 32344 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.215543 28332 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:19:53.158138 32089 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:53.150925 32261 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:53.241040 27731 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 12:19:53.234647 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.235339 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2431) took 1131 ms. Trace:
I20260709 12:19:53.235872 27655 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:53.245491 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 723) took 1099 ms. Trace:
I20260709 12:19:53.236224 27588 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 7. (term mismatch)
I20260709 12:19:53.246263 27588 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 6
I20260709 12:19:53.246693 27588 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 12:19:53.241415 27661 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:53.246073 27580 rpcz_store.cc:276] 0709 12:19:52.145722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.347028 (+201306us) service_pool.cc:224] Handling call
0709 12:19:53.245260 (+898232us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b54001fe5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:53.245414 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":85068544}
I20260709 12:19:53.247781 27588 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:53.245091 27581 rpcz_store.cc:276] 0709 12:19:52.103531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.141278 (+1037747us) service_pool.cc:224] Handling call
0709 12:19:53.141574 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.229934 (+ 88360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.235263 (+  5329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.248559 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2443) took 1105 ms. Trace:
I20260709 12:19:53.248911 27588 rpcz_store.cc:276] 0709 12:19:52.143497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.232534 (+1089037us) service_pool.cc:224] Handling call
0709 12:19:53.232807 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.248316 (+ 15509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.248509 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.241628 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:53.250036 27588 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:53.234205 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:53.238127 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.253549 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:53.143314 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:53.258836 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.259584 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:53.272274 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:53.169036 32138 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 12:19:53.230170 32105 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:53.269922 32324 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:53.276906 32126 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:53.169189 32291 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:19:53.183460 32218 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 12:19:53.284478 32089 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:53.259958 32160 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:53.238809 32311 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:19:53.275686 32267 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:19:53.286296 32316 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.260514 32222 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:19:53.358398 32262 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:19:53.287168 27664 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:53.266537 32181 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.243273 32248 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.287336 32327 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.273494 31908 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:19:53.289122 32326 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:53.290832 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2422) took 1196 ms. Trace:
I20260709 12:19:53.409363 27589 rpcz_store.cc:276] 0709 12:19:52.094634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.543475 (+448841us) service_pool.cc:224] Handling call
0709 12:19:52.543883 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:52.628471 (+ 84588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.290558 (+662087us) spinlock_profiling.cc:243] Waited 81.3 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.290770 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186900864}
I20260709 12:19:53.270504 32323 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:53.169227 32294 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:19:53.238600 31831 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.295197 32149 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.432539 32149 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.296196 32111 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.297209 27735 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:53.295238 32344 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:53.433750 32181 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:53.403503 32222 consensus_queue.cc:1243] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:19:53.297904 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:53.298396 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44955.3us
I20260709 12:19:53.434749 27657 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:53.435206 28295 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13.
I20260709 12:19:53.368320 32160 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.436074 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:53.300559 32295 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.437079 32295 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:53.438496 32103 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.500s	user 0.012s	sys 0.000s
I20260709 12:19:53.234915 31559 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.308418 27655 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
W20260709 12:19:53.325208 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.331421 27583 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
W20260709 12:19:53.331995 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.330840 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 719) took 1186 ms (client timeout 1530 ms). Trace:
W20260709 12:19:53.333474 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 164716us
I20260709 12:19:53.332074 31253 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:19:53.442416 27582 rpcz_store.cc:269] 0709 12:19:52.144792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.187852 (+ 43060us) service_pool.cc:224] Handling call
0709 12:19:53.330630 (+1142778us) spinlock_profiling.cc:243] Waited 93 ms on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:53.330776 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213959040}
I20260709 12:19:53.443183 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:53.445462 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.450369 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:53.442130 28336 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:53.446285 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:53.453495 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:53.455674 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:53.333492 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:53.345409 27580 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:53.352142 32259 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:53.356366 32105 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:53.360980 32089 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:53.378640 28335 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:53.359597 32324 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.459182 27658 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:53.381126 27586 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:19:53.460414 32100 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:19:53.460670 31559 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:53.389137 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2427) took 1293 ms. Trace:
I20260709 12:19:53.461593 27584 rpcz_store.cc:276] 0709 12:19:52.095704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.081449 (+985745us) service_pool.cc:224] Handling call
0709 12:19:53.227359 (+145910us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:53.227986 (+   627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.388859 (+160873us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54001892d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:53.389059 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":334736256}
I20260709 12:19:53.463028 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:53.464273 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:53.464836 27584 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:19:53.465821 27584 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:53.466336 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2437) took 1015 ms. Trace:
I20260709 12:19:53.466620 27584 rpcz_store.cc:276] 0709 12:19:52.450877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.465403 (+1014526us) service_pool.cc:224] Handling call
0709 12:19:53.465653 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.466124 (+   471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.466290 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:53.415655 28292 leader_election.cc:336] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.826s (SENT)
W20260709 12:19:53.469688 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.399312 32262 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.380898 27585 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:53.437392 32160 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.318727 31849 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.460855 32100 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:53.472831 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:53.475126 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:53.381596 27663 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
W20260709 12:19:53.418773 31798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.015s	sys 0.037s
I20260709 12:19:53.391923 32060 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.415583 32323 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:53.439266 32295 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:53.347137 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:53.468741 32089 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:53.584743 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.565595 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92849.1us
W20260709 12:19:53.537124 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 86859.2us
I20260709 12:19:53.589033 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:53.523185 32360 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:53.523288 31253 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:53.590904 31253 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:53.534159 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91063.5us
W20260709 12:19:53.524305 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.593760 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.567590 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92628.3us
I20260709 12:19:53.567602 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 703) took 1491 ms. Trace:
I20260709 12:19:53.469225 32169 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:53.568068 27702 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.002s	sys 0.019s
W20260709 12:19:53.598702 27702 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.002s	sys 0.019s
W20260709 12:19:53.599140 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 560293us
W20260709 12:19:53.599864 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.534277 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 200875us
I20260709 12:19:53.601675 27663 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:53.536803 27586 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10. Ops: []
I20260709 12:19:53.602857 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2451) took 1086 ms. Trace:
I20260709 12:19:53.591786 31253 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:53.487603 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71980.4us
I20260709 12:19:53.604526 31253 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:53.485237 32382 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:19:53.594743 28332 rpcz_store.cc:276] 0709 12:19:52.075864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.076231 (+   367us) service_pool.cc:224] Handling call
0709 12:19:52.076641 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.239409 (+1162768us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.567275 (+327866us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.567521 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2618073344}
I20260709 12:19:53.475181 32327 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:53.468180 32133 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:19:53.468506 32137 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13.
I20260709 12:19:53.459870 32152 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:53.603221 27586 rpcz_store.cc:276] 0709 12:19:52.516442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.478826 (+962384us) service_pool.cc:224] Handling call
0709 12:19:53.536579 (+ 57753us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.602642 (+ 66063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.602806 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:53.654711 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:53.655437 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.533569 32294 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:19:53.590126 31798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.019s	sys 0.041s
I20260709 12:19:53.615824 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.662981 27729 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:53.617662 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.619685 32133 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:53.619796 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.607929 32038 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:19:53.604529 32262 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.624837 32111 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:53.589926 32160 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:53.633306 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2424) took 1538 ms. Trace:
I20260709 12:19:53.678642 27583 rpcz_store.cc:276] 0709 12:19:52.095105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.648911 (+553806us) service_pool.cc:224] Handling call
0709 12:19:53.331218 (+682307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.441998 (+110780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.633003 (+191005us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54000bccd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:53.633218 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1402434944}
I20260709 12:19:53.523224 31849 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.635191 32105 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.637492 32115 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:53.637956 32248 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.641637 32360 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.641666 31831 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:53.642990 28332 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:19:53.645639 32344 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.516664 32087 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:53.533732 31908 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:53.619616 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.652746 27738 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:19:53.652979 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.655254 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:53.611395 27656 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:53.655892 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 457737us
I20260709 12:19:53.713843 27661 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:53.714318 27661 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.715052 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:53.590025 32100 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:53.715179 27661 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.592288 32311 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:19:53.657418 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.016s	sys 0.011s
I20260709 12:19:53.716763 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:53.717494 27734 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 12:19:53.657429 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.717720 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.718405 27659 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:53.718729 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.718885 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.483228 32290 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.131s
I20260709 12:19:53.474364 32259 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.673623 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.648144 32295 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:53.677640 27655 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:53.727348 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.634492 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 716) took 1490 ms. Trace:
I20260709 12:19:53.728574 27580 rpcz_store.cc:276] 0709 12:19:52.144157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.248183 (+1104026us) service_pool.cc:224] Handling call
0709 12:19:53.345185 (+ 97002us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.456869 (+111684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.634262 (+177393us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b5400089dd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:53.634425 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222107520}
I20260709 12:19:53.729910 27580 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:53.731065 27580 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:53.609144 32169 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.681313 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:53.683071 27587 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:53.740351 27662 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:19:53.677914 32038 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:53.679272 32160 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:53.693397 27733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.010s	sys 0.063s
W20260709 12:19:53.777101 27733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.010s	sys 0.063s
I20260709 12:19:53.704779 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.778311 27731 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:53.706651 27583 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:53.780025 27657 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:19:53.780205 27663 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:53.780695 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2452) took 1151 ms. Trace:
I20260709 12:19:53.781100 27583 rpcz_store.cc:276] 0709 12:19:52.629144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.679648 (+1050504us) service_pool.cc:224] Handling call
0709 12:19:53.706406 (+ 26758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.780454 (+ 74048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.780643 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:53.716987 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.016s	sys 0.011s
I20260709 12:19:53.665829 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.720544 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.716441 27661 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:53.678076 32138 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:19:53.689697 32395 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:53.720793 32311 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:53.732913 27737 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:53.749259 27580 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:19:53.694335 32266 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.557s	user 0.019s	sys 0.016s
I20260709 12:19:53.751096 32038 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:53.723217 27729 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:53.751435 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:53.750897 32202 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:53.769666 27656 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:19:53.802747 27656 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 6
I20260709 12:19:53.803066 27656 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:19:53.803376 27656 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:53.727115 32087 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:53.782996 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:53.719437 32248 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:53.725457 32267 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:53.792418 32385 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:19:53.754536 32060 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.779693 32160 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:53.796494 27662 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:53.722359 32102 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:53.752513 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:53.814566 27738 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:53.736477 32290 consensus_queue.cc:1243] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:19:53.726845 31908 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.820376 27586 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:53.709153 32111 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 12:19:53.697791 32360 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:53.824540 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:53.803617 32138 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 12:19:53.801807 32266 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.671s	user 0.019s	sys 0.016s
I20260709 12:19:53.825702 31918 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:19:53.825932 32038 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.826624 32105 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.754874 32133 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.828379 32149 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.822293 32202 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.834213 32089 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.851892 32267 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:53.709810 32344 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.887493 32038 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.887636 32169 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:53.888787 32259 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.889137 32344 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:53.794977 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:53.736052 32228 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.093s
I20260709 12:19:53.802043 32311 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:53.754190 32295 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.894300 32089 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:53.821274 32324 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:53.875025 27627 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.610s (SENT) [suppressed 16 similar messages]
I20260709 12:19:53.853077 32248 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.865111 32102 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:53.890261 27581 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.023s	sys 0.018s
W20260709 12:19:53.914366 27581 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.023s	sys 0.018s
I20260709 12:19:53.890419 27735 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:19:53.889833 31849 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.894387 32233 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:53.897821 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 728) took 1059 ms. Trace:
I20260709 12:19:53.888801 32360 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:53.917855 27586 rpcz_store.cc:276] 0709 12:19:52.838468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.721407 (+882939us) service_pool.cc:224] Handling call
0709 12:19:53.721745 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.856709 (+134964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.897529 (+ 40820us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.897747 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226275712}
I20260709 12:19:53.921069 27586 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:19:53.867429 27588 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.015s	sys 0.023s
I20260709 12:19:53.903951 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2439) took 1449 ms. Trace:
I20260709 12:19:53.922391 27585 rpcz_store.cc:276] 0709 12:19:52.454632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.477232 (+1022600us) service_pool.cc:224] Handling call
0709 12:19:53.477704 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.785824 (+308120us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:53.785854 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:53.785914 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:53.785962 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:53.903647 (+117685us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b540011ba54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:53.903707 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:53.903875 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":708059008}
W20260709 12:19:53.924157 27588 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.015s	sys 0.024s
I20260709 12:19:53.910763 27661 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:19:53.925026 27661 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 7
I20260709 12:19:53.927657 32267 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.889874 32323 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:53.930938 32323 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:53.909175 32201 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:19:53.914335 31908 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.892557 32152 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:19:53.858830 27583 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:19:53.931108 32100 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.898320 32087 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:53.925318 27661 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:19:53.953591 28330 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:53.948734 27661 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:53.975484 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:53.935314 32316 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.536s	user 0.004s	sys 0.005s
I20260709 12:19:53.944329 32152 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:53.951478 32327 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.012169 32152 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:54.018061 32327 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:53.945645 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.011s	sys 0.047s
W20260709 12:19:54.096438 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.012s	sys 0.047s
W20260709 12:19:54.097487 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.007s	sys 0.016s
W20260709 12:19:54.098021 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.007s	sys 0.016s
W20260709 12:19:54.100426 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 511405us
I20260709 12:19:54.101341 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2436) took 1650 ms. Trace:
I20260709 12:19:54.105037 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2438) took 1653 ms. Trace:
I20260709 12:19:54.105500 27584 rpcz_store.cc:276] 0709 12:19:52.451178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.467283 (+1016105us) service_pool.cc:224] Handling call
0709 12:19:53.534446 (+ 67163us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:53.535495 (+  1049us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:53.535539 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:53.536072 (+   533us) log.cc:844] Serialized 26 byte log entry
0709 12:19:54.104176 (+568104us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:54.104357 (+   181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.104496 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:54.104581 (+    85us) raft_consensus.cc:1685] finished
0709 12:19:54.104594 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.104873 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":72213,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":41618,"thread_start_us":496797,"threads_started":1,"wal-append.queue_time_us":479253}
W20260709 12:19:54.107182 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 381592us
W20260709 12:19:54.107226 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.116705 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 399851us
W20260709 12:19:54.119753 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 439339us
I20260709 12:19:54.030179 32100 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:54.123845 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 374476us
I20260709 12:19:54.101970 27582 rpcz_store.cc:276] 0709 12:19:52.450550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.447656 (+997106us) service_pool.cc:224] Handling call
0709 12:19:53.448115 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:53.516059 (+ 67944us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:53.516091 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:53.568029 (+ 51938us) log.cc:844] Serialized 26 byte log entry
0709 12:19:54.100283 (+532254us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:54.100430 (+   147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.100594 (+   164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:54.100720 (+   126us) raft_consensus.cc:1685] finished
0709 12:19:54.100755 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.101068 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67506,"raft.queue_time_us":75309,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":1738,"thread_start_us":472380,"threads_started":1,"wal-append.queue_time_us":460931}
I20260709 12:19:54.129132 32102 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.129359 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2044) took 2886 ms. Trace:
I20260709 12:19:54.129891 32102 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.131731 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:54.132220 32105 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:54.133234 32105 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.136572 32202 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.136914 32152 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.138624 32228 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:54.139559 32228 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:54.140050 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.142042 32228 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.143649 31559 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:54.145910 32100 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:19:54.145684 31559 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.146901 32100 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:54.154965 32100 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.156589 32202 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:54.156821 32261 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:54.157343 32261 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:54.157596 32202 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.129750 27733 rpcz_store.cc:276] 0709 12:19:51.242662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.435813 (+1193151us) service_pool.cc:224] Handling call
0709 12:19:52.436205 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:52.467236 (+ 31031us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:52.525271 (+ 58035us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.016892 (+491621us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b180037c120. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:53.777734 (+760842us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:53.778348 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:19:53.796927 (+ 18579us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:54.128022 (+331095us) log.cc:844] Serialized 14 byte log entry
0709 12:19:54.128563 (+   541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.128697 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:54.128871 (+   174us) raft_consensus.cc:1685] finished
0709 12:19:54.128895 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.129130 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37206,"prepare.queue_time_us":398292,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":97,"raft.queue_time_us":115256,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":17003,"spinlock_wait_cycles":28065664,"thread_start_us":1129893,"threads_started":3,"wal-append.queue_time_us":1288670}
I20260709 12:19:54.159199 32089 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:54.145900 32105 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.160282 32089 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.161477 32152 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.161640 32202 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.162436 32202 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:54.163753 31918 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.163934 32202 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.164753 31918 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.174539 32105 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:54.176857 32105 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:54.177384 32149 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:54.179947 32201 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:54.180423 32201 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:54.180190 32160 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.182341 32201 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.131089 32102 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.184185 32160 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.177435 32105 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:54.186667 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68687us
I20260709 12:19:54.160223 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:54.187963 32201 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:54.192425 32105 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.188127 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.196805 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 726) took 1678 ms. Trace:
I20260709 12:19:54.196895 32181 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:54.197893 31918 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.198298 32181 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:54.199131 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.208448 27584 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:54.200347 32089 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.200500 32181 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.201730 27587 rpcz_store.cc:276] 0709 12:19:52.518028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.568811 (+1050783us) service_pool.cc:224] Handling call
0709 12:19:53.601752 (+ 32941us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.740673 (+138921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.196230 (+455557us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:54.196604 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186590464}
I20260709 12:19:54.210671 32181 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:54.210731 32126 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.211658 32126 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.214977 32126 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.159183 32261 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.218200 32261 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:54.219589 32105 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:54.223683 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.225709 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:54.226174 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:54.227202 27587 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:54.227566 27584 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:54.231915 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.233353 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:54.233829 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.234356 27587 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:54.234915 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.235742 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.236711 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.238173 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.245401 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54957.1us
I20260709 12:19:54.254506 32202 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:54.307924 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.316805 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.311059 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:54.323146 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.333916 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:54.339071 32316 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:54.340344 32104 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 12:19:54.343513 32406 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.344237 32406 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.351699 32398 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:54.353178 32295 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.359460 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:54.360553 27583 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:19:54.361558 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.322172 31918 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:19:54.364004 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.364213 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:54.365305 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:54.366019 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:54.366822 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 178990us
I20260709 12:19:54.366963 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:54.367722 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 108788us
I20260709 12:19:54.368942 32327 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.369635 32327 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.370258 32233 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:19:54.370242 32327 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.373510 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:54.375768 31908 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:54.377432 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 718) took 2232 ms. Trace:
W20260709 12:19:54.379108 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72906.3us
I20260709 12:19:54.379653 32060 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:19:54.380244 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.381433 32409 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:54.382061 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.382368 32409 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:54.383608 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:54.384112 32138 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:19:54.384723 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.385061 32138 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:19:54.385452 32138 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:54.386026 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.386879 27582 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:54.375982 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 116810us
I20260709 12:19:54.367084 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:54.389236 27585 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.390393 32248 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.388288 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:54.392617 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:54.394413 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.377815 27581 rpcz_store.cc:276] 0709 12:19:52.144580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.248979 (+1104399us) service_pool.cc:224] Handling call
0709 12:19:53.249300 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.249531 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:19:53.914847 (+665316us) log.cc:844] Serialized 14 byte log entry
0709 12:19:54.341393 (+426546us) log.cc:844] Serialized 14 byte log entry
0709 12:19:54.376759 (+ 35366us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:54.376832 (+    73us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:19:54.376922 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.377024 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.377257 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39076,"prepare.queue_time_us":820374,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":1550,"raft.queue_time_us":50625,"raft.run_cpu_time_us":680,"raft.run_wall_time_us":49021,"thread_start_us":955930,"threads_started":2,"wal-append.queue_time_us":339201}
I20260709 12:19:54.397099 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.368098 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2446) took 1961 ms. Trace:
I20260709 12:19:54.397692 27588 rpcz_store.cc:276] 0709 12:19:52.406260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.251021 (+844761us) service_pool.cc:224] Handling call
0709 12:19:53.251269 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:53.251539 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:53.924665 (+673126us) log.cc:844] Serialized 14 byte log entry
0709 12:19:54.319949 (+395284us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:54.329076 (+  9127us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:54.335151 (+  6075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.335262 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.367910 (+ 32648us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58737,"prepare.queue_time_us":788916,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":6466,"raft.queue_time_us":61815,"spinlock_wait_cycles":4579584,"thread_start_us":957540,"threads_started":2,"wal-append.queue_time_us":331391}
I20260709 12:19:54.399832 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:54.400158 32403 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:19:54.400993 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.401764 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:54.402837 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.343158 32344 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 12:19:54.403627 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:54.404354 27581 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:54.404536 32201 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:54.405159 32201 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:54.405526 32201 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:54.406158 32327 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:54.406450 32393 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.511s	user 0.000s	sys 0.004s
I20260709 12:19:54.404574 32344 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:19:54.401435 32405 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:54.407833 32344 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:54.409139 32344 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.370713 32202 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:54.415117 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:54.402930 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.367755 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:54.417588 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49968.7us
I20260709 12:19:54.418362 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.418695 32295 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:54.419130 32295 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:54.419543 32295 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.386727 32138 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.421671 32316 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:54.422576 32316 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:54.422634 32405 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:19:54.423801 32405 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:54.421161 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.424283 32405 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:54.424875 32202 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:54.425438 32405 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.427013 32405 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:54.428530 32388 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.515s	user 0.001s	sys 0.000s
I20260709 12:19:54.421700 32138 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:54.418334 32181 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:54.383247 32409 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:54.479481 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.482966 32266 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.435587 32104 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:19:54.483883 32266 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.436893 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:54.454259 32111 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:54.454944 32395 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:54.488948 32395 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:54.489992 32119 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:54.490190 32395 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.436779 31918 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:19:54.490742 32119 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.491180 27584 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:19:54.492076 27581 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:19:54.456297 32316 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:54.439035 32262 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:19:54.446760 32398 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:54.493270 32119 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.494850 32104 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:54.495496 32104 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.496397 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:54.497112 28336 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:54.498135 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.498500 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.498621 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:54.498755 27655 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:54.500172 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.500799 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.497494 32104 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.463904 27585 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:54.464393 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:54.466338 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 213502us
I20260709 12:19:54.435310 27704 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:19:54.503404 27704 leader_election.cc:403] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:19:54.469338 31559 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.470438 32324 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.471279 27658 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:19:54.471462 32102 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.472307 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:54.473289 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.518484 27663 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:19:54.474174 32392 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.587s	user 0.000s	sys 0.001s
W20260709 12:19:54.474377 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.347137 32406 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.505167 27584 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:19:54.524068 28337 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
W20260709 12:19:54.519253 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.529698 27586 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.463581 32202 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.061s
W20260709 12:19:54.535722 32294 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.832s	user 0.007s	sys 0.006s
I20260709 12:19:54.536403 32294 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:19:54.537616 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.538992 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:19:54.543876 28336 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:19:54.544238 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 149949us
I20260709 12:19:54.545521 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:54.543872 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.547425 32398 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:54.547495 32324 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.549196 32398 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:54.521111 32104 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.549721 32398 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:54.549990 32324 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.554359 28336 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:19:54.559545 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:19:54.559782 27588 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:54.561053 27588 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:19:54.550057 32104 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.564028 32104 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.550868 32398 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.519114 32105 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:54.533772 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:54.578027 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 142965us
I20260709 12:19:54.523675 32266 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.588994 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:54.591524 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:54.596628 27587 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:19:54.577972 32105 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:54.617960 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:54.619748 31559 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.620441 31559 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.622730 31559 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.623265 32104 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:54.623471 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2267) took 4465 ms. Trace:
I20260709 12:19:54.623719 32104 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:54.624873 32104 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.623880 27736 rpcz_store.cc:276] 0709 12:19:50.128094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.828786 (+700692us) service_pool.cc:224] Handling call
0709 12:19:51.916759 (+1087973us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:51.976134 (+ 59375us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:52.071592 (+ 95458us) log.cc:844] Serialized 14 byte log entry
0709 12:19:53.125122 (+1053530us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b1800152860. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:19:54.131906 (+1006784us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:54.132764 (+   858us) log.cc:844] Serialized 26 byte log entry
0709 12:19:54.132994 (+   230us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:54.133043 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.192631 (+ 59588us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:54.192731 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:54.427488 (+234757us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:54.476802 (+ 49314us) raft_consensus.cc:1685] finished
0709 12:19:54.476829 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.587683 (+110854us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001d42d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:54.593321 (+  5638us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8368,"prepare.queue_time_us":161201,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":11842,"raft.queue_time_us":109660,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":9037,"spinlock_wait_cycles":2630980480,"thread_start_us":1081584,"threads_started":3,"wal-append.queue_time_us":1770067}
I20260709 12:19:54.626160 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.626906 27730 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:54.629305 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:54.629901 32323 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.696s	user 0.014s	sys 0.000s
I20260709 12:19:54.631007 27589 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 12:19:54.586376 32294 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:19:54.631357 27589 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 6
I20260709 12:19:54.631649 27589 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:19:54.631937 27589 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:54.632050 32097 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:54.632576 32097 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:54.633000 32097 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.633464 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2450) took 2223 ms. Trace:
W20260709 12:19:54.633944 32311 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.724s	user 0.008s	sys 0.006s
I20260709 12:19:54.633870 27589 rpcz_store.cc:276] 0709 12:19:52.410005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.410708 (+1000703us) service_pool.cc:224] Handling call
0709 12:19:53.411104 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:54.632449 (+1221345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.633187 (+   738us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54002ba7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:54.633406 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2803933952}
I20260709 12:19:54.634586 32311 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.613453 32105 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:54.637426 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:54.638413 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.638864 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.587635 32406 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:19:54.641216 31918 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:54.645977 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2459) took 1474 ms. Trace:
I20260709 12:19:54.648061 27660 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:19:54.647163 28337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 707) took 1768 ms (client timeout 2160 ms). Trace:
W20260709 12:19:54.657248 28337 rpcz_store.cc:269] 0709 12:19:52.878796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:52.879535 (+   739us) service_pool.cc:224] Handling call
0709 12:19:54.646868 (+1767333us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:54.647054 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3474922368}
I20260709 12:19:54.616891 32316 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.660959 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.662070 27586 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:54.626185 32104 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:54.670855 32344 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:19:54.673072 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.673712 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.674688 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.675457 32202 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:19:54.680071 27656 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
W20260709 12:19:54.686395 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.599198 32181 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.691704 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.693794 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.660967 32100 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:19:54.659638 31918 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:54.696630 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:54.698575 27730 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:54.671191 32311 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.705292 27587 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:19:54.656407 27580 rpcz_store.cc:276] 0709 12:19:53.171060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:53.795791 (+624731us) service_pool.cc:224] Handling call
0709 12:19:53.796149 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:53.873713 (+ 77564us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:53.873741 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:53.874266 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 12:19:54.387407 (+513141us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:54.387858 (+   451us) log.cc:844] Serialized 14 byte log entry
0709 12:19:54.388181 (+   323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:54.645410 (+257229us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:54.645454 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:54.645501 (+    47us) raft_consensus.cc:1685] finished
0709 12:19:54.645612 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:54.645785 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":189,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":2399,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":1442,"spinlock_wait_cycles":177483648,"thread_start_us":434965,"threads_started":1,"wal-append.queue_time_us":372766}
I20260709 12:19:54.658671 32119 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:54.660543 32233 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.710637 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:19:54.714224 27614 heartbeater.cc:646] Failed to heartbeat to 127.25.202.62:43715 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.25.202.62:43715 timed out after 15.000s (SENT)
I20260709 12:19:54.719667 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:54.723527 27662 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:19:54.719532 32201 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.646781 32295 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.732124 27661 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:54.733551 27581 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:54.742585 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:54.746906 27586 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:19:54.735580 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:54.759418 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.731381 27660 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:19:54.766353 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77031.5us
W20260709 12:19:54.769929 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.759136 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.746007 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.777789 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:54.778837 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.782346 32201 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.744932 32181 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.784442 32295 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.755509 32294 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.785379 32294 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.786885 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.773758 27589 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:54.788756 32233 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.791494 32404 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:54.735937 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.751847 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.822005 27657 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:19:54.731732 31918 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:54.818392 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
W20260709 12:19:54.844444 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:54.823931 32404 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:54.849859 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:54.851119 27729 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:54.857564 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.840996 32181 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:54.856227 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.852789 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.849227 32404 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:54.860910 27738 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 14 because replica has last-logged OpId of term: 13 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 12:19:54.863451 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138988us
I20260709 12:19:54.864279 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:54.846096 32406 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.860232 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:54.852684 32262 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.870613 27734 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:54.859391 32138 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:54.860195 32316 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
W20260709 12:19:54.878165 32427 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:19:54.878515 27734 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:19:54.879338 32169 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:54.879768 32169 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:54.880767 32169 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.881824 32169 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:54.883095 27731 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:54.884125 32311 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:54.884863 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.885943 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:54.895807 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:54.897909 27738 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:54.897481 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:54.902508 32233 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.903748 32267 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:54.904397 32267 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.904608 32105 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:54.905304 27736 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:19:54.906663 32105 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:54.885636 32181 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.893060 32406 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.909494 32181 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.910060 32406 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.874188 32262 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.910805 32428 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:19:54.912446 32262 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.906683 32267 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:54.927996 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63832.6us
W20260709 12:19:54.950914 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.965830 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:54.960927 32441 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.008s	sys 0.000s
I20260709 12:19:54.971344 32105 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:54.971395 32436 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.010s	sys 0.000s
I20260709 12:19:54.971951 32105 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.974334 32105 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:54.975466 32432 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:19:54.897555 32138 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.979902 32138 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:54.983408 32097 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:54.986410 32097 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:54.988969 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:54.990155 32097 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:54.991254 32097 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:54.991770 32097 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:55.000902 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:55.006460 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.004711 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:55.009285 27729 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:55.003895 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:55.013893 31918 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:55.018172 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 141755us
I20260709 12:19:55.011795 27737 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:55.031893 31831 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:19:55.032449 31831 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:55.032894 31831 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:55.035194 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:55.046629 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:55.054731 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:55.055721 27662 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:55.056165 31918 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:55.066900 32405 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.601s	user 0.009s	sys 0.001s
I20260709 12:19:55.072213 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:55.077054 27483 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 10 to 12, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:55.081254 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:19:55.082468 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.084319 27580 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:19:55.084851 32426 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.412s	user 0.000s	sys 0.002s
I20260709 12:19:55.085192 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.085565 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.086095 27587 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:55.086952 32439 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 12:19:55.084416 27581 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:19:55.086952 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.086854 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.097856 27583 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:19:55.098867 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.100044 32294 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.079203 27483 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 11 to 12, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 12 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:55.112818 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:55.114619 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:55.115767 32201 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:19:55.116335 32102 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:55.116822 32102 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:55.117004 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:55.117964 32100 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:55.118546 32100 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:55.119573 32100 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.117223 32102 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:55.162115 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 131357us
I20260709 12:19:55.162758 32428 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:19:55.209649 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.010056 32398 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:19:55.268671 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:55.116344 32201 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:55.093926 27738 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:55.114634 32087 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:19:55.275763 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.275787 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.163497 32395 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.292662 27657 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:19:55.301916 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:55.312911 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:55.272078 32119 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:55.308033 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:55.316123 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47598.8us
I20260709 12:19:55.316190 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.315160 27582 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:55.304003 32087 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:55.326648 32395 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:55.340572 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 188469us
I20260709 12:19:55.343550 32138 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:55.342912 32395 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.321653 31918 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:19:55.325019 32323 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.324388 32311 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:55.333992 32294 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:55.367559 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:55.312439 32233 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:19:55.366680 27586 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:19:55.351333 32398 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:19:55.375787 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:55.376958 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75584.7us
W20260709 12:19:55.380230 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.382843 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:55.337666 32149 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 12:19:55.382012 27659 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:19:55.375355 32126 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.013s	sys 0.010s
I20260709 12:19:55.392696 27655 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:19:55.375145 32398 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:55.399631 32311 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.400326 32149 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.402048 32097 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.374027 32323 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:55.397928 32126 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.013s	sys 0.010s
I20260709 12:19:55.390008 32267 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:55.420387 32087 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.435202 31831 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.484022 32087 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:55.485114 31831 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:55.487637 32181 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.447839 32398 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:55.488265 32181 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.490113 32138 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.490703 32138 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.421845 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.423270 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:55.428020 32119 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:55.431810 32105 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:55.432549 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:55.436076 32443 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.429s	user 0.003s	sys 0.010s
I20260709 12:19:55.436368 32262 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:55.443552 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51170.6us
I20260709 12:19:55.444075 32327 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:19:55.444730 32201 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:55.445770 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58306.6us
I20260709 12:19:55.384752 32294 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:55.392879 32406 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:19:55.415745 32434 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
W20260709 12:19:55.447166 32404 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.000s
W20260709 12:19:55.462030 32435 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.462s	user 0.000s	sys 0.004s
W20260709 12:19:55.471911 32445 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:19:55.474406 32149 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.481066 32323 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.482733 32097 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:55.487761 32311 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:55.496495 32138 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.541595 32327 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 12:19:55.498301 27588 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:55.512632 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:55.556119 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43585.7us
I20260709 12:19:55.557469 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:55.558130 27589 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:55.558225 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:55.513633 32395 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.560931 32395 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.559262 27732 raft_consensus.cc:606] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:55.562005 27732 raft_consensus.cc:614] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:55.502230 32406 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:55.563575 32406 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:55.563477 32453 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:19:55.564555 32181 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.498579 32119 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:55.496941 32267 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:55.514147 27483 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:55.561162 32344 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:19:55.515730 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:55.571071 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 280007us
I20260709 12:19:55.563136 32395 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:55.515848 32428 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.552090 32323 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.566267 32267 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.533754 32294 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.546171 32398 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.514472 32404 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.593669 27589 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:55.513175 32105 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.585429 27483 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 9 to 12, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:55.594749 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:55.601697 27588 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:55.551440 32262 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.568681 32102 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.582736 32344 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:55.585917 32395 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:55.597527 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.604646 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.612432 27583 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
W20260709 12:19:55.606767 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 238047us
W20260709 12:19:55.612497 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.614385 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:55.630781 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 246256us
W20260709 12:19:55.632113 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.633985 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.634819 27732 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:19:55.636312 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.637275 27738 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:19:55.599244 32323 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.630861 32087 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:55.646171 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.647996 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.648633 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:55.652175 27589 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:19:55.654263 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:55.655166 27735 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:19:55.650697 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.637511 32409 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.659571 27731 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:55.663846 32105 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:55.664943 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.665645 28328 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:55.667438 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.668095 32398 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.671309 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.673211 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:55.676016 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.676267 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.646858 32181 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:19:55.680498 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:19:55.681084 28337 raft_consensus.cc:606] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:55.671648 32452 raft_consensus.cc:2764] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:19:55.677953 32087 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.687564 27483 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 11, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 11 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:55.627358 32428 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.650756 32395 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.698161 32233 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.698717 32233 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.693147 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.670557 32409 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.674969 32138 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 12:19:55.704270 27483 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 3 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 11 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:55.710180 27586 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:19:55.689174 32087 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.710332 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:55.711598 27483 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 2 to 14, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 14 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:55.701849 32149 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.724190 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:55.727983 28337 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:55.680109 32344 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.731935 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.717679 27483 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 6 to 11, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:55.737337 27731 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:55.717973 32404 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:55.739787 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:55.744702 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:55.745424 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 145247us
I20260709 12:19:55.722828 32201 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:55.745880 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 231001us
I20260709 12:19:55.750030 32316 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.750749 32316 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.753767 32316 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:55.756028 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:55.733188 32100 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.013s	sys 0.009s
I20260709 12:19:55.761046 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.725853 32087 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:55.727339 32262 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:55.759934 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.759934 27483 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:55.757728 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.759858 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.709718 31559 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 12:19:55.709546 32311 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 12:19:55.765321 27584 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:55.732640 32102 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:55.716235 32406 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:55.761262 32100 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.013s	sys 0.009s
I20260709 12:19:55.762943 32201 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:55.769047 27586 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:55.769663 27734 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:55.775048 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.781407 27738 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:19:55.778995 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:55.782191 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:55.803728 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:55.783080 27731 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:55.784471 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:55.822326 27735 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:55.824347 27582 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:19:55.764436 32087 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:55.745347 32169 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:19:55.828052 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:55.831450 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64741.8us
I20260709 12:19:55.835593 27483 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 10, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 10 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:19:55.839749 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 191211us
I20260709 12:19:55.842854 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.782588 32458 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:55.847571 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.832026 32097 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:19:55.838322 32457 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:55.849251 32457 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:55.849619 32457 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:55.854158 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 149852us
I20260709 12:19:55.857111 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:55.858690 27659 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 12:19:55.833931 32087 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:55.876057 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:55.859061 27659 pending_rounds.cc:85] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 4
I20260709 12:19:55.880580 27659 pending_rounds.cc:107] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:19:55.880897 27659 raft_consensus.cc:2889] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:55.881680 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:55.883420 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.781641 32452 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 12:19:55.891887 27663 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:19:55.891942 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53274.4us
I20260709 12:19:55.884255 27731 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:55.896914 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.898561 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.900804 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:55.844465 32323 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:55.905861 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:55.908360 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:55.914815 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:55.918740 27655 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:55.922257 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.842002 32102 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 12:19:55.865006 32097 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:55.927260 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47827.7us
I20260709 12:19:55.915725 27738 raft_consensus.cc:606] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:55.942435 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.943995 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:55.878584 32458 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:55.951215 27663 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:19:55.891808 32453 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.959592 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.964003 27588 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:55.953310 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.966387 27734 raft_consensus.cc:2368] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:19:55.966542 27729 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 6.
I20260709 12:19:55.954849 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:55.952509 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 186993us
I20260709 12:19:55.969297 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:55.970086 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:55.970498 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:55.960249 27731 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:55.971673 27737 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:19:55.965675 28337 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:55.955122 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.954023 27483 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 12, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:55.967388 28332 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:55.975873 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:55.969398 27733 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:19:55.952078 32323 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:55.986922 27735 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:55.989404 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.951373 32454 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:19:55.946430 32138 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:19:55.992569 27483 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:56.003690 27729 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:55.971516 32233 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:55.974499 32428 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.014000 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:56.012814 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.014627 27734 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:19:56.016029 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 171949us
W20260709 12:19:56.017743 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53172us
I20260709 12:19:56.018797 32465 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 12:19:56.007041 32100 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:56.028482 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.024312 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.036892 27729 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
W20260709 12:19:56.038312 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62494us
W20260709 12:19:56.029564 27486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 222) took 30153 ms (client timeout 15000 ms). Trace:
I20260709 12:19:56.042986 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.049193 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:55.989162 32323 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.066030 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:56.066653 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.067798 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.068666 27729 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:56.031085 32267 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:56.075299 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.071775 32233 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:56.037755 32406 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.035782 32100 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:56.051924 32428 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.053969 32398 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:56.082674 27735 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:56.089421 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.022927 27483 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 10 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 11 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:56.064698 32395 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:56.123979 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:56.124336 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:56.078562 32267 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:56.128937 27731 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:56.080303 32233 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:56.088433 32100 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:56.136875 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.138485 27729 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:19:56.139438 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.134526 27483 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 8 to 9, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 9 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:56.156463 27729 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:19:56.042474 32097 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:56.164181 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:56.165908 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:56.166703 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 130789us
I20260709 12:19:56.133687 32406 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:56.131762 32267 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:56.171020 27483 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 9, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 9 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:56.166600 28332 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:56.136782 32233 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:56.242125 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81825.1us
W20260709 12:19:56.235708 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 169726us
I20260709 12:19:56.244143 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:56.249751 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:56.251796 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.252669 27729 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:56.133121 32404 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:56.256878 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:56.094967 32428 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.164098 32478 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.129458 32398 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.163581 32395 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:56.166106 32138 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.239487 32406 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:56.288823 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44789.7us
I20260709 12:19:56.275007 32481 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.276774 32102 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:56.309113 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.297405 27483 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 9 to 10, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 10 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:56.272492 32404 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:56.274520 32479 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:56.293653 32406 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:19:56.317487 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 177888us
I20260709 12:19:56.348436 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.362499 27659 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:56.356010 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:56.305060 32102 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:56.367858 27655 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:56.299995 32476 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:56.286245 32395 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.354249 27664 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 12:19:56.371511 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.362596 27483 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:19:56.373301 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 128726us
I20260709 12:19:56.376828 27659 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:56.362151 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:56.360220 32469 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.325s	user 0.001s	sys 0.000s
I20260709 12:19:56.310644 32316 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 12:19:56.374646 32102 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:56.376148 27664 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 7
I20260709 12:19:56.460412 27664 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 12:19:56.460907 27664 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:56.365170 32461 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.346s	user 0.000s	sys 0.001s
I20260709 12:19:56.163957 32457 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.283339 32478 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.401069 27483 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:56.320492 32485 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:19:56.039909 27486 rpcz_store.cc:269] 0709 12:19:25.678612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.678882 (+   270us) service_pool.cc:224] Handling call
0709 12:19:55.823495 (+30144613us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:49.157987 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:49.559992 (+402005us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:49.569408 (+  9416us) write_op.cc:435] Acquired schema lock
0709 12:19:49.572327 (+  2919us) tablet.cc:662] Decoding operations
0709 12:19:49.593809 (+ 21482us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:49.593923 (+   114us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:49.593963 (+    40us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:19:49.594961 (+   998us) tablet.cc:701] Row locks acquired
0709 12:19:49.595016 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:19:49.595221 (+   205us) write_op.cc:270] Start()
0709 12:19:49.646438 (+ 51217us) write_op.cc:276] Timestamp: P: 1783599589595143 usec, L: 0
0709 12:19:49.646492 (+    54us) op_driver.cc:348] REPLICATION: starting
0709 12:19:49.747279 (+100787us) log.cc:844] Serialized 2830 byte log entry
0709 12:19:51.215707 (+1468428us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:51.883210 (+667503us) op_driver.cc:464] REPLICATION: finished
0709 12:19:52.162621 (+279411us) write_op.cc:301] APPLY: starting
0709 12:19:52.226421 (+ 63800us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:52.274253 (+ 47832us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:52.274302 (+    49us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:52.432979 (+158677us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:52.525832 (+ 92853us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=14,mrs_lookups=0
0709 12:19:52.579723 (+ 53891us) write_op.cc:312] APPLY: finished
0709 12:19:52.638805 (+ 59082us) log.cc:844] Serialized 73 byte log entry
0709 12:19:52.638999 (+   194us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:52.639139 (+   140us) write_op.cc:454] Released schema lock
0709 12:19:52.639206 (+    67us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:52.639673 (+   467us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:52.639707 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":277058,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":188150,"apply.run_cpu_time_us":14269,"apply.run_wall_time_us":506008,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33387,"num_ops":7,"prepare.queue_time_us":268537,"prepare.run_cpu_time_us":33373,"prepare.run_wall_time_us":1081124,"raft.queue_time_us":229976,"raft.run_cpu_time_us":45176,"raft.run_wall_time_us":298244,"replication_time_us":2192386,"spinlock_wait_cycles":111364864,"thread_start_us":858343,"threads_started":4,"wal-append.queue_time_us":528568,"wal-append.run_cpu_time_us":39324,"wal-append.run_wall_time_us":3329968}]]}
I20260709 12:19:56.289081 32138 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:55.977288 32453 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.463800 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.321372 32473 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:56.461706 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:56.465133 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 170533us
I20260709 12:19:56.468881 27661 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:56.469575 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.352625 32454 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 12:19:56.471935 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.310763 32480 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.164s
W20260709 12:19:56.476366 32475 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.396s	user 0.000s	sys 0.001s
W20260709 12:19:56.476985 32119 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.901s	user 0.004s	sys 0.007s
I20260709 12:19:56.479409 27483 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:56.474733 27658 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:56.484022 27661 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:19:56.072114 32458 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.489140 32138 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:56.489500 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112101us
I20260709 12:19:56.490749 32428 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.491411 27655 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:56.520814 27655 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:56.491916 32473 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.404359 32476 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.496430 32457 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.308725 32481 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.499661 32323 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.501896 32478 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:56.502358 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 221622us
I20260709 12:19:56.503005 32485 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:56.505681 32453 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:56.395304 32490 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:56.508827 32395 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.517422 27656 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:56.522598 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:56.518525 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 156181us
I20260709 12:19:56.594380 27661 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:56.594551 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.361649 32479 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.530042 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:56.530853 27483 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 9, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 9 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:19:56.514433 32474 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.437s	user 0.000s	sys 0.001s
I20260709 12:19:56.541818 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:56.528753 32483 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 12:19:56.547282 32467 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:19:56.570793 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.545449 32104 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:56.551719 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.009s	sys 0.030s
I20260709 12:19:56.569784 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.570485 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:56.636157 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.010s	sys 0.031s
I20260709 12:19:56.636217 32104 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:56.637006 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 66622.9us
I20260709 12:19:56.637840 31918 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:19:56.638345 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:56.638595 32104 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:56.639125 32104 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:56.639755 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:56.533324 32428 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 12:19:56.533445 32468 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 12:19:56.521385 32119 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:56.627588 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:56.633529 27661 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:56.641502 32201 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.642684 32119 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:56.642958 27655 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260709 12:19:56.645282 32472 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.513s	user 0.000s	sys 0.001s
I20260709 12:19:56.646559 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:56.583153 32490 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:56.649410 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:56.622191 32262 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:56.667418 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132304us
I20260709 12:19:56.667524 27483 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:56.526980 32404 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.678534 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:56.686331 27734 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.671s	user 0.010s	sys 0.000s
I20260709 12:19:56.686810 27734 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:56.689561 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.694433 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.597523 32102 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.695924 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:56.696753 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 227173us
I20260709 12:19:56.666049 32398 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:56.694105 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47932.9us
I20260709 12:19:56.674309 32316 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:56.690928 27483 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:56.715065 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:56.698657 32404 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:56.698402 32480 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.689958 32496 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:56.697125 32489 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:56.731878 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.699393 32097 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:56.728657 27483 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 9 to 10, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 10 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:19:56.676040 32201 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
W20260709 12:19:56.734182 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:56.680176 32262 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.707309 32316 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:56.707965 32491 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:56.746573 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:56.748262 27661 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
W20260709 12:19:56.758175 32087 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.837s	user 0.009s	sys 0.000s
W20260709 12:19:56.758515 32100 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.002s
W20260709 12:19:56.759748 32508 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:19:56.764465 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.768919 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.752027 27483 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:19:56.761214 32495 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
W20260709 12:19:56.761871 32499 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.165s	user 0.000s	sys 0.006s
W20260709 12:19:56.762562 32487 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:19:56.740114 32395 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.792850 27483 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 7 to 9, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 9 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:19:56.777931 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.751080 32316 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:56.821491 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.789742 32511 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:56.793679 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.766665 32453 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.826119 32453 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.826802 32479 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:56.828024 32504 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:19:56.800597 32267 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.523s	user 0.010s	sys 0.000s
I20260709 12:19:56.774165 32344 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:56.767807 32473 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.804628 32102 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.807685 32506 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:56.737425 32480 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.753101 32490 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:56.809083 32233 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.524s	user 0.009s	sys 0.000s
I20260709 12:19:56.810338 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.764536 32489 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:56.811586 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65091.2us
I20260709 12:19:56.817934 32406 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.766472 32097 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:56.820907 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:56.758656 32087 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.825130 27661 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:56.865814 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:56.766821 32104 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:56.829233 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.870944 27737 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:56.830443 27729 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:19:56.767015 32119 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:56.872890 32119 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:56.873821 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88117.9us
W20260709 12:19:56.874783 27627 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.610s (SENT)
I20260709 12:19:56.832703 32262 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.833105 32511 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.875283 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:56.774786 32100 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.836649 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:56.880793 27658 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:56.717829 32485 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.852568 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:19:56.854056 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 84077.2us
W20260709 12:19:56.865453 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44630.1us
I20260709 12:19:56.885972 27701 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.828910 32395 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.878993 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:56.835269 32481 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.876022 27734 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:56.887172 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:56.887565 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.885262 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.892547 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:56.893164 27588 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:56.894047 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:56.895227 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:56.895810 27580 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:56.866442 32476 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.876749 32506 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:56.899035 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:56.905833 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:56.906432 27661 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:19:56.900681 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:56.837323 32267 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.903076 28337 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:19:56.898900 32104 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:56.904663 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:56.899525 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:56.912921 27582 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:19:56.864028 32233 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:56.910507 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:56.917202 27731 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:56.917619 27731 raft_consensus.cc:614] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:56.886785 32491 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.884640 32473 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:56.895027 32467 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:56.916435 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:56.911620 27589 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:19:56.934363 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:56.903712 32097 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.922111 27584 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:56.922151 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:56.936411 28335 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:56.937536 27663 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:19:56.888870 32478 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.943485 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:56.944291 27587 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:56.945564 28335 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:19:56.923920 27661 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:19:56.890522 32452 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:56.914785 27586 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:19:56.911487 32481 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:56.911952 32323 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:56.950006 28333 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:56.865545 32489 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:56.910944 32344 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:56.928359 32409 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:56.931512 32119 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:56.939821 32467 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:56.960697 32490 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:56.912102 32479 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
W20260709 12:19:56.979202 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 104497us
I20260709 12:19:56.933007 32491 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:56.958484 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59536.7us
W20260709 12:19:56.995146 32460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.044s	user 0.024s	sys 0.061s
I20260709 12:19:56.996659 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:57.011220 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:56.964200 32323 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:56.938413 32473 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.017933 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:19:57.018589 27731 raft_consensus.cc:606] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:57.018915 27731 raft_consensus.cc:614] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:56.990520 32467 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:56.989813 32409 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:56.988662 32453 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:56.954986 32478 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:57.013160 32460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.025s	sys 0.061s
I20260709 12:19:56.951246 32506 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:19:57.039049 32169 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:19:57.043028 27627 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.692s (SENT)
W20260709 12:19:57.040597 32521 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:19:56.961825 32452 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:57.041798 32477 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.502s	user 0.002s	sys 0.000s
I20260709 12:19:56.986478 32344 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.020231 32323 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:57.016021 32490 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:57.101100 27588 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:57.041790 32511 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.092274 32505 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:19:57.035717 32516 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:19:57.112239 32524 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:19:57.123447 32406 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.124500 32490 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.069525 32476 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.037984 32409 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.094843 32458 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.132403 32087 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.183928 27589 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:19:57.186780 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:19:57.187757 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 155501us
I20260709 12:19:57.160001 32480 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.161899 32398 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:57.168733 32404 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:19:57.194186 27627 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.863s (SENT)
I20260709 12:19:57.194054 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.200219 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.172879 32516 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:57.180603 32267 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.164952 32476 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.167876 32458 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.193017 32404 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:57.192940 32395 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.214612 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:19:57.196033 32398 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:57.227807 32527 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.124s	user 0.000s	sys 0.002s
I20260709 12:19:57.229699 32489 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.215656 32102 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:19:57.314064 32452 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.314780 32511 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:19:57.315310 32471 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.264s	user 0.010s	sys 0.000s
I20260709 12:19:57.316083 32515 raft_consensus.cc:993] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:19:57.316211 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.317057 32476 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.317559 27580 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
W20260709 12:19:57.318174 32530 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.148s	user 0.000s	sys 0.003s
I20260709 12:19:57.316113 32473 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.320453 32491 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.321017 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.335716 32479 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.322503 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:57.225121 32522 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.323207 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.258630 32516 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:57.332206 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 368106us
W20260709 12:19:57.336225 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147806us
I20260709 12:19:57.371517 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:57.338729 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.342414 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:19:57.350622 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.320019 32404 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.353034 28334 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:57.353201 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.374509 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:57.322068 32478 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.350188 32201 raft_consensus.cc:1081] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:19:57.378044 28335 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:57.356581 27584 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:19:57.363709 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:57.376037 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:57.374594 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:57.375484 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:57.379401 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:57.352283 32452 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:19:57.386261 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 457048us
I20260709 12:19:57.386181 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:57.352485 32454 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.391149 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:57.396780 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.397547 28335 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:19:57.398520 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.398543 27655 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:57.399514 32473 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:57.400599 32473 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:57.396658 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.335737 32316 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.341661 32511 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:57.396816 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.429991 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.433040 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.433401 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.430860 27583 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:19:57.376457 32398 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.435386 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.437011 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:57.438591 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.441924 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 745) took 1723 ms. Trace:
I20260709 12:19:57.398916 32454 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.445575 32398 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:57.445803 32478 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:19:57.446798 32536 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:57.446867 32480 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:57.447546 32485 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:57.441104 32102 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:57.448308 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.448576 32531 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:19:57.395148 32453 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:57.442435 27585 rpcz_store.cc:276] 0709 12:19:55.718457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:55.718593 (+   136us) service_pool.cc:224] Handling call
0709 12:19:55.718890 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:55.732137 (+ 13247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:55.732172 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:55.766150 (+ 33978us) log.cc:844] Serialized 26 byte log entry
0709 12:19:55.829497 (+ 63347us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:55.829557 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:55.829632 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:57.314665 (+1485033us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:57.429204 (+114539us) raft_consensus.cc:1685] finished
0709 12:19:57.429234 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:57.441832 (+ 12598us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7569,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":9935,"spinlock_wait_cycles":25438592,"thread_start_us":59325,"threads_started":1,"wal-append.queue_time_us":234010}
W20260709 12:19:57.469343 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 458167us
I20260709 12:19:57.376289 32522 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.471110 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:57.472589 27627 leader_election.cc:336] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.534s (SENT)
I20260709 12:19:57.366600 32479 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.497959 32406 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:19:57.498406 32406 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.432698 32476 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:19:57.494035 32409 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.498900 32406 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:57.499428 32395 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:57.499827 32395 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.500104 32316 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.500172 32395 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:57.433939 32528 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 12:19:57.426891 32473 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:57.502254 32473 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.503535 32473 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:57.505113 32479 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.505505 32476 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:57.505894 32476 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:57.506764 32476 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.435410 32404 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:57.401024 32452 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.464933 32398 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:57.494019 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45835.5us
I20260709 12:19:57.535995 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:57.537384 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:57.490077 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 97180.7us
I20260709 12:19:57.493464 32485 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:57.497414 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.469866 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:57.510958 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:57.401371 32470 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:19:57.532850 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.491874 32536 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:57.539215 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69467.7us
I20260709 12:19:57.554368 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:57.540719 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68159us
I20260709 12:19:57.513959 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.541733 28331 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:57.557199 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:57.543222 32479 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:57.542995 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.543527 32476 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:57.541599 32485 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:57.539966 32398 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:57.546115 32406 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.536181 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:57.561892 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.548698 32480 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:57.551687 32458 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.532158 32201 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:19:57.552340 32395 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.541584 32531 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:19:57.553452 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 174166us
W20260709 12:19:57.559122 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.073s	sys 0.115s
W20260709 12:19:57.560002 27627 leader_election.cc:336] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.704s (SENT)
I20260709 12:19:57.573264 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:57.543118 32104 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.001s
I20260709 12:19:57.541025 32323 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:57.565641 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:57.577441 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.555974 32102 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:19:57.568538 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.556205 32536 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:57.558629 32479 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:57.574427 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.073s	sys 0.115s
I20260709 12:19:57.563068 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:19:57.575384 28292 leader_election.cc:336] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.027s (SENT)
I20260709 12:19:57.568184 31918 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:19:57.547848 32532 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:19:57.612584 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:57.613430 27583 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:57.578456 32470 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:19:57.615461 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73173.9us
I20260709 12:19:57.595890 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 734) took 1020 ms. Trace:
I20260709 12:19:57.580791 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:57.596269 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:57.569520 32201 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.622187 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.622465 28333 rpcz_store.cc:276] 0709 12:19:56.575212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.575348 (+   136us) service_pool.cc:224] Handling call
0709 12:19:56.625856 (+ 50508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:56.960595 (+334739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:57.595604 (+635009us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:57.595814 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":727439616}
I20260709 12:19:57.591378 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.606876 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.569855 32511 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.618475 32119 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.576705 32463 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:57.621402 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2033) took 1072 ms. Trace:
W20260709 12:19:57.631183 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71199.5us
W20260709 12:19:57.620726 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60420.4us
W20260709 12:19:57.621788 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2086) took 1722 ms (client timeout 1734 ms). Trace:
W20260709 12:19:57.580452 32467 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.004s
I20260709 12:19:57.625406 27589 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:57.621157 31559 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:57.620024 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.628125 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.578838 32491 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:57.591529 32526 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:19:57.630085 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.615311 32479 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:57.592260 32100 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:57.629926 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49258.7us
I20260709 12:19:57.579087 32104 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:57.637621 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.592636 32550 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:19:57.641896 27735 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
W20260709 12:19:57.632387 27730 rpcz_store.cc:269] 0709 12:19:55.894592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:55.907408 (+ 12816us) service_pool.cc:224] Handling call
0709 12:19:57.616750 (+1709342us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54001d15d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:57.616885 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1653407104}
I20260709 12:19:57.631613 27664 rpcz_store.cc:276] 0709 12:19:56.545011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.545148 (+   137us) service_pool.cc:224] Handling call
0709 12:19:57.617086 (+1071938us) spinlock_profiling.cc:243] Waited 74.2 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:57.617193 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170682752}
I20260709 12:19:57.621212 32489 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.644721 27732 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:19:57.664455 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:57.628176 32462 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:19:57.645836 27627 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.161s (SENT)
I20260709 12:19:57.648691 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.669955 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.670746 27737 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:57.579507 32528 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:19:57.650934 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:19:57.612155 31918 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.648141 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:57.639544 32087 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:57.649111 32467 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.661626 32555 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:19:57.628598 32201 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:19:57.666689 32323 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.667305 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.629324 32102 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:19:57.647588 32511 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:57.679306 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.707297 27736 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:57.670235 32462 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.639124 32119 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.651610 32526 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.683768 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.684357 27731 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:19:57.683156 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.710633 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.686698 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.713939 28328 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:57.670078 32458 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:19:57.686573 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50660.5us
I20260709 12:19:57.713999 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:19:57.715198 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140278us
I20260709 12:19:57.677210 32468 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:57.677193 31918 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:57.720642 27731 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 12:19:57.652877 31559 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.727046 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.727273 27738 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:19:57.682258 32555 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:19:57.730997 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96991.5us
W20260709 12:19:57.732538 27627 leader_election.cc:336] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.460s (SENT)
I20260709 12:19:57.663273 32491 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:57.773056 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:57.727275 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:19:57.716535 32311 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.397s	user 0.007s	sys 0.029s
I20260709 12:19:57.735011 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.736886 31559 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.737262 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.709254 32454 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.739426 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:57.740823 32119 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.743484 27581 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:19:57.744596 32522 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.672401 32489 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.724399 32102 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:19:57.780303 32311 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.475s	user 0.007s	sys 0.030s
I20260709 12:19:57.746089 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2447) took 1433 ms. Trace:
I20260709 12:19:57.747146 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:57.796648 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:57.798799 27589 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:19:57.800020 32102 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.801460 32494 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.802047 32494 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.804721 32491 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.804838 32119 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:19:57.805709 32454 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:57.806121 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:57.806303 32454 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.807603 27581 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:57.808727 32454 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.809437 32454 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.795861 31559 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:57.747146 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.810117 31559 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.812418 31559 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.798375 27657 rpcz_store.cc:276] 0709 12:19:56.312879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.313031 (+   152us) service_pool.cc:224] Handling call
0709 12:19:56.357060 (+ 44029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:56.569840 (+212780us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:56.570182 (+   342us) log.cc:844] Serialized 14 byte log entry
0709 12:19:56.661936 (+ 91754us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:56.662008 (+    72us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:56.662075 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:57.223785 (+561710us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b18000bea20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:19:57.745845 (+522060us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:57.745891 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:57.746010 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11145,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":1297,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":9744,"spinlock_wait_cycles":442916992,"thread_start_us":90734,"threads_started":1,"wal-append.queue_time_us":119115,"wal-append.run_cpu_time_us":1675,"wal-append.run_wall_time_us":1077526}
I20260709 12:19:57.796080 32409 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.808823 27581 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:57.813359 32409 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:57.816059 32509 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.056s	user 0.004s	sys 0.000s
I20260709 12:19:57.714900 32526 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:57.755697 32556 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:57.714706 32462 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:57.731132 32452 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.757014 27736 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:19:57.758270 32398 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.791766 32468 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:57.764628 32467 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.707928 32087 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:57.842614 32087 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.765249 32406 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:57.845527 32470 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:19:57.765376 32344 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:57.766325 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:57.767879 32551 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:19:57.783778 32104 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:19:57.787310 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50144.6us
I20260709 12:19:57.787971 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.789875 27583 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:19:57.679023 32463 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:19:57.793035 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.794139 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.800434 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.808458 31918 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.747808 32395 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.864459 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.748409 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.750043 32233 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:19:57.827162 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:57.832232 32556 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:57.874676 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.876327 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.877377 27734 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:57.822953 32479 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.836274 27729 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:19:57.763861 32555 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.838660 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.885814 27736 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:19:57.806852 32201 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:57.838240 27737 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:57.838083 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:57.838660 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:57.888056 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.848271 28333 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:19:57.888974 27737 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:19:57.809943 32454 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.893021 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:57.894304 27655 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:57.881670 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.883847 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:57.886852 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2096) took 1076 ms. Trace:
W20260709 12:19:57.889858 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 89493.2us
I20260709 12:19:57.890663 27732 raft_consensus.cc:2368] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:19:57.891542 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:57.897001 27733 rpcz_store.cc:276] 0709 12:19:56.810263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.821163 (+ 10900us) service_pool.cc:224] Handling call
0709 12:19:57.886624 (+1065461us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540044e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:57.886799 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400990464}
I20260709 12:19:57.827263 32485 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:57.895728 27735 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:19:57.887818 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:19:57.896581 27737 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:57.899416 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:57.904812 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:57.905103 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:57.960263 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:57.905660 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:57.926957 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 213004us
I20260709 12:19:57.880136 32556 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:57.864863 32470 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:57.963304 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:57.954828 27738 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:19:57.884970 32406 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:19:57.874446 32344 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:57.870874 32316 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:57.963450 27585 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:57.964694 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:57.902765 32452 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:57.961891 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:57.974756 27660 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:57.873040 32526 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:57.999871 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 267347us
I20260709 12:19:58.004766 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:19:58.006672 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:58.007663 27662 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:19:58.007905 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:58.009979 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 226396us
W20260709 12:19:58.013499 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 163092us
I20260709 12:19:58.013715 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:58.014515 27657 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:57.965852 32404 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.018206 27738 pending_rounds.cc:85] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 9
I20260709 12:19:58.018801 27738 pending_rounds.cc:107] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:19:57.965745 32409 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:58.000360 32104 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.003700 32555 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:58.022907 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.064399 27660 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.013128 32323 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:19:58.022552 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.043154 27738 raft_consensus.cc:2889] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:57.963855 32564 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:19:57.999984 32344 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.021660 32516 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.047811 32491 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.049576 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.050298 27662 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:19:58.010048 32395 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:58.056541 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:58.066469 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:58.020506 32463 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:58.019886 32494 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.999855 32316 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.083559 32463 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:58.085009 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67061.4us
I20260709 12:19:58.085388 32316 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.085780 32463 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:57.968189 32569 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:19:58.076642 27661 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:57.972486 32470 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.040891 32311 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.048296 32567 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:58.129300 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.124573 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.130308 27661 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:58.082706 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:58.087517 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2460) took 1376 ms. Trace:
I20260709 12:19:58.087656 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.132050 27731 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.091109 28330 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:58.122851 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:58.132038 27658 rpcz_store.cc:276] 0709 12:19:56.711191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.711306 (+   115us) service_pool.cc:224] Handling call
0709 12:19:56.731645 (+ 20339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:56.881983 (+150338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.087268 (+1205285us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:58.087453 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":342520576}
W20260709 12:19:58.088073 32476 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 12:19:58.078565 32409 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:19:58.022321 32468 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:19:58.016150 32404 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.069312 32323 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:58.140854 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 731) took 1562 ms. Trace:
W20260709 12:19:58.145885 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58299.7us
W20260709 12:19:58.148121 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89663.8us
I20260709 12:19:58.149359 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:58.146160 28330 rpcz_store.cc:276] 0709 12:19:56.572393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.573248 (+   855us) service_pool.cc:224] Handling call
0709 12:19:58.090608 (+1517360us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:19:58.090868 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.134000 (+ 43132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.134251 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2433408}
I20260709 12:19:58.067792 32104 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:58.175227 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50750.7us
I20260709 12:19:58.176745 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.141424 32267 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:19:58.137030 32542 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:19:58.091012 32566 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:19:58.079836 32516 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.138965 32561 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.194080 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:58.196075 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46780.7us
I20260709 12:19:58.155431 32323 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:19:58.156410 32494 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:19:58.156960 32489 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.165939 32398 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.163195 32409 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:58.088932 32491 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.133329 32458 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:19:58.133605 32311 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.198690 32515 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:58.225666 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.226804 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2490) took 1336 ms. Trace:
I20260709 12:19:58.202363 32463 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:58.253580 32463 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:58.254235 32463 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.199738 32481 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:19:58.256783 32454 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.219856 32479 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:58.225647 32489 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:58.260478 32489 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.264480 32344 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.210598 31918 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.266557 32404 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.268396 32201 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:19:58.205154 32406 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 12:19:58.212745 32561 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.273072 32406 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:19:58.273630 32406 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:58.273960 32561 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:58.275938 32104 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.281267 32516 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.234573 32476 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:19:58.220883 32409 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:58.283444 32409 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.285354 32409 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.285203 32463 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.289042 32491 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.291903 32515 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:58.293690 32515 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.294569 32485 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.298144 32552 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:19:58.298704 32552 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:58.249531 32311 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:58.300089 32502 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.532s	user 0.005s	sys 0.000s
I20260709 12:19:58.302332 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2489) took 1411 ms. Trace:
I20260709 12:19:58.305111 32323 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.306470 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:58.236608 27587 rpcz_store.cc:276] 0709 12:19:56.892038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.959687 (+ 67649us) service_pool.cc:224] Handling call
0709 12:19:56.979425 (+ 19738us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:56.996001 (+ 16576us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:19:57.588880 (+592879us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b1800435220. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:19:57.715244 (+126364us) log.cc:844] Serialized 14 byte log entry
0709 12:19:57.898370 (+183126us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:57.963259 (+ 64889us) log.cc:844] Serialized 52 byte log entry
0709 12:19:57.964049 (+   790us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:57.966948 (+  2899us) log.cc:844] Serialized 14 byte log entry
0709 12:19:58.004550 (+ 37602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.004665 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:58.226463 (+221798us) raft_consensus.cc:1685] finished
0709 12:19:58.226502 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:58.226717 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":177640,"prepare.queue_time_us":620151,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":2104,"raft.queue_time_us":120340,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":23503,"spinlock_wait_cycles":12452608,"thread_start_us":236985,"threads_started":2,"wal-append.queue_time_us":144934}
I20260709 12:19:58.199514 32566 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:58.310463 27581 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:19:58.320212 27588 rpcz_store.cc:276] 0709 12:19:56.890473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.900473 (+ 10000us) service_pool.cc:224] Handling call
0709 12:19:58.302101 (+1401628us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400087ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:58.302223 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444494208}
I20260709 12:19:58.326202 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.326898 27580 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
W20260709 12:19:58.243212 32545 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.748s	user 0.002s	sys 0.000s
I20260709 12:19:58.330524 32323 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:58.342803 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:19:58.331012 27660 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:58.337289 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:58.345690 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:58.347023 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.348310 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.346462 27588 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:58.353775 27580 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:19:58.356383 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:58.356976 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.379400 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.359609 32567 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:19:58.325426 32566 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:58.379106 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:58.380187 27587 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:19:58.383085 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:58.388350 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:58.392182 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.384143 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:19:58.395651 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.394585 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.402575 27731 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:58.355185 32489 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.406481 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:19:58.400036 27580 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15.
I20260709 12:19:58.422858 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.399188 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:58.424351 32542 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:58.392678 32458 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:19:58.425002 32542 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:58.425423 32458 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:19:58.425844 32458 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:58.426067 32542 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.428282 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.430250 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:58.432090 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.433368 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.381997 32526 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.441179 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 735) took 1119 ms. Trace:
I20260709 12:19:58.443687 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:58.401259 32528 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:58.391762 32406 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.450450 32528 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:58.444968 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2485) took 1555 ms. Trace:
I20260709 12:19:58.460062 27582 rpcz_store.cc:276] 0709 12:19:56.889534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.913483 (+ 23949us) service_pool.cc:224] Handling call
0709 12:19:56.925297 (+ 11814us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:56.925717 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:56.925755 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:56.926315 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:19:57.176340 (+250025us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:57.176398 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:57.176501 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:58.242166 (+1065665us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:58.444622 (+202456us) raft_consensus.cc:1685] finished
0709 12:19:58.444669 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:58.444906 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":837,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":8523,"spinlock_wait_cycles":56638720,"thread_start_us":191472,"threads_started":1,"wal-append.queue_time_us":431989}
I20260709 12:19:58.446296 27587 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:19:58.401074 32463 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.448191 28328 rpcz_store.cc:276] 0709 12:19:57.322100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.322233 (+   133us) service_pool.cc:224] Handling call
0709 12:19:58.440985 (+1118752us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54006bc6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:58.441117 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":812519808}
W20260709 12:19:58.448191 32511 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.739s	user 0.008s	sys 0.004s
I20260709 12:19:58.405815 32505 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:58.469640 27587 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:19:58.474269 27588 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:19:58.475193 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2505) took 1025 ms. Trace:
I20260709 12:19:58.426812 32458 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.479002 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:58.478511 27588 rpcz_store.cc:276] 0709 12:19:57.450320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.470878 (+1020558us) service_pool.cc:224] Handling call
0709 12:19:58.473916 (+  3038us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.474804 (+   888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.475136 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:58.464039 32528 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:58.470602 32406 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.485148 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:58.488585 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.489565 27731 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:19:58.489645 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:58.405464 32552 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.478312 27582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:56296
W20260709 12:19:58.478140 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 252527us
W20260709 12:19:58.501415 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.503932 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:19:58.501823 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:58.510262 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:19:58.518816 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 192613us
I20260709 12:19:58.520953 32473 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:58.521545 32473 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:58.522081 32473 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.524850 32453 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:58.525552 32453 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:58.530813 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 330644us
I20260709 12:19:58.531968 32557 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:19:58.538622 32323 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:58.539534 27737 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
W20260709 12:19:58.539616 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 331710us
I20260709 12:19:58.540113 27737 pending_rounds.cc:85] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:19:58.540442 27737 pending_rounds.cc:107] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:19:58.540716 27737 raft_consensus.cc:2889] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:58.544977 32585 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.545339 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:58.545581 32585 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.546192 28336 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:58.545008 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:58.548280 32585 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.549541 32515 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:19:58.552212 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.553412 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:58.553969 27734 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.550557 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:58.555846 28331 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:58.556532 32470 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.560564 27582 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:58.560689 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 757) took 1188 ms. Trace:
I20260709 12:19:58.561084 27582 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:58.564496 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2500) took 1381 ms. Trace:
I20260709 12:19:58.564976 27585 rpcz_store.cc:276] 0709 12:19:57.183619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.562850 (+379231us) service_pool.cc:224] Handling call
0709 12:19:58.564321 (+1001471us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:58.564439 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":695745792}
I20260709 12:19:58.566752 27585 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: [15.13-16.14]
W20260709 12:19:58.570254 32575 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.522s	user 0.000s	sys 0.002s
I20260709 12:19:58.570271 32585 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:58.571118 32585 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.573336 32585 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:58.561101 27583 rpcz_store.cc:276] 0709 12:19:57.372705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.868098 (+495393us) service_pool.cc:224] Handling call
0709 12:19:57.898447 (+ 30349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:57.898677 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:19:57.940318 (+ 41641us) log.cc:844] Serialized 14 byte log entry
0709 12:19:58.089312 (+148994us) log.cc:844] Serialized 14 byte log entry
0709 12:19:58.530184 (+440872us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b180014a880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9e21c58f 00007f0e9725b571 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd
0709 12:19:58.536700 (+  6516us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b180014a880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:58.552053 (+ 15353us) spinlock_profiling.cc:243] Waited 9.8 ms on lock 0x7b180014a880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1
0709 12:19:58.552552 (+   499us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b180014a880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:58.559935 (+  7383us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b180014a880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:19:58.560157 (+   222us) log.cc:844] Serialized 14 byte log entry
0709 12:19:58.560357 (+   200us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:58.560404 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:58.560450 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.560519 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:58.560636 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":105904,"prepare.queue_time_us":505025,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":4015,"raft.queue_time_us":103956,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":23689,"spinlock_wait_cycles":192512256,"thread_start_us":134250,"threads_started":1,"wal-append.queue_time_us":227156}
I20260709 12:19:58.582228 27583 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:58.583276 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.586217 27729 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:58.587119 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:58.587190 27736 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:19:58.587791 27736 pending_rounds.cc:85] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:19:58.588151 27736 pending_rounds.cc:107] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:19:58.588481 27736 raft_consensus.cc:2889] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:58.590863 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.592171 27583 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:19:58.592279 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.596083 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:58.596156 32460 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:19:58.597442 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.598598 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:58.599140 27732 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:19:58.592213 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:19:58.594471 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.604585 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.605227 27657 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:19:58.607654 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:58.608289 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:58.609906 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:58.610500 27655 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:19:58.611640 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.612246 27663 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:58.614202 27583 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:58.613067 27657 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:58.616101 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.616882 27655 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:58.618184 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2493) took 1667 ms. Trace:
I20260709 12:19:58.619398 27729 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:58.621348 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:58.622061 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.627622 27657 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:58.628571 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:58.629196 27657 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:19:58.587133 27582 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.623934 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:58.624091 27729 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:19:58.618561 27586 rpcz_store.cc:276] 0709 12:19:56.950533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:56.951811 (+  1278us) service_pool.cc:224] Handling call
0709 12:19:56.972010 (+ 20199us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:57.127262 (+155252us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:57.628081 (+500819us) log.cc:844] Serialized 14 byte log entry
0709 12:19:57.832720 (+204639us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:57.833671 (+   951us) log.cc:844] Serialized 26 byte log entry
0709 12:19:58.597233 (+763562us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b54000d32a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a2bf6b8 00007f0e9a34892b 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c
0709 12:19:58.597292 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:58.617621 (+ 20329us) log.cc:844] Serialized 14 byte log entry
0709 12:19:58.617825 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.617893 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:58.618008 (+   115us) raft_consensus.cc:1685] finished
0709 12:19:58.618022 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:58.618132 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":525684,"prepare.queue_time_us":493955,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":131072,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":27606,"spinlock_wait_cycles":59088768,"thread_start_us":304886,"threads_started":3,"wal-append.queue_time_us":74016}
I20260709 12:19:58.592226 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.639827 27734 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
W20260709 12:19:58.509088 32102 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.016s	sys 0.007s
W20260709 12:19:58.647979 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55831.6us
I20260709 12:19:58.648509 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.649034 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.650902 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.495160 32493 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:19:58.658726 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 114969us
I20260709 12:19:58.659693 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:58.661229 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:58.494022 32458 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:19:58.665759 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:58.671361 27704 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15.
I20260709 12:19:58.671864 27704 leader_election.cc:403] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:19:58.645143 32505 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:58.666335 32467 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:19:58.674320 32505 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.676198 32344 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:58.676787 32344 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:58.677278 32344 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:58.678575 32505 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:58.679311 28332 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16. Ops: []
I20260709 12:19:58.682127 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:19:58.651106 32102 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.016s	sys 0.007s
I20260709 12:19:58.682284 27657 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:19:58.716480 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.652771 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:58.643543 32476 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:19:58.643822 32536 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.015s	user 0.013s	sys 0.000s
I20260709 12:19:58.655397 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:58.644415 32526 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.650580 32511 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:19:58.651664 32589 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.655148 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:19:58.725216 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54102.3us
W20260709 12:19:58.725975 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.012s	sys 0.030s
I20260709 12:19:58.734232 27586 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 12:19:58.728116 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75448.1us
I20260709 12:19:58.644807 32478 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:58.656948 32493 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:58.516803 32481 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:19:58.735391 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.012s	sys 0.031s
I20260709 12:19:58.735193 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:58.666548 32201 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:19:58.654505 31831 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:58.733304 31559 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15.
I20260709 12:19:58.666769 32561 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:58.735309 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:19:58.653029 32491 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:58.718940 32592 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:58.655224 32585 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:19:58.725690 32476 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:19:58.895665 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160536us
W20260709 12:19:58.795629 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:58.901037 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 165769us
I20260709 12:19:58.655432 32569 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:19:58.746055 32493 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:19:58.733757 32511 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:58.813302 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:58.953612 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.738093 32467 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:19:58.735741 32560 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:19:58.960142 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57147.6us
W20260709 12:19:58.829835 32462 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.989s	user 0.008s	sys 0.004s
I20260709 12:19:58.740357 32262 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:58.969609 31559 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
W20260709 12:19:58.991961 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 471668us
I20260709 12:19:58.993088 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:58.994009 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 473875us
I20260709 12:19:58.908074 32592 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:58.910252 32476 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:19:58.998447 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:58.998353 28333 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:19:58.910979 32395 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:19:58.738652 32526 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.911181 32516 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:59.024108 27582 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:58.745582 32102 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.270s
I20260709 12:19:58.954782 32511 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:19:58.802130 32087 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.230s
I20260709 12:19:58.733384 32536 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:58.911855 32537 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.027347 27586 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:19:58.912276 32491 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.024487 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:58.996803 32262 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:59.048316 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:56296 (request call id 759) took 1367 ms. Trace:
I20260709 12:19:59.045956 32490 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:19:59.134896 27582 rpcz_store.cc:276] 0709 12:19:57.680891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.467958 (+787067us) service_pool.cc:224] Handling call
0709 12:19:58.956596 (+488638us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b54001fefd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a33fa3d 00007f0e9f30fcd0 00007f0e98894254 00007f0e988941e2 00007f0e98894111 00007f0e9889408c 00007f0e98893342 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:59.042861 (+ 86265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151079168}
I20260709 12:19:59.023772 32592 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:59.043463 32556 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.042s	user 0.008s	sys 0.004s
I20260709 12:19:59.015537 32522 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:19:58.967653 32583 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:19:58.774555 32404 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.282s
I20260709 12:19:59.078401 28328 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:19:59.079824 27589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.023s	sys 0.025s
W20260709 12:19:59.162245 27589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.023s	sys 0.025s
I20260709 12:19:59.033300 32395 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:59.137116 32262 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:59.002588 32542 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.044s	sys 0.070s
I20260709 12:19:58.988920 31831 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.996280 32453 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:58.740485 32589 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.025977 32473 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.136710 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.510s (SENT) [suppressed 26 similar messages]
I20260709 12:19:59.014348 32462 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.138132 32599 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:19:59.171824 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 139587us
I20260709 12:19:59.186379 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:59.152384 32583 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:59.049437 32516 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.161154 32555 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.088s	user 0.004s	sys 0.010s
I20260709 12:19:59.175979 31559 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:19:59.181306 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2104) took 1446 ms. Trace:
I20260709 12:19:59.131237 32537 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.167466 32395 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:59.189476 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:19:59.171958 32542 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.044s	sys 0.071s
I20260709 12:19:59.200306 27730 rpcz_store.cc:276] 0709 12:19:57.735269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.390770 (+655501us) service_pool.cc:224] Handling call
0709 12:19:58.391129 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.396531 (+  5402us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:58.743712 (+347181us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.180764 (+437052us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:59.180981 (+   217us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:59.181035 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.181105 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.181245 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":749666,"threads_started":2}
W20260709 12:19:59.198452 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:59.153326 32556 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.197438 31559 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:19:59.215576 32567 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.216177 32567 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.218079 32406 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.734s	user 0.008s	sys 0.000s
I20260709 12:19:59.218374 32567 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.218147 32395 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.219533 32594 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:19:59.220252 32536 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.220649 32395 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.221612 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:59.222178 28329 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:19:59.222518 28329 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:19:59.222740 32596 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.226s	user 0.000s	sys 0.000s
I20260709 12:19:59.223611 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2344) took 1545 ms. Trace:
I20260709 12:19:59.224393 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2051) took 1248 ms. Trace:
W20260709 12:19:59.225154 32600 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.271s	user 0.000s	sys 0.001s
I20260709 12:19:59.225900 32201 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.226541 32201 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.229302 32201 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.230069 32481 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:19:59.230613 32481 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:59.231902 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2499) took 1879 ms. Trace:
I20260709 12:19:59.232311 27581 rpcz_store.cc:276] 0709 12:19:57.352070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.814342 (+462272us) service_pool.cc:224] Handling call
0709 12:19:58.310131 (+495789us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.327929 (+ 17798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.231703 (+903774us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b5400087ad0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:59.231843 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139248000}
I20260709 12:19:59.233258 32462 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.236719 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2102) took 1502 ms. Trace:
W20260709 12:19:59.242444 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:59.195613 32583 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:59.223980 27735 rpcz_store.cc:276] 0709 12:19:57.678091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.716640 (+ 38549us) service_pool.cc:224] Handling call
0709 12:19:57.889995 (+173355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:57.962863 (+ 72868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.223377 (+1260514us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54000a4bd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:19:59.223546 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364934016}
I20260709 12:19:59.245723 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:59.237361 27738 rpcz_store.cc:276] 0709 12:19:57.734628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.015422 (+280794us) service_pool.cc:224] Handling call
0709 12:19:58.017946 (+  2524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.071333 (+ 53387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.071368 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.071770 (+   402us) log.cc:844] Serialized 26 byte log entry
0709 12:19:58.894982 (+823212us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:59.176679 (+281697us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.176887 (+   208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.176973 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.236435 (+ 59462us) raft_consensus.cc:1685] finished
0709 12:19:59.236465 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.236639 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":43985,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":85,"thread_start_us":942936,"threads_started":2,"wal-append.queue_time_us":912873}
I20260709 12:19:59.224881 27664 rpcz_store.cc:276] 0709 12:19:57.975642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.975887 (+   245us) service_pool.cc:224] Handling call
0709 12:19:58.010717 (+ 34830us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.017150 (+  6433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.017179 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.017612 (+   433us) log.cc:844] Serialized 26 byte log entry
0709 12:19:58.515618 (+498006us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:19:58.515680 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.515782 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.222108 (+706326us) spinlock_profiling.cc:243] Waited 7.04 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:59.224036 (+  1928us) raft_consensus.cc:1685] finished
0709 12:19:59.224064 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.224323 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5984,"raft.queue_time_us":6682,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":21080,"spinlock_wait_cycles":16183552,"thread_start_us":434744,"threads_started":1,"wal-append.queue_time_us":748186}
I20260709 12:19:59.219098 32567 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:59.261067 27662 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:19:59.268215 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:59.256088 31559 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.272166 31559 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:19:59.243515 32344 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.005s
I20260709 12:19:59.283576 32567 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:59.279088 32481 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.323802 32611 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
W20260709 12:19:59.427920 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 429904us
I20260709 12:19:59.187645 32622 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.123s
I20260709 12:19:59.316848 32625 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:19:59.287616 32344 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.518147 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:59.519014 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:59.470343 32493 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.565s	user 0.009s	sys 0.004s
W20260709 12:19:59.478374 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 453921us
W20260709 12:19:59.472450 32614 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:19:59.455013 32526 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.442605 32617 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:59.522236 32623 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:19:59.544636 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:59.565969 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:19:59.534526 32597 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.496s	user 0.000s	sys 0.001s
I20260709 12:19:59.481302 32473 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.579784 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2119) took 1030 ms. Trace:
I20260709 12:19:59.493587 32625 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:59.581205 32625 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.580341 27729 rpcz_store.cc:276] 0709 12:19:58.549522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.634574 (+ 85052us) service_pool.cc:224] Handling call
0709 12:19:58.634867 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.733829 (+ 98962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.733864 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.747074 (+ 13210us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.168754 (+421680us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:59.168811 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.168900 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.579436 (+410536us) raft_consensus.cc:1685] finished
0709 12:19:59.579470 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.579693 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126011,"raft.queue_time_us":99180,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":7060,"thread_start_us":325069,"threads_started":1,"wal-append.queue_time_us":714682}
W20260709 12:19:59.554718 32626 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:19:59.555529 32602 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.354s	user 0.000s	sys 0.001s
W20260709 12:19:59.556332 32603 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
I20260709 12:19:59.556507 32567 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.559671 32453 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.560009 32556 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:59.564203 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 763) took 1811 ms. Trace:
I20260709 12:19:59.645352 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 758) took 2272 ms. Trace:
I20260709 12:19:59.645603 27588 rpcz_store.cc:276] 0709 12:19:57.752984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.481078 (+728094us) service_pool.cc:224] Handling call
0709 12:19:58.484145 (+  3067us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.537184 (+ 53039us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.537220 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.537763 (+   543us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.003135 (+465372us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:59.003193 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.003277 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.563969 (+560692us) raft_consensus.cc:1685] finished
0709 12:19:59.563994 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.564133 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52603,"raft.queue_time_us":52947,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":464957,"threads_started":1,"wal-append.queue_time_us":995786}
I20260709 12:19:59.572033 32463 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:59.549669 32493 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.573309 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 742) took 1174 ms. Trace:
I20260709 12:19:59.652067 27736 rpcz_store.cc:276] 0709 12:19:58.398808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.602978 (+204170us) service_pool.cc:224] Handling call
0709 12:19:58.603304 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.662499 (+ 59195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.662546 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.663169 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.043502 (+380333us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:59.043570 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.043800 (+   230us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.573008 (+529208us) raft_consensus.cc:1685] finished
0709 12:19:59.573041 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.573228 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65748,"raft.queue_time_us":80076,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9981,"thread_start_us":363675,"threads_started":1,"wal-append.queue_time_us":884045}
I20260709 12:19:59.564057 32485 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.572103 32516 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:19:59.574640 32566 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.155s	user 0.006s	sys 0.006s
I20260709 12:19:59.395238 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 762) took 1713 ms. Trace:
I20260709 12:19:59.544615 32491 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:59.552233 32615 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
W20260709 12:19:59.618086 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52205.8us
I20260709 12:19:59.682391 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:59.622397 32583 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.628206 32537 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:59.628810 32557 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:19:59.628279 32344 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:59.733799 27664 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:19:59.573696 32262 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:59.643468 32458 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.959s	user 0.014s	sys 0.000s
I20260709 12:19:59.552109 32617 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:59.565222 32623 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:19:59.655916 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.003s	sys 0.005s
I20260709 12:19:59.675889 27587 rpcz_store.cc:276] 0709 12:19:57.681950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.475706 (+793756us) service_pool.cc:224] Handling call
0709 12:19:58.475988 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.476223 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:19:59.001907 (+525684us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.392698 (+390791us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:59.392777 (+    79us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:19:59.392838 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.392941 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.394817 (+  1876us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91979,"prepare.queue_time_us":92507,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":22210,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":822633,"threads_started":2}
W20260709 12:19:59.742362 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.004s	sys 0.005s
I20260709 12:19:59.742859 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.743508 27587 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
W20260709 12:19:59.680203 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.002s	sys 0.008s
W20260709 12:19:59.745256 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.002s	sys 0.009s
I20260709 12:19:59.743111 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2059) took 1136 ms. Trace:
W20260709 12:19:59.745637 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 750499us
W20260709 12:19:59.746784 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:59.683488 27660 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.008s	sys 0.003s
W20260709 12:19:59.747720 27660 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.008s	sys 0.003s
I20260709 12:19:59.748634 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2056) took 1144 ms. Trace:
I20260709 12:19:59.684520 32606 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:19:59.646066 27589 rpcz_store.cc:276] 0709 12:19:57.372980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.434132 (+1061152us) service_pool.cc:224] Handling call
0709 12:19:58.434487 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.434959 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:19:58.568019 (+133060us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.644961 (+1076942us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:59.645027 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:19:59.645084 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.645155 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.645285 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":151902,"prepare.queue_time_us":133521,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":45024,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":20886,"thread_start_us":1051674,"threads_started":2,"wal-append.queue_time_us":628595}
I20260709 12:19:59.692099 32555 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:59.692711 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:59.693502 32556 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:19:59.707231 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 743) took 1194 ms. Trace:
I20260709 12:19:59.759827 28328 rpcz_store.cc:276] 0709 12:19:58.512441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.512629 (+   188us) service_pool.cc:224] Handling call
0709 12:19:58.665838 (+153209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:59.156790 (+490952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.706989 (+550199us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:19:59.707171 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933316992}
I20260709 12:19:59.694339 28329 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.715356 32478 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:59.715116 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.716899 27581 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 10. Current term is 11. Ops: []
I20260709 12:19:59.719524 27585 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:19:59.719671 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:59.720630 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:19:59.620167 31831 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:59.727880 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:59.728892 27661 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:19:59.651895 32463 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:59.658341 32467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.001s	sys 0.008s
I20260709 12:19:59.732637 32557 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:19:59.735154 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.675436 32485 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:19:59.682575 32476 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.000s
I20260709 12:19:59.768138 27584 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:59.766947 27585 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:19:59.744153 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.093s	sys 0.154s
W20260709 12:19:59.769873 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.094s	sys 0.155s
I20260709 12:19:59.762096 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
W20260709 12:19:59.771616 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51090.7us
W20260709 12:19:59.772488 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:59.751895 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.757153 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.775816 27589 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
W20260709 12:19:59.754037 27701 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.009s	sys 0.020s
I20260709 12:19:59.745947 27656 rpcz_store.cc:276] 0709 12:19:58.607173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.607297 (+   124us) service_pool.cc:224] Handling call
0709 12:19:58.607634 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.607838 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:58.717707 (+109869us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.742779 (+1025072us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:59.742827 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:19:59.742870 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.742951 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.743062 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109562,"prepare.queue_time_us":129768,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":758,"thread_start_us":1024731,"threads_started":1}
W20260709 12:19:59.758843 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66248.1us
W20260709 12:19:59.776907 27701 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.010s	sys 0.020s
I20260709 12:19:59.777922 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:59.778887 28328 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:59.777925 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:59.773619 32476 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.783550 32561 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:19:59.784015 32476 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.784205 32561 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:59.776213 32632 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:19:59.785152 32632 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:19:59.785495 32632 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:59.785521 32561 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.786888 32493 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:59.779151 32623 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.788616 32623 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:19:59.737295 27731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.000s	sys 0.013s
I20260709 12:19:59.682761 32566 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.764999 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:59.801868 32566 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:19:59.803038 32610 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.648s	user 0.001s	sys 0.009s
I20260709 12:19:59.803494 27580 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
W20260709 12:19:59.755686 32511 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.000s
I20260709 12:19:59.804450 32511 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.741454 32617 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:19:59.762102 32409 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.476s	user 0.017s	sys 0.000s
I20260709 12:19:59.820899 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.776119 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2110) took 1998 ms. Trace:
I20260709 12:19:59.743600 32589 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:59.822266 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:19:59.791440 27731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.000s	sys 0.013s
I20260709 12:19:59.793625 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:59.823539 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:59.824033 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2105) took 2088 ms. Trace:
I20260709 12:19:59.824149 27655 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:19:59.794277 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:19:59.825278 27662 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:59.794462 27657 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 12:19:59.763641 32467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.001s	sys 0.008s
I20260709 12:19:59.762224 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:59.824388 27731 rpcz_store.cc:276] 0709 12:19:57.735565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.511518 (+775953us) service_pool.cc:224] Handling call
0709 12:19:58.668074 (+156556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.668331 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:19:59.170518 (+502187us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.823735 (+653217us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:59.823785 (+    50us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:19:59.823827 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.823888 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.823987 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":746322,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"thread_start_us":1106423,"threads_started":2}
W20260709 12:19:59.829831 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67691.6us
I20260709 12:19:59.830619 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:19:59.752480 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.011s	sys 0.018s
W20260709 12:19:59.831619 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.011s	sys 0.018s
W20260709 12:19:59.831976 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 626185us
I20260709 12:19:59.749159 27660 rpcz_store.cc:276] 0709 12:19:58.604550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.604969 (+   419us) service_pool.cc:224] Handling call
0709 12:19:58.605271 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:58.605464 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:19:58.717196 (+111732us) log.cc:844] Serialized 14 byte log entry
0709 12:19:59.748189 (+1030993us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:59.748262 (+    73us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:59.748328 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.748414 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.748564 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98693,"prepare.queue_time_us":127380,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":3946,"raft.queue_time_us":183177,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":8455,"thread_start_us":994991,"threads_started":2,"wal-append.queue_time_us":468561}
W20260709 12:19:59.796890 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 660365us
I20260709 12:19:59.833575 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:59.834187 27664 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:59.834455 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.821595 27737 rpcz_store.cc:276] 0709 12:19:57.777483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.541544 (+764061us) service_pool.cc:224] Handling call
0709 12:19:58.541889 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.655966 (+114077us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.656001 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.681530 (+ 25529us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.131165 (+449635us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:19:59.131217 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.131285 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.775880 (+644595us) raft_consensus.cc:1685] finished
0709 12:19:59.775903 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.776048 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":133374,"raft.queue_time_us":116359,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":13970,"thread_start_us":390228,"threads_started":1,"wal-append.queue_time_us":485241}
I20260709 12:19:59.790340 32201 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:19:59.769091 32406 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:19:59.838326 32406 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:19:59.838385 32552 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:59.838842 32406 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:19:59.798911 32485 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:59.805574 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:59.820294 32262 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:19:59.763823 32557 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:59.807972 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:59.809602 27663 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:59.811134 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 741) took 1412 ms. Trace:
I20260709 12:19:59.851032 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:19:59.812354 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:59.761866 32636 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:59.768226 32478 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:19:59.816365 32592 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.680s	user 0.010s	sys 0.000s
I20260709 12:19:59.858556 32485 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:19:59.858819 32592 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.861222 32592 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:59.858600 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:19:59.862545 27733 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:19:59.863492 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:59.841080 32522 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:19:59.818564 32528 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.326s	user 0.005s	sys 0.005s
I20260709 12:19:59.864822 32478 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.864640 32528 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.865639 32478 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:19:59.849435 32623 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.866302 32623 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.854645 32462 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 12:19:59.866902 32528 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:59.768903 32556 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:19:59.867501 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:59.868319 32623 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:19:59.817667 32583 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.763445 32463 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:19:59.870865 32511 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.854266 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:19:59.828740 32467 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:19:59.851512 27732 rpcz_store.cc:276] 0709 12:19:58.398443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.600914 (+202471us) service_pool.cc:224] Handling call
0709 12:19:58.601299 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.601517 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.601544 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.602018 (+   474us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.002344 (+400326us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:19:59.002397 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.002483 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.802272 (+799789us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:19:59.806062 (+  3790us) raft_consensus.cc:1685] finished
0709 12:19:59.806083 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:59.811071 (+  4988us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12279680,"thread_start_us":399904,"threads_started":1,"wal-append.queue_time_us":442366}
I20260709 12:19:59.856972 27663 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:19:59.762001 32606 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:19:59.849629 32404 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:19:59.864029 27733 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:19:59.867956 32556 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.757846 32458 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:19:59.869046 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:19:59.889513 27731 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:19:59.870814 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:59.871826 27587 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:19:59.856050 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:59.890988 27656 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:19:59.872524 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:59.762663 32555 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.839883 32406 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:19:59.877617 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:59.881402 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:59.898790 27655 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:19:59.881321 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:19:59.882750 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:19:59.901336 28332 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:59.891589 27660 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:19:59.876436 32262 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:19:59.923110 27661 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:19:59.929528 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:59.930373 27585 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:59.886070 32267 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:19:59.877324 32560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.003s	sys 0.023s
I20260709 12:19:59.936388 27586 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:19:59.938084 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:59.944785 27584 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:19:59.952972 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:19:59.942128 27588 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
W20260709 12:19:59.958382 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87632.4us
I20260709 12:19:59.960290 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:19:59.960502 28328 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:19:59.957065 27664 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:19:59.952241 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:19:59.959046 27587 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:19:59.951052 27655 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:19:59.968017 27587 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 8
I20260709 12:19:59.968434 27587 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:19:59.968860 27587 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:59.887308 32462 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:19:59.969404 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:19:59.975430 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:59.976680 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:59.977720 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:19:59.968065 32406 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:19:59.980156 27732 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
W20260709 12:19:59.961038 32560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.004s	sys 0.023s
I20260709 12:19:59.983757 27733 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
W20260709 12:20:00.000715 27735 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.006s	sys 0.018s
I20260709 12:19:59.944893 32555 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:19:59.888329 32404 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:00.009389 27735 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.006s	sys 0.019s
I20260709 12:20:00.009416 27656 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
W20260709 12:19:59.962272 32599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.008s	sys 0.011s
I20260709 12:19:59.939164 32644 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:20:00.018914 32637 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.943640 32632 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.023639 28332 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:00.025907 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 728) took 1380 ms. Trace:
I20260709 12:20:00.026374 28333 rpcz_store.cc:276] 0709 12:19:58.645569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.647980 (+  2411us) service_pool.cc:224] Handling call
0709 12:19:58.672898 (+ 24918us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:59.032949 (+360051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:00.025601 (+992652us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540024a254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:00.025840 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":729922816}
I20260709 12:20:00.006685 32645 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:59.959919 32463 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:00.019819 32599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.009s	sys 0.011s
I20260709 12:20:00.007437 32201 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:00.011535 32485 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:00.085474 27585 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:00.088658 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:00.101640 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2128) took 1506 ms. Trace:
I20260709 12:20:00.104341 27729 rpcz_store.cc:276] 0709 12:19:58.595042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.582930 (+987888us) service_pool.cc:224] Handling call
0709 12:19:59.583655 (+   725us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:59.584056 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:19:59.629450 (+ 45394us) log.cc:844] Serialized 14 byte log entry
0709 12:20:00.101306 (+471856us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:00.101366 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:20:00.101411 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:00.101481 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:00.101591 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":45187,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":201,"thread_start_us":471491,"threads_started":1}
I20260709 12:20:00.118680 27589 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:00.089339 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 129117us
W20260709 12:20:00.092960 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63491.4us
I20260709 12:19:59.946847 32617 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:00.098353 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47518.9us
W20260709 12:20:00.146590 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:00.147934 32523 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.264s	user 0.016s	sys 0.059s
W20260709 12:20:00.148377 32523 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.309s	user 0.017s	sys 0.060s
W20260709 12:20:00.149097 32647 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260709 12:20:00.150382 32604 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.948s	user 0.001s	sys 0.001s
I20260709 12:20:00.118693 32201 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:00.156610 32404 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:00.156889 32201 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.159637 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2048) took 2258 ms. Trace:
I20260709 12:20:00.160082 27659 rpcz_store.cc:276] 0709 12:19:57.901561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:57.954768 (+ 53207us) service_pool.cc:224] Handling call
0709 12:19:57.955132 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:57.955388 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:57.955414 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:59.207700 (+1252286us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:19:59.270287 (+ 62587us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.878277 (+607990us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:59.878330 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.878461 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:00.159269 (+280808us) raft_consensus.cc:1685] finished
0709 12:20:00.159293 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:00.159556 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20774144,"thread_start_us":485008,"threads_started":1,"wal-append.queue_time_us":841634}
I20260709 12:20:00.034466 32637 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:00.164863 32644 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:00.165524 32644 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:00.166478 32637 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.167616 32598 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:00.168025 32644 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.168807 32644 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:00.105437 32646 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.169955 32646 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.171473 27627 leader_election.cc:400] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
W20260709 12:20:00.173569 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:59.934145 32522 consensus_queue.cc:1175] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 16 status { last_received { term: 15 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:20:00.177449 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2127) took 1618 ms. Trace:
I20260709 12:20:00.135602 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:00.179019 32644 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:00.178014 27730 rpcz_store.cc:276] 0709 12:19:58.558989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.271110 (+712121us) service_pool.cc:224] Handling call
0709 12:19:59.289156 (+ 18046us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:59.315802 (+ 26646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:59.315846 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:59.365665 (+ 49819us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.877447 (+511782us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:59.877496 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.877562 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:00.177152 (+299590us) raft_consensus.cc:1685] finished
0709 12:20:00.177172 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:00.177359 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2575,"raft.queue_time_us":34007,"raft.run_cpu_time_us":6215,"raft.run_wall_time_us":116338,"thread_start_us":334433,"threads_started":1,"wal-append.queue_time_us":638652}
W20260709 12:20:00.179924 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44416.1us
I20260709 12:20:00.181051 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.181602 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:00.182319 32644 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:00.185643 32463 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.183024 32644 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:00.187713 32644 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:00.193186 32490 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:00.193909 32490 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:00.198305 32612 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.030s	user 0.000s	sys 0.003s
I20260709 12:20:00.207378 32473 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:00.208225 32473 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:00.096184 32645 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.100291 32631 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.140s
I20260709 12:20:00.101263 32599 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:00.019881 32467 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:00.225824 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2506) took 1917 ms (client timeout 2280 ms). Trace:
W20260709 12:20:00.288141 27580 rpcz_store.cc:269] 0709 12:19:58.308332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.422497 (+114165us) service_pool.cc:224] Handling call
0709 12:20:00.225585 (+1803088us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:00.225737 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3173237632}
I20260709 12:20:00.236033 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:00.269963 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.359618 27738 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:00.253244 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.361600 27731 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:00.251175 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.252172 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.362323 27730 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:00.363633 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.364697 27730 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:00.365826 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.275363 27737 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:20:00.404867 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.405946 27658 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:00.243268 32595 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.111s	user 0.006s	sys 0.000s
I20260709 12:20:00.251804 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:00.255849 32648 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:20:00.287047 27733 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:20:00.331120 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:00.370076 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 146411us
I20260709 12:20:00.370471 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:00.383173 27659 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:20:00.383778 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.371841 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 769) took 1707 ms. Trace:
I20260709 12:20:00.248272 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:00.411314 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:00.238175 32619 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.873s	user 0.003s	sys 0.000s
W20260709 12:20:00.252815 32643 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:20:00.418922 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:00.252293 27729 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:00.229518 32104 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:20:00.234352 32593 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.390s	user 0.000s	sys 0.006s
W20260709 12:20:00.496672 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.052s	sys 0.044s
I20260709 12:20:00.428318 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.497819 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.497949 27660 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:00.497176 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:00.464282 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.499295 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:00.497812 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.052s	sys 0.044s
W20260709 12:20:00.501423 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 693473us
I20260709 12:20:00.502063 27585 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:20:00.502218 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:00.503244 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:00.504026 27585 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:00.503867 27583 rpcz_store.cc:276] 0709 12:19:58.664843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.664997 (+   154us) service_pool.cc:224] Handling call
0709 12:19:58.672785 (+  7788us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.673136 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.673168 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.673686 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:19:58.997223 (+323537us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:19:58.997267 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:58.997336 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:00.197358 (+1200022us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:00.371474 (+174116us) raft_consensus.cc:1685] finished
0709 12:20:00.371514 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:00.371769 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":659,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":13512960,"thread_start_us":323132,"threads_started":1,"wal-append.queue_time_us":404277}
I20260709 12:20:00.507462 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.508397 27733 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:20:00.510524 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:00.511375 27584 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:00.211269 32473 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.249686 32635 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.235229 32511 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:00.502071 27737 raft_consensus.cc:2368] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:00.573521 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.445622 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.595439 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:00.596813 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.454679 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.599082 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.600514 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.601787 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:00.273111 32267 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:00.508044 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:00.603582 32267 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:00.604282 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96303.9us
I20260709 12:20:00.605268 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:00.607167 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:00.265348 32632 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:00.536935 27657 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.034s	sys 0.047s
I20260709 12:20:00.547358 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
W20260709 12:20:00.608029 27657 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.036s	sys 0.047s
I20260709 12:20:00.514741 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.608940 27655 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:00.609782 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.610538 27731 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:20:00.611482 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.612112 27731 raft_consensus.cc:2376] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12. Re-sending same reply.
I20260709 12:20:00.613189 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.613982 27731 raft_consensus.cc:2368] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:00.615826 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:00.531584 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.013s	sys 0.036s
I20260709 12:20:00.549471 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:00.619603 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.013s	sys 0.038s
I20260709 12:20:00.620021 27589 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
W20260709 12:20:00.536954 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:00.528889 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.550457 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.622548 27659 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:00.513711 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.623644 27730 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:20:00.624424 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.624784 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:00.625155 27730 raft_consensus.cc:2368] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:00.629594 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.629879 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:00.229727 32516 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.283s
I20260709 12:20:00.633517 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.634347 27729 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:00.636997 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.637779 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
W20260709 12:20:00.552495 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.009s	sys 0.021s
I20260709 12:20:00.359298 32646 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:00.553618 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:00.274233 32617 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.359012 32645 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:00.672075 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.010s	sys 0.021s
I20260709 12:20:00.371484 32463 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.673069 32463 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:00.513543 27662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.019s	sys 0.022s
W20260709 12:20:00.427815 32531 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.007s	sys 0.043s
W20260709 12:20:00.675509 27662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.022s	sys 0.022s
W20260709 12:20:00.675701 32531 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.950s	user 0.007s	sys 0.052s
I20260709 12:20:00.676496 32531 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:00.676905 32463 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.677381 32531 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:00.677686 32463 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:00.678206 32463 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:00.678668 32463 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:00.603310 27661 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:00.441591 32496 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.721222 27583 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: []
I20260709 12:20:00.708662 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:00.444427 32616 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.010s	sys 0.014s
I20260709 12:20:00.497236 32104 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:00.719436 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:00.512502 32489 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:00.639165 32557 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.771s	user 0.011s	sys 0.000s
I20260709 12:20:00.743176 27584 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:00.764688 27580 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
W20260709 12:20:00.747642 32552 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.909s	user 0.002s	sys 0.008s
I20260709 12:20:00.255208 32599 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:00.751166 28329 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:20:00.753882 32406 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.743s	user 0.008s	sys 0.000s
I20260709 12:20:00.753744 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.779798 27733 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:00.757292 32606 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:00.775475 32587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.006s	sys 0.015s
W20260709 12:20:00.781736 32587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.227s	user 0.007s	sys 0.015s
I20260709 12:20:00.791190 27658 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:20:00.763471 32638 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:00.727725 32616 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.907s	user 0.014s	sys 0.014s
I20260709 12:20:00.798347 28329 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
W20260709 12:20:00.804273 28329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 633) took 1595 ms (client timeout 1871 ms). Trace:
I20260709 12:20:00.806418 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2121) took 2256 ms. Trace:
I20260709 12:20:00.793576 32453 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.808161 27734 rpcz_store.cc:276] 0709 12:19:58.550097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:58.641416 (+ 91319us) service_pool.cc:224] Handling call
0709 12:19:58.641894 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:58.683888 (+ 41994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:58.683937 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:58.737601 (+ 53664us) log.cc:844] Serialized 26 byte log entry
0709 12:19:59.169130 (+431529us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:19:59.169182 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:59.169264 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:59.784664 (+615400us) spinlock_profiling.cc:243] Waited 5.9 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:20:00.237366 (+452702us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:00.806198 (+568832us) raft_consensus.cc:1685] finished
0709 12:20:00.806221 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:00.806362 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24489,"raft.queue_time_us":71522,"raft.run_cpu_time_us":14204,"raft.run_wall_time_us":1066719,"spinlock_wait_cycles":77908736,"thread_start_us":332310,"threads_started":1,"wal-append.queue_time_us":445761}
I20260709 12:20:00.632397 32635 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:00.810307 28329 rpcz_store.cc:269] 0709 12:19:59.208491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.208976 (+   485us) service_pool.cc:224] Handling call
0709 12:19:59.693785 (+484809us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b4c000e5940. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98614a20 00007f0e98625a29 00007f0e985eb644 00007f0e98612f2d 00007f0e98612b93 00007f0e9a34318b 00007f0e9a343cb5 00007f0e9a33ef30 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17
0709 12:20:00.750569 (+1056784us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b580075f5d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c
0709 12:20:00.804194 (+ 53625us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1265421440}
I20260709 12:20:00.771270 32489 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.820907 32453 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.822431 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:00.823164 27733 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:00.791980 32557 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:00.825356 32557 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:00.741885 32496 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.832376 27730 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:00.836627 32605 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:00.837073 32605 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:00.837978 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:00.837890 32605 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.839208 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:00.841508 32567 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:20:00.844620 32567 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:00.845788 32567 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.846498 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.847261 27734 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:00.848157 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:00.849416 27734 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:00.849284 32406 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:00.850033 32406 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:00.851946 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:00.717033 32511 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.839120 32605 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:20:00.814395 32485 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.706s	user 0.008s	sys 0.002s
I20260709 12:20:00.870159 27737 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:00.868191 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:00.868698 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.871834 27660 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:00.874325 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.876648 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:00.877056 27661 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:00.881479 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:00.883008 27585 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:00.883808 27589 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:00.881148 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:00.884557 28332 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:00.885250 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:00.886703 28334 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:20:00.886615 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:00.887548 28331 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:00.864204 32458 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.806401 32638 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:00.868770 32606 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.972144 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.875578 32485 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:20:00.886389 32625 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.152s	user 0.089s	sys 0.114s
I20260709 12:20:00.987512 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:00.883114 32516 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:00.977185 27663 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:00.992061 27664 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:00.889747 32406 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:00.922065 32569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.202s	user 0.018s	sys 0.040s
W20260709 12:20:00.975997 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.201s	user 0.005s	sys 0.070s
I20260709 12:20:00.888332 32560 raft_consensus.cc:2764] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 12:20:00.987057 32625 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.253s	user 0.089s	sys 0.115s
W20260709 12:20:01.022128 32428 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.236s	user 0.024s	sys 0.055s
I20260709 12:20:00.982705 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:00.833940 32104 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:00.997449 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:01.028784 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s	user 0.017s	sys 0.036s
I20260709 12:20:01.029471 27655 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:01.058797 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.017s	sys 0.036s
I20260709 12:20:01.029371 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.059442 27660 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:20:01.030589 28334 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:01.031962 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.257s	user 0.006s	sys 0.070s
I20260709 12:20:01.039577 27661 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:00.987788 32561 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.198s	user 0.006s	sys 0.043s
W20260709 12:20:01.072362 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 1.29725e+06us
W20260709 12:20:01.031242 32569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.311s	user 0.018s	sys 0.041s
W20260709 12:20:01.058090 32428 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.024s	sys 0.056s
I20260709 12:20:01.080567 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:01.089627 28328 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.022s	sys 0.029s
I20260709 12:20:00.975704 32458 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.083904 27655 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:00.975520 32606 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.087124 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.096909 27660 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 6.
I20260709 12:20:01.020550 32637 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:01.091761 28328 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.022s	sys 0.029s
W20260709 12:20:01.074733 32561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.285s	user 0.011s	sys 0.043s
I20260709 12:20:00.967015 32267 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:01.085443 32523 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.033s	sys 0.019s
I20260709 12:20:01.101899 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.105427 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.116830 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.117481 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:01.091398 32621 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:01.110321 28329 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:01.027238 32516 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:01.119009 32654 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:20:01.120574 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 745) took 2438 ms. Trace:
I20260709 12:20:01.168445 27736 rpcz_store.cc:276] 0709 12:19:58.681703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.659733 (+978030us) service_pool.cc:224] Handling call
0709 12:19:59.731771 (+ 72038us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:59.735659 (+  3888us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:59.735689 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:59.736169 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:20:00.620093 (+883924us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:00.620134 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:00.620199 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:01.120299 (+500100us) raft_consensus.cc:1685] finished
0709 12:20:01.120323 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.120508 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3336,"raft.queue_time_us":4834,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":2071,"thread_start_us":883591,"threads_started":1,"wal-append.queue_time_us":1363791}
I20260709 12:20:01.169500 32516 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:01.120832 32652 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:20:01.121574 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:01.121997 32523 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s	user 0.033s	sys 0.019s
I20260709 12:20:01.171633 32661 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:01.171928 32523 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:01.172171 32661 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.172575 32523 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.170830 32516 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:01.174233 32661 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.174775 32523 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.175308 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:01.127805 32651 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:20:01.142308 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 750) took 1261 ms. Trace:
I20260709 12:20:01.149459 32453 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.153015 32458 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.097339 32428 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:20:01.154568 32606 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.157619 27664 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 12:20:01.161239 32490 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.967s	user 0.009s	sys 0.000s
I20260709 12:20:01.162551 32635 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:01.106602 32496 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.197172 32627 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 12:20:01.181301 32473 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.196043 32428 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:01.165205 32511 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.166193 32638 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.111227 32637 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.238531 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.239373 27658 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:01.167768 32621 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.171247 27661 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:01.176698 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.250233 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
W20260709 12:20:01.131956 32656 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:20:01.181860 27662 rpcz_store.cc:276] 0709 12:19:59.880453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.881464 (+  1011us) service_pool.cc:224] Handling call
0709 12:19:59.881764 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:59.883024 (+  1260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:59.883050 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:59.883510 (+   460us) log.cc:844] Serialized 26 byte log entry
0709 12:20:00.675975 (+792465us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:00.676017 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:00.676079 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:01.142103 (+466024us) raft_consensus.cc:1685] finished
0709 12:20:01.142121 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.142248 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":867,"raft.queue_time_us":972,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":10315,"thread_start_us":787667,"threads_started":1,"wal-append.queue_time_us":1188348}
I20260709 12:20:01.184249 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:01.254772 27585 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:01.189018 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.201148 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.201349 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:01.258446 27586 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:01.163653 32598 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.163643 32489 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:01.191392 32644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.000s	user 0.007s	sys 0.042s
I20260709 12:20:01.201046 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.220397 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.218282 32627 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:01.275213 27581 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
W20260709 12:20:01.275250 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:01.221976 32490 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.197633 32523 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:20:01.277413 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.000s	user 0.032s	sys 0.027s
W20260709 12:20:01.277990 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.032s	sys 0.027s
I20260709 12:20:01.258602 27660 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:01.248836 32458 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.292237 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.293298 27658 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:01.274549 32644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.007s	sys 0.043s
I20260709 12:20:01.201984 32473 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.299711 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:01.304845 27663 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:01.276794 32627 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:01.306807 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:01.308238 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.269982 27585 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
W20260709 12:20:01.237708 32463 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.559s	user 0.012s	sys 0.000s
I20260709 12:20:01.309835 32463 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.278795 32523 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:01.311895 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.312539 27583 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:01.273294 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:01.252362 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.261049 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.316332 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.321169 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.322434 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:01.322970 27661 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:01.324625 32473 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.274789 32598 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.315220 27589 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:01.319041 27662 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:01.317294 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:01.287223 32526 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.024s	user 0.007s	sys 0.025s
I20260709 12:20:01.340129 27655 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:01.344717 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.349562 32104 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:01.351981 27664 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:01.296674 32458 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.353039 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:01.354128 27735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.342s	user 0.020s	sys 0.024s
I20260709 12:20:01.319450 32617 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:01.352116 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:01.346103 32526 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.007s	sys 0.025s
W20260709 12:20:01.361176 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.349s	user 0.021s	sys 0.025s
I20260709 12:20:01.375425 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2126) took 2816 ms. Trace:
W20260709 12:20:01.362751 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.021s	sys 0.031s
I20260709 12:20:01.367507 27662 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:20:01.366952 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.017s	sys 0.044s
I20260709 12:20:01.295738 32428 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.363422 32617 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:01.376897 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.021s	sys 0.032s
I20260709 12:20:01.380728 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2075) took 1508 ms. Trace:
I20260709 12:20:01.382030 32598 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:01.377086 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.017s	sys 0.045s
I20260709 12:20:01.368264 32463 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.385047 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.385655 27663 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 10.
I20260709 12:20:01.386745 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.386917 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:01.377166 32599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.015s	sys 0.020s
W20260709 12:20:01.387557 32599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.015s	sys 0.021s
I20260709 12:20:01.387682 27659 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:20:01.388626 32496 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.389106 32496 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.391220 32496 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.392042 32496 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.392640 32496 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.394644 32496 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:01.382015 27657 rpcz_store.cc:276] 0709 12:19:59.872024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.872679 (+   655us) service_pool.cc:224] Handling call
0709 12:19:59.873020 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:59.873278 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:20:00.608392 (+735114us) log.cc:844] Serialized 14 byte log entry
0709 12:20:01.380371 (+771979us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:01.380424 (+    53us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:01.380477 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:01.380545 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.380657 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1232594,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":39,"raft.queue_time_us":82,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":1494141,"threads_started":2}
I20260709 12:20:01.400230 32621 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:01.402199 27659 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:01.403137 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.403759 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.404336 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.404790 27664 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:20:01.405233 27657 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:20:01.406016 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.406639 27662 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:01.411754 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:01.414004 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:01.414268 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:01.417677 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.419339 32606 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:01.420182 32658 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:01.420827 32606 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.408170 32428 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:01.416709 32489 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.423810 32489 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.426473 32489 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.427172 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.402534 32617 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:01.428412 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2519) took 1447 ms. Trace:
I20260709 12:20:01.435889 27588 rpcz_store.cc:276] 0709 12:19:59.980938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.981864 (+   926us) service_pool.cc:224] Handling call
0709 12:19:59.982223 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:00.007091 (+ 24868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:00.007134 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:00.018463 (+ 11329us) log.cc:844] Serialized 26 byte log entry
0709 12:20:01.059248 (+1040785us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:01.059290 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:01.059554 (+   264us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:01.428228 (+368674us) raft_consensus.cc:1685] finished
0709 12:20:01.428246 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.428364 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2135,"raft.queue_time_us":122,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":11884,"thread_start_us":964345,"threads_started":1,"wal-append.queue_time_us":1374489}
I20260709 12:20:01.429342 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.424150 32606 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.437763 27663 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:01.366662 32627 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.380312 32267 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:01.458794 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.013s	sys 0.050s
I20260709 12:20:01.369594 32523 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:01.459400 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.014s	sys 0.050s
I20260709 12:20:01.437620 32404 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:20:01.462524 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:01.465612 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 858448us
I20260709 12:20:01.429178 32552 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:20:01.467006 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:01.471639 32627 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:01.475265 27731 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.025s	sys 0.044s
W20260709 12:20:01.476956 27731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.025s	sys 0.044s
I20260709 12:20:01.472743 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:01.438290 32561 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:01.486882 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:01.508906 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.509639 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:01.472810 32637 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.477941 32463 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:01.484375 32661 raft_consensus.cc:2764] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260709 12:20:01.535512 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.375850 27735 rpcz_store.cc:276] 0709 12:19:58.558740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.260529 (+701789us) service_pool.cc:224] Handling call
0709 12:19:59.280473 (+ 19944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:59.280763 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:20:00.012156 (+731393us) log.cc:844] Serialized 14 byte log entry
0709 12:20:01.375075 (+1362919us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:01.375134 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:20:01.375182 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:01.375248 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.375375 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":190060,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":84,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":24092,"thread_start_us":2074055,"threads_started":2}
I20260709 12:20:01.484484 32490 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.484437 32552 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:01.543896 27588 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:20:01.543301 27630 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.017s	sys 0.024s
I20260709 12:20:01.515692 32569 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:01.516022 32561 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.544281 32458 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:01.527624 32404 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:20:01.560681 32605 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.680s	user 0.008s	sys 0.003s
W20260709 12:20:01.606057 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.016s	sys 0.012s
W20260709 12:20:01.606532 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.016s	sys 0.012s
I20260709 12:20:01.544770 32635 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
W20260709 12:20:01.565614 27630 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.018s	sys 0.024s
I20260709 12:20:01.566896 32490 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:20:01.579957 32267 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:01.608175 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.015s	sys 0.015s
I20260709 12:20:01.608316 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
W20260709 12:20:01.610309 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 880173us
I20260709 12:20:01.603996 32404 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:01.613157 27734 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.012s	sys 0.020s
W20260709 12:20:01.621796 27734 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.012s	sys 0.020s
I20260709 12:20:01.546834 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:01.617785 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.015s	sys 0.015s
I20260709 12:20:01.622601 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:01.617514 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:01.623680 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:01.605077 32496 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:20:01.546586 32463 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:01.626281 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 780665us
I20260709 12:20:01.586424 32569 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.643440 27585 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:20:01.611685 32616 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.019s	sys 0.009s
I20260709 12:20:01.639694 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:01.647673 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:01.648743 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.007s	sys 0.028s
W20260709 12:20:01.651305 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.007s	sys 0.028s
I20260709 12:20:01.653960 27585 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:20:01.621421 32201 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.023s	sys 0.006s
W20260709 12:20:01.627961 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81218.2us
I20260709 12:20:01.657402 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:01.657205 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:01.656116 32616 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.019s	sys 0.010s
I20260709 12:20:01.662245 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.669939 27588 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:01.669947 27585 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:01.636163 32496 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:01.656687 32201 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.023s	sys 0.006s
I20260709 12:20:01.672155 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:01.674000 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.674221 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.674731 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.679129 27585 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:20:01.675714 32496 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:20:01.677691 32670 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20260709 12:20:01.679962 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:01.650960 32625 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:01.624914 32267 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.683681 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.678370 32598 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:01.685519 32598 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:01.686324 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 749) took 1698 ms. Trace:
I20260709 12:20:01.686861 27585 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:01.687562 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:01.680534 32644 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:01.685949 32598 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:01.686729 28328 rpcz_store.cc:276] 0709 12:19:59.987805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.988309 (+   504us) service_pool.cc:224] Handling call
0709 12:19:59.988717 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:59.989199 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:59.989243 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:00.032839 (+ 43596us) log.cc:844] Serialized 26 byte log entry
0709 12:20:01.100055 (+1067216us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:01.100097 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:01.100159 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:01.686120 (+585961us) raft_consensus.cc:1685] finished
0709 12:20:01.686138 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.686265 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9835,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":11504,"thread_start_us":990158,"threads_started":1,"wal-append.queue_time_us":1466568}
I20260709 12:20:01.691642 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:01.698726 32404 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.699414 32490 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:01.699806 32644 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:01.701481 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.724967 27588 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
W20260709 12:20:01.703955 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:01.688406 32556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.439s	user 0.004s	sys 0.033s
I20260709 12:20:01.704173 32561 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:01.704545 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:01.736310 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:01.667795 32605 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:20:01.712574 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.712949 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.715597 32676 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:01.625236 32552 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.703208 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:01.706235 32267 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:01.742159 27737 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:01.743221 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:01.756233 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:01.756415 27730 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:20:01.756985 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:01.757350 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:01.736184 32556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.487s	user 0.004s	sys 0.034s
I20260709 12:20:01.697952 32635 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:01.715924 32606 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:01.751593 27733 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:20:01.766995 27733 pending_rounds.cc:85] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 6
I20260709 12:20:01.767297 27733 pending_rounds.cc:107] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:20:01.767607 27733 raft_consensus.cc:2889] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:01.689494 32625 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.753257 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:01.772688 27735 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:01.754171 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:01.759056 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:01.757972 27730 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:01.762015 27732 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:01.734952 32490 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:01.739423 32644 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:01.773910 27736 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
W20260709 12:20:01.768476 28292 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.008s	sys 0.017s
W20260709 12:20:01.778131 28292 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.008s	sys 0.017s
W20260709 12:20:01.769634 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.267s	user 0.012s	sys 0.013s
I20260709 12:20:01.753189 32676 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:01.746546 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:01.774042 27737 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
W20260709 12:20:01.778522 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 926256us
W20260709 12:20:01.779199 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.277s	user 0.012s	sys 0.013s
I20260709 12:20:01.766690 32635 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:01.755676 32404 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:01.783149 27702 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:20:01.783092 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:01.823734 27702 leader_election.cc:403] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
W20260709 12:20:01.775594 28334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.008s	sys 0.016s
I20260709 12:20:01.820212 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.817133 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:01.826352 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
W20260709 12:20:01.827389 28334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.009s	sys 0.016s
I20260709 12:20:01.834434 27732 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:01.834965 27732 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:01.837193 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53412.2us
I20260709 12:20:01.840679 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:01.782044 32490 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:20:01.848848 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 237) took 22297 ms (client timeout 15000 ms). Trace:
I20260709 12:20:01.857671 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:01.858709 28334 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:01.786823 32676 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:01.833950 32616 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:01.867163 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2528) took 2137 ms (client timeout 2025 ms). Trace:
W20260709 12:20:01.871948 27582 rpcz_store.cc:269] 0709 12:19:59.796183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:59.796325 (+   142us) service_pool.cc:224] Handling call
0709 12:20:01.866882 (+2070557us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:01.867072 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4484569856}
W20260709 12:20:01.857026 32675 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:20:01.855674 27732 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:01.836881 32511 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:01.835266 32635 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:01.875739 32523 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:01.876262 27487 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:01.881042 27582 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:01.887897 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:01.889536 32516 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:20:01.899518 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:01.858371 32569 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:01.908857 27732 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:01.909925 27587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.017s	sys 0.037s
I20260709 12:20:01.856976 32679 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
W20260709 12:20:01.917090 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.018s	sys 0.038s
I20260709 12:20:01.913861 32516 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:01.936842 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51510.4us
I20260709 12:20:01.937992 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:01.901222 32616 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.944280 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 753) took 1919 ms. Trace:
W20260709 12:20:01.860541 32567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.010s	sys 0.014s
I20260709 12:20:01.944782 27656 rpcz_store.cc:276] 0709 12:20:00.024457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:00.024582 (+   125us) service_pool.cc:224] Handling call
0709 12:20:00.048212 (+ 23630us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:00.276121 (+227909us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:00.276171 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:00.276753 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 12:20:01.279049 (+1002296us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:01.279104 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:01.398422 (+119318us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b5400291b54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:01.398487 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:01.935935 (+537448us) raft_consensus.cc:1685] finished
0709 12:20:01.935955 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:01.944212 (+  8257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36285,"raft.queue_time_us":37801,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":42263,"spinlock_wait_cycles":439501056,"thread_start_us":1001815,"threads_started":1,"wal-append.queue_time_us":1413068}
W20260709 12:20:01.853358 27483 rpcz_store.cc:269] 0709 12:19:39.570174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:39.574265 (+  4091us) service_pool.cc:224] Handling call
0709 12:19:41.396372 (+1822107us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a350f3a 00007f0ea0106e9c 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc 00007f0e9e6c5392 00007f0e9868348c 00007f0e986828c6
0709 12:20:01.848776 (+20452404us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:57.541062 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:57.593112 (+ 52050us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:57.593163 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:19:57.593181 (+    18us) tablet.cc:662] Decoding operations
0709 12:19:57.677402 (+ 84221us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:57.714517 (+ 37115us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:57.716812 (+  2295us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:19:57.753783 (+ 36971us) tablet.cc:701] Row locks acquired
0709 12:19:57.753832 (+    49us) write_op.cc:260] PREPARE: finished
0709 12:19:57.754013 (+   181us) write_op.cc:270] Start()
0709 12:19:57.754187 (+   174us) write_op.cc:276] Timestamp: P: 1783599597753939 usec, L: 0
0709 12:19:57.754232 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:19:57.754953 (+   721us) log.cc:844] Serialized 9801 byte log entry
0709 12:19:58.135178 (+380225us) op_driver.cc:464] REPLICATION: finished
0709 12:19:58.342658 (+207480us) spinlock_profiling.cc:243] Waited 6.48 ms on lock 0x7b1800240c40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a2c5190 00007f0e9a2beae7 00007f0e9a2bddd9
0709 12:19:58.593067 (+250409us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b1800240c40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:19:58.593303 (+   236us) write_op.cc:301] APPLY: starting
0709 12:19:58.593407 (+   104us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:58.611284 (+ 17877us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:58.611374 (+    90us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:59.151958 (+540584us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:59.300070 (+148112us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=50,mrs_lookups=0
0709 12:19:59.364335 (+ 64265us) write_op.cc:312] APPLY: finished
0709 12:19:59.554477 (+190142us) log.cc:844] Serialized 220 byte log entry
0709 12:20:00.782321 (+1227844us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:00.782608 (+   287us) write_op.cc:454] Released schema lock
0709 12:20:00.782715 (+   107us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:00.783282 (+   567us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:00.783339 (+    57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":793735,"reported_tablets":51,"spinlock_wait_cycles":2139360128,"child_traces":[["op",{"apply.queue_time_us":150605,"apply.run_cpu_time_us":65633,"apply.run_wall_time_us":2189929,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108627,"num_ops":25,"prepare.queue_time_us":244877,"prepare.run_cpu_time_us":16535,"prepare.run_wall_time_us":472439,"raft.queue_time_us":136067,"raft.run_cpu_time_us":37750,"raft.run_wall_time_us":101151,"replication_time_us":380790,"spinlock_wait_cycles":69896832,"thread_start_us":1698214,"threads_started":5,"wal-append.queue_time_us":1566417,"wal-append.run_cpu_time_us":21568,"wal-append.run_wall_time_us":1368150}]]}
W20260709 12:20:01.951764 32567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.010s	sys 0.014s
W20260709 12:20:01.956288 32467 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.017s	sys 0.023s
W20260709 12:20:01.975909 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.014s	sys 0.015s
W20260709 12:20:01.976341 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.014s	sys 0.015s
W20260709 12:20:01.960525 27580 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.013s	sys 0.017s
W20260709 12:20:01.962078 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 289989us
W20260709 12:20:01.977067 27580 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.013s	sys 0.017s
I20260709 12:20:01.977677 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:01.928740 32685 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
I20260709 12:20:01.929311 32463 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:01.962666 27584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.013s	sys 0.037s
I20260709 12:20:01.916253 32617 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:01.949460 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 5 to 6, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 6 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:01.938148 32516 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:02.009642 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:02.013830 27584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.013s	sys 0.037s
W20260709 12:20:02.000000 32467 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.018s	sys 0.023s
I20260709 12:20:01.948480 32621 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:20:01.975842 32526 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.014s	sys 0.022s
I20260709 12:20:01.928819 32458 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.073s
I20260709 12:20:01.952546 32201 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:02.032649 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2523) took 1830 ms. Trace:
I20260709 12:20:02.035205 32511 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:02.044754 32677 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260709 12:20:02.045643 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:02.047170 32681 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 12:20:02.027829 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:02.049763 32617 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:02.052403 27587 rpcz_store.cc:276] 0709 12:20:00.202683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:00.205956 (+  3273us) service_pool.cc:224] Handling call
0709 12:20:00.279499 (+ 73543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:00.279765 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:20:00.408942 (+129177us) log.cc:844] Serialized 14 byte log entry
0709 12:20:01.183854 (+774912us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b18002b8520. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:20:01.943662 (+759808us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:01.943724 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:01.943772 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:01.961391 (+ 17619us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:02.032426 (+ 71035us) spinlock_profiling.cc:243] Waited 73.1 ms on lock 0x7b54002ba7d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:02.032578 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25916,"prepare.queue_time_us":128970,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":15771,"raft.queue_time_us":26059,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":24354,"spinlock_wait_cycles":186717824,"thread_start_us":733560,"threads_started":1}
I20260709 12:20:02.052541 32267 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:02.097357 32267 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:20:02.022212 32682 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:20:02.063158 32511 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:02.084540 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56782.3us
I20260709 12:20:02.108111 27627 leader_election.cc:400] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:02.108465 27627 leader_election.cc:403] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:02.087649 32567 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:02.111666 32625 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:02.112107 32567 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.113508 32625 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:02.114537 32567 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:02.099066 32267 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:20:02.111197 27627 leader_election.cc:400] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:02.116441 27627 leader_election.cc:403] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:20:02.117620 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:02.118261 32267 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:02.079173 32458 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.042s
W20260709 12:20:02.122136 32671 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.535s	user 0.006s	sys 0.000s
I20260709 12:20:02.125851 27732 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:20:02.126354 27732 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:20:02.126673 27732 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:20:02.127014 27732 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:02.127449 27732 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:20:02.127774 27732 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:20:02.148869 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.026s	sys 0.041s
I20260709 12:20:02.132138 32267 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:20:02.134088 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:02.132134 27487 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:20:02.145748 32104 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.790s	user 0.010s	sys 0.002s
W20260709 12:20:02.150358 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:02.026168 32526 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.014s	sys 0.023s
I20260709 12:20:02.157120 27487 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 9 to 15. New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.130915 32511 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:02.151810 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.026s	sys 0.041s
I20260709 12:20:02.150393 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
W20260709 12:20:02.162945 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 652346us
I20260709 12:20:02.164287 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:02.164670 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 339135us
I20260709 12:20:02.165141 32567 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:02.165647 32567 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:02.166009 32567 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:02.160494 27487 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 8 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 11 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.114565 32625 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:02.153303 32616 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:02.163041 27627 leader_election.cc:400] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:02.167761 27487 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.167837 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:02.190667 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:02.195281 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.016s	sys 0.041s
W20260709 12:20:02.196476 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.016s	sys 0.041s
W20260709 12:20:02.214390 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 552612us
I20260709 12:20:02.199755 27627 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:02.215270 27627 leader_election.cc:403] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:02.150707 32598 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:02.194765 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 261084us
I20260709 12:20:02.194892 32463 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:20:02.216640 27731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.011s	sys 0.045s
I20260709 12:20:02.194975 32637 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:02.216195 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.238502 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:02.234046 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.000s	sys 0.014s
W20260709 12:20:02.239924 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.000s	sys 0.014s
W20260709 12:20:02.226258 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 388104us
W20260709 12:20:02.229512 27731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.011s	sys 0.045s
I20260709 12:20:02.233199 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:02.242319 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:02.240339 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:02.245879 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.246173 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.247071 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:02.240350 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 542060us
W20260709 12:20:02.250165 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:02.252288 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.253364 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:02.255759 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:02.256769 27663 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13.
I20260709 12:20:02.259291 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:02.260896 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:02.261615 27660 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:02.262961 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.263993 27664 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:02.266484 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 10 to 11, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 11 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.268656 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.270231 27487 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 7 to 10, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 10 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.267146 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.275406 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.241277 32625 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:02.276346 27736 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:02.251652 32685 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:02.280323 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.318572 32627 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:02.303663 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.339468 27662 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:02.251641 32104 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:02.310758 27657 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:20:02.280220 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:02.241880 32267 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:02.315236 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:02.331660 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 385215us
I20260709 12:20:02.325073 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.255407 32605 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:02.298759 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:02.324124 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.255817 32702 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:02.261801 32463 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:02.276324 32637 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:02.343286 27664 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:02.348018 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:02.356586 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.360610 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.316287 32711 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:02.454936 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.345963 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:02.359112 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:02.393496 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:02.465531 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:02.337666 32708 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:02.391746 27657 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:02.340683 32627 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:02.350661 32685 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:02.483278 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:02.483213 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 137339us
I20260709 12:20:02.322971 32516 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:02.360064 32702 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:02.493091 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:02.498705 27664 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
W20260709 12:20:02.475144 32644 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.658s	user 0.005s	sys 0.006s
I20260709 12:20:02.339378 32710 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.149s
I20260709 12:20:02.462759 32605 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:20:02.508607 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 456579us
I20260709 12:20:02.361456 32467 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.131s
W20260709 12:20:02.489499 32676 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.627s	user 0.008s	sys 0.004s
I20260709 12:20:02.361853 32617 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.134s
I20260709 12:20:02.475996 32708 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:02.467710 32701 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:02.518316 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:02.489533 32685 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:02.363214 32267 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:02.545114 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0028076999999485s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
I20260709 12:20:02.563537 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:02.565279 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 298205us
I20260709 12:20:02.546993 27738 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:02.566043 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:02.451402 32625 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
W20260709 12:20:02.552647 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:02.553243 32404 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.718s	user 0.007s	sys 0.004s
I20260709 12:20:02.548065 32511 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:02.501276 32704 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:20:02.454428 32706 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.150s
I20260709 12:20:02.501065 32702 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:02.543844 32678 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:02.512843 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 256073us
I20260709 12:20:02.520364 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:02.588943 27581 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:02.568110 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:02.569770 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.594189 27737 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:02.548559 32605 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:02.555948 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:02.569633 32701 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:02.560883 32685 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:02.549194 32561 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:02.551867 32708 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:20:02.585388 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 322480us
I20260709 12:20:02.586581 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.562175 32552 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:20:02.570258 32490 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.702s	user 0.010s	sys 0.001s
I20260709 12:20:02.514660 32644 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.598052 27587 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:02.547812 32676 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.574874 27656 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:02.577755 32712 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:02.568267 32598 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:02.566637 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.606441 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:02.607681 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.609414 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.609616 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:02.599666 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.618871 27730 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:02.566649 32463 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:02.623701 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:02.624174 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57603.1us
I20260709 12:20:02.625140 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:02.626597 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:02.602397 32701 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:02.627594 32523 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.710s	user 0.006s	sys 0.006s
W20260709 12:20:02.629519 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:02.630383 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:02.633981 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.634162 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:02.632717 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:02.640367 27585 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:02.641443 27733 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:20:02.642678 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.640623 27730 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:20:02.566779 32637 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.649778 27582 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:02.650789 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:02.660405 27656 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:02.621246 32712 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:02.603441 32678 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:02.670037 27730 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:20:02.671057 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.634855 32706 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:02.622651 32552 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:02.674530 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:02.709059 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:02.613564 32599 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:20:02.740123 27735 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:02.638813 32701 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:02.612231 32561 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:02.717259 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 224232us
I20260709 12:20:02.693004 27730 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:20:02.765200 27730 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:20:02.765532 27730 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:02.664636 32523 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:02.765426 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:02.715205 27587 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:02.618777 32490 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.666363 32712 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:02.632992 32709 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
I20260709 12:20:02.771538 27584 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:02.663261 32729 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:02.626379 32732 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.125s
I20260709 12:20:02.781092 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.782608 27736 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:02.782789 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:02.783100 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 217173us
I20260709 12:20:02.785647 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.746793 32706 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:02.769371 32496 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.082s	user 0.012s	sys 0.001s
I20260709 12:20:02.745744 32723 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:02.749853 32428 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:02.801585 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.783226 32511 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:02.783910 32516 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:02.797590 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 153896us
I20260709 12:20:02.829864 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:02.830456 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 179843us
I20260709 12:20:02.831128 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:02.779500 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.017s	sys 0.014s
W20260709 12:20:02.831877 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.017s	sys 0.014s
I20260709 12:20:02.832224 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:02.832404 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 579085us
I20260709 12:20:02.833559 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:02.836107 32729 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:02.836347 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:02.837240 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.838369 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:02.800760 32700 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 12:20:02.791106 32467 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:02.782708 32728 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:20:02.795246 32706 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:02.847288 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:02.799032 32723 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:02.793835 32731 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:02.803110 32730 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:02.750773 32605 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.804776 32428 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:02.803957 32523 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:02.879031 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:02.859997 32723 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:02.898993 32720 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
W20260709 12:20:02.899761 32716 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.237s	user 0.000s	sys 0.000s
W20260709 12:20:02.901057 32721 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.281s	user 0.000s	sys 0.000s
I20260709 12:20:02.901458 32722 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:02.901994 32722 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:02.902377 32635 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.981s	user 0.012s	sys 0.000s
W20260709 12:20:02.905365 32719 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:20:02.892246 27730 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:02.908972 32741 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:02.911005 32738 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:02.902792 32635 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.911417 32738 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:02.912269 32738 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.912773 32404 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:02.913094 32635 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:02.913712 32635 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:02.914016 32635 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:02.914395 32635 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:02.914865 32710 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:02.914441 32404 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:02.915773 32404 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:02.915979 32644 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:02.916497 32710 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:02.917243 32710 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:02.916698 32404 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.917896 32710 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:02.918552 32710 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:02.918918 32710 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:02.851392 32728 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:02.919602 32728 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:02.921957 32561 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:02.922534 32561 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:02.849002 32552 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:02.924239 32490 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:02.924160 32712 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:02.926712 27581 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:02.926950 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2384) took 2082 ms. Trace:
I20260709 12:20:02.803370 32496 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.927476 27734 rpcz_store.cc:276] 0709 12:20:00.844551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:00.850299 (+  5748us) service_pool.cc:224] Handling call
0709 12:20:00.850587 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:01.622533 (+771946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:01.622564 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:01.622998 (+   434us) log.cc:844] Serialized 26 byte log entry
0709 12:20:02.153550 (+530552us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:02.153611 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:02.153709 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:02.926660 (+772951us) raft_consensus.cc:1685] finished
0709 12:20:02.926688 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:02.926876 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":320878,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":1262695,"threads_started":2,"wal-append.queue_time_us":705057}
I20260709 12:20:02.933295 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:02.934265 28329 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:02.877324 32267 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:02.891785 32428 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:02.851214 32516 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
I20260709 12:20:02.916687 32644 raft_consensus.cc:2764] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 12:20:02.953001 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 120831us
I20260709 12:20:02.923000 32561 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:02.958002 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:02.955451 27729 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 7: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:02.944351 32567 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.778s	user 0.008s	sys 0.004s
I20260709 12:20:02.976078 32712 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:02.952781 32104 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.606s	user 0.009s	sys 0.001s
W20260709 12:20:03.002517 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 164037us
I20260709 12:20:03.004524 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:03.007855 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48050.3us
I20260709 12:20:02.931037 32523 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:02.914381 32702 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.000379 32676 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:02.945840 32725 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:20:03.025871 32516 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:03.029834 32724 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:20:03.027387 32706 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.029131 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.029567 32561 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:03.023367 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63151.2us
I20260709 12:20:03.026283 28331 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:20:03.016662 32727 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:03.016744 32104 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.017201 32526 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:03.000130 32567 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.018985 32644 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.044605 32678 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.019248 32599 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:03.060920 32496 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.063040 32702 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.002671 32710 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.064932 32708 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:03.068189 32739 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:20:03.070726 32523 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:03.071642 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 242652us
W20260709 12:20:03.074829 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 254264us
I20260709 12:20:03.115103 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.076056 27704 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13.
W20260709 12:20:03.077811 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 230607us
I20260709 12:20:03.071568 32516 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:03.094969 27730 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:03.072070 32724 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:03.100778 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71735.3us
I20260709 12:20:03.078145 32727 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:03.063143 32701 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.083010 32526 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:03.087396 32722 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.121335 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:03.140407 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:03.124163 27587 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:20:03.126223 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:03.130156 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:03.131866 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 773) took 1952 ms. Trace:
I20260709 12:20:03.132843 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:03.104298 32625 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.133864 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:03.144209 27580 rpcz_store.cc:276] 0709 12:20:01.179981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:01.180317 (+   336us) service_pool.cc:224] Handling call
0709 12:20:01.188112 (+  7795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:01.276921 (+ 88809us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:01.978059 (+701138us) log.cc:844] Serialized 14 byte log entry
0709 12:20:02.456695 (+478636us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:02.456777 (+    82us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:02.456846 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:03.131675 (+674829us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x7b54001fe5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:03.131722 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:03.131822 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49663,"prepare.queue_time_us":683702,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":88492,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":8917,"spinlock_wait_cycles":203499904,"thread_start_us":1084651,"threads_started":2,"wal-append.queue_time_us":634939}
I20260709 12:20:03.151626 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:03.152556 27657 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:03.144757 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:03.081395 32685 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.115559 32756 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:03.136510 32724 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:03.160653 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.156165 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:03.126876 32516 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:03.158604 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.163968 27660 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.164054 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.097424 32644 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.161000 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.162148 27552 leader_election.cc:400] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13.
I20260709 12:20:03.220924 27552 leader_election.cc:403] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
W20260709 12:20:03.221765 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59760.8us
I20260709 12:20:03.172199 32759 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:03.159276 32743 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:03.188481 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:03.225788 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.138746 32702 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:20:03.152323 32104 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.103996 32599 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.211488 27737 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:03.153033 32496 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:03.212124 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56029.5us
I20260709 12:20:03.253675 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.164767 27732 raft_consensus.cc:2376] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14. Re-sending same reply.
I20260709 12:20:03.147716 32561 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:20:03.157572 32523 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:03.154707 32710 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.229120 32706 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.229828 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2372) took 2941 ms. Trace:
I20260709 12:20:03.230572 32755 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:20:03.231221 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66021.2us
I20260709 12:20:03.212409 32711 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:03.233150 32516 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.234148 32728 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.234934 32759 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:03.253165 32496 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:03.161536 32637 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:03.153824 32753 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:03.139030 32678 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 12:20:03.158739 32756 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:03.219794 32489 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:03.247735 32708 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.227276 32727 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.249396 32428 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.250720 32701 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.291194 32496 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.259351 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:03.259992 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.152622 32201 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.224107 32729 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:03.277539 27731 rpcz_store.cc:276] 0709 12:20:00.288066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:00.625401 (+337335us) service_pool.cc:224] Handling call
0709 12:20:00.625693 (+   292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:01.478329 (+852636us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:20:01.621746 (+143417us) log.cc:844] Serialized 14 byte log entry
0709 12:20:02.258259 (+636513us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:03.158787 (+900528us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:03.182833 (+ 24046us) log.cc:844] Serialized 26 byte log entry
0709 12:20:03.219527 (+ 36694us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:03.219586 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:03.219703 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:03.229446 (+  9743us) raft_consensus.cc:1685] finished
0709 12:20:03.229616 (+   170us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:03.229769 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19529,"prepare.queue_time_us":143180,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":561421,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":50359,"spinlock_wait_cycles":44214016,"thread_start_us":1424288,"threads_started":2,"wal-append.queue_time_us":608556}
I20260709 12:20:03.285207 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:03.160826 32757 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:03.288832 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 170979us
I20260709 12:20:03.254261 32760 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13.
I20260709 12:20:03.276926 32725 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:20:03.292037 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69606.3us
I20260709 12:20:03.336912 32744 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.278182 32561 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:03.253384 32104 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.243637 32635 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.252887 32463 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:20:03.298301 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 756) took 1047 ms. Trace:
I20260709 12:20:03.288684 32711 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:03.245029 32526 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.341068 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:03.345924 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 86029us
I20260709 12:20:03.345816 32706 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.298580 32516 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.292618 32755 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:03.373212 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2385) took 1634 ms (client timeout 1508 ms). Trace:
I20260709 12:20:03.373104 32755 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:03.373674 27729 rpcz_store.cc:269] 0709 12:20:01.738377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:01.738945 (+   568us) service_pool.cc:224] Handling call
0709 12:20:03.372977 (+1634032us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54001d47d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:03.373120 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2755816192}
I20260709 12:20:03.357822 32701 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:20:03.348402 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 186230us
I20260709 12:20:03.353327 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:03.276530 32523 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:03.354434 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69379.7us
I20260709 12:20:03.388535 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:03.365911 32711 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.347221 32567 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.360386 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:03.299237 32637 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.360950 27738 rpcz_store.cc:276] 0709 12:20:02.251077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:02.274319 (+ 23242us) service_pool.cc:224] Handling call
0709 12:20:03.298042 (+1023723us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:03.298225 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":466189568}
I20260709 12:20:03.363963 32760 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:03.352957 32727 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.351405 32757 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:03.369071 32599 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.371487 32627 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.362105 32685 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.396314 28331 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:03.362317   304 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.391971 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:03.386140 32701 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:03.416649 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:03.397814 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.399693 32567 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:03.358284 32744 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:03.400492 32761 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:20:03.401086 32428 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:03.401966   300 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:20:03.390206 32523 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:20:03.410507 32723 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.515s	user 0.002s	sys 0.011s
I20260709 12:20:03.396044 32679 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.368189 32605 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:20:03.399693 32644 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.409557   306 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:03.434471 27729 pending_rounds.cc:85] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 12
I20260709 12:20:03.470482 27729 pending_rounds.cc:107] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 12:20:03.412767 32637 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.367878 32724 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:03.426288   302 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:20:03.424391 32701 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:03.413367 32757 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.409698   309 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:03.423899 32567 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:03.362124 32759 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:03.485947 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:03.361498 32756 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.415601 32760 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.436972 32627 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.438205 32463 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:20:03.493350 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76793.8us
I20260709 12:20:03.470921 27729 raft_consensus.cc:2889] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:03.435431 32201 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
W20260709 12:20:03.446210 32404 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.009s	sys 0.009s
I20260709 12:20:03.442067 32753 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:03.433427 32678 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:20:03.526319 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:03.474573   311 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:03.425891 32561 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.428972   304 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.501929 32567 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:03.497409 32728 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:03.553053 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:03.440869 32723 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:03.536547   307 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:20:03.538095 32404 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.014s	sys 0.011s
I20260709 12:20:03.525976 32463 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:03.517516 32496 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.567268 32556 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:03.573017   311 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:03.577809 32463 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.506269   309 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.577454 32104 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.589736 32556 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:03.591641 32753 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.604417 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.605808 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:03.597350   305 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.173s	user 0.001s	sys 0.001s
I20260709 12:20:03.607187 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.612568 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.614178 27732 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:03.620854 32496 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.652690 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.653312 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.654683 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.655516 27585 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:03.656270 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.656688 27589 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.657691 27589 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:03.653415 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:03.658893 28329 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:20:03.661753 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:03.665174 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93567.5us
I20260709 12:20:03.665755 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.666452 27729 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:03.667644 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.665871 27582 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:20:03.671847 27583 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
W20260709 12:20:03.675069 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 271157us
I20260709 12:20:03.647732   309 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.658345 32569 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.687526 32698 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:03.680397   314 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:20:03.688361 32698 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.690815 32698 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.680868 32556 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.650913 32744 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.693194 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.662357 32724 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.694500 27589 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:03.694837 27583 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:20:03.695305 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:03.697075 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.702814 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.703030 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.703848 27589 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:03.687563 32428 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.707818 32428 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.709527   308 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:03.710045 32428 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.696012 32569 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.712576 32569 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.713222   324 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.713831   324 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.715986 32701 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.716447   324 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.717499 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.718449 27737 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:03.722215 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.723083 27586 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:03.712704 32751 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.656s	user 0.001s	sys 0.003s
I20260709 12:20:03.724622 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.727591 27729 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:03.728080 27588 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:03.719300 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.728781 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.732758 27588 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:20:03.733417 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.734123 27583 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.734738 27737 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.737694 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.730353 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:03.740955 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:03.710072   308 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:03.744786 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:03.744349   324 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.733611 32724 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.728662 27589 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:03.754611 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.757591 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.759534 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.738125 32723 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.742247 32267 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:03.764286 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:03.764683 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:03.749428   308 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.758993 32702 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:03.768517 32701 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:03.770205 27583 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:03.778924 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.677088   311 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.767654 32267 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:03.794598 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:03.783748 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.805799 27729 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:03.755792 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.784601 27737 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:03.810478 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.811393 27737 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:03.812530 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.813616 27737 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:03.814884 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.816489 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:03.817560 27737 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:03.820837 27737 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:03.822666 27737 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20260709 12:20:03.828861 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91884.5us
I20260709 12:20:03.762476 32724 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.831943 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:03.833297 27729 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:03.833606 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.833865 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.834421 27729 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:03.835003 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.836090   324 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:03.836862 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.836990 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.810200 32267 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:03.828697   319 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:20:03.845670 27737 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:20:03.828430 32702 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:03.845424 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.847432 27737 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:20:03.848785 27737 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:03.848582 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.836716 32569 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:03.849897 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.846829 32678 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:03.856065 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.838181 32567 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.852054   324 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:03.838984   321 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:20:03.863229 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 790) took 1345 ms. Trace:
I20260709 12:20:03.863703 27584 rpcz_store.cc:276] 0709 12:20:02.517808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:02.518014 (+   206us) service_pool.cc:224] Handling call
0709 12:20:03.863046 (+1345032us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:03.863165 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":507457280}
I20260709 12:20:03.865861 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:03.866608 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2087) took 1538 ms. Trace:
I20260709 12:20:03.866947 27656 rpcz_store.cc:276] 0709 12:20:02.328003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:02.328311 (+   308us) service_pool.cc:224] Handling call
0709 12:20:03.866408 (+1538097us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400292554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:03.866563 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406640512}
I20260709 12:20:03.866293 32561 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:03.869647 32759 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:03.870204 32759 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:03.870628 32759 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.872278 27589 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:03.873340 32567 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.850030 32702 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.865880 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.878566 27585 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:03.851308 32523 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.883481 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:03.884370   328 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 12:20:03.886351   327 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:20:03.860884 32723 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:03.892865 32724 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.899788 32702 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:03.915454 27733 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.931047 27587 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [15.13-17.14]
I20260709 12:20:03.931447 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.940785 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:03.941473 27587 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:03.942095 32267 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.942741 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.943881 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:03.943980 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:03.944919 32723 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:03.945339 32723 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:03.930841 32738 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.945467 32722 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:03.945875 32723 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:03.953527   304 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.953584 32723 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:03.954702 32711 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.541s	user 0.008s	sys 0.003s
I20260709 12:20:03.956058 32723 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.945302 27587 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:03.957653 32463 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.967099 32702 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:03.968081 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:03.967810 32702 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:03.968674 27660 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:03.969161 27660 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:03.969765 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.970417 27658 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:03.971050 27660 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:20:03.971457 32725 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:03.972188 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:03.971948 32725 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:03.972872 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:03.973451 27655 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:03.974215 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.974347 32725 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:03.974714 27655 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:20:03.970379 27662 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:03.976365 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.977027 27662 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:03.978749 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.979499 27663 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:03.983974 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.984731 27661 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:03.989306 27658 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:03.992766 27659 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:20:03.993733 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:03.926050 32724 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:20:03.995343 32724 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:03.995791 32724 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:03.991657 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:03.992380 27662 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:03.996917 27656 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:03.998387 32724 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.006124 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:04.006681 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:04.007987 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:04.008291 27659 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:04.009606 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:04.011353 27662 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:04.012586 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:04.007414 27658 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:04.015658 27658 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 12:20:04.016983 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75981.5us
I20260709 12:20:04.028050 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:04.029122 27662 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:04.030036 27655 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 7.
W20260709 12:20:04.030823 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 85476.8us
I20260709 12:20:04.028285 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:04.032050 27663 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:04.039512 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:04.040827 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:04.042930 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:04.044482 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:04.045508 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:03.970347 32702 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.049824 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:04.047720 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:04.054533 27656 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:04.046319 28334 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:04.056509 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:04.047464 32605 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:20:04.015640 32724 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:04.059479 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:04.053339 27661 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:04.067839 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:04.040131 32598 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.067198 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:04.064085 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:04.073354 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:04.042613 32489 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:04.039625 32627 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.076067 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:04.113672 32627 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.115720 32627 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.116243 32627 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:04.042286 32725 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:20:04.123207 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:04.124336 27661 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14.
I20260709 12:20:04.048203 32473 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:04.059430 32727 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:04.119694 27738 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:04.112572 27580 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:04.053198 32729 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:04.069844 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:04.066084 32710 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:04.065965 32702 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.019649 32711 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.117190 32496 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:04.138727 32725 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:04.139140 32725 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:04.132311 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 62552.7us
I20260709 12:20:04.133313   311 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:04.141644   311 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:04.142280   311 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:04.151244 27588 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:04.141481 32759 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.154776 28334 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:04.151316 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:04.158341 27661 raft_consensus.cc:606] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:04.158763 27661 raft_consensus.cc:614] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:04.122710 32404 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:04.160230 32404 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.161954 32759 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:04.162566 32404 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.164886 32463 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:04.170816 32757 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.664s	user 0.010s	sys 0.000s
I20260709 12:20:04.076730 32598 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.173770 32496 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:04.173908 32598 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.077704 32605 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:04.077030 32755 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:04.066368 32701 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:20:04.174989 32496 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.175848 32701 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:04.176222 32701 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:04.077805   308 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.176971 32701 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.178269 32701 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:04.135380 32729 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:04.179845 32729 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:04.180226 32523 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.133939 32267 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.071034 32744 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
W20260709 12:20:04.136545   329 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 12:20:04.068140 32516 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:20:04.137123 32702 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.178856   308 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.178861 32755 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:04.184782 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:04.179724 27583 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:04.133771 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:04.298924 28329 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:04.238636 27738 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:04.269855 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:04.274636 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:04.292820 32735 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:04.310428 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:04.395221 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 261463us
I20260709 12:20:04.397127 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:04.398178 27738 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 11: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:04.381137 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2146) took 1739 ms. Trace:
I20260709 12:20:04.398207 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:04.400846 27736 rpcz_store.cc:276] 0709 12:20:02.641506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:02.641819 (+   313us) service_pool.cc:224] Handling call
0709 12:20:04.380601 (+1738782us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:04.381009 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205875968}
I20260709 12:20:04.408062 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:04.409667 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:04.415656 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:04.417497 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:04.418871 27738 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:20:04.406663 32473 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.407270 32404 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:04.444620 32404 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:04.445195 32404 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:04.406572 32627 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.448511 32627 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.424475 32735 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.171272 32757 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.456107 32711 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.457450 32757 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.395862 32598 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:04.458531 32598 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.459273 32725 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.460798 32725 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:04.461218 32598 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.403134 32744 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:04.461975 32744 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:04.423132 32516 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:20:04.428839 32710 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:04.465780 32760 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.466964 32635 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:04.428849 32473 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.469628 32526 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:04.470211 32473 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.437837 32755 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:04.443797 32758 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.383s	user 0.007s	sys 0.004s
I20260709 12:20:04.447781   306 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:04.448945   301 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:04.437137 32702 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.441084 32523 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:20:04.483943   335 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.360s	user 0.012s	sys 0.019s
I20260709 12:20:04.451190 32627 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.439258 32708 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:04.439272 32756 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:04.503589   322 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.722s	user 0.006s	sys 0.006s
I20260709 12:20:04.467131 32729 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.488559   301 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:04.512085 32729 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.512248   301 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.509476   311 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.513689   301 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:04.515573   311 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.491088   306 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:04.526358 32733 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.602s	user 0.002s	sys 0.012s
W20260709 12:20:04.709337 32736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.653s	user 0.015s	sys 0.015s
W20260709 12:20:04.710119 32736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.654s	user 0.015s	sys 0.015s
I20260709 12:20:04.711889 32735 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.712320 32744 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.714201 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 766) took 1551 ms. Trace:
W20260709 12:20:04.567384 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:04.570462 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:04.575157 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 455865us
I20260709 12:20:04.578886 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:04.585904 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:04.580122 27660 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:04.587018 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:04.721182 27738 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:20:04.721815 27737 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:04.581357 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2398) took 1041 ms. Trace:
I20260709 12:20:04.722828 27734 rpcz_store.cc:276] 0709 12:20:03.540337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:03.542853 (+  2516us) service_pool.cc:224] Handling call
0709 12:20:03.543349 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:03.543793 (+   444us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:03.543839 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:03.544439 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 12:20:03.649197 (+104758us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:03.673843 (+ 24646us) log.cc:844] Serialized 14 byte log entry
0709 12:20:03.674081 (+   238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:03.674172 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:04.493058 (+818886us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:04.580665 (+ 87607us) raft_consensus.cc:1685] finished
0709 12:20:04.580826 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:04.581171 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24382,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":49,"raft.queue_time_us":209,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":15357,"spinlock_wait_cycles":5932288,"thread_start_us":68068,"threads_started":1,"wal-append.queue_time_us":159630}
I20260709 12:20:04.589638 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:04.727207 27661 raft_consensus.cc:606] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:04.727880 27661 raft_consensus.cc:614] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:04.730520 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:04.731453 28331 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:04.491639 32523 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:04.669057 32706 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:04.676467 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2159) took 1007 ms. Trace:
I20260709 12:20:04.680620 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:04.517906 32708 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:04.688419 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.007s	sys 0.006s
I20260709 12:20:04.674779 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:04.695957 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2156) took 1838 ms. Trace:
W20260709 12:20:04.771682 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.007s	sys 0.006s
I20260709 12:20:04.714816 27733 rpcz_store.cc:276] 0709 12:20:03.162630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:03.225441 (+ 62811us) service_pool.cc:224] Handling call
0709 12:20:04.713628 (+1488187us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:04.713920 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1584022784}
I20260709 12:20:04.765832 27730 rpcz_store.cc:276] 0709 12:20:02.857890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:02.908612 (+ 50722us) service_pool.cc:224] Handling call
0709 12:20:02.908924 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:03.127110 (+218186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:04.695191 (+1568081us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54001d38d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:04.695716 (+   525us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":427215232}
I20260709 12:20:04.749879 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:04.750192 27729 rpcz_store.cc:276] 0709 12:20:03.668607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:03.835075 (+166468us) service_pool.cc:224] Handling call
0709 12:20:03.835355 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:03.835537 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:20:04.077796 (+242259us) log.cc:844] Serialized 14 byte log entry
0709 12:20:04.675699 (+597903us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:04.675812 (+   113us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:04.675920 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:04.676034 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:04.676287 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110257,"prepare.queue_time_us":24207,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":116807,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":404,"thread_start_us":689157,"threads_started":2}
I20260709 12:20:04.787127 32744 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:04.795076 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:04.799448 32706 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.804950 32523 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:04.806084 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71027.5us
W20260709 12:20:04.807849 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87006.3us
I20260709 12:20:04.810782 32489 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:04.814038   344 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:20:04.814520   306 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:04.817274 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 967358us
I20260709 12:20:04.821965 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:04.822856 28333 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:04.811386 32489 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:04.828920 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:04.834105   349 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.829604 32766 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:04.832470 32489 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:04.845101 32708 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:04.847050 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:04.835003   349 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:04.853192 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:04.855876 27584 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:20:04.861799   306 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:04.864831   344 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.865692   344 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:04.871490 32523 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:04.877418 32732 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:04.878014 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:04.878263 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:04.881729 32703 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:04.882527 32703 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.884984 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:04.868904 32706 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.893841 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:04.894761 27627 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:04.894987 27581 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:04.895188 27627 leader_election.cc:403] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20260709 12:20:04.884364 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.009s	sys 0.045s
I20260709 12:20:04.876787   306 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:04.885133 32766 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.902405 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:04.905627 32616 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.905746 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:04.906454 32616 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.907469 32766 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.908147 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:04.912127 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49417.5us
I20260709 12:20:04.913565 32741 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:04.914098 32741 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:04.914527 32741 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:04.924624 28331 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:04.914335 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:04.922153 27730 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:20:04.928992 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:04.913530 32616 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:04.935883 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:04.936421 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:04.937355   311 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:04.940340   349 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:04.940901   311 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:04.941471   311 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:04.941789 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47664.9us
I20260709 12:20:04.938112 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:04.943763 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:04.946259 32637 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:04.946712 32637 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:04.947122 32637 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:04.948238 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:04.949030 28330 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:04.950610 32746 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:04.951123 32644 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14.
I20260709 12:20:04.951390 32746 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.952854 32744 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:04.953603 32744 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.954082 32746 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:04.955564 32569 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:04.956256 32569 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:04.951424 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:04.940933 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:04.961462 27732 raft_consensus.cc:606] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:04.970582 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:04.982349 27732 raft_consensus.cc:614] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:04.975560 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:04.957667 32569 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:04.994962 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:04.995460   349 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:04.995885 27661 raft_consensus.cc:606] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:04.996317   349 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:04.996802 27661 raft_consensus.cc:614] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:04.997130   349 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:05.002842 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:05.004153 28330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:20:05.005435 32746 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:05.009960 32746 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.011564 32732 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:05.007112 32644 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.006161 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.013412 32644 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.014307 32746 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.014914 28330 raft_consensus.cc:606] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:05.015686 28330 raft_consensus.cc:614] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:05.005496 32625 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.016778 32625 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.021634 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:20:05.024451 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 616311us
I20260709 12:20:05.026041 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:05.027457 32523 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:05.027858 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.027882 32523 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.028960 32523 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:05.029644 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:04.942965   311 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.046962 32625 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.059664   306 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.063948 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:05.064175 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:05.064551   311 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:05.065403   323 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:05.065750 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:05.067047 32755 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.067520 32744 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.067800 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:05.058485 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:05.071884 28330 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:05.073441 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87211.7us
I20260709 12:20:05.006995 32267 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.029565 32644 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.090631 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:05.103142 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44839us
I20260709 12:20:05.077881 32703 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.108011 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:05.115589 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.103206 32746 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:20:05.118507 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 92756us
I20260709 12:20:05.129091 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.134136 32599 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.135834 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.136932 27660 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:05.067822 32755 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.147249 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.150264 32561 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:05.151204 32561 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:05.152706 32561 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:05.154685 32727 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.955s	user 0.018s	sys 0.004s
I20260709 12:20:05.155562 27661 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
I20260709 12:20:05.155898 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:05.115296 32625 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:05.174825 32644 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:05.174896 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.155207 32727 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:05.182013 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:05.182956 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.179589 27583 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:05.135017 32599 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.197175 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.117808 32267 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.160286 32625 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:05.164722 32746 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:05.204953 32703 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:05.205857 27657 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:05.206532 32703 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:05.208487 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:05.209534 27588 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:05.209904 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.210649 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.209980 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.211434 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:05.212419 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115314us
I20260709 12:20:05.201829 27660 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:05.207105 32703 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:05.205595 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.216760 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.217250 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.193697 32644 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.223238 27588 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:05.216823 32703 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.214390 32625 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:05.257771 27583 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:05.264030 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.247826 32644 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:05.273959 32756 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.773s	user 0.016s	sys 0.000s
I20260709 12:20:05.275671 27657 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:05.253858 32708 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:05.276497 27660 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
W20260709 12:20:05.287657 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47161.3us
W20260709 12:20:05.288956 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132216us
I20260709 12:20:05.296707 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:05.296952 32744 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:05.297657 32744 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.299515 27588 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
W20260709 12:20:05.303289   353 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.410s	user 0.003s	sys 0.003s
I20260709 12:20:05.304728   344 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:05.305970   361 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260709 12:20:05.308260   360 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:20:05.308522 32703 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:05.314815 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:05.346429 27738 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:05.298658 32637 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.316370 27657 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:05.348657 27657 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:05.349216 32637 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.317203 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:05.350131 27660 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:05.318231 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.319746 27656 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:20:05.320997 27658 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:05.351609 32746 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.328400 32744 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:05.330222 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.335193 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:05.356345   359 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:20:05.356959 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.356213 27588 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:20:05.336575   350 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.229s	user 0.001s	sys 0.001s
I20260709 12:20:05.336359 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.336717 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.337858 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:05.361343 27729 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:05.361872 27730 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:05.339059 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.340240 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:05.340903   301 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.827s	user 0.008s	sys 0.012s
I20260709 12:20:05.341442 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.342842 27737 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:05.357702 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:05.335592   352 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:05.299495 32627 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.358417 27588 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:05.284483 32708 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.368479 27733 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
W20260709 12:20:05.365558 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73405.7us
I20260709 12:20:05.367574 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:05.363116 27734 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:05.368747 27582 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:05.376044 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.377620 27738 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
W20260709 12:20:05.384130 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.384608 27658 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:05.391829 32727 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:20:05.391973 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.392385 27656 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:20:05.393575 27583 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:20:05.393661 32725 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.932s	user 0.015s	sys 0.011s
I20260709 12:20:05.393927 27736 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:20:05.395527 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.396370 27729 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:05.379819 32708 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:05.398443 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.405934 32561 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:05.409337 27661 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:20:05.414652 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.415735 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:05.417761 27664 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
W20260709 12:20:05.422200 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54812.2us
I20260709 12:20:05.415371 27732 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:05.427017 27736 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:05.432508 32755 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:05.435632 32760 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:05.436390 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.437825 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.439436 32561 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.439622 27588 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.441145 27588 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:05.442246 32561 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.443753 32637 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:05.445760 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2411) took 1311 ms. Trace:
I20260709 12:20:05.439038 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.379637 32627 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:05.451269 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.460310 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:05.446528 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:05.463810 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:05.418912 32703 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:20:05.465523 32698 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:20:05.465965 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.466868 27736 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:05.467236 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.468009 27580 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:05.468545 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:05.470068 32741 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.000s
I20260709 12:20:05.470412 27580 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:05.470582 32741 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.471832 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:05.472364 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:05.472589 32526 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:05.416179   352 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.474195   308 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:05.410629 32727 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:20:05.467934 27737 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:05.477802 32523 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.480155 32698 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:20:05.481868 32404 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.036s	user 0.018s	sys 0.014s
I20260709 12:20:05.483518 27662 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:20:05.484217 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:05.525815 27660 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:05.485313 27657 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:05.485498 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.486752 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:05.452165 32599 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.460444 27735 rpcz_store.cc:276] 0709 12:20:04.134805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:04.161546 (+ 26741us) service_pool.cc:224] Handling call
0709 12:20:04.161991 (+   445us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:04.162469 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:04.162518 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:04.163200 (+   682us) log.cc:844] Serialized 52 byte log entry
0709 12:20:04.984868 (+821668us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:20:05.023946 (+ 39078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:05.024079 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:05.445463 (+421384us) raft_consensus.cc:1685] finished
0709 12:20:05.445483 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:05.445675 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14622,"raft.queue_time_us":839,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":590065,"threads_started":1,"wal-append.queue_time_us":798953}
I20260709 12:20:05.487708 32569 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:20:05.487756 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:05.490401 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:05.490744 32489 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.625s	user 0.011s	sys 0.003s
W20260709 12:20:05.491926 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79408.3us
I20260709 12:20:05.492367 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:05.493120 32724 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:20:05.534315 27586 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:05.499882   308 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:05.498380 32703 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:05.494359 32267 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.523147 32698 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:05.547609 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:20:05.549044 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:05.550768 28333 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:05.564513 32267 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:05.565547 32267 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.570461 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.574340 27661 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:05.584735 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.537889 32756 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.498668 32766 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:20:05.609985 27580 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:05.546665 32703 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:05.537058 32489 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.640530 32766 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:05.642537 32766 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:05.646736 32523 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:05.649364 32627 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.640415 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:05.652813 32489 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.653668 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:05.654204 32489 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:05.655933   308 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.656783 32725 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:05.657428 32725 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.657641   308 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:05.657876 32725 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:05.660195 32625 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.537177 32404 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.662650 32404 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.663293 32404 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:05.663195 32703 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.666868 27656 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:05.667969 27731 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:05.669021 32741 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.670574   352 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.633685 32756 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:05.701140 27588 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:20:05.715932 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:05.643090 32766 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.628696 32732 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:05.654963 32489 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:05.734319 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59264.5us
W20260709 12:20:05.736846 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.671270 32755 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:05.607476 32708 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.741099 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:05.623680   323 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.097s
I20260709 12:20:05.722215 32766 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:05.734220   349 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.736s	user 0.006s	sys 0.008s
I20260709 12:20:05.755862 27660 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
W20260709 12:20:05.748651 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 315516us
I20260709 12:20:05.746275 32627 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:05.731264   352 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:20:05.756840 32703 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:05.785637 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 293340us
I20260709 12:20:05.760056 32724 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:20:05.790854 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49877.2us
I20260709 12:20:05.781009 32627 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.747871 32732 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.762266   349 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.786356 32267 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:05.816681 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.801337 32627 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:05.823383 27587 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
W20260709 12:20:05.839768 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:05.840246 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:05.837682 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 365945us
W20260709 12:20:05.862998 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 390667us
I20260709 12:20:05.867765   385 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:05.868815 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:05.878774 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83511us
I20260709 12:20:05.869439 32599 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:05.890066 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:05.887161 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70865.9us
I20260709 12:20:05.901410 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:05.887764 32644 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.002s
I20260709 12:20:05.909595 32644 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:05.905365   367 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:20:05.906044   383 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:05.908586   382 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:05.910310 32732 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.915061 32746 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:05.882639 32267 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:20:05.915602 32746 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:05.916076 32746 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:05.916204 32644 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:05.922878   378 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:05.927717   385 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:05.929018 32746 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.930161 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 775) took 1352 ms. Trace:
I20260709 12:20:05.930550 27731 rpcz_store.cc:276] 0709 12:20:04.577671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:04.708428 (+130757us) service_pool.cc:224] Handling call
0709 12:20:04.708978 (+   550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:05.668385 (+959407us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:05.929900 (+261515us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:05.930097 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196600704}
I20260709 12:20:05.931143 32637 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:05.931600 32637 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:05.932547 32637 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:05.933442 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:05.933929 32637 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:05.935933 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:05.954380 32746 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:05.955065 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:05.901252   352 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:05.988515 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:05.991163 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:06.010757 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:05.993366 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55280.6us
I20260709 12:20:05.961167 32489 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:06.025679 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 89988.2us
I20260709 12:20:06.026233 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:06.028404 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.015151 32727 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:06.030074 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:06.030872 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 185326us
I20260709 12:20:06.034173 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.035543 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:06.036098   349 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.037606 32732 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:06.037317 32644 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
W20260709 12:20:06.038146   347 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:20:06.038034 32267 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:06.038676 32267 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:06.038784   393 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:06.039124 32267 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.039232   393 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:06.039722   393 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:06.038064 32732 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:06.041965 32732 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.030468 32746 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:06.054853   392 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:06.055382   392 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:06.055781   392 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:06.059384   368 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:20:06.060355 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:06.060525   390 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:20:06.061872   372 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:20:06.053542 32627 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.063874   311 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:20:06.065330 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:06.065670 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.064287   311 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:06.032235   391 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:06.070007   352 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:06.070286   391 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:06.071909   391 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.073099   378 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:06.073741 32698 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.074882 32698 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:20:06.076511 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.077215 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:06.077289 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.078608 32725 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.079797 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.080681 27655 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 12:20:06.080618 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 167310us
I20260709 12:20:06.081569 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.078644 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.078953 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.083612   384 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:06.084602   370 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.298s	user 0.000s	sys 0.002s
I20260709 12:20:06.084260   384 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.076440 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.085218   352 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:06.085678 27656 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:06.083189 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.086575   384 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:06.086951 27663 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:06.089725 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:06.090852 28334 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:06.092422 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.081677   385 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:06.095714 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:06.098948 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:06.099936 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:06.098973   385 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.102011   385 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.103403   311 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.104856   311 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:06.105563 32756 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:06.069211   390 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:06.109637   390 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.111225 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.112012 27737 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:20:06.112255 27731 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:06.112798 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 120138us
I20260709 12:20:06.116055 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.116806 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.117616 27660 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:06.119072 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.119968 27658 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
I20260709 12:20:06.120587 27656 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:06.066578 32724 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.115130 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:06.121877 27655 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 15. Ops: []
W20260709 12:20:06.123034 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57683.5us
I20260709 12:20:06.123201 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:06.123097 28334 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.122161 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:06.124223 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.124383 27663 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:20:06.130445 32756 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.131242 32756 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.134670 32756 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.135670 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.137107 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:06.139653 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.143639 27662 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:06.143589   397 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:20:06.145334 27733 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:20:06.144665 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:06.160673 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.177727 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.150683 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.178898 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.173218 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.184054 27736 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:20:06.173870 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:06.171787 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.064893 32727 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.135658   384 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.122234 32489 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:06.189985 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65840us
I20260709 12:20:06.149004 32703 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:06.190908 27661 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:20:06.191545 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:06.192202 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 99864.1us
I20260709 12:20:06.192190 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.193655 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.199730 27736 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.194047 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.191808 27732 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:06.112603 32766 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.176694 32708 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:06.182202 32755 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:06.204075 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.204787 27735 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:06.176735   383 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:06.175896   306 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:20:06.199577 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.136185 32724 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.178517   352 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:20:06.209951 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:06.197329 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.136842 32732 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.219707 32404 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:06.220144   401 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:06.220983 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:06.225786   392 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.227979 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.235810 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:06.237812 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.253204 28334 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:06.237322 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.199769   397 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:20:06.267735 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.268113   397 consensus_queue.cc:1243] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:20:06.276500 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.283478 27660 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:06.284057 27661 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
W20260709 12:20:06.290769   394 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.227s	user 0.000s	sys 0.003s
W20260709 12:20:06.292861 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82312.8us
I20260709 12:20:06.294486 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:06.206791 27552 connection.cc:256] RPC call timeout handler was delayed by -1.933411426000248s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20260709 12:20:06.296152 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0473512260002735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20260709 12:20:06.296682 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0410512260002633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
I20260709 12:20:06.299152 27662 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:06.299502 27661 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:06.215821 32703 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:06.300865 27733 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:06.301774 27736 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:06.204018 32489 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.234113   401 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.208341   384 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.311916 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:06.306597 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70867.7us
I20260709 12:20:06.308975 27660 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:06.314210 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:06.315989 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 755498us
W20260709 12:20:06.316710 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.310134 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:06.318660 27730 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:20:06.320564 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.318101 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:06.307821 32744 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:06.323446 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:06.325722 27586 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:06.312391   382 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:06.292284 32267 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.314153   402 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:06.330139 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.330724 27733 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:06.330549 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.316857 32644 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:06.317814 32627 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.340993 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.342036 27736 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:06.322711 32599 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:06.346814 32599 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:06.347255 32599 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:06.336010 32637 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:06.348304 32637 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:06.349304 32637 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.349905 32724 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:06.351701 32625 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:06.335798   402 raft_consensus.cc:2764] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:20:06.358237 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.361809 27662 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:20:06.364796 27738 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:20:06.367290 27660 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:20:06.375149   389 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:06.377547   396 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:06.377954   396 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.380965 32727 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:06.384040 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:06.388151   403 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 12:20:06.390393   398 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:06.393512 32724 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:20:06.395370 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.397255 27586 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:06.397459 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.399341 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:06.342511 32725 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.401736 32708 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:06.402251 32644 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:06.402308 27730 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
W20260709 12:20:06.403592   405 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
W20260709 12:20:06.411793 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89946.1us
I20260709 12:20:06.412503 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:06.418169 32703 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:06.442188 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:06.442672 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:06.420521 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.343005 32744 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:06.426129 32267 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:06.434082 27731 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:20:06.451579 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:06.439990 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 133887us
I20260709 12:20:06.362489 32766 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.363409   382 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.357244   306 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:20:06.431301   385 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.463446 32744 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.468709   382 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.469229 32746 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:06.469810 32746 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:06.471973 27735 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 12:20:06.378867   396 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.474542   306 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:06.476166   384 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:06.367416   401 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.472460 27735 pending_rounds.cc:85] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:20:06.479051   391 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.481848 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.483677   323 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:06.503605   323 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.504730   323 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.505638   401 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:06.506168   323 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:06.486658 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:06.476001 32644 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:06.502450 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:06.470916 32746 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.519644 32644 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.525888 27733 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:06.495016 27735 pending_rounds.cc:107] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:20:06.528219 27735 raft_consensus.cc:2889] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:06.529129   398 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:06.529161 32756 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:06.529920   398 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:06.529435 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57684.3us
I20260709 12:20:06.534689 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:06.561115 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.494575   306 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.523058   391 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:06.590929 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 370025us
I20260709 12:20:06.523654 32732 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:06.599427 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:06.599875 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.594357 27736 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
W20260709 12:20:06.600306 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:06.600721 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.602818 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:06.610222 27738 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
W20260709 12:20:06.611518 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75785.6us
I20260709 12:20:06.589491 32756 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:06.615069 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 297020us
I20260709 12:20:06.596343 32569 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:06.619125 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 252968us
W20260709 12:20:06.628434 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.618363 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.628999 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:06.630771 27701 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14.
I20260709 12:20:06.631047 27701 leader_election.cc:403] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:20:06.631562   411 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:06.632174   411 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:06.626325 27738 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:20:06.634439 27738 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:20:06.628461 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.634778 27738 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:06.618046 32756 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:06.617566   391 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.616298 32489 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:06.626552 32732 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.643422 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:06.646487 27736 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:06.644534 27664 pending_rounds.cc:85] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 13
I20260709 12:20:06.650537 27664 pending_rounds.cc:107] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:20:06.650909 27664 raft_consensus.cc:2889] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:06.651784   391 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.640156 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:06.646785 32756 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.636906   311 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:06.637036 32725 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:06.663383 32616 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:06.640547   392 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.647365   390 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14.
W20260709 12:20:06.729372 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91587us
I20260709 12:20:06.564057   393 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.627980 32569 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.653328 32732 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.733222 27736 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:06.638698 32526 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.070s
W20260709 12:20:06.740486 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60948us
I20260709 12:20:06.671443 32703 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:20:06.742923 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.745569 27731 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 6. Current term is 7. Ops: []
W20260709 12:20:06.746327 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.746477 32703 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:06.747601 32703 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.633081   411 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.709347 32725 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:06.749408 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.531104   398 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.759099 27657 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:06.738276   416 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.743158 32489 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.761766 27738 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
W20260709 12:20:06.769220   417 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:20:06.770412 32569 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.771505 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:06.775465 32489 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.776985 32732 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.734768   306 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.778677   306 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:06.744649   383 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:20:06.759627 32725 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.779891   383 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:06.780299   383 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.781376   383 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.782744   383 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:06.792825 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.767719   416 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.810022   416 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.799017   391 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:06.791487 32766 consensus_queue.cc:1175] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 12 status { last_received { term: 12 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 12 index: 10 } }
I20260709 12:20:06.812098 32698 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:06.812770 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.813478 32569 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:06.814030 32569 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:06.814119   311 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:06.767449 32526 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:06.815033 32526 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:06.815294 32569 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.813309 32698 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:20:06.816517   420 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:20:06.816176 32526 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.791808   390 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:06.817569 32526 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:06.810551 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.814415 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:06.812924 32489 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:06.823269 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.828476 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.832683 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.833436 27738 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:06.834273 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.810057   393 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.829653 32698 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:06.819203   392 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
W20260709 12:20:06.857776 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86390.3us
I20260709 12:20:06.858404 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.860684 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:06.864180 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:06.866359 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111614us
I20260709 12:20:06.861187 32625 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:06.831393 32599 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:06.870235 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.871572 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.873600 27730 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:06.875475   433 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:06.878716 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.879033 32599 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:06.889499 32725 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:06.867075 32569 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:06.892915 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.909230 27730 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:06.895874   392 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:06.894783 32698 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:06.892441 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.902736 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.881784 32267 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:06.915580 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:06.911895 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.916538 27730 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:06.920853 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.922338 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.930744 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.931591 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:06.946453 32644 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:06.946700 27662 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:06.957267 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.958038 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.958128 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.958806 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:06.960960 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.961935 27658 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:20:06.967859 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:06.974761 27731 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:06.975646 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:06.976650 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:06.976918 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:06.977815 28330 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:20:06.979020 27731 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 12:20:06.914412 32724 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:20:06.967135 27663 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:06.931658   393 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:06.984432 32724 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:06.984517 32725 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:06.984911 32724 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:06.984668 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.985855 32724 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:06.986304 27663 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:06.986737 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:06.987023 32724 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:06.987248 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.988081 27663 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:06.989080 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.989739 27663 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:06.990514 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:06.990628   390 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:06.973906 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.979468 32599 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:06.992780 27663 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:06.994639   396 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:06.930783 27730 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:06.964046 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.979954 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:06.997996 27662 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:06.999168 32599 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:06.999589 32724 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:07.000002 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:07.000921 27737 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.005573 27660 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:07.022171 27738 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:07.016108 27661 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:07.023941 32724 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.024942   416 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.025418   416 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.027572 32703 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.028137 32703 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.034483 27660 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:20:07.039498 27737 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.041894 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.039090 32599 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.051491 27586 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:07.053139 32599 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.049026 27730 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:07.054306 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:07.055130 28330 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:07.056231 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.056847 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.061697 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.062404 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:07.068295 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.069113 27588 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.070128 27664 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:20:07.076937 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.078161 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.079190 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.079730 27582 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:07.080049 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.080519 32746 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.081027 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.079751 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.082172 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.082792 27581 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 7.
I20260709 12:20:07.082831 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.083666 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.084271 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.062914 32644 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:07.085624 32644 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.085686 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.086081 32644 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:07.086346 27585 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:07.090966 27582 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:07.092634   406 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.764s	user 0.005s	sys 0.001s
I20260709 12:20:07.093145 32756 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.095906 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2591) took 1140 ms. Trace:
I20260709 12:20:07.096542 27659 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:07.096416 27584 rpcz_store.cc:276] 0709 12:20:05.955479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:05.989339 (+ 33860us) service_pool.cc:224] Handling call
0709 12:20:06.014246 (+ 24907us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:06.027476 (+ 13230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:06.027508 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:06.032914 (+  5406us) log.cc:844] Serialized 26 byte log entry
0709 12:20:06.198821 (+165907us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:06.198857 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:06.198921 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:07.091898 (+892977us) spinlock_profiling.cc:243] Waited 8.69 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:07.095486 (+  3588us) raft_consensus.cc:1685] finished
0709 12:20:07.095518 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:07.095805 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30285,"raft.queue_time_us":2974,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":7757,"spinlock_wait_cycles":19990400,"thread_start_us":111208,"threads_started":1,"wal-append.queue_time_us":223954}
I20260709 12:20:07.098976 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.099630 27583 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.095578 32756 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.104346 27589 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.115841 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.116720 27581 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:20:07.128108 28334 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:20:07.133291 27588 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.134037 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.135188 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.135460 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:07.135816 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:07.137775 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.139671 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.139775 27589 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.149367   440 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.150007 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:07.152779 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.154608 27583 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:20:07.168510 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:07.180845 27581 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:07.172777 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.187925 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.188658 27586 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.180461   391 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:07.187717   306 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 12:20:07.171257 32732 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:20:07.205065   391 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:07.205610   391 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.206058   391 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:07.180981 32599 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:07.160197   392 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:07.208405 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.209189 27584 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:07.221041   416 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.223194 32644 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.226090 32644 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.227240 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:07.227870 28330 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:07.238278 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65561.3us
I20260709 12:20:07.239037 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.248924 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.251756 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:07.253999 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 242516us
I20260709 12:20:07.253993 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.150065   440 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.263712 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.259148 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.274639 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.275353 27733 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:07.275200 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.275542 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.275882 27738 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.273391 27586 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.255467   429 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:07.279122   429 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.263515   402 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:07.281150   402 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.267107   416 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:07.281729   416 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.282058   416 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.282112   402 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.263648 32644 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:20:07.283646 32644 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.284008 32644 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:20:07.285084 32746 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:07.285547 32746 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.285905 32746 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.288151   390 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.288739   390 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.291060 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:07.292062 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:07.294999 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.295651 27731 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:07.296468 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:07.300120 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45374.9us
I20260709 12:20:07.298151 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 786) took 1143 ms. Trace:
I20260709 12:20:07.299585 27738 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.302812 27736 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:20:07.305207 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:07.314247 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:07.300980 27732 rpcz_store.cc:276] 0709 12:20:06.154339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:06.212698 (+ 58359us) service_pool.cc:224] Handling call
0709 12:20:06.291438 (+ 78740us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:06.764282 (+472844us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:06.838058 (+ 73776us) log.cc:844] Serialized 14 byte log entry
0709 12:20:07.246741 (+408683us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:07.253575 (+  6834us) log.cc:844] Serialized 52 byte log entry
0709 12:20:07.253732 (+   157us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:07.253932 (+   200us) log.cc:844] Serialized 14 byte log entry
0709 12:20:07.297748 (+ 43816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:07.297848 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:07.297902 (+    54us) raft_consensus.cc:1685] finished
0709 12:20:07.297930 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:07.298083 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":398230,"prepare.queue_time_us":75031,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":5463,"raft.queue_time_us":314847,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":17303,"thread_start_us":533617,"threads_started":2,"wal-append.queue_time_us":104067}
I20260709 12:20:07.291807 32698 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:07.317137 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.317963 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:07.319587 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:07.321555 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43561.7us
I20260709 12:20:07.293412   400 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:07.329517 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.331851 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.311721   449 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:07.311615 32756 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.335875 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:07.336509 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.319458   392 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.320199 32267 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:07.356117 32724 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.358860 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 791) took 1018 ms. Trace:
I20260709 12:20:07.359850   441 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:07.360316   441 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:07.360677   441 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:07.333662 32625 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:07.363528 32625 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.363885 32625 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:07.365698   391 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.349820 32698 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:07.350060 32756 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.370081 32756 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.351310   451 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:07.371623   429 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.356060 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.359289 27734 rpcz_store.cc:276] 0709 12:20:06.340001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:06.772909 (+432908us) service_pool.cc:224] Handling call
0709 12:20:06.773175 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:07.034878 (+261703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:07.034905 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:07.034952 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:07.034998 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:07.358573 (+323575us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:07.358624 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:07.358770 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":601380608}
I20260709 12:20:07.380506 27658 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:20:07.382239 28330 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:07.383502 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.355096   452 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:07.423863 27583 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:07.427471 27730 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:07.428810   429 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.429361   429 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.429653 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:07.430989 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:07.390462   451 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:07.389986   396 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:20:07.434059 32724 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.338763   400 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.435272 32724 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.436271 32644 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.437170   400 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.437235 32724 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.438014   429 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.438588 32644 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.439710 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:07.434405   451 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.440846 27734 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:20:07.444305 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:07.393344 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57555.2us
I20260709 12:20:07.444962 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.446228 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:07.349720   390 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.454152 28334 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:07.333465 32725 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:07.442552   416 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.455253 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:07.337410   449 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:07.462704 27588 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:20:07.387705   409 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.018s	user 0.005s	sys 0.000s
I20260709 12:20:07.380393 32267 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.389170   440 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.385068   392 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.466552   395 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:20:07.467476 27552 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
I20260709 12:20:07.472021 27552 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:20:07.472263 27581 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:20:07.472669 27581 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 8
I20260709 12:20:07.458324   411 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:07.469218 27587 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:20:07.472891 27581 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:20:07.476320 27581 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:07.455902 32637 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:07.419803   323 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:20:07.464146 32727 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:07.467870   449 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:07.466709   451 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.469617 32625 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.486181 32637 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:07.487383 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 124) took 25442 ms (client timeout 15000 ms). Trace:
I20260709 12:20:07.487946 32625 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.466581   391 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.481142 32755 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
W20260709 12:20:07.487754 27487 rpcz_store.cc:269] 0709 12:19:41.943621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.943905 (+   284us) service_pool.cc:224] Handling call
0709 12:19:42.302596 (+358691us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b54001a0590. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e972240b2 00007f0ea019d759 00007f0ea0135a76 00007f0ea0106d58 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc
0709 12:20:07.374880 (+25072284us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:02.916974 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:02.917526 (+   552us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:02.917565 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:20:02.917586 (+    21us) tablet.cc:662] Decoding operations
0709 12:20:02.918522 (+   936us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:02.918579 (+    57us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:02.918608 (+    29us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:20:02.929942 (+ 11334us) tablet.cc:701] Row locks acquired
0709 12:20:02.931651 (+  1709us) write_op.cc:260] PREPARE: finished
0709 12:20:02.947437 (+ 15786us) write_op.cc:270] Start()
0709 12:20:02.958758 (+ 11321us) write_op.cc:276] Timestamp: P: 1783599602945134 usec, L: 0
0709 12:20:02.968553 (+  9795us) op_driver.cc:348] REPLICATION: starting
0709 12:20:03.055641 (+ 87088us) log.cc:844] Serialized 3574 byte log entry
0709 12:20:04.477950 (+1422309us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e9723e0ac 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b
0709 12:20:04.816417 (+338467us) op_driver.cc:464] REPLICATION: finished
0709 12:20:05.059498 (+243081us) write_op.cc:301] APPLY: starting
0709 12:20:05.059647 (+   149us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:05.137384 (+ 77737us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:05.137429 (+    45us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:05.478612 (+341183us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:05.479175 (+   563us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=0
0709 12:20:05.479242 (+    67us) write_op.cc:312] APPLY: finished
0709 12:20:05.528179 (+ 48937us) log.cc:844] Serialized 89 byte log entry
0709 12:20:05.528436 (+   257us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:05.528733 (+   297us) write_op.cc:454] Released schema lock
0709 12:20:05.528835 (+   102us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:05.529431 (+   596us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:05.529461 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2417579,"reported_tablets":26,"spinlock_wait_cycles":31481728,"child_traces":[["op",{"apply.queue_time_us":172997,"apply.run_cpu_time_us":21282,"apply.run_wall_time_us":469932,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41747,"num_ops":9,"prepare.queue_time_us":371559,"prepare.run_cpu_time_us":36731,"prepare.run_wall_time_us":1794128,"raft.queue_time_us":273133,"raft.run_cpu_time_us":14169,"raft.run_wall_time_us":70484,"replication_time_us":1847490,"spinlock_wait_cycles":17135872,"thread_start_us":2213026,"threads_started":4,"wal-append.queue_time_us":1484214,"wal-append.run_cpu_time_us":31527,"wal-append.run_wall_time_us":1996871}]]}
I20260709 12:20:07.462105 32489 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:20:07.500267   411 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:07.500772   411 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:07.501857   411 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.477860   390 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:07.503818   390 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:07.504504   442 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:20:07.504366   390 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:07.511448 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:07.512253 28334 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:07.513670 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:07.530186 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 75026.9us
I20260709 12:20:07.503193   411 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:20:07.543542 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103903us
I20260709 12:20:07.481813   440 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
I20260709 12:20:07.548926 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2177) took 1422 ms. Trace:
I20260709 12:20:07.550938 27729 rpcz_store.cc:276] 0709 12:20:06.126179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:06.126380 (+   201us) service_pool.cc:224] Handling call
0709 12:20:06.174710 (+ 48330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:06.177835 (+  3125us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:06.177863 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:06.186065 (+  8202us) log.cc:844] Serialized 26 byte log entry
0709 12:20:06.308180 (+122115us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:06.308236 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:06.308435 (+   199us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:07.387092 (+1078657us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:07.528456 (+141364us) raft_consensus.cc:1685] finished
0709 12:20:07.528477 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:07.548830 (+ 20353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60537216,"thread_start_us":95800,"threads_started":1,"wal-append.queue_time_us":152885}
I20260709 12:20:07.553542 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:07.528178   391 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:07.556981   391 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.557165   454 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.557574   391 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.557752   454 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.560003   454 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:07.525538   392 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.570474   382 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:20:07.574156 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:07.577219 27655 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:07.580873 32703 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:07.583420 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116198us
W20260709 12:20:07.584295 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:07.591272 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.594372 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.595053 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:07.596774 27588 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:20:07.598803 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 133087us
I20260709 12:20:07.599378 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:07.586727 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.601537 27587 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:20:07.604722 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.607034 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:07.610376 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.610884 27582 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:20:07.610980 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 274524us
I20260709 12:20:07.612584 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:07.613730 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.614320 27589 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:07.615085 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.615659 27588 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:07.617395 27552 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
I20260709 12:20:07.617699 27552 leader_election.cc:403] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:20:07.618016 27580 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:07.477164 32267 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.622648 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.623917 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:07.625312 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:07.626969 27581 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:07.603287   454 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:07.625890 27585 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 12:20:07.606153   393 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:07.630028 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.630612 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:07.617197 32404 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.627128   402 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.636659   456 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:07.637248   456 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.609866   416 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.637822   391 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.638134   456 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.643442   451 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:07.644399   382 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:07.621302   440 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:07.656162   440 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:07.656586   440 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:07.657104   391 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.611446   441 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.661296   441 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.661931   441 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:07.662823 27584 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:07.623247   443 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:20:07.664067   443 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:07.664390   443 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:07.667501 32637 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.668190   311 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:07.630676 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.632342 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:07.674638 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44029us
I20260709 12:20:07.675252 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.634799   402 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.676277 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.681221 27581 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.677655 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:07.679735 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:07.682621 28330 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.690902 27582 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 8.
I20260709 12:20:07.626222   445 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
W20260709 12:20:07.698793 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65430.3us
I20260709 12:20:07.706477 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.627864 32698 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:20:07.717196 28330 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.679040 32644 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20260709 12:20:07.687304   391 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:07.681800   393 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.719183 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.721624 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.722396 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.722991 27738 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:07.719282 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.695813   451 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:07.696056 32625 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:07.720551   391 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:07.735873   391 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:07.739826 27655 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:07.720124 32756 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:20:07.740722 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:07.742923   382 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:07.727075 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.731760 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.733021 32404 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:07.735183 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.743160 27736 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:07.754640 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:07.696966   392 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:07.725239 27730 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:07.743201 27738 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:07.720521 32599 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:20:07.747053 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:07.744231 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.752722 27663 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:20:07.761431 27657 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:07.766680 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132798us
I20260709 12:20:07.711519   400 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:20:07.772850 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.769745 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:07.774935   445 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:07.776078   445 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:07.776463   445 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:07.777292   445 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.778551   445 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:07.782342 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:07.783608 27659 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:07.757052 32724 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:07.786909 27661 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:07.721271   416 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
W20260709 12:20:07.798041 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:07.805703   458 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.347s	user 0.003s	sys 0.000s
I20260709 12:20:07.812618   451 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:07.767958 32746 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.817808 32746 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.766130   390 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.825922   390 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:07.826838   393 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.833395 27663 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:07.766356   392 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.834707 27731 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:20:07.771538 32756 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:07.837456 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:07.837558 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:07.844925 27588 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.839113 27659 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:20:07.835557 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:07.768359 32599 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.869710 27588 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.873131 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:07.878091 27732 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:07.875373 27662 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:20:07.880247 32724 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.881234 32724 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:07.883538 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43785.5us
I20260709 12:20:07.886576 32756 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:07.887940 32756 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.892566   392 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:07.895911 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44479.1us
I20260709 12:20:07.850672   311 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:07.857605 32698 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.913956 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.927824   456 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:07.912269   311 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.916580 32698 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.942924 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.945076 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:07.935426   402 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.935117 32569 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:07.947402 32637 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.008s
W20260709 12:20:07.957823 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72685.5us
I20260709 12:20:07.947628   472 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:07.961583   472 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:07.972249 27581 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:07.934878   456 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.956336   414 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:07.932853 32599 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.962361 32698 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.982656   311 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:07.977478   414 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:07.987600   414 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:07.988940   414 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:07.989068   474 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:20:07.993556   391 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:07.996871   461 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:07.997375   461 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:07.997591 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 373682us
I20260709 12:20:07.998472 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:07.999694 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.000726 32744 raft_consensus.cc:2764] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:20:08.001416 32744 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:08.001894 32489 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:08.004050   391 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.004438   456 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:08.006558 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:08.007675 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:07.997767   461 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.011375 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:08.012601 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.013506 27730 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:08.014796 27735 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:08.009131 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.011054 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.016335 27729 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 15. Ops: []
I20260709 12:20:08.016805 27733 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:08.020987 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:08.021658 27733 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:20:08.022735 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:08.023232 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78291.3us
I20260709 12:20:08.023460 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.023715 28330 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.024055 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.024977 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.025597 27733 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:20:08.025915   456 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:08.026726   472 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:08.026839   402 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:07.959762   449 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.028069 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:08.030197 27731 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:08.029228 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.031546 27733 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:08.035059 27734 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:08.033767 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:08.044390 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.044384 27586 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:08.027141   472 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:08.047525 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.047217 32755 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:08.055752   462 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:08.055850 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:08.059264 28330 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:08.060552 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:08.043498 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:08.056592   454 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.079293   449 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.033046 32732 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:08.041260 32644 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
W20260709 12:20:08.081441 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:08.081646 32644 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:08.082130 32644 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:08.083490 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:08.084312 28334 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:08.086843   472 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.088655   472 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:08.099771 28333 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:08.100732 27585 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:08.101531 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:08.034686   429 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.102001 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:08.035151   311 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:08.102581 27580 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:08.069566 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.103821 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.068938 32489 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:08.105444   352 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:08.105827 32489 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.071791   453 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:08.106312   463 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:08.106828   453 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:08.106456   352 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:08.048255 32625 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:08.072224 32755 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:08.107674 32625 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.101655 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:08.108402 32625 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:08.108570 27585 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:08.072413   452 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:08.105882   464 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:08.102262   429 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.109390   452 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.102240 27583 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:20:08.109813   452 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.106393 32489 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.105399 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:08.079471   454 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.116828   454 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.107743   453 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.123183   461 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.084352 28331 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:08.029773   402 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:08.141139 28334 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:20:08.142786 27580 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:08.033982   440 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:08.181479 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 99413us
W20260709 12:20:08.182173 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76847.4us
I20260709 12:20:08.187795 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.189611   463 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:08.153427 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:08.183663 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:08.184113 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 136685us
I20260709 12:20:08.199497 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.176721   352 consensus_queue.cc:1243] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:20:08.204746 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.206089 27664 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:20:08.206987 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:08.208148 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.206427 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.209789 27661 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:08.210711 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:08.214048 27582 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:08.181497 32746 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.181746   392 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:08.196518 32703 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.186192   451 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.237706 27660 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: []
I20260709 12:20:08.195799   454 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:08.239348 27661 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:20:08.201907 32637 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:08.189951   453 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:20:08.083174 32644 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.215258   391 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:08.213475 32708 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:08.267158 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:08.209335 32755 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 12:20:08.213634 32404 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.115s
I20260709 12:20:08.339807   443 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:08.343918 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:08.230876   449 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:08.255782   480 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:20:08.256204   482 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:20:08.263801   396 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.256234   440 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.259847   413 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:08.366950 32744 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:08.369755   413 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:08.253806   454 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.355017 32625 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.355491   494 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:08.256376   402 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.263403   451 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.244331 32698 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.103s
I20260709 12:20:08.256716 32732 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.363473   453 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:08.251713 32746 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.255601   484 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:08.347924   391 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:08.351025 32644 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.238472   463 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:08.399529 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 191456us
W20260709 12:20:08.409420 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 210049us
I20260709 12:20:08.357618   464 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.330055   392 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.333019 32708 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.254415 32703 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.272185   461 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.432374 32744 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:08.434346 27732 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:08.371646   443 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:08.418700   454 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:08.441738 27656 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:08.370010   396 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.366339   449 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.447847 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.424727 32267 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:20:08.444448   484 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:08.464066 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:08.477991 27731 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:08.443353 32732 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.491243 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:08.447724   453 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.487207 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:08.482645   413 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.485414 32744 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.506601 32708 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.506742   311 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:08.508134   440 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:08.509027   445 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:08.509377   483 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:20:08.508306 32744 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:08.510705 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58123.6us
I20260709 12:20:08.512967 32637 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:08.514658 32746 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.518013   392 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.517525 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:08.522507 27731 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:08.451956   463 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.498368 32404 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:08.524953 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.545017 27663 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:08.459395 32489 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.526772 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:08.530014   488 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:20:08.482697   484 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.530417 27733 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:08.554423 27733 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:20:08.531428 32267 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.531618   453 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.532723   392 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:08.533063 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:08.569051 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:08.569489 28334 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:08.533672 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.534582 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.541178 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:08.482766   494 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:08.529856 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.549727 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.455216   464 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.575402 27737 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:08.568638 32267 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.576546 27737 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:08.576887 27737 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:08.577291   464 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.577663 32267 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.577587 27737 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.578372 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:08.578722 27737 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:08.503314   461 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:08.579265   461 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.579669 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:08.580380 27661 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:08.580401   499 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:20:08.581416   461 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.582377 32625 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.547374 32404 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.567440 27732 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:20:08.573266 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.573354 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.588889 27729 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:08.574184 27736 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:20:08.571303 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:08.579571   453 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.560524 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:08.565119 27733 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:08.570636 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:08.583142 32708 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:08.588188 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:08.587468   495 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:08.599182   496 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260709 12:20:08.600199 27733 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:20:08.600636 32489 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.599993   501 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:20:08.524235   391 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.602507 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.603256 27664 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:08.553277   311 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:20:08.607518 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:08.557299   445 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:20:08.591209 32404 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:08.618777 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:08.619711 28331 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:08.616125 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.621181 27729 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:08.570593   440 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:20:08.597155   441 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:20:08.607687   472 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.518s	user 0.000s	sys 0.012s
I20260709 12:20:08.594383 32708 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.607463 32267 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:08.716637 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.592587   464 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:20:08.721385 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:08.622682 32698 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:08.622524   400 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:08.597757   453 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.715866 32708 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:08.737859 27737 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:20:08.744328 27737 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:20:08.744568 27737 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:20:08.744807 27737 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:08.739979 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:08.734295 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 151553us
I20260709 12:20:08.746891 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
W20260709 12:20:08.741642 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148906us
I20260709 12:20:08.720474 32267 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:08.720144   396 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.755390 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.766491 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:08.769021 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.769843 27661 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:08.777424 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.765465 32267 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:08.788344 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2568) took 1064 ms. Trace:
I20260709 12:20:08.788887 27660 rpcz_store.cc:276] 0709 12:20:07.723657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:07.724029 (+   372us) service_pool.cc:224] Handling call
0709 12:20:07.724378 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:08.248000 (+523622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:08.787954 (+539954us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b54001e5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:08.788236 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1180033792}
W20260709 12:20:08.787487   452 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.677s	user 0.007s	sys 0.005s
I20260709 12:20:08.746428   400 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.746279   454 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.793421 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.794694   512 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:08.798092 27731 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:08.803613 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:08.793454   449 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.806011   514 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:08.805775   512 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.806533   449 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:08.806752   514 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:08.806999   449 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:08.817099   352 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 12:20:08.805955   504 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:08.830166   504 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:08.830394   454 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.831343 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.833170   400 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.833765   400 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:08.834163   396 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:08.834718   396 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.835829   461 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:08.836246   461 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:08.837018   461 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.838471   461 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:08.839089 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.839733 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:08.841104 27657 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:20:08.842185   453 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.842706   396 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.818678 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2132) took 1038 ms. Trace:
I20260709 12:20:08.844370 27662 rpcz_store.cc:276] 0709 12:20:07.780286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:07.787468 (+  7182us) service_pool.cc:224] Handling call
0709 12:20:07.787764 (+   296us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:07.897582 (+109818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:08.818392 (+920810us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:08.818612 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":200895360}
I20260709 12:20:08.820263   383 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:08.845391   383 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.845752   383 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:08.796726   452 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.821071   463 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:08.809904   502 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:20:08.807991   500 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:20:08.879932   445 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:08.808362   512 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.807336   514 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.781852   495 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:08.827724 27738 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:20:08.815294   520 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:08.847309 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.891556 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2622) took 1074 ms. Trace:
I20260709 12:20:08.868831 32267 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.831180   504 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.834233   400 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.827800   449 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:08.875735 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:08.907264 27660 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:08.876894   452 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.877506 32404 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.878712   515 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:08.889186 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:08.904059 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:08.904881   514 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.912035   495 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:08.913410   484 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.891917 27587 rpcz_store.cc:276] 0709 12:20:07.816892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:07.833301 (+ 16409us) service_pool.cc:224] Handling call
0709 12:20:07.833694 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:07.846043 (+ 12349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:07.846073 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:07.857374 (+ 11301us) log.cc:844] Serialized 26 byte log entry
0709 12:20:08.341444 (+484070us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b18002893c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:08.494293 (+152849us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:08.608507 (+114214us) log.cc:844] Serialized 14 byte log entry
0709 12:20:08.742132 (+133625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:08.742247 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:08.882785 (+140538us) raft_consensus.cc:1685] finished
0709 12:20:08.891363 (+  8578us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:08.891498 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16453,"prepare.queue_time_us":243993,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":140508,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":25378,"spinlock_wait_cycles":33233280,"thread_start_us":374967,"threads_started":3,"wal-append.queue_time_us":704439}
I20260709 12:20:08.926813 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:08.915691   411 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:08.899940   483 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:20:08.916179   515 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:08.956230 32267 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:08.957842 32404 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:08.962356   383 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:08.888044   518 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.075s
W20260709 12:20:08.973294 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 43715.5us
I20260709 12:20:08.978539 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:08.990350 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:08.933436   397 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:08.990937 27732 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:08.991209   397 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:08.991611   397 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:08.982486   508 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.176s	user 0.004s	sys 0.000s
I20260709 12:20:08.986056   383 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.989573 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:08.994627 27662 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:08.997011   400 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.998804 27732 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:08.999811 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.001902 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.002192   515 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.003696   515 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:09.004181 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.005311   383 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:09.005458 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:09.005877   383 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:09.007437   383 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.009696   323 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:09.011749   483 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:09.012308   483 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:09.013399   483 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.017479 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:09.018817   443 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.019375   443 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.020061   445 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:09.020759   445 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:20:09.021582   443 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.022075 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.022811   397 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.023396 27582 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:20:09.024737 27584 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:09.025184 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.026039 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.026652 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:09.027880 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.028609 27583 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:09.029764 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.034125 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.034736 27662 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.035014 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:09.035755 27731 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.038610 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.022167   443 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:09.015017   483 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:09.048712   525 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:20:09.043202   397 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:08.965832   411 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:09.052474 32703 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.053704 27662 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:09.060052 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.060745 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:09.053889   511 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
I20260709 12:20:09.061563 27584 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:20:09.063082 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:08.966480   517 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:09.065533 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.066386 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:09.083310 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:09.069919   395 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:09.088171   395 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:09.089074   395 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.100116   484 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:09.101544   411 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.102629   411 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:09.079667   517 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:09.104250   517 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.086321 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:09.123185 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.124974 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.127909 27582 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:09.129441 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.051466   443 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.131557 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.134289 27587 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:09.130543   520 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.160823 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.159492 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.132802   514 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:09.164124   520 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.156019 32708 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.171046 27585 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:09.135252   414 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:20:09.108438   449 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:09.189718 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:09.207820 27735 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:09.212030   449 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.212931 27583 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:09.216734   520 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.220779   443 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.224881   311 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:09.225478   311 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:09.235757 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 109072us
I20260709 12:20:09.236430 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2573) took 1031 ms. Trace:
I20260709 12:20:09.236929 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:09.207799 27582 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:09.236825 27657 rpcz_store.cc:276] 0709 12:20:08.204692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:08.206785 (+  2093us) service_pool.cc:224] Handling call
0709 12:20:09.202602 (+995817us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b5400291b54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:09.236358 (+ 33756us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456277376}
I20260709 12:20:09.247391 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.248735 27657 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:09.249902 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.250762 27656 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
W20260709 12:20:09.258638 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:09.264384 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:09.263520 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.266323 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.268904 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:09.226599   311 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.261497   443 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:09.276691   397 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:09.275808 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.286633 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:09.271109 32569 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:09.280344   495 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 12:20:09.289748 32569 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:09.290305 32569 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.295403   461 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:09.295898   461 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:09.296284   461 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:09.279633   429 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:09.278344 32404 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.299787 32404 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.302186 32404 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.303637   311 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.286674 27660 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:09.308028 27588 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:20:09.308871 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.310421 32489 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:09.310873 32489 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:09.311870 32489 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.313055 32489 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:09.313287 28334 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:09.315831   440 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:09.316293   440 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:09.317459 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:09.288302   505 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:09.322845   526 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:20:09.324659   495 consensus_queue.cc:1243] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:20:09.326388   449 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.326804 32708 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.326929   449 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.329636 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:09.330307 28330 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:09.330173 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43612.5us
I20260709 12:20:09.327589 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.334348 27584 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:20:09.337393 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:09.340906 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.342306 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:09.343348 28330 raft_consensus.cc:2368] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15.
I20260709 12:20:09.342336 27587 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:09.311161   451 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.345721 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:09.349006 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:09.345036 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:09.334028 32708 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:09.353281 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.353873 27587 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:09.335201 32756 raft_consensus.cc:2764] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:20:09.346933   443 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:09.348407   514 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:09.348131   462 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:09.334105   505 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.382671 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:09.384145 28330 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 13. (term mismatch)
I20260709 12:20:09.384595 28330 pending_rounds.cc:85] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 12
I20260709 12:20:09.384855 28330 pending_rounds.cc:107] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260709 12:20:09.385102 28330 raft_consensus.cc:2889] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:09.367841 28333 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:09.377439 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.388213 27702 leader_election.cc:400] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15.
I20260709 12:20:09.388569 27702 leader_election.cc:403] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:09.350056 32404 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:20:09.389540 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:09.393024 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.394217 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.396888 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.399740 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:09.401309 27587 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:09.402364 28335 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:20:09.403169 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.403939 27588 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:09.405198 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.405925 27588 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:09.408799 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:09.411698 28334 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:09.383782   462 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:09.414156 27587 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:09.383482   443 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:09.372848 32732 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.392917 32708 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:20:09.391513   514 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:09.423880 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2604) took 2089 ms. Trace:
I20260709 12:20:09.424341 27586 rpcz_store.cc:276] 0709 12:20:07.334685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:07.334840 (+   155us) service_pool.cc:224] Handling call
0709 12:20:07.335126 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:08.045491 (+710365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:09.423653 (+1378162us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54000898d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:09.423821 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1630557952}
I20260709 12:20:09.415625 32569 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.430172 32569 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.412662   429 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.415912   462 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:09.415582   515 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.439934   512 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:09.462548   514 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.491356 27587 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:09.510838 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 111176us
I20260709 12:20:09.511739 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.478086   449 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.441016   429 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.504453   414 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.517815 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.512764 32569 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.444675   515 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.517761   512 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:09.532621   548 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.506596   397 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 12:20:09.534340   549 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:20:09.538852 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 125382us
I20260709 12:20:09.530134 32569 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.545924 32569 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.547472   461 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.529140   414 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.550460   414 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.509927   443 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.533113   548 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.553965   461 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:09.554178   462 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.556124   548 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.559347 27584 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:09.528370 32404 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:09.558007   462 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:09.566496 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48809.9us
W20260709 12:20:09.564070   544 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:20:09.577987 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.579221 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.577992 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.582506 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.583135 27730 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:09.584003 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.584542 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:09.584633 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:09.585448 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:09.585897 27731 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:09.566159 32489 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.587507 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.591291 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.592481 27586 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:09.591573 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:09.593624 28333 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:09.563705   505 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.566510   546 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.589278 27733 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:09.594276 27731 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:09.567389   461 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.599210   461 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.567508   512 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:09.597491 27737 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:09.573351   551 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:09.603076   491 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:09.601734   547 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:09.605682   413 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:09.609145 32732 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:09.588096   416 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.613310   515 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.611001   512 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:09.616384 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:09.638528 27587 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
W20260709 12:20:09.610877   543 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.244s	user 0.002s	sys 0.000s
I20260709 12:20:09.600867   501 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:20:09.596352 32489 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.647966   551 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:09.657212   551 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.589116 32404 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:20:09.650116   512 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.623960   546 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.665419   546 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:09.652222   515 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.666405   515 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.666903 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2491) took 1076 ms. Trace:
I20260709 12:20:09.667537 27735 rpcz_store.cc:276] 0709 12:20:08.590664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:08.591726 (+  1062us) service_pool.cc:224] Handling call
0709 12:20:09.666656 (+1074930us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:09.666830 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1346763136}
I20260709 12:20:09.668649   515 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.669214   515 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.669754   515 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.648976   416 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:09.672562   507 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.879s	user 0.000s	sys 0.002s
I20260709 12:20:09.615284   505 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:20:09.674312 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2576) took 1468 ms. Trace:
I20260709 12:20:09.674659   441 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:09.675235   441 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:09.686368   441 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.674664 27659 rpcz_store.cc:276] 0709 12:20:08.205535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:08.206118 (+   583us) service_pool.cc:224] Handling call
0709 12:20:08.206462 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:08.461099 (+254637us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:08.461151 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:08.461589 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 12:20:08.617324 (+155735us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:08.682555 (+ 65231us) log.cc:844] Serialized 14 byte log entry
0709 12:20:08.935654 (+253099us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:08.935767 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:09.223253 (+287486us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b180034ea00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:09.639747 (+416494us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:09.674103 (+ 34356us) raft_consensus.cc:1685] finished
0709 12:20:09.674123 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:09.674253 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8918,"prepare.queue_time_us":53197,"prepare.run_cpu_time_us":9084,"prepare.run_wall_time_us":46124,"raft.queue_time_us":63728,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":10762,"spinlock_wait_cycles":52257024,"thread_start_us":575743,"threads_started":3,"wal-append.queue_time_us":265959}
I20260709 12:20:09.603909   456 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:20:09.692723 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:09.693099 27735 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:09.693774   501 consensus_queue.cc:1243] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:20:09.694041 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.652134   494 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.695313   494 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.641381   547 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:09.694734 27735 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:09.717730 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.720162 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.722386 27659 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:20:09.723848 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115611us
I20260709 12:20:09.729307 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.724792 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.734089 27588 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:09.734452 27732 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.730118 27663 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.735971 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:09.729353 28333 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:09.736954 27581 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.738459 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:09.739321 27587 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:20:09.696647   472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.014s	sys 0.015s
I20260709 12:20:09.747452 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2205) took 1197 ms. Trace:
I20260709 12:20:09.745962 27659 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:09.747416 27735 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:20:09.747872 27738 rpcz_store.cc:276] 0709 12:20:08.550237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:08.551033 (+   796us) service_pool.cc:224] Handling call
0709 12:20:08.551348 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:08.890928 (+339580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:09.747162 (+856234us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:09.747388 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":634999680}
I20260709 12:20:09.756655 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:09.679275   443 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.765053   492 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:09.765141   443 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.765698   492 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:09.766146   492 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:09.770414 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:09.773402   503 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:09.773926   503 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:09.774304   503 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:09.774972 27663 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:09.721407   456 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:09.754774   472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.014s	sys 0.015s
I20260709 12:20:09.762306   416 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.764575   505 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.760143 27582 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:20:09.721275   547 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:09.798832 27582 pending_rounds.cc:85] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 8
I20260709 12:20:09.799202 27582 pending_rounds.cc:107] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:20:09.799522 27582 raft_consensus.cc:2889] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:09.730057   518 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.784538   393 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:09.785104 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.801673 27664 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:09.785566   515 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:09.786875   484 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.788178   491 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:09.748270   501 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:09.789522 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.808956 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:09.789700   504 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.747380   441 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.735075   414 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.814565 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.814901 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.815550 27587 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:09.790081   469 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:09.790916 27729 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:20:09.816577 27729 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:09.791478 32489 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.792966 32708 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15.
I20260709 12:20:09.798321 27581 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:09.800379   505 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.810220 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:09.809808 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 212949us
I20260709 12:20:09.813032 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.816592 27664 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:09.820923 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.817408 27729 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.824365 27729 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:20:09.801029   445 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.778304   512 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.828984 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.813768   416 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:09.817456   501 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:09.832744 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61194.3us
I20260709 12:20:09.835470 27659 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:09.837368 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.837901 27734 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:20:09.839494 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.814342   414 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.876859   549 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.876971   383 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:09.877588   383 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:09.877514   549 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.881557   549 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.847684   503 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.838562   557 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:20:09.884783   503 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.837729 32489 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.853032   512 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.853713 27736 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:09.886555 32489 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.832551   445 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.889073 32489 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.890249   445 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.890475 27586 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:09.860096 27660 raft_consensus.cc:1217] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 8.6->[9.7-14.9]   Dedup: 11.8->[14.9-14.9]
I20260709 12:20:09.891295 27660 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [9.7-14.9]
I20260709 12:20:09.891438   494 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.892746 32616 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.893352 32616 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.869087   491 raft_consensus.cc:2764] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:20:09.870812   505 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.796506   456 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.871748 32698 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.810497   518 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.872974 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:09.873322 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:09.908089 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:09.816771   504 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.809955   484 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.874195   501 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.877755   414 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.933751 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.851646 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.900357 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.940438 27663 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:09.904177 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.940986 27657 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:09.899914   551 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.909168 32744 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:09.929878 32616 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:09.930392 27655 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:09.931252 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.951103 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.878851   383 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.934580 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.952643 27730 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:09.909335   512 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:09.953902 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.942126 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.937418 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:09.956696 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.923934 27729 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:20:09.957432 27729 pending_rounds.cc:85] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 5
I20260709 12:20:09.957361 27661 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:09.957756 27729 pending_rounds.cc:107] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:20:09.958015 27729 raft_consensus.cc:2889] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:09.944979 27660 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:20:09.945384 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 136490us
I20260709 12:20:09.945590 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.945683 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.946190 27587 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:20:09.909512   549 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:09.962080 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.949505 27735 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:20:09.909747 32698 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.954720 27730 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:09.965656 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.968333 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.970803 27729 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:09.969641 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.930877   541 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:09.969704 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:09.972755 27730 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:09.969285 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.975306 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:09.978060 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.948383 32744 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:09.979431 27738 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:09.982482 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.980404 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:09.908150   492 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.987994 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:09.985266 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.988598 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.988449 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:09.990983 27660 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:09.990540 27661 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:09.990381 27738 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
W20260709 12:20:09.973598   411 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:20:09.994985 27729 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:20:09.969393   503 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:09.946552   494 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:20:10.008694 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.009500 27661 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:10.012581 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:09.947507   491 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:20:10.017339   505 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:10.018195 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.019217   505 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:10.022339   503 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:10.023555   503 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.985765   449 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:09.954679   551 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:09.981287   520 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:20:09.996186   514 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:10.032359 27656 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:09.972965   452 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
W20260709 12:20:09.985546   396 consensus_queue.cc:1175] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 13 status { last_received { term: 13 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 13 index: 10 } }
I20260709 12:20:10.035175 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:10.047556 27660 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:10.037009 27657 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
W20260709 12:20:10.048403 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58793.4us
I20260709 12:20:09.961416   512 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:09.980654 32569 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:20:10.058929 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.043799 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2625) took 1010 ms. Trace:
I20260709 12:20:10.060189 27583 rpcz_store.cc:276] 0709 12:20:09.033510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.039602 (+  6092us) service_pool.cc:224] Handling call
0709 12:20:10.043567 (+1003965us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:10.043719 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":342567296}
I20260709 12:20:10.076627 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:09.989771   401 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:20:10.042182 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.049675 27730 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:20:10.049810 27657 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 12:20:09.980355   549 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.082924 27657 pending_rounds.cc:85] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 11
I20260709 12:20:10.083241 27657 pending_rounds.cc:107] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:20:10.083369 27738 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:09.993719   397 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:20:10.082945 27661 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:10.084534   549 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.084636 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:10.080022 27735 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:10.086247   549 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:10.086892 27588 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:20:10.086796   549 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.087435 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.088044 27735 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:10.088307 27588 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:10.088936   549 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:10.080801   496 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.083509 27657 raft_consensus.cc:2889] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:10.089761   496 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.045472 32616 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.090217   461 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:09.991683   416 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:10.090694 32616 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.078979 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:10.091960   496 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.091651   416 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.093025 32616 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:10.093315 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.093971 27730 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:10.094851 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.095477 27657 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:10.096818 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:10.105177 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95353.2us
I20260709 12:20:10.044292   514 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:10.112962   565 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:20:10.114300   564 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:20:10.035796   445 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:20:10.047084 32708 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:10.121989 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:10.129575 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.047350   449 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:10.127789 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.133638 27657 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:10.130862 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:10.139813 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.136567 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:10.118237   514 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:10.128247 32616 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:10.131292   524 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.127892   397 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:10.166847 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.162647 27660 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:10.047212 32744 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.142233   445 consensus_queue.cc:1243] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:20:10.167752 27735 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:20:10.139729   568 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:10.193019 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.174832 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.156215 32616 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:10.142365 32708 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.143935   557 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:20:10.168617   416 raft_consensus.cc:2764] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 12:20:10.229589 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54850.5us
I20260709 12:20:10.230453 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.168521   518 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:10.232601 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:10.252418 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.353821 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.354812 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.355455 27730 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:10.164526   429 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:10.255023   551 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.256210   566 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.267545   440 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.360291   566 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.361136   440 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:10.228891   484 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:10.233153 32744 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:10.240631   501 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:10.263849   492 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:10.193423   461 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.204711 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.173179   397 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.283087 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 682) took 1013 ms. Trace:
I20260709 12:20:10.371630 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:10.297724   413 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:10.358079   547 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.540s	user 0.008s	sys 0.000s
I20260709 12:20:10.390236 28331 rpcz_store.cc:276] 0709 12:20:09.269787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.270043 (+   256us) service_pool.cc:224] Handling call
0709 12:20:09.281783 (+ 11740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:09.336548 (+ 54765us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:09.336619 (+    71us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:09.336688 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:09.336737 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:10.282743 (+946006us) spinlock_profiling.cc:243] Waited 54.5 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:10.282808 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:10.282999 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125331200}
I20260709 12:20:10.170105   524 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.281515 32756 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:10.358167   514 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.169720   562 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:10.198877   568 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:10.198563   520 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:10.368956 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:10.255178   567 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:20:10.198238   504 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:10.277948   512 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:10.265517 32708 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.396708   484 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.227252 32616 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:10.401345 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:10.412443 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43573.5us
W20260709 12:20:10.419543 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 290041us
I20260709 12:20:10.278847 32698 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.402379   413 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.241559   449 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.491714 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:10.399845   561 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:10.483517 27585 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:10.494323 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:10.500265 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99038.6us
I20260709 12:20:10.504315 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.365036   461 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.416980   520 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:10.402712   492 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:20:10.413465   568 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:10.537662 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.499601 32708 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.405019 32627 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260709 12:20:10.418211   501 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:10.542115 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:10.551715 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:10.414383   451 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.128s
I20260709 12:20:10.582015 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.404844 32756 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.546275 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.507249   512 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:10.402349   547 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.542783 27584 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
W20260709 12:20:10.519168 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124181us
I20260709 12:20:10.536921   561 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:10.566354 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:10.596030 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:10.539506   429 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:10.541961   520 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:10.505326   484 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.598661 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.570060 32404 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:10.553346   584 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:10.606695 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:10.604600 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:10.586711   501 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:10.626470 27736 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
W20260709 12:20:10.710870 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 104249us
I20260709 12:20:10.604260   562 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:10.621738   512 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:10.603493 32489 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:10.579981 32708 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.588925   581 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:10.596418   515 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:10.597329   561 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:10.723639 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:10.789517 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:10.790370 28334 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:10.596647   441 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.041s
W20260709 12:20:10.769840 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 134416us
I20260709 12:20:10.621351   570 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:10.792524   570 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:10.793051   570 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:10.621518   584 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:10.795611   584 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:10.627733   501 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:10.604754   496 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.113s
I20260709 12:20:10.805342   496 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.829391   501 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.712926   469 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:10.837507 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:10.722086 32404 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.844305 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:10.768514   483 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:10.843839   397 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:10.843976   581 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.896409   581 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.854050 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:10.855203   583 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:20:10.901247 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:10.856380 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:10.902535 27735 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:20:10.904554 27735 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:20:10.858525   578 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.358s	user 0.000s	sys 0.001s
I20260709 12:20:10.917001 27735 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
W20260709 12:20:10.859622   585 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:20:10.873245 32627 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:10.861382   580 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.233s	user 0.000s	sys 0.000s
I20260709 12:20:10.776394 32489 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:10.854100 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:10.833523   515 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:10.710017   587 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.222s
I20260709 12:20:10.876267 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:10.890157   469 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:10.840395   512 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:10.805965   496 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.890889 28332 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:10.863579   514 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:10.860378   575 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.326s	user 0.000s	sys 0.001s
I20260709 12:20:10.874102   524 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:10.919559 32698 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:10.942119 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 66299.7us
I20260709 12:20:10.888199 28330 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:10.940963   413 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:10.914999   570 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.907887   483 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:10.922966 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:10.943527 32489 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:10.918051   584 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.941406   515 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:10.960218 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 413980us
I20260709 12:20:10.919502   598 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:10.943769   461 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.950456   469 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:10.950989   597 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.983601   572 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:10.983788   496 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.984539   602 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.040124   602 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.985230 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.042122 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.042555   602 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.043207 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:10.985386   570 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:10.977285   483 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:10.986778   551 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:10.989689   592 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:10.992496   501 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:10.993093   598 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:11.058539   524 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:11.058777   598 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:11.059121   524 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.061219   524 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.061699   524 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:10.908093 32404 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:10.994355   512 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:10.996721 32744 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.634s	user 0.010s	sys 0.000s
I20260709 12:20:11.000316   595 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:11.003737   600 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.020016   484 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.020097   472 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:11.021214   572 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.895290   562 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.026125 32708 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.004323 32698 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.029407   561 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.030278   469 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:10.944286 32627 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.032636   397 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:11.033610   597 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.049881 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.015359 32756 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.045722   547 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.062109   524 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.056331   551 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:11.078279 27627 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.501s (SENT) [suppressed 17 similar messages]
I20260709 12:20:11.097694 27729 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:20:11.101713 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119812us
W20260709 12:20:11.106670 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 140672us
I20260709 12:20:11.075798   494 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.074759   581 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.099534   472 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:11.125216 27662 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:11.119989 27663 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:11.100986 32746 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.152292 32746 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.153385   561 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.155817   494 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.156240 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.156821 28332 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:11.074137   592 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.159955 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2229) took 1036 ms. Trace:
I20260709 12:20:11.160364 27735 rpcz_store.cc:276] 0709 12:20:10.123701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.905510 (+781809us) service_pool.cc:224] Handling call
0709 12:20:10.906820 (+  1310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:10.917696 (+ 10876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.159683 (+241987us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b5400046da0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34ba02 00007f0e9a347dd6 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:11.159898 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6728832}
I20260709 12:20:11.155030 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.162613 27731 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:11.133733 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.135075   472 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.141229   547 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.143102 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2661) took 1089 ms. Trace:
I20260709 12:20:11.165496 27586 rpcz_store.cc:276] 0709 12:20:10.053512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.590843 (+537331us) service_pool.cc:224] Handling call
0709 12:20:10.603608 (+ 12765us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:10.604069 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:10.604101 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:10.621500 (+ 17399us) log.cc:844] Serialized 26 byte log entry
0709 12:20:10.971812 (+350312us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:20:11.108089 (+136277us) log.cc:844] Serialized 14 byte log entry
0709 12:20:11.114593 (+  6504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.114671 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:11.142859 (+ 28188us) raft_consensus.cc:1685] finished
0709 12:20:11.142882 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:11.143036 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":114170,"raft.queue_time_us":7222,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":25078,"thread_start_us":249542,"threads_started":2,"wal-append.queue_time_us":383306}
I20260709 12:20:11.144284 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.099938   595 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.145812 32404 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.150695 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.170663 27733 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:11.164657 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.170665 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.172130 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.175230 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:11.178985 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.179684 27663 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:11.180477 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.124339   551 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:11.185767 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:11.186813 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:11.182742 27733 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 12:20:11.182754 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.187517 27733 pending_rounds.cc:85] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:20:11.187799 27733 pending_rounds.cc:107] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:20:11.109395 32698 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.188023 27733 raft_consensus.cc:2889] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:20:11.173060   429 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.016s	sys 0.004s
I20260709 12:20:11.196393 27731 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:11.197708 28335 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:20:11.198247 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.201257 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:11.204219 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.100394   600 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.208690 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 826) took 1008 ms. Trace:
I20260709 12:20:11.168612   397 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:11.169636   548 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:11.174937   400 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:11.184244   472 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:11.185554   606 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:11.182871 32708 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:11.222761 32746 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.173188 32489 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.204109   429 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:11.238716 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.223515   484 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:11.223728 32698 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:11.214496 32616 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.707s	user 0.006s	sys 0.005s
I20260709 12:20:11.185050   469 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.209244 27663 rpcz_store.cc:276] 0709 12:20:10.200523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.207700 (+1007177us) service_pool.cc:224] Handling call
0709 12:20:11.208050 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:11.208272 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:11.208309 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:11.208375 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:11.208419 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.208491 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:11.208621 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:11.189203   581 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:11.197788   608 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.204494   524 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:11.221292   440 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.856s	user 0.012s	sys 0.000s
W20260709 12:20:11.240516 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53775.8us
I20260709 12:20:11.273471   443 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.252763   494 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:11.244020   520 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.619s	user 0.011s	sys 0.000s
I20260709 12:20:11.229957 32404 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:20:11.236123 32708 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.257068 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.240578   613 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.268385 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2231) took 1144 ms. Trace:
I20260709 12:20:11.229769   400 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.274497 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.254482   616 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:11.225819   397 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.251907   484 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:11.276306 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2511) took 1174 ms. Trace:
I20260709 12:20:11.220084   548 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:11.271064   592 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.253793   606 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:11.256268   572 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.279301 27586 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:11.183374   547 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.274777 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.290553 27583 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:11.291721 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.292464 27583 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:11.280946 27584 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:11.265190   584 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.283998 27730 rpcz_store.cc:276] 0709 12:20:10.102204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.540853 (+438649us) service_pool.cc:224] Handling call
0709 12:20:10.541274 (+   421us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:10.959270 (+417996us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:20:11.101798 (+142528us) log.cc:844] Serialized 14 byte log entry
0709 12:20:11.233293 (+131495us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:11.234107 (+   814us) log.cc:844] Serialized 52 byte log entry
0709 12:20:11.234297 (+   190us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:11.234820 (+   523us) log.cc:844] Serialized 14 byte log entry
0709 12:20:11.235979 (+  1159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.236075 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:11.275944 (+ 39869us) raft_consensus.cc:1685] finished
0709 12:20:11.276104 (+   160us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:11.276251 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5474,"prepare.queue_time_us":194094,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":35754,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":19832,"thread_start_us":659077,"threads_started":3,"wal-append.queue_time_us":155148}
I20260709 12:20:11.293490 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.294989 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.283643   613 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.296633 27584 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:11.297585 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.298147 27584 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:11.298405   613 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.277624   608 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.303486   615 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.304356 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2650) took 1486 ms. Trace:
I20260709 12:20:11.304826 27587 rpcz_store.cc:276] 0709 12:20:09.817904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.818075 (+   171us) service_pool.cc:224] Handling call
0709 12:20:09.818364 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:09.959691 (+141327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.304117 (+1344426us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54002bacd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:11.304297 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293329152}
I20260709 12:20:11.306607 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.264886 32616 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.314235 27583 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:11.280920 32708 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:11.314168 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88623.1us
I20260709 12:20:11.328377 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.285805   484 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:11.331990 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:11.333856 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 255591us
I20260709 12:20:11.336362 27586 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:11.337536 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.282027 32744 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.287748   548 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:11.304085   615 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.288056   597 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.350884 27584 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:11.278681   520 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.270030   581 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.342886 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.355418 28333 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:20:11.359791   501 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:11.280614 27735 rpcz_store.cc:276] 0709 12:20:10.124321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.169752 (+1045431us) service_pool.cc:224] Handling call
0709 12:20:11.170040 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:11.170231 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:20:11.190941 (+ 20710us) log.cc:844] Serialized 14 byte log entry
0709 12:20:11.268068 (+ 77127us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:11.268119 (+    51us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:20:11.268163 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.268226 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:11.268329 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15492,"prepare.queue_time_us":20539,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":4681,"raft.queue_time_us":17309,"spinlock_wait_cycles":6637312,"thread_start_us":61087,"threads_started":1}
I20260709 12:20:11.360268   501 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.360641   501 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:11.360488 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.361646 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.354347 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.362498 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.357620   581 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.340624   613 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.365032 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:11.367230   568 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.814s	user 0.006s	sys 0.006s
I20260709 12:20:11.350195 32744 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.335407 32627 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.363387 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.367719   568 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.370397   397 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.371140   568 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.374838   400 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:11.376127 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.382711 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:20:11.383188 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:11.384307   600 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.384462   608 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.385473 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.386320 27586 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:11.387691 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.398272 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.398880 27584 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:11.358609 32616 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.399672 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.399974 27735 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:11.400847 27731 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:11.407083 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.407302 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.410486 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.366161 32746 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:20:11.305012   606 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.402045   608 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:11.341197   429 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.365278   598 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.408799   595 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.460536 27731 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:11.396036   613 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.443634 32404 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.469413 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.449788 32627 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:11.450088   608 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.471423 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.472721 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.473424 27731 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:11.443235   572 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:20:11.480176 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.412963   483 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.465199 32489 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.476573   608 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:11.492462   615 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.493809   600 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:11.492959 32627 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:11.478912 32404 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.510131 32627 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:11.515350 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:11.495236   515 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.524361   547 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.525249   515 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.526051 27589 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:20:11.527674 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2664) took 1351 ms. Trace:
I20260709 12:20:11.528075 27589 rpcz_store.cc:276] 0709 12:20:10.176402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.950678 (+774276us) service_pool.cc:224] Handling call
0709 12:20:10.968956 (+ 18278us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:11.526517 (+557561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.527389 (+   872us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:11.527611 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1280556928}
I20260709 12:20:11.530154 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.542076   484 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.530882 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.560330 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.509370   600 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.569851 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.572677 27581 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:20:11.573047 27584 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:11.570562 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:11.556663   520 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.575625 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.576120 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.521039   551 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.585670 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.586592 27581 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:11.574515   451 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.590756 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.591832 27581 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:11.596204 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.596805 27659 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:20:11.597481 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.598634 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.599146 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.599786 27584 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:11.601922 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.604203   484 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.582341   501 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.607549   501 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.591478   452 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.608134 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:11.608904 28334 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:11.609861 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:11.614495 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.614692 27730 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:11.580176   572 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:11.621722 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.588346 32746 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.630010 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.631196 27584 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:11.636477 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.638137 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.638885 27657 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:11.640492 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.640972 27663 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 12:20:11.667102 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2591) took 1801 ms (client timeout 1837 ms). Trace:
I20260709 12:20:11.669332 28334 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
W20260709 12:20:11.667932 27656 rpcz_store.cc:269] 0709 12:20:09.839080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.839205 (+   125us) service_pool.cc:224] Handling call
0709 12:20:11.639906 (+1800701us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:11.640080 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432537856}
I20260709 12:20:11.671936 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.672048 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.629050   598 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.674082 27656 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:11.677796 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:11.681757   591 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.893s	user 0.006s	sys 0.006s
I20260709 12:20:11.681815 27657 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:11.683669 32404 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:11.684800   632 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:20:11.608182   452 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.690687 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.691231   452 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.691622 32644 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:11.692239 32644 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:11.692341 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.693336 32644 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.694694   472 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:20:11.695173   472 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.695573   472 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:20:11.698181 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.667039 32746 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.699397 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.674292 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:11.700922 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.701510 27657 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:11.689126 32627 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:11.702648 27659 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:11.702961 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.705106 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.600376   451 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.641688   514 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:11.671689   633 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:20:11.705793 27586 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:11.705889 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:11.667234   613 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.674154 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.714406 27655 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:11.668763   562 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.671197   606 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:11.708922 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.672840   520 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:20:11.708082   568 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:11.641605   572 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.719144 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.719611 27656 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:11.719868 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2500) took 2128 ms (client timeout 1501 ms). Trace:
W20260709 12:20:11.725478 27732 rpcz_store.cc:269] 0709 12:20:09.584216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.584334 (+   118us) service_pool.cc:224] Handling call
0709 12:20:11.712661 (+2128327us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:11.712766 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343234304}
I20260709 12:20:11.725952 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.726567 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2647) took 1917 ms. Trace:
I20260709 12:20:11.726840 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.727025 27582 rpcz_store.cc:276] 0709 12:20:09.809306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.809489 (+   183us) service_pool.cc:224] Handling call
0709 12:20:09.817331 (+  7842us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:11.271783 (+1454452us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:20:11.271837 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:11.271895 (+    58us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:11.271937 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.726334 (+454397us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54000be5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:11.726395 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:11.726507 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":494,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":3343976192}
I20260709 12:20:11.728298 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.718514 32404 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:11.729195 32746 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.689229   548 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.730161 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.731148 27589 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:20:11.731279   548 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.710862   601 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:11.732082   601 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.732019   472 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.732411 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:11.732673   601 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:11.733093 28332 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:11.728953 27732 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:11.716245   483 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.736663 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.745321 27732 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:11.725317 27655 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:11.740711 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.741067   456 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.742727 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.727411 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.744473   551 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.745591 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:11.746678 27735 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:11.747769 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.749171 27735 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:11.712071 32627 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.746801 28334 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:11.750553 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.753413 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.754822 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.755316 28332 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:11.754684   568 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:11.756434   548 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:11.756851 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.757056   568 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.758090 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.758225   584 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:11.758647   584 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:11.746814 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.755868   452 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.759224 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.777163 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.717105   484 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:20:11.768540 28334 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:11.715631   514 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.767264 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.789582 27735 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:11.769043 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.791770 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.784431 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.786355 27656 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:11.794296 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.794473 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:11.749912 27657 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:11.742959 32708 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.796339 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:11.798043 28330 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:11.720702   562 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.776644   600 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.801072 27733 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:11.718354   608 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:11.756863   548 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.760627   456 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:11.806221 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.807806 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:11.817023 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.817517 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.813292 27589 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:11.817618 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:11.814680 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.820961 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.801613   613 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:11.802440   551 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:11.819031 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:11.827399 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.808310   484 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:20:11.875442 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56475.6us
I20260709 12:20:11.875684 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.829816 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.832602 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.817327   456 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:11.786887   452 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.828054   581 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:11.816004   548 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:11.839644 32708 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.788544   584 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.842386 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:11.856355 28332 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:11.918164 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:11.844950   566 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.845508 32627 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:11.849192 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.850208 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2171) took 1134 ms. Trace:
I20260709 12:20:11.857546 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.921823   396 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:11.926653   615 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:11.926952 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.927634 27582 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:11.928598 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.929682 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.930366 27582 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:11.851053 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:11.852447 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.934584 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.935761 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:11.936231 28337 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:11.938541 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.853209   601 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.847991 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.855726 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.862089   451 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.851904 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:11.811767   501 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:11.815456   603 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:11.827858   520 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:11.863189 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:11.917260   547 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.888801 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:11.846747   514 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.895357 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.888903   613 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:11.862385 32746 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:20:11.896660 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.897059 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.888032   411 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:11.897817 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2652) took 1937 ms. Trace:
I20260709 12:20:11.898855 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 777) took 1109 ms. Trace:
I20260709 12:20:11.888290   456 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:11.902413   383 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:11.904577 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.976886 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:11.977986 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:11.896579 27583 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:11.890888 27664 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:11.911440 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2497) took 2295 ms. Trace:
I20260709 12:20:11.979516 27734 rpcz_store.cc:276] 0709 12:20:09.583265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.583393 (+   128us) service_pool.cc:224] Handling call
0709 12:20:09.595823 (+ 12430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:09.838243 (+242420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.878780 (+2040537us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b54000a4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:11.878977 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":556007168}
I20260709 12:20:11.980301 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2173) took 1262 ms. Trace:
I20260709 12:20:11.914130 27735 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:11.981819 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.983222 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.984038 27735 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:11.985073 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:11.915777 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:11.918920 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76614.4us
I20260709 12:20:11.987030 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:11.988583 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:11.919853 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:11.989727 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69973.9us
I20260709 12:20:11.862929   472 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.854344   572 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:11.927093   615 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:11.858556   579 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.321s	user 0.001s	sys 0.001s
I20260709 12:20:11.925246 27663 rpcz_store.cc:276] 0709 12:20:10.716197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.274917 (+558720us) service_pool.cc:224] Handling call
0709 12:20:11.640681 (+365764us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:11.654666 (+ 13985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.850030 (+195364us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b5400210a50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:11.850154 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840355584}
I20260709 12:20:11.912618   581 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:11.907894   551 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:12.015439 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:12.016125 27580 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:12.016345 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2608) took 1142 ms. Trace:
I20260709 12:20:12.016674 27663 rpcz_store.cc:276] 0709 12:20:10.874508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.015622 (+1141114us) service_pool.cc:224] Handling call
0709 12:20:12.015883 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.016055 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:12.016080 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:12.016118 (+    38us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:12.016153 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.016201 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.016297 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:12.017620 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:12.018080 27663 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:12.018600 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2609) took 1132 ms. Trace:
I20260709 12:20:11.816771   483 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.018879 27663 rpcz_store.cc:276] 0709 12:20:10.886247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.017563 (+1131316us) service_pool.cc:224] Handling call
0709 12:20:12.017810 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.018399 (+   589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.018554 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:12.019665 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:12.020238 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:12.020802 27663 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:20:11.931077 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.964015 27657 raft_consensus.cc:2410] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 12:20:11.964712 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 101595us
I20260709 12:20:11.921839   512 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:11.972530 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:11.972329 27585 rpcz_store.cc:276] 0709 12:20:09.960290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.967493 (+  7203us) service_pool.cc:224] Handling call
0709 12:20:11.897600 (+1930107us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54001248d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:11.897724 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":989714688}
I20260709 12:20:11.973300 28333 rpcz_store.cc:276] 0709 12:20:10.789395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.790239 (+   844us) service_pool.cc:224] Handling call
0709 12:20:10.864569 (+ 74330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:11.360243 (+495674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.898575 (+538332us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:11.898762 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1128155904}
I20260709 12:20:11.945011   429 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:11.998888 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:12.025815 27733 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:20:11.980679 27664 rpcz_store.cc:276] 0709 12:20:10.718360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.979570 (+1261210us) service_pool.cc:224] Handling call
0709 12:20:11.979843 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:11.980008 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:11.980043 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:11.980083 (+    40us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:11.980113 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:11.980162 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:11.980253 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:11.923281 32644 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:20:11.963816   520 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:11.964038   603 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:12.000716 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:11.965672   451 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:12.004796 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:12.035159 27587 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:12.007973 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:12.037359 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.005869 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:11.969709   501 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:12.013844 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.046587 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.047145 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:12.047997 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.048518 27735 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:12.049265 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.049758 27735 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:12.050448 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:12.029071 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.059580   608 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.033005   483 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:11.969455   547 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.061527 27584 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:12.062579   483 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.062891   547 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.063519   547 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.012624   587 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:12.064191 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2616) took 1074 ms. Trace:
I20260709 12:20:12.064399   587 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:12.064584 27657 rpcz_store.cc:276] 0709 12:20:10.989532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.063405 (+1073873us) service_pool.cc:224] Handling call
0709 12:20:12.063715 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.063882 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:12.063911 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:12.063953 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:12.063985 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.064047 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.064140 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:12.065477   587 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.012246   615 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:12.067754 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:12.066213 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:12.068774 27582 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:20:12.069183 27663 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:12.032666 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:12.072443 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:12.073102 27580 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:12.006582 32489 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:11.999933   492 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:12.041970 27737 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:12.004516   396 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:12.043080 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:12.051424 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:12.069836 27664 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:20:12.069895 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:12.071041 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:12.075524 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 226275us
I20260709 12:20:12.015334   581 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:12.014189   601 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:12.083658 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:12.084395 27737 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:12.080540 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:12.081508 27733 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:20:12.087702 27733 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:20:12.033600   603 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:12.089272 27582 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:12.089612 27582 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:12.041615   520 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:12.014247 27482 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 8 to 11, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 11 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:12.030015 28337 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:12.094671 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.096168 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:12.098080 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.082731 27735 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:12.100109 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:12.100448 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:12.101372 27586 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:12.069264   414 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.069106   549 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:12.100097 27587 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:12.041117   561 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:20:12.108037 27737 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:12.108937   561 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:12.109467   561 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:12.109905   561 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:12.111931   549 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:12.112504   549 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:12.113752   549 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:12.088379   579 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:20:12.116540 27734 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:12.033480   572 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:20:12.041615 32732 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:20:12.041491   555 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 12:20:12.090060   566 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:12.086370   483 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.133141   483 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.080760   396 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:12.128815   491 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:12.083948   492 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.121917   601 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:12.136067   566 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:20:12.136358   601 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:12.136524   566 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:12.135525   396 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:12.137279   601 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.137487   566 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.119638   472 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:12.139782   472 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.141788   472 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.145167   456 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.067158   501 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.099547   600 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:12.032783   512 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.090194   411 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:20:12.158746 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2596) took 2209 ms. Trace:
I20260709 12:20:12.135182   483 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.160784 27658 rpcz_store.cc:276] 0709 12:20:09.944931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.945341 (+   410us) service_pool.cc:224] Handling call
0709 12:20:12.153629 (+2208288us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:12.153790 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321085312}
I20260709 12:20:12.074100   429 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.165230 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:12.137282   492 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.169762 27731 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:12.153755   456 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.084250   613 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.184706 28331 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.185734 28331 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 12:20:12.181877 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:12.195889 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 283562us
W20260709 12:20:12.197008 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.172816   547 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:12.222082 27482 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 10 to 11, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:12.220816 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.186151   617 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.171293   548 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:12.246678 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2216) took 2311 ms (client timeout 2046 ms). Trace:
I20260709 12:20:12.222384   615 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:12.264750 27736 rpcz_store.cc:269] 0709 12:20:09.935532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:09.962380 (+ 26848us) service_pool.cc:224] Handling call
0709 12:20:12.246398 (+2284018us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:12.246589 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1460559616}
I20260709 12:20:12.266510 27736 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 12:20:12.243734   582 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.458s	user 0.003s	sys 0.007s
I20260709 12:20:12.246852 27738 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
W20260709 12:20:12.260442 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:12.241812 27662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2174) took 1522 ms (client timeout 1562 ms). Trace:
I20260709 12:20:12.240540 27658 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
W20260709 12:20:12.269773 27662 rpcz_store.cc:269] 0709 12:20:10.718951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.900849 (+181898us) service_pool.cc:224] Handling call
0709 12:20:12.241559 (+1340710us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54001e4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:12.241738 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512913792}
I20260709 12:20:12.229121 27482 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 12, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:12.271064 27736 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:20:12.271260 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 829) took 1093 ms. Trace:
I20260709 12:20:12.236743   396 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.269321   572 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.271672 27658 rpcz_store.cc:276] 0709 12:20:11.178004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.166999 (+988995us) service_pool.cc:224] Handling call
0709 12:20:12.240285 (+ 73286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.270362 (+ 30077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.271094 (+   732us) spinlock_profiling.cc:243] Waited 72.5 ms on lock 0x7b54001e3cd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:12.271208 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166835584}
W20260709 12:20:12.287688 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61510.9us
W20260709 12:20:12.223160   632 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:20:12.262466 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2226) took 2223 ms. Trace:
I20260709 12:20:12.261565 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.276561 27734 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:12.307916 27729 rpcz_store.cc:276] 0709 12:20:10.039546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.039751 (+   205us) service_pool.cc:224] Handling call
0709 12:20:10.057106 (+ 17355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:11.123131 (+1066025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.262226 (+1139095us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540044e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:12.262396 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2392007040}
I20260709 12:20:12.271710 27482 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 9 to 11, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:12.260828 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2613) took 1373 ms. Trace:
I20260709 12:20:12.293730 32744 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.159044   551 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.295200 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 824) took 2204 ms. Trace:
I20260709 12:20:12.297098   548 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:12.266850   512 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:12.224790   547 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:12.338218 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46880.5us
I20260709 12:20:12.261065   429 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.336530 27482 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 9 to 11, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:12.346072 27664 rpcz_store.cc:276] 0709 12:20:10.887541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.026878 (+1139337us) service_pool.cc:224] Handling call
0709 12:20:12.027172 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:12.081374 (+ 54202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.260565 (+179191us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b5400291b54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:12.260768 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97573120}
I20260709 12:20:12.324409   608 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:12.306306   383 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:12.349884   396 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:12.340663 27661 rpcz_store.cc:276] 0709 12:20:10.085745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.085858 (+   113us) service_pool.cc:224] Handling call
0709 12:20:10.086163 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:10.115589 (+ 29426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:10.115636 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:10.116202 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:20:10.492779 (+376577us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:10.492814 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:10.492872 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:12.243020 (+1750148us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:12.279076 (+ 36056us) raft_consensus.cc:1685] finished
0709 12:20:12.279111 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.289980 (+ 10869us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47980,"raft.queue_time_us":30360,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9811,"spinlock_wait_cycles":27246848,"thread_start_us":376160,"threads_started":1,"wal-append.queue_time_us":475320}
I20260709 12:20:12.288435   615 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:12.308899 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47418.9us
I20260709 12:20:12.354163 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:12.383176 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.368585 28332 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:12.367295 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 775) took 1567 ms. Trace:
I20260709 12:20:12.386272 28331 rpcz_store.cc:276] 0709 12:20:10.788592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.788774 (+   182us) service_pool.cc:224] Handling call
0709 12:20:12.355154 (+1566380us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54005d93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:12.355288 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2956971392}
I20260709 12:20:12.386924 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:12.354451 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 256475us
W20260709 12:20:12.358556 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 261109us
W20260709 12:20:12.357524 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 330313us
I20260709 12:20:12.355792   441 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.393011 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.393472 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.394928 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:12.395272 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:12.393366 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.376483 28335 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:12.374060 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:12.352429   608 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:12.399847 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:12.325287   572 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.379530 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:12.383093 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 345828us
I20260709 12:20:12.467206 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:12.397706 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.397714 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:12.290118   617 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.467408 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:12.437543 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44192.3us
I20260709 12:20:12.471388 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:12.371978   484 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:12.471939 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.442629 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:12.446867 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:12.474757 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52429.2us
I20260709 12:20:12.443315 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:12.487607 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 79539us
I20260709 12:20:12.502693 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:12.507107 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.512009 27656 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:20:12.513031 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:12.513796 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:12.504186 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.502602   608 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:12.516621 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.518543 27482 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 17, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 17 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:12.527918 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 508269us
I20260709 12:20:12.539690 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.541314 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.392117   383 raft_consensus.cc:2764] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:20:12.436784   396 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:20:12.582777 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111121us
I20260709 12:20:12.593473 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:12.584112 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:12.468657   555 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:12.607529 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2526) took 1475 ms. Trace:
I20260709 12:20:12.608903 28329 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:12.505625 32708 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.589017   600 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:12.519769   615 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:12.513827   592 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:12.436539   411 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.607986 27732 rpcz_store.cc:276] 0709 12:20:11.131992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.803467 (+671475us) service_pool.cc:224] Handling call
0709 12:20:11.804832 (+  1365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:11.805257 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:11.805299 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:11.815805 (+ 10506us) log.cc:844] Serialized 26 byte log entry
0709 12:20:12.174694 (+358889us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:12.174750 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.174839 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:12.607284 (+432445us) raft_consensus.cc:1685] finished
0709 12:20:12.607303 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.607469 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2443,"raft.queue_time_us":5710,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":2369,"thread_start_us":341451,"threads_started":1,"wal-append.queue_time_us":639533}
I20260709 12:20:12.537281   311 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.619125 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2622) took 1168 ms. Trace:
I20260709 12:20:12.634840 27659 rpcz_store.cc:276] 0709 12:20:11.451040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.238703 (+787663us) service_pool.cc:224] Handling call
0709 12:20:12.239106 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.239387 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:12.239768 (+   381us) log.cc:844] Serialized 14 byte log entry
0709 12:20:12.618072 (+378304us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:12.618216 (+   144us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:12.618351 (+   135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.618637 (+   286us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.618943 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1888,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"thread_start_us":377706,"threads_started":1}
I20260709 12:20:12.615144 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2538) took 1160 ms. Trace:
I20260709 12:20:12.436481   494 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.151s
I20260709 12:20:12.626681 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:12.642560 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:12.469188   441 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.502437   393 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.110s
I20260709 12:20:12.637775 27730 rpcz_store.cc:276] 0709 12:20:11.454890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.258819 (+803929us) service_pool.cc:224] Handling call
0709 12:20:12.259130 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.259359 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:20:12.259677 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 12:20:12.614252 (+354575us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:12.614525 (+   273us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:12.614740 (+   215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.614911 (+   171us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.615070 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27240,"prepare.queue_time_us":158,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":31886,"thread_start_us":326516,"threads_started":1}
I20260709 12:20:12.597399   472 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:12.639650 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 170816us
I20260709 12:20:12.520124   484 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.515333   520 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.519897 32644 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.097s
I20260709 12:20:12.721724 32644 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:12.723109   656 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:12.723634   656 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:12.724547   656 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.726794   653 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:12.727296   653 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:12.590838   514 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.084s
I20260709 12:20:12.727506   472 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.728219   653 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.728626 27482 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 10 to 11, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:12.730017   472 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:12.704852 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.613873   541 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:12.730858   541 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.733232   541 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:12.733335 28336 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:20:12.733971 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2172) took 2016 ms. Trace:
I20260709 12:20:12.734367 27660 rpcz_store.cc:276] 0709 12:20:10.717705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.931986 (+1214281us) service_pool.cc:224] Handling call
0709 12:20:11.932322 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:11.932707 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:11.932738 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:11.933194 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:20:12.380581 (+447387us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:12.380655 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.380783 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:12.733757 (+352974us) raft_consensus.cc:1685] finished
0709 12:20:12.733780 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.733912 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11985,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":17678,"thread_start_us":447023,"threads_started":1,"wal-append.queue_time_us":709119}
I20260709 12:20:12.737457 28336 raft_consensus.cc:606] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:12.737947 28336 raft_consensus.cc:614] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:20:12.747845   581 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.655s	user 0.010s	sys 0.000s
I20260709 12:20:12.748769   572 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.709161 28331 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:12.749627   311 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:12.750123   311 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:12.750916   311 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.752522   311 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:12.753322   615 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.754714 28331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:20:12.755481 28331 raft_consensus.cc:606] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:12.757735   520 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:12.709561   645 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:20:12.470357   547 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.714445   665 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:12.632004 32404 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:12.748490   581 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.731132 27482 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 11 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:12.706969 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:12.788554 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.644713 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:12.741528   561 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.003s
W20260709 12:20:12.780853 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 387775us
I20260709 12:20:12.791651 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.617215   657 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.640267   512 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:12.794567 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:12.624572 32756 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.158s
I20260709 12:20:12.623960 32708 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:12.787117 32489 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.706s	user 0.009s	sys 0.003s
I20260709 12:20:12.703245   382 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.137s
I20260709 12:20:12.799134 28331 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:12.784058 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.811239 27482 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 4 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 11 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:12.640328   411 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.805923   514 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:12.807785 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.785192 32404 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:12.825294 27482 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 7 to 8, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:12.790828   555 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
W20260709 12:20:12.804741   603 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.700s	user 0.005s	sys 0.006s
I20260709 12:20:12.817915 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.811671 32489 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:20:12.806983   594 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.012s	sys 0.020s
I20260709 12:20:12.838739   541 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:12.825619 27730 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:12.798633   512 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:12.815229 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:12.829155 27737 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 17 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 12:20:12.823300 27731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.003s	sys 0.026s
I20260709 12:20:12.802006   441 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.807263   383 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:20:12.799152 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 154526us
I20260709 12:20:12.788281   494 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:20:12.794700   561 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:12.845476 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 305350us
I20260709 12:20:12.839089 32404 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:12.831151   411 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:12.931293 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 123618us
I20260709 12:20:12.876502 27482 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 5 to 13, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 13 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:12.886602 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
W20260709 12:20:12.952919 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 368905us
I20260709 12:20:12.959280 27660 raft_consensus.cc:606] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:12.959564 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.959781 27660 raft_consensus.cc:614] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:20:12.958720 27731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.004s	sys 0.031s
I20260709 12:20:12.959553 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.953604 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 814) took 2790 ms. Trace:
W20260709 12:20:12.957738 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 142625us
W20260709 12:20:12.962646 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.958326 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:12.963382 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.962703 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2539) took 1264 ms. Trace:
I20260709 12:20:12.964825   672 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:12.928045   541 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:12.844154   603 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:12.989506   603 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.961338   512 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:12.968314 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.968936   551 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.969568 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2541) took 1271 ms. Trace:
I20260709 12:20:12.884817   673 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:12.996119 27729 rpcz_store.cc:276] 0709 12:20:11.698644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.309001 (+610357us) service_pool.cc:224] Handling call
0709 12:20:12.343001 (+ 34000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:12.343508 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:12.343553 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:12.344115 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 12:20:12.762995 (+418880us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:12.763052 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.763144 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:12.969345 (+206201us) raft_consensus.cc:1685] finished
0709 12:20:12.969368 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.969507 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1354,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":418485,"threads_started":1,"wal-append.queue_time_us":584411}
I20260709 12:20:12.997865 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:12.998062 27664 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:12.957674   581 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.999060 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2540) took 1300 ms. Trace:
I20260709 12:20:12.943042   484 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:12.997879   561 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.001494 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.003676   680 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:12.978505   441 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:13.004251   551 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:13.004565   441 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:13.005007   441 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.005977   541 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.006688 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:12.980412   667 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:20:13.007928 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:12.845834   613 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.024s	sys 0.015s
W20260709 12:20:13.008531   613 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.026s	sys 0.024s
I20260709 12:20:13.008750   680 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:12.984542   672 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.009186   613 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:13.009508   672 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:13.010820 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.011641   613 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:13.012048 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.012782   613 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:13.013172 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:13.013980   613 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:13.013962   441 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:12.985080 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 167239us
W20260709 12:20:12.883878   594 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.013s	sys 0.021s
I20260709 12:20:12.961023 28328 rpcz_store.cc:276] 0709 12:20:10.150544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.150715 (+   171us) service_pool.cc:224] Handling call
0709 12:20:10.151032 (+   317us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:10.151396 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:20:10.151679 (+   283us) log.cc:844] Serialized 14 byte log entry
0709 12:20:10.230723 (+ 79044us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:10.231179 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:20:10.346523 (+115344us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:10.346623 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:10.346722 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:11.857981 (+1511259us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:12.939373 (+1081392us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b2400318008. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:20:12.940359 (+   986us) raft_consensus.cc:1685] finished
0709 12:20:12.940379 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.940541 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24157,"prepare.queue_time_us":4499,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":781,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":21457,"spinlock_wait_cycles":30766720,"thread_start_us":78724,"threads_started":1,"wal-append.queue_time_us":252151}
I20260709 12:20:12.942770 32708 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:12.995805 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:12.999446 27736 rpcz_store.cc:276] 0709 12:20:11.698371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.272097 (+573726us) service_pool.cc:224] Handling call
0709 12:20:12.344393 (+ 72296us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:12.354696 (+ 10303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:12.354731 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:12.355488 (+   757us) log.cc:844] Serialized 52 byte log entry
0709 12:20:12.931756 (+576268us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:12.984366 (+ 52610us) log.cc:844] Serialized 14 byte log entry
0709 12:20:12.998767 (+ 14401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.998834 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:12.998873 (+    39us) raft_consensus.cc:1685] finished
0709 12:20:12.998888 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.999008 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15266,"prepare.queue_time_us":51552,"raft.queue_time_us":369,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":12384,"spinlock_wait_cycles":2434432,"thread_start_us":493821,"threads_started":1,"wal-append.queue_time_us":586790}
I20260709 12:20:13.003816   681 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:20:12.981272   663 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:20:13.042752 28330 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:13.014307 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:12.999094 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.044446 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.031251   678 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:20:12.875922   393 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:12.931558 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 314012us
I20260709 12:20:12.976169 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:13.064839 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88749.5us
I20260709 12:20:13.065922 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:12.976874 27731 rpcz_store.cc:276] 0709 12:20:11.698048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.270735 (+572687us) service_pool.cc:224] Handling call
0709 12:20:12.271041 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.271244 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:12.271602 (+   358us) log.cc:844] Serialized 14 byte log entry
0709 12:20:12.962233 (+690631us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:12.962285 (+    52us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:12.962393 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.962459 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:12.962649 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":116,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":145,"thread_start_us":688630,"threads_started":1,"wal-append.queue_time_us":615037}
I20260709 12:20:13.068441 27731 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:13.016175 27482 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 12 to 13, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 13 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:13.069443 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:13.070623 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92718.4us
I20260709 12:20:13.071128 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:13.071964 27585 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:20:13.003662   512 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:12.995136   608 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.077549   551 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.078573 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2619) took 2033 ms. Trace:
I20260709 12:20:13.022087   441 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.078936 27657 rpcz_store.cc:276] 0709 12:20:11.046231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.065570 (+1019339us) service_pool.cc:224] Handling call
0709 12:20:12.367781 (+302211us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.367933 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:12.367958 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:12.367996 (+    38us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:12.368023 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:12.368083 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.078414 (+710331us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b5400344c50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:13.078507 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":694264064}
I20260709 12:20:13.079581   608 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.025045   547 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.084210   681 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:13.084888   681 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.086937 27585 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:20:13.031109   561 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:20:13.088304 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:13.022107 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:13.029597 32627 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:13.088858   657 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:13.089434 32627 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:13.030552 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.090427 32627 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.091216   512 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.023844 32404 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.091784 32627 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.072582 32708 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:13.092585 32708 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:13.093039 32708 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.093354 32404 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:13.046700 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2544) took 1347 ms. Trace:
I20260709 12:20:13.094099 27729 rpcz_store.cc:276] 0709 12:20:11.699488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.045849 (+1346361us) service_pool.cc:224] Handling call
0709 12:20:13.046246 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.046420 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:13.046443 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.046486 (+    43us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:13.046518 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.046569 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.046650 (+    81us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.032042 27662 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:20:12.957125   383 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:13.053640 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:13.096333   561 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:13.055646 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.075544   581 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.097537 27583 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:13.097673   383 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:13.098011   451 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:13.098438   451 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:13.056357 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.099789 27585 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:13.057143 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:13.031746   588 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:13.070616 27482 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:13.069458 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2545) took 1369 ms. Trace:
I20260709 12:20:13.105815 27731 rpcz_store.cc:276] 0709 12:20:11.699751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.067745 (+1367994us) service_pool.cc:224] Handling call
0709 12:20:13.068173 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.069105 (+   932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.069367 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.072606 27586 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:13.076581 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.023793 28336 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:20:13.088416   484 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:12.872076   675 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:20:13.095819 27729 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:20:13.118572 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2546) took 1418 ms. Trace:
I20260709 12:20:13.119055 27729 rpcz_store.cc:276] 0709 12:20:11.700071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.095090 (+1395019us) service_pool.cc:224] Handling call
0709 12:20:13.095466 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.118209 (+ 22743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.118498 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.121353 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 833) took 1280 ms. Trace:
I20260709 12:20:13.121825 27729 rpcz_store.cc:276] 0709 12:20:11.841152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.120177 (+1279025us) service_pool.cc:224] Handling call
0709 12:20:13.120628 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.120906 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:13.120948 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.121006 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:13.121057 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.121136 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.121281 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.096637 27587 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:13.053988 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.100023 27588 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:13.091255 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60778.8us
I20260709 12:20:13.109268 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.126273 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:13.127493   678 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:13.128244 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.128295 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.126420 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.129355 27663 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:20:13.130249 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:13.126827 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2242) took 1273 ms. Trace:
I20260709 12:20:13.127056   568 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:13.137156   568 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.137606   568 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:13.138732   561 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:13.139217   561 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:13.139696   561 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.104883   451 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.140853   451 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:13.115599 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 832) took 1274 ms. Trace:
I20260709 12:20:13.143682 27731 rpcz_store.cc:276] 0709 12:20:11.840772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.114487 (+1273715us) service_pool.cc:224] Handling call
0709 12:20:13.114826 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.115104 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:13.115148 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.115210 (+    62us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:13.115261 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.115374 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.115525 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.145305 27731 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:13.145929 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2243) took 1292 ms. Trace:
I20260709 12:20:13.146275 27731 rpcz_store.cc:276] 0709 12:20:11.853739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.144783 (+1291044us) service_pool.cc:224] Handling call
0709 12:20:13.145115 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.145688 (+   573us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.145875 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.130995 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.061874   483 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:20:13.148449 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.130947 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.149675   483 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:20:13.150343   483 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:13.152125   601 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.153839   311 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:13.068014   393 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.107755   581 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:13.155333   603 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:13.156356   581 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.156653   603 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:13.152151   483 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.158466   603 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.160746   603 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:13.161855 27581 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:13.113646   681 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:13.163667   512 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:13.132723 27662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:13.132918 27657 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:13.164601 27662 raft_consensus.cc:606] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:13.133734 27656 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:20:13.164862 32756 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:13.134265   441 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:13.134719   655 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:13.165820 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.166404   655 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:13.166913 32756 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:13.134810   608 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:13.167394 32756 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:13.167333   674 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:13.168073 32756 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:13.169755   393 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.170233 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.131171 27729 rpcz_store.cc:276] 0709 12:20:11.853467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.123019 (+1269552us) service_pool.cc:224] Handling call
0709 12:20:13.126318 (+  3299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.126515 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:13.126542 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.126584 (+    42us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:13.126616 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.126672 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.126776 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:13.170720 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.172435 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.104924   588 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:13.175920   675 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:13.176532 32756 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.177261   562 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.178519   566 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:20:13.127183   677 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:20:13.165695   441 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.117353   518 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:13.183804 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:13.159031   581 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.180979 27587 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.198596 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.197935 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.214567 27662 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:13.215224 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.216511 27659 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:13.202108 27583 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:13.186882 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.218951 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:13.219726 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:13.214926 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.222576 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.209563 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.184867   608 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:13.236651 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:13.231498 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.232806 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.196154   674 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.015s
W20260709 12:20:13.239094 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.241823 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.242584 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2230) took 3118 ms. Trace:
I20260709 12:20:13.185264   678 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:13.208308   441 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:13.265733 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0760676139998395s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1562s for a response
I20260709 12:20:13.237030 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.267244 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.266907 27738 rpcz_store.cc:276] 0709 12:20:10.124026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:10.993326 (+869300us) service_pool.cc:224] Handling call
0709 12:20:12.246619 (+1253293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.269397 (+ 22778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.242353 (+972956us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400317250. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:13.242504 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2881393408}
I20260709 12:20:13.258176 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2543) took 1559 ms. Trace:
W20260709 12:20:13.268970 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59525.2us
I20260709 12:20:13.315703 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:13.316711 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.316936 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.267974 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:13.231307 27587 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:13.278421 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.320390 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:13.232380   452 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.281539   674 consensus_queue.cc:1243] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:20:13.245445 32708 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.210783   494 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20260709 12:20:13.167522   655 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.324621   617 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:13.324937   655 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:13.288266   518 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:13.327644   518 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.292387   588 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.233013   393 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:13.294891 27656 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:20:13.277976 27736 rpcz_store.cc:276] 0709 12:20:11.699187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.040454 (+1341267us) service_pool.cc:224] Handling call
0709 12:20:13.040871 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.041143 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:13.062632 (+ 21489us) log.cc:844] Serialized 14 byte log entry
0709 12:20:13.247557 (+184925us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.247622 (+    65us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:13.247681 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.247769 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.247897 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":114364,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":17642,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":10630,"thread_start_us":205350,"threads_started":2}
I20260709 12:20:13.341540 27656 pending_rounds.cc:85] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 13
I20260709 12:20:13.341961 27656 pending_rounds.cc:107] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:20:13.341775 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.188354   512 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 12:20:13.303175   584 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:13.304725 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.345047 27657 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.312510 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:13.312722 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90234.8us
I20260709 12:20:13.346364 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.321264 27586 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:13.320406 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:13.279453   501 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.321305 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.336786 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.338915 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:13.342341 27656 raft_consensus.cc:2889] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:13.342283 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.315912   678 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.342531 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:13.359715 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:13.360477 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.361639 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.339464   548 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:13.364274 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.362326 27662 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:13.386623   548 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:13.386775 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.387401 27662 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:13.387578 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.366446   594 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:13.388126 27738 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.363226   411 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:13.388352   684 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:13.388777   411 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:13.388818   684 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:13.389895   411 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.390013   684 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.367957   601 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:13.392172   656 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:13.388299   594 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:13.394627   594 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.395437 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2634) took 1001 ms. Trace:
I20260709 12:20:13.395926 27661 rpcz_store.cc:276] 0709 12:20:12.394249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.396629 (+  2380us) service_pool.cc:224] Handling call
0709 12:20:12.470768 (+ 74139us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:12.843372 (+372604us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:20:12.843442 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:12.876406 (+ 32964us) log.cc:844] Serialized 26 byte log entry
0709 12:20:13.245365 (+368959us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:13.245415 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.326321 (+ 80906us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54001e50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:13.326369 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:13.394987 (+ 68618us) raft_consensus.cc:1685] finished
0709 12:20:13.395019 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.395357 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16583,"raft.queue_time_us":143945,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":247296256,"thread_start_us":577402,"threads_started":2,"wal-append.queue_time_us":479927}
I20260709 12:20:13.398057   594 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.404431 28336 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:20:13.404121   441 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.406291   441 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.329771   547 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:20:13.408000 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:13.410075 28330 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
W20260709 12:20:13.418057 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.372325 27655 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:13.373203   588 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:13.376284   518 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.376645 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.435468 27737 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:13.436805 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.383014 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.384536 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 824) took 2205 ms. Trace:
I20260709 12:20:13.439627 27733 rpcz_store.cc:276] 0709 12:20:11.179347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.102317 (+922970us) service_pool.cc:224] Handling call
0709 12:20:12.102631 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:12.591499 (+488868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:12.591578 (+    79us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:12.613531 (+ 21953us) log.cc:844] Serialized 26 byte log entry
0709 12:20:13.383928 (+770397us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b18002efe40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1
0709 12:20:13.384095 (+   167us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:13.384135 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.384207 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:13.384255 (+    48us) raft_consensus.cc:1685] finished
0709 12:20:13.384266 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.384462 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18727,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":24035,"spinlock_wait_cycles":49212800,"thread_start_us":711247,"threads_started":2,"wal-append.queue_time_us":252700}
I20260709 12:20:13.441601 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.442305 27733 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
I20260709 12:20:13.430701 27657 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:13.443064 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.446718 27738 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:13.447074 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.448946 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.451200 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.454001 27660 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:13.454249 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.360843   681 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:13.455138 27733 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:13.457469 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:13.456624   681 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.464177   672 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.446367   684 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.360584   652 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 12:20:13.445611   393 raft_consensus.cc:2764] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 12:20:13.467562 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.434034   452 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:13.464028   681 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:13.488345 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:13.500078 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117122us
I20260709 12:20:13.451006 32489 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:20:13.466517   677 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:20:13.464020   548 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.434456   595 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:13.504333 27737 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:13.503607 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:13.526731 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44401.5us
I20260709 12:20:13.527213 27737 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:13.520313 27587 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:13.520970 27738 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:13.521764 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.525061 27664 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:20:13.500072   678 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.532860 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.487087   382 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:20:13.523788 32756 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.506774   615 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 12:20:13.531678 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.544961 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.532305   395 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:13.532092   518 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:13.518541   568 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.530089   617 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:13.550675 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.551630 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.552129 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2553) took 1651 ms. Trace:
I20260709 12:20:13.517650   383 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.519491   561 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.552503   494 raft_consensus.cc:2764] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:20:13.534035   441 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:13.562546 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.528571   654 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:20:13.530534   595 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.541509   608 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:13.655498   608 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.547195   678 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:13.658803   678 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.659236   678 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:13.577086   518 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.536770   594 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:13.550376   674 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.587754 27731 rpcz_store.cc:276] 0709 12:20:11.900723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.147118 (+1246395us) service_pool.cc:224] Handling call
0709 12:20:13.175474 (+ 28356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.176556 (+  1082us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:20:13.178568 (+  2012us) log.cc:844] Serialized 14 byte log entry
0709 12:20:13.551813 (+373245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.551861 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:13.551899 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.551956 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.552070 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3166,"prepare.queue_time_us":1359,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":37,"raft.queue_time_us":4007,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":369701,"threads_started":1}
I20260709 12:20:13.556119   395 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.531929   396 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:20:13.544435   653 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.553535   549 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:13.554131 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2542) took 1855 ms. Trace:
I20260709 12:20:13.626783   383 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:13.667999   395 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:13.668299   594 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:13.669512   594 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.670073 32627 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:13.670457 32627 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:13.671416 32627 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.672080   396 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:13.672472   396 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:13.673609 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 826) took 2476 ms. Trace:
I20260709 12:20:13.673440   396 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.674566   396 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.675032   555 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:13.675426   555 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:13.627821   694 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:20:13.628274   672 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.628749   561 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:13.629426 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51129.6us
I20260709 12:20:13.682346 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.684749 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.685879 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:13.686723 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:13.631656 28335 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:13.638003 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.632437 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.687726 27736 raft_consensus.cc:2368] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:20:13.687848 27581 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:13.642839 27737 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
W20260709 12:20:13.649461   693 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:20:13.648164 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.610737   541 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:20:13.694871   669 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.743s	user 0.005s	sys 0.001s
I20260709 12:20:13.676203   555 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.666795 27731 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:20:13.696480 27731 pending_rounds.cc:85] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 11
I20260709 12:20:13.696735 27731 pending_rounds.cc:107] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:20:13.696964 27731 raft_consensus.cc:2889] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:13.650574   452 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:13.628186 27659 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.673985 27735 rpcz_store.cc:276] 0709 12:20:11.199243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.106847 (+907604us) service_pool.cc:224] Handling call
0709 12:20:12.107141 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:12.167908 (+ 60767us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:12.168071 (+   163us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:12.168681 (+   610us) log.cc:844] Serialized 26 byte log entry
0709 12:20:12.520536 (+351855us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:12.520604 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.673211 (+1152607us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:13.673255 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:13.673304 (+    49us) raft_consensus.cc:1685] finished
0709 12:20:13.673318 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.673540 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5378,"raft.queue_time_us":8437,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11905,"spinlock_wait_cycles":126046464,"thread_start_us":349135,"threads_started":1,"wal-append.queue_time_us":565398}
I20260709 12:20:13.677645 27734 rpcz_store.cc:276] 0709 12:20:11.698920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.367443 (+668523us) service_pool.cc:224] Handling call
0709 12:20:12.367751 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.367933 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:20:12.377146 (+  9213us) log.cc:844] Serialized 14 byte log entry
0709 12:20:13.329398 (+952252us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b18003bdee0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:20:13.553224 (+223826us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:13.553285 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:13.553330 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.553939 (+   609us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:13.554076 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5370,"prepare.queue_time_us":11046,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":1940,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":15418,"spinlock_wait_cycles":25943808,"thread_start_us":221217,"threads_started":1,"wal-append.queue_time_us":1185986}
I20260709 12:20:13.701984   595 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.660550   441 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.666139 27589 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:20:13.702479 27735 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:20:13.701344 27734 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:13.703933   697 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:13.679160 27738 raft_consensus.cc:1217] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Deduplicated request from leader. Original: 15.11->[16.12-16.12]   Dedup: 16.12->[]
I20260709 12:20:13.705240   514 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:13.608743   617 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.705377   672 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:13.705271   698 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:13.746559 28331 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:20:13.709988 27584 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:13.710608 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 695) took 1353 ms. Trace:
I20260709 12:20:13.747751 27584 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:13.711831 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.748739 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:13.713939 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.716866 27659 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:13.719071 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:13.633147   568 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.705335 27733 raft_consensus.cc:606] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:13.756104 27657 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:20:13.672613 32627 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.719555 27581 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
W20260709 12:20:13.724735 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:13.729521 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48329.5us
I20260709 12:20:13.732589   483 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:13.733632 27732 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
W20260709 12:20:13.737622 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:13.760212 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.748716 27584 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.738206   441 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.743232 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:13.709143   518 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.749938 27738 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:20:13.751389 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.754647 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.785800 27587 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:13.785869 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.756356 27733 raft_consensus.cc:614] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:13.756553 27657 pending_rounds.cc:85] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:20:13.762763 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:13.786800 27657 pending_rounds.cc:107] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:20:13.786563 27738 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:13.787216 27657 raft_consensus.cc:2889] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:13.787154 27661 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:13.747807 28329 rpcz_store.cc:276] 0709 12:20:12.357346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.357835 (+   489us) service_pool.cc:224] Handling call
0709 12:20:13.710399 (+1352564us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54005a2654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:13.710550 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":446673792}
I20260709 12:20:13.759222 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.782125 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:13.775820 27580 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:20:13.768757 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.762814 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.768203 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.742519   697 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.728163   603 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 12:20:13.742741   614 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:13.798321 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84445.8us
I20260709 12:20:13.788393 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.786520 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:13.757694 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.792376 28331 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:13.792320 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.800149 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.802250 27552 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15.
I20260709 12:20:13.802845 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.803812 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:13.768165 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:13.790813 27581 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:13.805675 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.756101   672 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.796015 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.739393   395 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.760188   520 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:13.810070 28337 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:20:13.812032 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.813776 27661 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:13.816335 28334 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:13.820225 27732 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:13.827845 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.785132   594 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.756163   698 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.833462 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.824895 27738 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:13.830964 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:13.867218   484 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:13.867964   568 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:13.868911   652 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:13.869448   652 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.870967   383 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:13.871577   383 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:13.871742   652 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:13.872063   698 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:13.872810 32489 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:13.872570   383 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.873872 32489 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:13.874756   520 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:13.874878   501 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:13.875370   501 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:13.875761   520 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.876219   501 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.877560   501 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.878253   617 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.787288   615 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
W20260709 12:20:13.880585 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68627.3us
I20260709 12:20:13.880731   484 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:13.881346 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:13.795361   595 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 12:20:13.881692   484 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.882969   609 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.883625   311 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:13.884047   311 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:13.884404   311 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:13.884704   709 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
I20260709 12:20:13.812489   512 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 12:20:13.882509 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.890849 27735 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:13.891428 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57593us
I20260709 12:20:13.796440   697 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:13.894045 28336 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:13.894279 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:13.895202 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61814.1us
I20260709 12:20:13.898530 27586 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:13.899411 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.781339   483 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.905225 27734 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:20:13.798097   518 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.808525   672 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.813571   601 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:13.819211   655 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:13.820664   608 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:13.818082   441 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.910614 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.907066   494 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:20:13.901304   584 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:20:13.904232   566 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:13.826467   491 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:13.909492 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:13.889755   603 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:13.893821   548 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:13.887045   678 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.835551   681 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:13.933207 27582 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.942425 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:13.925159   588 raft_consensus.cc:993] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:13.953029 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2561) took 1106 ms. Trace:
I20260709 12:20:13.917516   547 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:13.948683 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:13.950246 27627 leader_election.cc:400] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:20:13.808651   614 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:13.953428 27737 rpcz_store.cc:276] 0709 12:20:12.847014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.528465 (+681451us) service_pool.cc:224] Handling call
0709 12:20:13.528754 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.688854 (+160100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:13.952788 (+263934us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:13.952970 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261900544}
I20260709 12:20:13.962563 27737 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:20:13.915812   393 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:20:13.956517 27627 leader_election.cc:403] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:20:13.917893   652 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:20:13.967382 27736 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:13.945762   584 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:13.968089 27660 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:20:13.969911 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:13.924937   656 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:20:13.959260   414 raft_consensus.cc:1081] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:13.979286 28328 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:20:13.979694 27737 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:13.960938   441 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:13.947059   494 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:20:13.966250   598 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:13.965696   609 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:20:13.971781   674 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:20:13.952915   449 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:20:13.950019   548 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.015033 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:13.944686   491 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.060190   491 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.008757 27588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:56296
I20260709 12:20:14.063759   395 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.003532 27581 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:14.068440 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.069609 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:14.070632 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.005941 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.072464 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.073029   681 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.074450 27580 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:20:13.971854   594 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17.
I20260709 12:20:14.014361 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.063220 27588 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:14.076274 27588 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:14.076967 27583 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:20:14.062566   548 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.086345 27580 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:20:14.088145 27580 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 9
I20260709 12:20:14.089486 27580 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:20:14.089941 27580 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:14.078109 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.093811 27583 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:13.992287 27737 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:14.095155 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:14.098311 27583 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:14.103024 27583 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:14.104420 27588 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.095654 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:14.092947 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:13.983008   547 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.127s
I20260709 12:20:14.112767 27588 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:14.114369 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:13.982297   595 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.113776   541 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:20:14.115656 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2685) took 2277 ms (client timeout 1840 ms). Trace:
I20260709 12:20:14.042685   615 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.116259 28335 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:14.056676   449 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:20:14.118448 27589 rpcz_store.cc:269] 0709 12:20:11.828986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:11.829117 (+   131us) service_pool.cc:224] Handling call
0709 12:20:14.105753 (+2276636us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:14.105900 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4107892992}
I20260709 12:20:14.139240 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:14.139652 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:14.119692 28335 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:14.004616   614 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.143517 27658 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:14.123955 27630 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.840s (SENT)
I20260709 12:20:14.125169 32708 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.131439   678 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:13.981289   653 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.135s
I20260709 12:20:14.106603   494 consensus_queue.cc:1243] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:20:14.085299   441 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:14.144163 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:14.144191 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:14.128515 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.145517 27589 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:14.162249 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:14.105619   681 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:20:14.164149 27589 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:14.118211   541 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:14.141871   547 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:13.981421   561 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.138s
I20260709 12:20:14.152506 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:14.154827 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:14.184229 27661 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:14.184801   594 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:14.181169   547 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:14.191531   547 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:14.119611   595 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.198359   595 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.181030   614 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:14.147006   311 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.209517   311 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.146023 28335 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:14.168119 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44198.7us
I20260709 12:20:14.211628 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.211699   441 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.213610 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.116883   520 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.170358 27589 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:14.178872 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:14.179992   717 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:20:14.180414 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.140877   615 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.215283 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.217103 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.234282 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:14.230813 27730 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:20:14.232537 27738 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:14.217891 27589 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:14.229357 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:14.225513 28335 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:14.249691 27589 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 8
I20260709 12:20:14.250535 27589 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:20:14.239668 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:14.251183 27589 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:14.234766 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:14.243976   311 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:20:14.219048   491 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:14.222255   483 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.254658 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2261) took 1052 ms. Trace:
I20260709 12:20:14.221966 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:14.257488 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.258662 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:14.221446   601 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:14.280289 27663 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:14.266876 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:14.288964 27658 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 12:20:14.300004 27737 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:20:14.301726 27664 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:20:14.301914 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2203) took 1085 ms. Trace:
I20260709 12:20:14.302547 27659 rpcz_store.cc:276] 0709 12:20:13.216739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.234329 (+ 17590us) service_pool.cc:224] Handling call
0709 12:20:14.301685 (+1067356us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b5400291154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:14.301836 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":809976064}
W20260709 12:20:14.304271 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89094.9us
I20260709 12:20:14.184069   653 consensus_queue.cc:1243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:20:14.329567 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:14.217415   672 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.307595 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:14.332899 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71195us
I20260709 12:20:14.175099   697 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.331094 27664 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:14.336263 27659 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:14.336361 27731 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:20:14.341495 27659 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:14.217926   514 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:14.219202   594 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:14.222146   614 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:14.339635 27583 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:14.244632   725 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.131s	user 0.003s	sys 0.000s
I20260709 12:20:14.342362 27659 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.406695 27659 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:14.389392 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:14.346932 27664 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:14.409821 27731 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:14.349385 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:14.410982 27663 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
W20260709 12:20:14.383451 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75949.1us
I20260709 12:20:14.384315 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:14.255049 27736 rpcz_store.cc:276] 0709 12:20:13.202334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.688300 (+485966us) service_pool.cc:224] Handling call
0709 12:20:13.688598 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.968904 (+280306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:14.254387 (+285483us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:14.254597 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":640475392}
I20260709 12:20:14.346015 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:14.417047 27581 raft_consensus.cc:2368] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:14.419400   672 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.324450   561 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:14.423305 27585 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:20:14.430773   411 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:14.431602   547 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.431807   548 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:14.432996   686 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.225462   698 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.435038   615 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.404806   603 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:20:14.436120   653 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.403378   675 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:14.443493 32756 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:20:14.443221 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 105362us
I20260709 12:20:14.558861 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.446173 28335 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:14.344512 32746 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.102s
I20260709 12:20:14.344066   518 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:14.343765   566 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.109s
I20260709 12:20:14.344838 32644 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:14.305971   483 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.142s
I20260709 12:20:14.215723   520 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.298887   608 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:14.288317   652 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.117s
W20260709 12:20:14.456634 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:14.463230 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:14.604779 27738 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:14.487617 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:14.607699 28334 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:14.492693   548 consensus_queue.cc:1243] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:20:14.494880   672 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:14.495837   547 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.496469   411 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:14.506213 27659 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:14.566308   657 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:14.596725 32489 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.513420 32627 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:20:14.540400 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:14.338284   541 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:14.543305   723 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.411s	user 0.004s	sys 0.000s
I20260709 12:20:14.537043 27587 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:20:14.545463   697 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.539414   656 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.511735   686 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.537724   698 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.578189 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:14.601132 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:14.610199 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:14.610381 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:14.610936 28334 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:20:14.679916 28332 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:14.680240 28334 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:14.680546 27582 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:14.680581 28334 raft_consensus.cc:614] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:20:14.613700 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126216us
I20260709 12:20:14.619453 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:14.507885 27731 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:14.683892 27731 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:14.572774   615 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:14.514228   561 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260709 12:20:14.572949   603 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:14.511873 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:14.615011 32404 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:20:14.603219   518 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:14.546720   653 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.662746 27588 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:20:14.618845   614 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:14.612287 32756 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:20:14.603240 32746 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:14.696341 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:14.698104 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:14.677973   652 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:14.663450   549 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:14.679490   547 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:14.720014   547 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.717082   549 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:14.721314 32627 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:14.721606   549 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.722224 32627 consensus_queue.cc:1243] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:20:14.722510   547 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:14.722947   549 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:14.723472   311 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:14.704512   603 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.725862 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:20:14.623068   672 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.720353   548 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:20:14.719182   728 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:20:14.702073 27734 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:14.728248 27580 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 12:20:14.690069   609 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:20:14.731566 27659 raft_consensus.cc:1217] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 8.6->[9.7-16.10]   Dedup: 11.8->[14.9-16.10]
I20260709 12:20:14.728331 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:14.746556 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:14.747735 27582 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:20:14.748279 27582 pending_rounds.cc:85] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 11
I20260709 12:20:14.748687 27582 pending_rounds.cc:107] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:20:14.748983 27582 raft_consensus.cc:2889] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:14.659122 32489 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.657331   657 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.723120   547 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:14.745352 27659 pending_rounds.cc:85] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 8
I20260709 12:20:14.753399 27659 pending_rounds.cc:107] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:20:14.753727 27659 raft_consensus.cc:2889] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:14.701213   681 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:20:14.690593   656 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.725608   655 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:14.701714   615 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:14.753471   616 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:20:14.763448 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2660) took 2856 ms. Trace:
I20260709 12:20:14.772171 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:14.773101 28328 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:14.767992   520 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:14.770066 32489 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.776505 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:14.778745 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 376866us
I20260709 12:20:14.783896 27661 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:14.737378   652 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:14.780601 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 220210us
I20260709 12:20:14.761541   548 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:14.735402   411 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.778602   674 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:14.782469   568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.019s	sys 0.022s
I20260709 12:20:14.812752   657 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.754823   547 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:14.814667   568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.019s	sys 0.025s
I20260709 12:20:14.818327   548 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.822068 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:14.772421 27582 rpcz_store.cc:276] 0709 12:20:11.893530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.076101 (+182571us) service_pool.cc:224] Handling call
0709 12:20:12.089165 (+ 13064us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:12.846641 (+757476us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b540041a8a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c606f 00007f0e9a33ef60 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62
0709 12:20:13.904807 (+1058166us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b540041a8a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c5520 00007f0e9a33ef70 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62
0709 12:20:14.678447 (+773640us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b540041a8a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bd597 00007f0e9a33ef9e 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62
0709 12:20:14.749485 (+ 71038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:14.749720 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355923328}
I20260709 12:20:14.788694 32489 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.790612 27586 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:14.802490   520 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.802711   566 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:14.809188   603 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:14.823994   562 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.022s	sys 0.019s
I20260709 12:20:14.815743   441 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.817925   674 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.838115   484 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:14.837759   382 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.865403 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:14.868885 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:14.834440   449 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:20:14.843631   566 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:14.857378   562 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.023s	sys 0.019s
I20260709 12:20:14.870596 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:14.790025   518 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:14.866109 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 265091us
I20260709 12:20:14.841857 32489 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.873306   382 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:14.864470   484 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.886369 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:14.877240   731 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:20:14.868212   686 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.847195   520 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.883948   742 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:20:14.858189   441 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.885833   514 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:20:14.961833   514 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:20:14.962157   520 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.924363   657 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:14.963546   657 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:14.964375   657 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:14.964973   441 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.966727   742 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.932142   746 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
W20260709 12:20:14.932427 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:14.896883   484 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:14.969691   740 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
W20260709 12:20:14.970247   741 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:20:14.974754   441 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.976315 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:14.977294 27733 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:14.979753   737 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:20:14.985433 27733 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:14.986521 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:14.987824   738 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:20:14.987566   441 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:14.989400 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 849) took 1220 ms. Trace:
I20260709 12:20:14.992146 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:14.967442   742 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:14.995558 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 207187us
I20260709 12:20:14.876579   613 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:20:14.956290 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57909.6us
W20260709 12:20:14.889698   733 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:20:15.002842 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:14.989850 27656 rpcz_store.cc:276] 0709 12:20:13.769443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.794648 (+ 25205us) service_pool.cc:224] Handling call
0709 12:20:13.794858 (+   210us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:13.808920 (+ 14062us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:13.893185 (+ 84265us) log.cc:844] Serialized 14 byte log entry
0709 12:20:14.310945 (+417760us) spinlock_profiling.cc:243] Waited 9.74 ms on lock 0x7b18003a1560. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:20:14.711598 (+400653us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:14.712079 (+   481us) log.cc:844] Serialized 26 byte log entry
0709 12:20:14.712206 (+   127us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:14.712236 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:14.712304 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:14.989090 (+276786us) raft_consensus.cc:1685] finished
0709 12:20:14.989203 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:14.989338 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56973,"prepare.queue_time_us":84115,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":99741,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":10088,"spinlock_wait_cycles":22393216,"thread_start_us":395326,"threads_started":1,"wal-append.queue_time_us":931103}
W20260709 12:20:15.005704 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 315387us
I20260709 12:20:14.884799 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:14.956910 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86387.2us
I20260709 12:20:15.008371 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:14.989452   656 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:14.995524   483 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:14.998548   615 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.086174   547 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:14.896195 32404 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.190s
I20260709 12:20:15.079536   652 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:14.949028   566 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.089072   672 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.090106   751 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.078881   561 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.104660   753 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:20:15.083122   483 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:15.173758 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 336136us
I20260709 12:20:15.234344 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:15.175024 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 336716us
I20260709 12:20:15.188458 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2558) took 2669 ms. Trace:
I20260709 12:20:15.131364   561 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.235756 27729 rpcz_store.cc:276] 0709 12:20:12.519194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.185411 (+666217us) service_pool.cc:224] Handling call
0709 12:20:13.222203 (+ 36792us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:13.241641 (+ 19438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:13.241687 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:14.200716 (+959029us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:14.201292 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 12:20:14.508785 (+307493us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:14.576767 (+ 67982us) log.cc:844] Serialized 14 byte log entry
0709 12:20:14.577138 (+   371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:14.577217 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:15.187430 (+610213us) raft_consensus.cc:1685] finished
0709 12:20:15.187737 (+   307us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:15.188127 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10889,"prepare.queue_time_us":53607,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":67,"raft.queue_time_us":22712,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":24083,"spinlock_wait_cycles":5280512,"thread_start_us":306794,"threads_started":1,"wal-append.queue_time_us":476129}
I20260709 12:20:15.207224   382 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.240670   561 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.240839   562 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:15.241402   562 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.122526   441 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.208406 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.224596 32489 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.243214 27589 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:15.243263   562 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.245126   562 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:15.245855 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:15.246488 27589 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:15.129822   566 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:15.081871   613 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:15.260656   675 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:15.260882   613 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.113075   615 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:15.263907   615 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.262087   675 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:15.143384   734 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:20:15.145901   742 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.145699   747 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:15.112828   748 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:15.384207 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.132704   754 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:20:15.386305   754 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:15.386878   754 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.285459   615 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:15.358506 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 124401us
I20260709 12:20:15.396018   613 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.365023   764 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
W20260709 12:20:15.367167   678 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:20:15.367431   758 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:15.383729   748 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.123531   751 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.243832 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:15.541867 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 298385us
I20260709 12:20:15.542936 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:15.388007   656 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:15.388630   747 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:15.367233   763 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:15.390071   767 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.435482   760 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:15.417904   675 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:15.421325 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 332922us
I20260709 12:20:15.423817 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.551709 27729 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:15.551992   770 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.552721   770 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.555028 32756 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:15.561408   393 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:15.562983   562 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:15.563426   562 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.564632   562 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.565778   767 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.566699   562 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:15.569900   757 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:15.570500   757 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:15.472628 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:15.418782   769 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:15.436350   484 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:15.414116 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 378402us
W20260709 12:20:15.443738 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 558861us
W20260709 12:20:15.466490 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 369186us
W20260709 12:20:15.431412   549 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.707s	user 0.014s	sys 0.015s
I20260709 12:20:15.419615   771 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.422652 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.415722 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2692) took 1403 ms. Trace:
I20260709 12:20:15.460191 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2700) took 1427 ms. Trace:
I20260709 12:20:15.469233 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2213) took 1229 ms. Trace:
I20260709 12:20:15.423529   777 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.471779 27660 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:15.486476 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2578) took 1007 ms. Trace:
I20260709 12:20:15.467705 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.484753 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2574) took 1255 ms. Trace:
W20260709 12:20:15.527979 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.529987 32746 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:15.532915   566 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:15.624500   566 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.625720   566 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.539610   518 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.591398 32627 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.627835   518 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.614775 27738 rpcz_store.cc:276] 0709 12:20:14.478540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.605780 (+127240us) service_pool.cc:224] Handling call
0709 12:20:14.606171 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:14.782913 (+176742us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:14.782963 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:14.783023 (+    60us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:14.783071 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.485912 (+702841us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540029fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:15.485987 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:15.486283 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405902720}
I20260709 12:20:15.546686   763 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.630223   763 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:15.631121 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:15.546185   747 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.617292 32746 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.632973 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:15.632663   747 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.633857 27661 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:15.633266 32746 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.634377   747 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:15.613453 27658 rpcz_store.cc:276] 0709 12:20:14.239665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.258447 (+ 18782us) service_pool.cc:224] Handling call
0709 12:20:14.288617 (+ 30170us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:14.317321 (+ 28704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.468755 (+1151434us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b540022b350. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:15.469046 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68565248}
I20260709 12:20:15.539799   758 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:15.637689 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:15.614634 27660 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:15.638654 27662 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:15.612944 27581 rpcz_store.cc:276] 0709 12:20:14.012286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.070409 (+ 58123us) service_pool.cc:224] Handling call
0709 12:20:15.409118 (+1338709us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54000b1154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:15.415468 (+  6350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":794856192}
I20260709 12:20:15.639577   757 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.640766   603 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:15.630254 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:15.641309   603 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:15.641825 27664 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:15.612174 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:15.613325 27588 rpcz_store.cc:276] 0709 12:20:14.032202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.115680 (+ 83478us) service_pool.cc:224] Handling call
0709 12:20:14.116338 (+   658us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:14.703796 (+587458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.459436 (+755640us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:15.459946 (+   510us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1235319040}
I20260709 12:20:15.624178 27733 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:20:15.623667   771 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.646324   781 raft_consensus.cc:993] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:15.627841 32732 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:15.648100 32732 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.649129   653 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.615417 27737 rpcz_store.cc:276] 0709 12:20:14.228857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.229048 (+   191us) service_pool.cc:224] Handling call
0709 12:20:15.484276 (+1255228us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b54001d47d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:15.484475 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117717376}
I20260709 12:20:15.650460 32732 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.630769   777 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.650827   768 raft_consensus.cc:1081] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:15.652709   655 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:15.653229   777 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.655746   674 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.637357   758 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.642539   603 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:15.592445 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 356334us
W20260709 12:20:15.592168 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46268.9us
W20260709 12:20:15.664307 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 129417us
I20260709 12:20:15.635069 32746 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:15.639065 27660 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.583738   770 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.572945   759 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:15.647125   771 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.578826   748 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.582823   751 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:15.714423   657 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.749s	user 0.016s	sys 0.006s
I20260709 12:20:15.644446   767 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.715688   566 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:15.715186   657 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.588893   555 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:15.653270   655 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:15.724404 27660 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:15.723393 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:15.732863   603 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:15.736465   763 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.666527   777 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:20:15.732962   615 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.740284 28337 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:15.740293 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:15.741468   615 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.743911   584 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:15.744405   584 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.744889   584 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:15.746706 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:15.747433 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.748724 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.749233 27732 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:20:15.749382 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:15.750051 28334 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:15.749500 27731 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:15.749876 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.755488 32627 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.757920 32627 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.758526 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:15.767371 27660 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:15.730612 27662 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:15.774206   655 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.779814   609 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.780895   609 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.781911   615 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:15.782730 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.782589 27737 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
W20260709 12:20:15.784404 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.782595   615 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.785212   512 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.785889   512 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.786432   615 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.786755   549 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:15.786339   771 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.788919   764 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:15.789291   771 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.789696   764 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.785996 28334 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:20:15.723176   657 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.793246   764 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.794471   655 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.799592   609 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.794628   771 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.801550 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2660) took 1656 ms. Trace:
I20260709 12:20:15.801748   483 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:15.801978   775 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:20:15.788422   512 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.810420 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.812770 27660 rpcz_store.cc:276] 0709 12:20:14.144939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.145074 (+   135us) service_pool.cc:224] Handling call
0709 12:20:14.145301 (+   227us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:15.471491 (+1326190us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b540028ba20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:20:15.767964 (+296473us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.801246 (+ 33282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":743408000}
I20260709 12:20:15.820935 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:15.821656 27658 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
W20260709 12:20:15.822432 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47860.8us
W20260709 12:20:15.824110   752 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.690s	user 0.004s	sys 0.000s
I20260709 12:20:15.825943 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2702) took 1749 ms. Trace:
I20260709 12:20:15.826355 27583 rpcz_store.cc:276] 0709 12:20:14.076400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.423574 (+347174us) service_pool.cc:224] Handling call
0709 12:20:14.424097 (+   523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:14.757743 (+333646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:14.757777 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:14.758243 (+   466us) log.cc:844] Serialized 26 byte log entry
0709 12:20:14.992318 (+234075us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:14.992352 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:14.992848 (+   496us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:15.823070 (+830222us) spinlock_profiling.cc:243] Waited 975 us on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:15.825405 (+  2335us) raft_consensus.cc:1685] finished
0709 12:20:15.825435 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:15.825768 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4228,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":2243456,"thread_start_us":566197,"threads_started":2,"wal-append.queue_time_us":362284}
I20260709 12:20:15.829460 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:15.839754 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.829281 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2705) took 1687 ms. Trace:
I20260709 12:20:15.728545   760 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.842864 27580 rpcz_store.cc:276] 0709 12:20:14.142292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.742856 (+600564us) service_pool.cc:224] Handling call
0709 12:20:14.743152 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:14.754016 (+ 10864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:14.754061 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:14.754116 (+    55us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:14.754161 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.828794 (+1074633us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:15.828858 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:15.829118 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24460160}
I20260709 12:20:15.835630   483 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:20:15.854534 27660 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:15.727344   769 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.863350 27664 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:15.859982 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:15.870868 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:15.872016 27664 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:15.873657   584 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.866384   484 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.878259 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.879258 27589 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:20:15.879608 32627 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:15.880391 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.880293 32627 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.881101 27589 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:15.882100   541 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:15.882794 32627 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:15.879772   773 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.346s	user 0.002s	sys 0.000s
I20260709 12:20:15.882114 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.884181 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.883247 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2699) took 1985 ms. Trace:
I20260709 12:20:15.885011 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.890365 27583 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:15.866394   518 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.887576 27587 rpcz_store.cc:276] 0709 12:20:13.898129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.898236 (+   107us) service_pool.cc:224] Handling call
0709 12:20:13.902078 (+  3842us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:14.660024 (+757946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.882821 (+1222797us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:15.883074 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1459442944}
I20260709 12:20:15.891386 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:15.904278 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.905086 27588 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:15.906994   759 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:15.907791   759 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.909648 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.909965   584 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.910356   759 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:15.910663   584 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.911270   584 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.913542   584 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.914094   484 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.914806   484 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.916271 27581 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:15.916990   484 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.911170 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:15.913378   674 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.920755   769 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:15.920559   674 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:15.923103   674 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.924052 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:15.924983 28331 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:15.926440 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:15.927875   763 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:15.928543   763 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:15.929055   763 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:15.935737 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:15.938465 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2707) took 1726 ms. Trace:
I20260709 12:20:15.938902 27582 rpcz_store.cc:276] 0709 12:20:14.212348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.824942 (+612594us) service_pool.cc:224] Handling call
0709 12:20:14.839472 (+ 14530us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:15.176754 (+337282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:15.176859 (+   105us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:15.178229 (+  1370us) log.cc:844] Serialized 26 byte log entry
0709 12:20:15.331983 (+153754us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:15.332039 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.332184 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:15.879028 (+546844us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:15.938054 (+ 59026us) raft_consensus.cc:1685] finished
0709 12:20:15.938074 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:15.938289 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48874,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":9574,"spinlock_wait_cycles":10823936,"thread_start_us":489436,"threads_started":2,"wal-append.queue_time_us":355046}
I20260709 12:20:15.940810 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.942085 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.942787 27582 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:15.942912 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.950624 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2292) took 1160 ms. Trace:
I20260709 12:20:15.952145 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:15.978448 27583 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:15.952754 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.906854   603 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.951140 27733 rpcz_store.cc:276] 0709 12:20:14.790119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.986254 (+196135us) service_pool.cc:224] Handling call
0709 12:20:15.947964 (+961710us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b54002e90d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:15.950458 (+  2494us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1449008512}
I20260709 12:20:15.983031 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:15.988384 27587 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:15.931219   763 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
W20260709 12:20:15.993157 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.001709 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.000955 27580 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:15.983584   555 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:15.978101   655 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 12:20:15.996821   382 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.024242   382 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.025791 28336 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:20:16.026773 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.027478 28333 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:16.001293 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.000041   657 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.020404 32708 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.023478 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:16.006541 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:16.045475 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57085.9us
W20260709 12:20:16.047739   787 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:20:16.048388 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.049273 28331 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:16.050190   762 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.051086   762 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.051190 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:16.052079   555 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.054034   762 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.054561 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.054888   762 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:16.055728 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.056485 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.051199 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:16.066823 28333 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:16.054208   555 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.027179   382 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.018898   751 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:16.058226 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:16.069422 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.003696   603 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:16.074424 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55682.2us
W20260709 12:20:16.078365 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74973.4us
I20260709 12:20:16.066437   762 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.080819 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.083495 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.084152 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.073276   748 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:16.086627 27662 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:16.065248 32627 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.071162   760 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.074633   763 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:16.090559 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:16.095628 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.088969 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.101441 32627 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.079457 32708 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.102464 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:16.085813   762 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.089605   555 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:16.119102   781 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.135509 28333 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:16.133805 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:16.135413 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.135689   781 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.119928   748 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:16.105758   763 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.079636   657 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.104635   603 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:16.147615 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61321.9us
I20260709 12:20:16.149070   452 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:20:16.149637   452 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:16.150193   452 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:20:16.152179 27663 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260709 12:20:16.152648 27663 pending_rounds.cc:85] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 5
I20260709 12:20:16.152947 27663 pending_rounds.cc:107] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260709 12:20:16.153154 32708 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.158582   657 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.160598 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:16.160894   708 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:16.162329   776 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:16.105559   751 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.162958   708 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:16.164997 32627 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.165372   751 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.165761 32627 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.166704 32627 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.160638 28329 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:16.153245 27663 raft_consensus.cc:2889] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:16.145569   763 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.179702   781 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:16.180647 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 176496us
I20260709 12:20:16.129334   555 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.181552   555 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:16.189193   801 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:20:16.145658 27661 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:16.192543 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2694) took 2414 ms. Trace:
I20260709 12:20:16.192976 27585 rpcz_store.cc:276] 0709 12:20:13.777544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.777691 (+   147us) service_pool.cc:224] Handling call
0709 12:20:13.797956 (+ 20265us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:14.468457 (+670501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:16.191915 (+1723458us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400088ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:16.192280 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1437545216}
I20260709 12:20:16.195240 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 805) took 1512 ms. Trace:
I20260709 12:20:16.127107   762 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.215523 28332 rpcz_store.cc:276] 0709 12:20:14.682991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.683185 (+   194us) service_pool.cc:224] Handling call
0709 12:20:14.683615 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:15.648044 (+964429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:15.648090 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:15.648606 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:20:15.833272 (+184666us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:15.833316 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:16.194611 (+361295us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:16.194676 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:16.194735 (+    59us) raft_consensus.cc:1685] finished
0709 12:20:16.194754 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:16.195009 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9079,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":27438,"spinlock_wait_cycles":2216533760,"thread_start_us":184159,"threads_started":1,"wal-append.queue_time_us":295018}
W20260709 12:20:16.248349 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160085us
I20260709 12:20:16.240178   603 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:20:16.258879 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 141461us
I20260709 12:20:16.253419 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:16.260170 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.261797 28335 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:16.263895   568 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.264066 32746 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.264509   568 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.265911 27589 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:20:16.264833 32746 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.270887 32746 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.273330 27661 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:16.276294 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:16.279780   767 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.280759   767 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.284029   767 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.284310 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2286) took 1948 ms. Trace:
I20260709 12:20:16.284723 27735 rpcz_store.cc:276] 0709 12:20:14.335632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.497605 (+161973us) service_pool.cc:224] Handling call
0709 12:20:15.487007 (+989402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:15.487866 (+   859us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:15.487907 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:15.488582 (+   675us) log.cc:844] Serialized 26 byte log entry
0709 12:20:15.535898 (+ 47316us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:20:15.535942 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:15.538783 (+  2841us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:15.569467 (+ 30684us) raft_consensus.cc:1685] finished
0709 12:20:15.569485 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:16.283931 (+714446us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54001d47d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:16.284145 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9313,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":211,"spinlock_wait_cycles":2274136832,"thread_start_us":46794,"threads_started":1,"wal-append.queue_time_us":78741}
W20260709 12:20:16.277498   747 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.643s	user 0.014s	sys 0.000s
I20260709 12:20:16.291955 27664 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:16.294818   653 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:16.295301   653 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:16.297837   771 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.298497   771 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.301416   771 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.302531 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.302394   763 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.296836   653 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.303589   698 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:16.291985 32744 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.305451 32744 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.308820 32744 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.314378   452 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.278734 32627 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.319059   785 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:16.319738   785 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:16.322250 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.322983 27663 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:16.323980 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.325541 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.326084 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:16.326622 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:16.327682 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.328455 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.329568   698 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:16.331168 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.332008 27662 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:20:16.332680   541 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.333176   541 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.334721 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.335391 27662 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:16.333925 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.266358 32708 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:16.336800 32708 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.337185 32708 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:16.340126 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73097.7us
W20260709 12:20:16.340391   491 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:20:16.304778   762 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.311385 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.330793   698 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.336527 27664 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:16.343953 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.305428   653 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:16.340181 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.328029 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.344841 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:16.335223   541 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.308928   555 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:16.361163 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 99017.5us
I20260709 12:20:16.361748 27663 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.362938 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.369071 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.378770 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.359658 32744 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:16.379253 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.379594 27662 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:16.380528   657 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:16.380921   657 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.381268   657 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.382925 28333 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:16.383080   555 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.380174 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.387940 28329 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:16.360428   382 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:20:16.398460 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.383761   555 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.393392 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.419942 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.420671 27661 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:16.363385   603 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.391960   771 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:16.394323   382 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:16.431691 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.438067   793 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:20:16.440089   777 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:16.440593   777 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.441061   777 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:16.432967 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46689.4us
I20260709 12:20:16.387984   763 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.443682 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:16.455505 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.456736 27661 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:20:16.419267   794 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.431263   771 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:16.428427   757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.035s	sys 0.028s
I20260709 12:20:16.470265 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.427667   748 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:16.475724 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47938.7us
I20260709 12:20:16.436406   382 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:16.481669 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:16.428090   555 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:16.477387   757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.035s	sys 0.028s
W20260709 12:20:16.491258 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.476413   771 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.500375   581 raft_consensus.cc:2764] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:20:16.415673   785 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.507959   785 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:16.511560   746 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.511719   615 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:16.512300   746 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.514905   746 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.515472   748 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.517341   754 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.518087   748 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.518622   754 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:16.467029   603 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.521346   603 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.523658   452 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.523856   555 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.526877 27737 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:16.527294 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.528425 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.529102 27732 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:16.530301 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.530748 32744 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:16.531134 27731 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:16.533886 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.534579 27738 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:16.536115 27732 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:16.485069   793 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
W20260709 12:20:16.541474 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.537919 32708 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.557379 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.563722   581 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.564574   581 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.574088 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.575435 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:16.576428 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.538394   555 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:20:16.512360   615 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:16.588135   581 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.588340 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:16.612710 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:16.539053   782 consensus_queue.cc:1175] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 18 status { last_received { term: 17 index: 12 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:20:16.596771   501 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:16.602329 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.603519 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.604971   793 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:16.606127 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.607167 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.558787   743 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:16.486116   794 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.612995 27729 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:16.613677 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.619292 27656 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:20:16.608798 27655 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:16.622520 27738 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:16.628512 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:16.629153 27738 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.629415 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:16.629503   793 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.630206 27734 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:16.617918   501 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:16.631743   793 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.632268 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.609612 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.635677 27661 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:16.636602 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.637881 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.636656 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.640398 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2708) took 2427 ms. Trace:
I20260709 12:20:16.640529 27661 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:20:16.640234   615 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.642279   568 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.640909 27586 rpcz_store.cc:276] 0709 12:20:14.212699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:14.837751 (+625052us) service_pool.cc:224] Handling call
0709 12:20:14.838012 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:14.838752 (+   740us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:14.854218 (+ 15466us) log.cc:844] Serialized 14 byte log entry
0709 12:20:16.332757 (+1478539us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b1800096e20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:20:16.639611 (+306854us) log.cc:844] Serialized 14 byte log entry
0709 12:20:16.639881 (+   270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:16.639949 (+    68us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:16.640027 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:16.640097 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:16.640226 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51862,"prepare.queue_time_us":29340,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":4927,"raft.queue_time_us":45201,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":10131840,"thread_start_us":154965,"threads_started":1}
I20260709 12:20:16.652904 32708 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.639014 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.645884 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.653889 28329 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:16.646015 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 809) took 1174 ms. Trace:
I20260709 12:20:16.654752 28337 rpcz_store.cc:276] 0709 12:20:15.471726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:15.472011 (+   285us) service_pool.cc:224] Handling call
0709 12:20:16.645795 (+1173784us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:16.645923 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":383809792}
I20260709 12:20:16.659714 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.660548 27737 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:16.661549 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.662636 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.623998 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:16.664220 27657 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:16.651934 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.638631   501 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.583166   760 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:16.682461 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60385.4us
I20260709 12:20:16.678025 27661 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:16.680218 27738 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:16.623173   743 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.689327 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:16.696803 27738 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.686923 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:16.695657 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.707449 27736 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:16.698491 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.710225 27734 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:16.711272 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.712199 27732 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:16.712616 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:16.724905 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.727556 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.664811   657 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.734737 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.710214   743 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:16.735700 27656 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:16.736774 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:16.742606 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.742692 27732 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
W20260709 12:20:16.744460 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 274541us
I20260709 12:20:16.750072 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:16.709437   760 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.751034 27737 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:16.754676 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.756588 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.762483 27738 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:16.760296 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.764225 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.764580 27656 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:16.764901 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:16.765795 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.742121 32756 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.748281   501 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.748925   555 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:16.769064 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:16.769508 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:16.773679 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.774390 27664 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:16.773293 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:16.778641 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66113.9us
I20260709 12:20:16.781530   547 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:16.784763   311 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.785398   311 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.786473 27657 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:16.788796 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:16.789661 27586 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:16.790892 27657 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:16.791874 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.792547 27662 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:16.793071   794 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.793131   681 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:16.793591 27662 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
I20260709 12:20:16.793466 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.798395 27584 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:16.799628 32756 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.800659   760 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.801725 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.801744   762 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:16.802915   657 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.805159   777 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.805688 27583 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:20:16.805996   793 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.779871 32746 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:16.808676 27585 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:20:16.862712 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.863549 27585 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:16.864409 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:16.864742 27584 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 12:20:16.865408   472 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 12:20:16.865576 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.866057   514 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.810297 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.866598   514 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.867286 27656 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:16.867807 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2690) took 3556 ms. Trace:
I20260709 12:20:16.811260 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.868146 27584 rpcz_store.cc:276] 0709 12:20:13.311426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.312020 (+   594us) service_pool.cc:224] Handling call
0709 12:20:13.312321 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:13.709893 (+397572us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54000d64a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:20:16.797642 (+3087749us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b58008bdfd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:20:16.865119 (+ 67477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:16.867755 (+  2636us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5028236288}
I20260709 12:20:16.868860 27663 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.869457 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:16.870007 27584 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:16.870620   603 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:20:16.873147 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48278.1us
I20260709 12:20:16.873311   382 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.874547 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:16.827172   681 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.875572   681 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:16.875977 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:16.876833 27729 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:16.878144 27729 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:16.878342 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:16.866329 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:16.815554 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.823350 27736 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:16.820240 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.791366   587 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.835412 27589 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:16.838613 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.883138 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.841815 27660 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:20:16.842639 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:16.834231   762 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.842916   771 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.827445   555 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:16.844663 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:16.845605 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.846753   777 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.891603 32644 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.892256 32644 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.893260   771 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.847477 27701 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:16.894167 27701 leader_election.cc:403] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:20:16.894467 27630 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 2.171s (SENT) [suppressed 5 similar messages]
I20260709 12:20:16.894711 32644 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:16.894907 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71531.7us
I20260709 12:20:16.895236   452 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.895727   452 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.896199 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:16.897814 27735 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:16.848218 27738 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 12:20:16.849260   814 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:20:16.850023 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.904495 27580 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:16.853124 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.855297 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.855669 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.858763 27655 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:16.859571   541 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:16.906831   541 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.907711   549 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:16.860296 32756 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.876770   794 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:16.881691 27664 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:16.913194 27663 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
W20260709 12:20:16.829692 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.883019 27737 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:16.861852 32746 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:16.887434   762 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:16.889420   616 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:16.889873 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:16.890365   555 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.903599   793 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.897871   452 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:16.917514 32746 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:16.928186 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.928619 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:16.930380 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:16.928893 27585 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:16.934587 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.935179 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:16.882628   587 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.935985 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:20:16.941035 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.948083 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:16.941952 27589 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:16.949613 27587 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:16.953896 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:16.962693 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.945724   777 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.963470 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.964102 27584 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:16.959373 28337 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:16.963680 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:16.966511 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:16.951020   794 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:16.953179   771 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:16.953092   501 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:16.970572   777 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:16.971629   501 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.972208 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:16.973875   777 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:16.974292   501 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.975462   548 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:16.975669   822 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:16.976138   822 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.976502   822 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:16.979171 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:16.975497   600 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:16.983891 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:16.984793 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 228308us
I20260709 12:20:16.985265   825 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:16.992220   825 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:16.993218   825 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.959100   562 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:16.995097 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.995739 27658 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:16.995643   562 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:16.995890 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.994822   514 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:16.997490 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.998865 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:16.999449 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:17.000648   587 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.004325 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.005205 27582 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:17.005306 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.005892 27702 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:17.006222 27702 leader_election.cc:403] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:17.006052 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.006559 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.007567 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:16.951588   681 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.007635 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:17.011647   818 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.218s	user 0.004s	sys 0.000s
I20260709 12:20:17.012077 27658 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:17.013760 32746 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.008733 27655 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:17.007228 27585 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:17.017246 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.018447 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.021133 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:17.025707 27582 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:17.029695 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.030630 27585 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:17.029989 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:17.032632 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.035454 27584 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:17.036110 27582 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.038187 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:17.041143 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77536.5us
W20260709 12:20:17.076656 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.106199 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.133934 27589 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:20:17.131877 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.119382 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:17.129354 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 159324us
I20260709 12:20:17.135241 27584 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.040493 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:20:17.136241 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95820.9us
W20260709 12:20:17.136816 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.137543 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2585) took 1568 ms. Trace:
W20260709 12:20:17.137811 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 151780us
I20260709 12:20:17.138497   587 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:17.138898   587 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:17.044046   549 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:17.139907   587 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:17.141469   501 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 12:20:17.159533 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.140458   777 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:17.166761 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:17.165014 32708 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:20:17.172325 32756 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:17.173070 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.173477   614 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:17.175635 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:17.177608 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:17.183830 32756 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:17.184445 32756 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.184916 32756 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:17.194599 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.200718 32746 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.201699   771 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:17.202250   771 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:17.202814   771 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:17.206387   762 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.207576 27657 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:17.208329   657 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:17.209142   657 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.212882   548 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:20:17.216308   681 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:17.217900 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.141619   613 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:17.219163   603 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:17.220065   657 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:17.220763 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.221382 27735 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:17.222148 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.224119 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:17.225078 27733 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:17.228264 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:17.229082   829 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:17.229530   829 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.229941   829 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.230161 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.230760 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:17.231673 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:17.231573   822 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.232292 27732 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:20:17.234337   822 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.235867 27737 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:17.237497 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60012.7us
W20260709 12:20:17.241511 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 347028us
I20260709 12:20:17.235178 27585 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:17.239248 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:17.265005 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:17.272957 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:17.275713 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.275887 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.284530 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:17.290614 28329 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:20:17.293294 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:17.294075 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:17.295599 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56436.3us
I20260709 12:20:17.294971 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.300455 27732 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 10.
I20260709 12:20:17.316367 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.317250 27735 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:17.176712   777 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:17.321511 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.321815 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.259361   746 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:17.335454 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.322669 27729 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:17.360483 27584 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:17.315374   653 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:17.370838 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.315860   793 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.378211 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.377216 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:17.328297   501 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.331928   777 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.413779 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:17.414605 27737 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:17.345839 32746 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:17.376304   828 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:17.379909 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:17.374650   743 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.624s	user 0.008s	sys 0.008s
I20260709 12:20:17.427033 27738 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:17.446473 27580 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
W20260709 12:20:17.460415   800 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.338s	user 0.000s	sys 0.004s
I20260709 12:20:17.461513 27586 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:17.464051 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2724) took 1584 ms. Trace:
W20260709 12:20:17.464855   812 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.673s	user 0.003s	sys 0.000s
I20260709 12:20:17.449862 27587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:20:17.469610 27587 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:17.470019 27587 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:17.473235   830 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.184s	user 0.000s	sys 0.000s
I20260709 12:20:17.138012 27730 rpcz_store.cc:276] 0709 12:20:15.469596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:15.477126 (+  7530us) service_pool.cc:224] Handling call
0709 12:20:15.477522 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:15.563931 (+ 86409us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:15.564496 (+   565us) log.cc:844] Serialized 14 byte log entry
0709 12:20:15.692495 (+127999us) log.cc:844] Serialized 14 byte log entry
0709 12:20:15.692691 (+   196us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:15.728043 (+ 35352us) log.cc:844] Serialized 26 byte log entry
0709 12:20:15.728180 (+   137us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:15.728214 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:16.644839 (+916625us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b180039c5e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:16.807838 (+162999us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x7b180039c5e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:17.037127 (+229289us) spinlock_profiling.cc:243] Waited 68.4 ms on lock 0x7b540017f2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:17.037180 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:17.037228 (+    48us) raft_consensus.cc:1685] finished
0709 12:20:17.037241 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:17.037386 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4720,"prepare.queue_time_us":1050,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":86,"raft.queue_time_us":2591,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":23355,"spinlock_wait_cycles":175441920,"thread_start_us":127193,"threads_started":1,"wal-append.queue_time_us":289233,"wal-append.run_cpu_time_us":14059,"wal-append.run_wall_time_us":1279940}
I20260709 12:20:17.412304   512 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 12:20:17.480580 27730 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 9. Current term is 10. Ops: []
W20260709 12:20:17.458127   827 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.604s	user 0.002s	sys 0.000s
W20260709 12:20:17.485632 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.464694 27588 rpcz_store.cc:276] 0709 12:20:15.879213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:15.905782 (+ 26569us) service_pool.cc:224] Handling call
0709 12:20:15.906190 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:16.020115 (+113925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:16.020181 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:16.021150 (+   969us) log.cc:844] Serialized 26 byte log entry
0709 12:20:16.070402 (+ 49252us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:16.397691 (+327289us) log.cc:844] Serialized 14 byte log entry
0709 12:20:16.397910 (+   219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:16.398005 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:17.459667 (+1061662us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:17.463617 (+  3950us) raft_consensus.cc:1685] finished
0709 12:20:17.463759 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:17.463962 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1400,"prepare.queue_time_us":30670,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":25212,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":7191,"spinlock_wait_cycles":12817152,"thread_start_us":476307,"threads_started":3,"wal-append.queue_time_us":100658}
I20260709 12:20:17.487946 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 718) took 1690 ms. Trace:
I20260709 12:20:17.488406 28334 rpcz_store.cc:276] 0709 12:20:15.797169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:15.797423 (+   254us) service_pool.cc:224] Handling call
0709 12:20:15.798074 (+   651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:15.903539 (+105465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:15.903577 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:15.903627 (+    50us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:15.903679 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:17.487685 (+1584006us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:17.487739 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:17.487871 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234203904}
I20260709 12:20:17.489372   794 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:20:17.487884 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 449745us
I20260709 12:20:17.493726 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.494916 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.488603 27586 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:17.363862   746 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:17.496313 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:17.507400 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.379102   653 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:17.515206 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.516541 27582 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:17.495242   581 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:17.522349 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:17.479053   512 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:17.520321 27737 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:17.428285   828 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.524721 27587 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.529768 27738 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:17.535006 27585 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:17.536849 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:17.538252 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:17.540409 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:17.542533 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:17.543905 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.545715 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.517728   754 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:17.524315   653 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.550604 27587 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:17.547745 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.566747 27586 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:17.535128   828 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:17.523905   614 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:17.523553   382 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:17.586288 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 242400us
I20260709 12:20:17.587415 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.429410   501 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.375746   829 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.546765   794 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:17.546586   822 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.514604   743 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.526485   512 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:17.559798   441 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:17.601282 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.550362   603 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:17.547355 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:17.551105 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.539453   764 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:17.591543   754 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:17.534776   776 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:17.605541 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:17.615839 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:17.608659 27736 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:17.547170   763 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:20:17.610534 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63258us
I20260709 12:20:17.621347 28337 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:20:17.603703   441 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:17.607985   603 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.631062 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.626061 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:17.638229 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:17.638974 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:17.635767 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:17.637184   472 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.642719   472 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.640373   568 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:20:17.604341   822 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.644917   472 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.646113   494 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:17.646250   822 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.646589   494 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:17.647100   494 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:17.648351   743 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.648936   746 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.614379   541 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:17.600811 32756 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.630133   794 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:17.628518   549 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:17.678009 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47056.4us
I20260709 12:20:17.636783   441 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:17.637822   603 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.681257 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.640165   615 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:17.708619   615 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:17.717335 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:17.625665   555 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:17.633183   763 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:17.708660 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.733202 27587 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.687734   608 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:20:17.665984   581 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:17.681135   512 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.651841 32744 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 12:20:17.679103   754 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:17.636796   613 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:20:17.677847   771 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:17.695314   837 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:20:17.706318   762 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:17.688697   829 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:17.694734   501 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:17.700668 32756 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:17.625775   776 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.797756 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:17.714751 32644 raft_consensus.cc:2764] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:20:17.799449 28332 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:17.720170   615 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.683014   549 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:17.807186   549 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:17.715663   653 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.725111 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:17.693236   587 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.000s
I20260709 12:20:17.736374   555 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:17.736989   828 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.737754   777 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.740903   311 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:17.819877 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 489322us
I20260709 12:20:17.746312 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.701457   494 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:17.753495 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 514248us
W20260709 12:20:17.760051 27581 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.038s	sys 0.060s
W20260709 12:20:17.821218 27581 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.038s	sys 0.060s
I20260709 12:20:17.731781 32746 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:17.831686 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.835039   828 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:17.837435   777 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:17.779629 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:17.840073 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60581.1us
I20260709 12:20:17.795415 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:17.762492   472 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:20:17.773418   613 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 16.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:20:17.757524   581 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.815421   382 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.724572   794 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:17.831544 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 106511us
I20260709 12:20:17.139154 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:17.859163 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.821496 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:17.861377 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:17.798811   776 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:17.762816   568 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:20:17.844236   501 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:17.855847 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:17.820509   512 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:17.844594   762 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:17.864650 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:17.868232 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:17.875090 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.879794   794 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.854609   581 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:17.844058   472 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:17.820919 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 499414us
I20260709 12:20:17.838836   555 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:17.908128 27580 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:17.909298 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44725.4us
I20260709 12:20:17.921691 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.909987 27585 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:17.913275 27587 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:17.819731   587 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.930250 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:17.918378 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:17.885453 32744 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:17.871877   547 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:17.907382   762 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:17.885236 32746 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:17.865096   382 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:17.942494 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:17.947913 28329 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:17.959280 27587 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:17.932147 32756 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:17.933040 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:17.965790 27585 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:17.935742 32744 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:17.999930 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:17.926455 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:18.006012 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 353730us
W20260709 12:20:18.014189 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87818.6us
I20260709 12:20:18.014816 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:18.018430 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:17.864475   763 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:18.023085 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 483956us
I20260709 12:20:17.954974   762 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:17.955413 32746 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:18.033094 28331 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:17.949483   501 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:18.018555 32744 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:18.034153   541 raft_consensus.cc:2764] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:20:18.031494   547 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:18.069460   603 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.070869 27582 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:18.044634 32746 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:18.037251 32644 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:18.049878 32756 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:18.022604   311 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:20:18.049213 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:18.071836   843 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 12:20:18.063738   568 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:18.078191 32644 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.086246   494 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.090171 32644 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:18.090523   494 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:18.090938   494 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:18.091362   494 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:18.091465   838 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 12:20:18.064031   452 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:20:18.099205   494 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.099680 32644 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.101439   494 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.102243 32644 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.103358 32644 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:18.087220 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:18.104571   494 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:18.104943   494 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.104998 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:18.105381   494 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:18.106022   541 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:18.106431   541 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:18.106791   541 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:18.121196 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:18.130983 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 264102us
W20260709 12:20:18.136189 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 196514us
I20260709 12:20:18.074782   771 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:18.140256   541 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.076932   547 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:18.082415   653 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.158368   653 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:18.159075   653 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.161854   653 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.060783   501 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.080797   844 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:18.146566 32644 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:18.150723   494 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.169019 32644 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.169945   501 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:18.170450   494 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:18.176185   847 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:20:18.144449 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:18.213295   549 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.221227 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.214884 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2734) took 1296 ms. Trace:
I20260709 12:20:18.143006   613 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:18.117564   776 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.101s
I20260709 12:20:18.187703   793 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:18.237262 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 92880.8us
I20260709 12:20:18.203842   514 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:18.241380 27581 rpcz_store.cc:276] 0709 12:20:16.848923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.228825 (+379902us) service_pool.cc:224] Handling call
0709 12:20:17.229099 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:17.229334 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:17.822073 (+592739us) log.cc:844] Serialized 14 byte log entry
0709 12:20:18.144859 (+322786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:18.144917 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:18.144963 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:18.145036 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:18.145159 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":304896,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":62761,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":8239,"thread_start_us":882352,"threads_started":2}
I20260709 12:20:18.244690 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:18.247596   568 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:18.254796 27731 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:20:18.222909   603 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.271736 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.271658   547 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.279951   501 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:18.280737   501 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.285037   859 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:18.261121 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:18.290122 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.290908 27655 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:18.263299 27587 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 7
I20260709 12:20:18.231827   600 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:18.291281 27655 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:18.166819   472 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.202874   441 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.307122 27655 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.298653 27587 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 16.8
I20260709 12:20:18.287549   616 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:18.331086   616 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:18.332206   616 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.313169 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.333730 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.333792 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:18.334628 27662 raft_consensus.cc:2368] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14.
I20260709 12:20:18.335139 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.335557 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.336655 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.337503 27664 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:20:18.337685 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.338344 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.338986 27664 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:18.339497   568 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:18.341355   568 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:20:18.342978 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 400558us
I20260709 12:20:18.245499   562 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.351346 27655 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:18.291342 27582 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:18.321494 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.226868   856 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 12:20:18.355418   494 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:18.330262 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2739) took 1480 ms. Trace:
I20260709 12:20:18.338658   603 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:20:18.377437   850 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 12:20:18.376727 27584 rpcz_store.cc:276] 0709 12:20:16.850153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.412673 (+562520us) service_pool.cc:224] Handling call
0709 12:20:17.418638 (+  5965us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:17.418865 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:17.952027 (+533162us) log.cc:844] Serialized 14 byte log entry
0709 12:20:18.329813 (+377786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:18.329883 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:18.329947 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:18.330035 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:18.330192 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22101,"prepare.queue_time_us":120149,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":38935,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":778453,"threads_started":2}
I20260709 12:20:18.379449 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2737) took 1402 ms. Trace:
I20260709 12:20:18.315018 27658 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:20:18.379859 27583 rpcz_store.cc:276] 0709 12:20:16.976702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.492068 (+515366us) service_pool.cc:224] Handling call
0709 12:20:17.492370 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:17.492735 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:17.492762 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:17.493218 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:20:17.913947 (+420729us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:17.958846 (+ 44899us) log.cc:844] Serialized 14 byte log entry
0709 12:20:17.959846 (+  1000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:17.960037 (+   191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:18.379168 (+419131us) raft_consensus.cc:1685] finished
0709 12:20:18.379210 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:18.379385 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77405,"prepare.queue_time_us":21285,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":2480,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":48474,"thread_start_us":420315,"threads_started":1,"wal-append.queue_time_us":588541}
I20260709 12:20:18.381368 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.344236 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.385888 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:18.386039 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 889) took 1540 ms. Trace:
I20260709 12:20:18.313787 27663 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:18.386626 27589 rpcz_store.cc:276] 0709 12:20:16.845904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.174167 (+328263us) service_pool.cc:224] Handling call
0709 12:20:17.174481 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:17.490307 (+315826us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:17.490342 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:17.490855 (+   513us) log.cc:844] Serialized 26 byte log entry
0709 12:20:17.957646 (+466791us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:17.957684 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:17.957750 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:18.385771 (+428021us) raft_consensus.cc:1685] finished
0709 12:20:18.385799 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:18.385973 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":315455,"raft.queue_time_us":316196,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"spinlock_wait_cycles":39168,"thread_start_us":466412,"threads_started":1,"wal-append.queue_time_us":831695}
I20260709 12:20:18.389225 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2707) took 1733 ms. Trace:
I20260709 12:20:18.389940   514 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:18.391559   859 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:20:18.392078   853 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:20:18.393195 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.389688 27656 rpcz_store.cc:276] 0709 12:20:16.655614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:16.868105 (+212491us) service_pool.cc:224] Handling call
0709 12:20:16.868399 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:16.911568 (+ 43169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:16.911599 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:16.917671 (+  6072us) log.cc:844] Serialized 26 byte log entry
0709 12:20:17.579582 (+661911us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b180041cc20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:17.963262 (+383680us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:17.963308 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:17.963387 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:18.388858 (+425471us) raft_consensus.cc:1685] finished
0709 12:20:18.388889 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:18.389154 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42794,"raft.queue_time_us":43027,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":4945,"spinlock_wait_cycles":10781056,"thread_start_us":383491,"threads_started":1,"wal-append.queue_time_us":1309589}
I20260709 12:20:18.395076 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.369388 27588 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:18.257241   613 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.404280 27552 leader_election.cc:400] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14.
I20260709 12:20:18.404593 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:18.404631 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.404878 27552 leader_election.cc:403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:20:18.407933 27660 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:18.406567 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:18.411264 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.328800 27587 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:18.412248 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:18.413686 28332 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.597s	user 0.012s	sys 0.000s
I20260709 12:20:18.414464 28332 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:18.414958 27581 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:18.416864 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.357394   382 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:18.414990 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 465954us
I20260709 12:20:18.426968 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.360004   472 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:20:18.400677   501 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.453621 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:18.454368 27586 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:18.322412   600 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:18.476230 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 229665us
I20260709 12:20:18.497058 27664 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:18.499366 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.507355 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.419947   865 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:18.403873   514 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.055s
W20260709 12:20:18.510335 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84200.5us
I20260709 12:20:18.513831 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.518112 27734 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:20:18.515089 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.519935 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:18.523782 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.524137 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.524675 27732 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:18.511605 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 467660us
I20260709 12:20:18.535208 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.535907 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.475278   562 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:18.502765   754 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:18.495600   868 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:18.551115 27732 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:18.496301   501 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:18.555365 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:18.569267 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 167207us
I20260709 12:20:18.561821 27660 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:18.517410   844 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:18.579375 27585 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.002s
I20260709 12:20:18.579949 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:18.581702 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.599165 27736 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
W20260709 12:20:18.613873 28331 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.566s	user 0.012s	sys 0.000s
I20260709 12:20:18.614722 28331 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:18.563745   793 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.620436 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:18.621109 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:18.626276 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 116254us
I20260709 12:20:18.526775   865 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.628805 27736 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:18.561101   562 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.601048   382 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:18.616562   653 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:18.638916 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.654603 27664 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:18.655941 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.639552 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.605500   541 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.659133   852 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:18.660082   541 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:18.659737   852 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.660624   541 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:18.661566   541 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.662101   852 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.649513 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.648958   770 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:18.663730   770 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.665681 27589 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:18.637692   858 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.314s	user 0.000s	sys 0.001s
I20260709 12:20:18.671833 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.676456 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.681352 27734 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:18.617244   822 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:20:18.619904   501 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.687049 27736 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
W20260709 12:20:18.688724 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53880.9us
I20260709 12:20:18.601505   868 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.619935   879 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:20:18.691825 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:18.697613 27734 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:20:18.593356   844 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.631515   793 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.692270   311 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:18.693500   653 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.714071   613 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:18.714334   873 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:20:18.714947   871 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:20:18.716926 32756 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.000s
I20260709 12:20:18.701083   549 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:18.718490   501 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.718611 27656 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:18.708451   884 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.720470   865 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.695247   855 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:18.721568 32746 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.640s	user 0.010s	sys 0.000s
I20260709 12:20:18.722144   881 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:18.672809   382 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.753072   884 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:18.760725 32744 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.689s	user 0.010s	sys 0.001s
I20260709 12:20:18.761199 32744 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.725769 27664 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:18.761925   884 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.755813   881 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:18.763418   881 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.764751   881 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:18.727357   890 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:20:18.765858   855 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.766454   890 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:18.767603   890 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.730520 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2592) took 2198 ms. Trace:
W20260709 12:20:18.717741   762 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.659s	user 0.006s	sys 0.005s
W20260709 12:20:18.769666   869 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:20:18.770133   770 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:18.720124   876 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:20:18.732450   653 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.769053 27731 rpcz_store.cc:276] 0709 12:20:16.532165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:16.552340 (+ 20175us) service_pool.cc:224] Handling call
0709 12:20:17.636858 (+1084518us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:20:18.254485 (+617627us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:18.259813 (+  5328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:18.730233 (+470420us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b54002e7cd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:18.730406 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1425524352}
I20260709 12:20:18.773083   653 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.734865   822 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:18.774060   822 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:18.728767   587 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.775384   822 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.776290   562 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:18.776510   581 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.827s	user 0.012s	sys 0.000s
W20260709 12:20:18.743804 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:18.748425 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.733310 32756 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:18.759209   844 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.758910 32746 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:18.768330   874 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:20:18.769279   762 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.777797 27736 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:18.779819   865 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.784477   770 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:18.797242   770 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:18.798151   861 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.240s	user 0.000s	sys 0.000s
I20260709 12:20:18.776896   581 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.801924   581 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.805681   653 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:18.806329   653 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.808708   653 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.768646   855 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.811208 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2750) took 1283 ms. Trace:
W20260709 12:20:18.758677   512 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.850s	user 0.008s	sys 0.004s
I20260709 12:20:18.732394 27655 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:18.811663 27588 rpcz_store.cc:276] 0709 12:20:17.527423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.551864 (+ 24441us) service_pool.cc:224] Handling call
0709 12:20:17.552208 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:18.396806 (+844598us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:18.810974 (+414168us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b54000be0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:18.811148 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1875379072}
W20260709 12:20:18.774238   867 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:20:18.816745   793 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.819187 32746 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.753217   794 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.822585   794 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.825153   794 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.827016 32744 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:18.828321   863 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.487s	user 0.002s	sys 0.000s
I20260709 12:20:18.828573   770 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.829813 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 736) took 1207 ms. Trace:
I20260709 12:20:18.830262 28332 rpcz_store.cc:276] 0709 12:20:17.622562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.622750 (+   188us) service_pool.cc:224] Handling call
0709 12:20:18.829648 (+1206898us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:18.829755 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369040128}
W20260709 12:20:18.784505 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:18.795490 27662 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:18.785930 27586 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:18.832980 27661 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:18.780015 28334 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:18.742681   866 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.754309   600 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:18.842417 27658 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:18.843101 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 323264us
I20260709 12:20:18.761921   382 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.836249 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:18.837496 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:18.844025 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:18.852051 27664 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:18.855118 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.856841 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:18.859095 28337 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:18.795292   441 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.795531   762 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.864277   653 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:18.842353   568 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14.
I20260709 12:20:18.857609   844 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:18.866413   844 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:18.866870   844 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:18.868234 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:18.788141   868 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.853950   600 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:18.853534 32756 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:18.883720   880 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.884368   880 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.886842   883 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:18.845731   866 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.887486   883 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.870105   512 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:20:18.889133   512 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:18.889536   512 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:18.890106   845 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:18.890738   845 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.891548   600 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:18.796614   865 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:18.896113 27659 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:18.898322 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.901474 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:18.901837 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 210073us
I20260709 12:20:18.903688 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.880577   897 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:18.895522 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.905802 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.890606   512 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:18.907563 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:18.910113 27663 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:18.911064 27737 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:20:18.893268   845 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:18.910272 27732 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:20:18.895099 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:18.914677 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.916406 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.918808 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:18.915927 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:18.920374 27730 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:18.921993 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:18.922606 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:18.923489 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:18.932092 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:18.938326 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.951946 27733 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:18.956199 27656 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:18.959501 27662 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:18.961577 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.896358   657 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:20:18.964493 27587 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:18.964855 27663 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:18.966867 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.967700 27657 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:18.970028 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.969849 27483 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 12 to 15, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.970999   793 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:18.971560   494 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:18.971854 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:18.972790 27589 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:18.971961   494 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:18.973583 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.973721   512 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:18.971230 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.917371   897 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:18.980466 27580 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:18.979630 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:18.985540 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:18.975558 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:18.951891   514 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:18.993742 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.998271 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:18.997174   514 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:19.003451   541 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:19.004129   541 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.006343   541 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.006807 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:19.006920   898 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:18.997368   859 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.008208   859 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.010367   859 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:18.994870   895 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:18.997202 32644 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:19.004488   514 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.016343 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:18.981302   866 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.017346 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 821) took 1698 ms. Trace:
I20260709 12:20:19.020481 27660 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:19.022647 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.023238 27661 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:19.020629 28329 rpcz_store.cc:276] 0709 12:20:17.319299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.335847 (+ 16548us) service_pool.cc:224] Handling call
0709 12:20:17.498299 (+162452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:17.964958 (+466659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:18.253247 (+288289us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54005a2654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:19.017165 (+763918us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54005a2650. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:19.017289 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1401446528}
I20260709 12:20:19.026391 27663 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:18.993682   657 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:19.037564 27657 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:19.014670   616 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.044430 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.045079 27660 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:19.021624   603 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:19.050803 27732 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:20:19.051955 27657 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:19.052981 27663 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:19.058175 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.060281   494 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.061547   494 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.031623   541 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:19.054322 27656 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:19.081027 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.082362 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.032050   657 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:19.042189   822 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:19.121059 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.122493 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.124498 27588 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:18.989902   600 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.051679   896 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.131940 27733 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:19.135473 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:19.136271 27580 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:19.137014 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.137616 27580 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:19.052107   581 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:20:19.133741   897 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:19.156018   901 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:20:19.122787   541 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:19.166131 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.172151 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.173363 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.174909 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.176677 28330 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:20:19.176945 27660 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:19.178691 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:19.194809 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.127421   822 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.192879   441 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:19.196029   865 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:19.233772 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48448.4us
I20260709 12:20:19.239989 27658 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:19.245016 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:19.161007   896 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.253019 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:19.255961 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 825) took 1317 ms. Trace:
I20260709 12:20:19.256356 28334 rpcz_store.cc:276] 0709 12:20:17.938976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:17.940301 (+  1325us) service_pool.cc:224] Handling call
0709 12:20:18.776963 (+836662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:18.843145 (+ 66182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:19.255772 (+412627us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x7b5400491bd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:19.255905 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1890888064}
I20260709 12:20:19.241230   441 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:19.258437 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.241099   311 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.213016   844 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.258733   763 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:19.264019   902 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 12:20:19.263837   763 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.265007   844 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.190595   657 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:19.263371   904 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:20:19.250239   472 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:19.265427   912 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:20:19.281306 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.267241   311 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.276310   866 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:19.279376   891 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:20:19.326469   911 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:20:19.329699 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48478.4us
I20260709 12:20:19.330484 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.331609 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:19.332285   915 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:20:19.332499   770 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:19.333225   770 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:19.333817   770 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:20:19.325424   825 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.337623   825 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.291396 32744 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:20:19.281543   562 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:20:19.339612 32744 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:19.339982 32744 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:19.340818 32744 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.338057   657 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.341986 32744 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:19.344018 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:19.349517   825 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:19.355590   906 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:20:19.358182 27586 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:19.280997   864 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.337072   896 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.362349 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.378660 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.367352 27582 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:19.367830 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:19.380086 27661 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:19.388813 27655 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:20:19.390992 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.413574 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.254164   541 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.419667 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:19.424502 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.426515 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.427172 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:19.365633   866 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:19.434522 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.421847   763 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:19.414347   828 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.430655 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.444181 27661 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
W20260709 12:20:19.414158   916 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:20:19.446951 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.447361 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:19.449783 27656 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:20:19.426910   494 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:19.450640 27655 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:19.450724 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.453784 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:19.451371 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 119844us
I20260709 12:20:19.413440   864 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.460238 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:19.466585 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:19.462420 27661 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:20:19.468492 27655 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:19.466395 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.450198   562 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.473730 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.474480 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:19.475433 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.473685 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.474956 27661 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:19.479589 27661 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:19.480083 27655 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:19.424367   441 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.481626 27587 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:19.483134 27661 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:19.449322   828 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.487768 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.472154   771 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.495954 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.476797   822 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.510349 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.501034 27664 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:19.479233   770 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:19.518793 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.504498 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:19.521546 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:19.532058 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:19.479755   562 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.537340 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.538177 27589 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:19.534511 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:19.546093 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:19.514287   615 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.546123 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:19.535925 32644 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:19.532855   897 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:19.552338 32644 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:19.553577 32644 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.550302   890 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.555876   890 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.558326   890 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.559047   754 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:19.559584   754 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:19.542248   884 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.563760   884 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.566494   884 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:19.566720   921 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260709 12:20:19.569154   512 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.000s
I20260709 12:20:19.570180   895 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.570557 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.546353   865 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:19.570772   864 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:19.542547   657 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:19.558379   880 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.571269   581 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.501607   771 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.574981 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:19.574846 27580 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:19.579191   657 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:19.580516 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.586869 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.586990 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:19.503273   866 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.587172 27585 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:19.588796 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.591293 27659 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
W20260709 12:20:19.589454 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 115051us
I20260709 12:20:19.592024 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.592027 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.592885 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:19.593888 27580 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:20:19.594924 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:19.593135 27589 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:19.595952 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:19.596280 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:19.598644 27585 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:20:19.551581   615 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.601454 27585 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:19.603943 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.604091 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:19.593660   865 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:19.606314 27585 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:20:19.592732   880 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:19.594216   864 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.612900   864 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.601374   512 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:19.615008   512 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:19.615447   512 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:19.617179 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2269) took 1285 ms. Trace:
I20260709 12:20:19.617545 27662 rpcz_store.cc:276] 0709 12:20:18.331708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:18.335281 (+  3573us) service_pool.cc:224] Handling call
0709 12:20:19.616998 (+1281717us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:19.617090 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056157056}
I20260709 12:20:19.579715   581 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.605021   845 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:19.620972   845 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:19.621198 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2744) took 1228 ms. Trace:
I20260709 12:20:19.621380   845 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:19.621578 27582 rpcz_store.cc:276] 0709 12:20:18.392430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:18.406346 (+ 13916us) service_pool.cc:224] Handling call
0709 12:20:19.621004 (+1214658us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54000bd1d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:19.621144 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2199012736}
I20260709 12:20:19.602901   884 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:19.623375   382 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:19.623754   884 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:19.624747   884 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.628161 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2267) took 1327 ms. Trace:
I20260709 12:20:19.628806   581 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.611256   890 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:19.629734   754 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.612315   562 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.637974 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.628545 27658 rpcz_store.cc:276] 0709 12:20:18.301129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:18.381037 (+ 79908us) service_pool.cc:224] Handling call
0709 12:20:19.627995 (+1246958us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:19.628106 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058522368}
I20260709 12:20:19.576836   895 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:19.646752 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47018us
I20260709 12:20:19.649575 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.650022 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:19.650215 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:19.608744   865 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.653126   865 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.651684   894 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:20:19.648882   925 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:19.654138   925 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.654515   925 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:19.656481   615 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.654680   828 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:19.658046   864 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.658514   883 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.609159   880 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.661870   880 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.661983 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:19.667779 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.677500 27580 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:19.678397 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.679776   764 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:19.680063   868 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:20:19.680927   894 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:19.701409 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.702064 27730 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.680985 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.682170 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:19.673202   931 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:19.695842 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:19.704597 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.705194 27731 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:19.706009 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.694504 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:19.699777 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.700433 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.706853 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.706947 27737 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:19.707494 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.708115 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.711246 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.720386 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.721166 27737 raft_consensus.cc:2368] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:19.721020 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.698537   764 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:19.748529 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.751916 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:19.701504   894 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:19.702183   754 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.759758   382 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:19.763595 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.689531   657 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.753157 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.771929   512 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.776690   895 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.788098 27729 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:19.782337 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:19.810537 27730 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:20:19.754089   764 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:19.782456   472 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.503s	user 0.006s	sys 0.003s
I20260709 12:20:19.814940 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.816159 27737 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:20:19.778103   311 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:19.826958   311 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:19.805725   382 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:19.831640   828 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:19.833197   887 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:19.834488 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.835736 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.833467 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.841142 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:19.838587 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.842167 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.846637 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.843071 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:19.822934   472 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:20:19.892851 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49890.4us
I20260709 12:20:19.895843 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:19.896785 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.899842   894 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.902225   894 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.903672 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.907627 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:19.908223 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:19.908876 27731 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:19.911612 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:19.831079   771 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:19.917460 27736 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:19.921497 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.924887 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:19.778674   845 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:19.922497 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.926180 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.926693 27734 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:19.897961 27730 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:19.838577   382 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:19.928797 28337 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:19.932617 27737 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:19.933920 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:19.831955   872 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
I20260709 12:20:19.936467 27587 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:19.893364   884 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:19.938588 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.916159   864 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:19.941759 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:19.916395   754 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:19.927850   828 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:19.935910   771 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:19.939013   615 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:19.998829 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2711) took 1101 ms. Trace:
I20260709 12:20:19.966552   864 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:19.959980   828 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:20.012457 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.016978 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:19.999548 27659 rpcz_store.cc:276] 0709 12:20:18.897861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:18.898009 (+   148us) service_pool.cc:224] Handling call
0709 12:20:19.998595 (+1100586us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b5400344254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:19.998750 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1057944960}
I20260709 12:20:20.005916   828 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:20.006026   616 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.964546   754 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.032764 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:20.034619 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:20.034467   616 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.036584   616 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:20.037336   864 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.046494 27733 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:20.017261   883 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:20.062412   941 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 12:20:20.064278 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:20.064998   932 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:20.065421   932 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:20.066028 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.067157   311 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:20.067597 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.068975   616 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:20.069669   865 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.070650   616 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:20.072100 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.049079   512 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:20.075554   865 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:20.076771 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 145112us
I20260709 12:20:20.076566 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.079607 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:20.080178 27659 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:20.086038 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:20.087697 27657 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:20.018909 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.020371   844 raft_consensus.cc:2764] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:20:20.089375 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:20.085775 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:20.091459 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.076395   311 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:20.093806 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:20.094606 27659 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260709 12:20:20.096040 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77552us
I20260709 12:20:20.096695 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:20.096244 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:20.084112   879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.003s	sys 0.010s
I20260709 12:20:20.098145 27662 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:20.100099   864 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:20.100521   864 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.100931   864 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:20.086037   793 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:20.105212 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.096768 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.097425 27659 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:20.099399 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:20.107820 27656 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:20.097165   568 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:20.074183   441 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:20:20.099298   794 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:20.111815 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:20.115185 27656 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:20.116019 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:19.934832   770 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
W20260709 12:20:20.120431   942 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:20:19.964252   884 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:20:20.158187 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.066228   932 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.126312   382 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.099673 32644 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:20.104291   879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.004s	sys 0.010s
I20260709 12:20:20.138820 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.139518 27732 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:20.155674 27662 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:20:20.193635 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95104.6us
I20260709 12:20:20.156808   944 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:20.091946   872 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:20:20.138844 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:20.222213 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 83441.4us
I20260709 12:20:20.160789 27733 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:20:20.189231 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:20.172777 27627 leader_election.cc:400] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:20.225915 27627 leader_election.cc:403] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:20:20.216137 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.128858   452 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:20.134088   794 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:20.236462 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98014us
I20260709 12:20:20.238364 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.130620   945 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:20.158828   883 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:20.159379   845 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:20.249735 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:20.245942   828 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.284896   382 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:20.294067 27733 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:20.295509 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.296340 27736 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:20.298166 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.299719 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.302194 27733 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 12:20:20.303031 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:20.254668   562 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:20.235579   829 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:20.255255 32644 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:20.256392   932 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:20.309131 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:20.310128 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:20.371095 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48734us
W20260709 12:20:20.310362   925 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.655s	user 0.008s	sys 0.001s
I20260709 12:20:20.247539   945 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.313712 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:20.377803   945 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:20.378293 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64673.3us
I20260709 12:20:20.379514   452 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:20.379552 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:20.381043   452 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:20.314711 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:20.382375 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67750.5us
W20260709 12:20:20.315918   936 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.497s	user 0.003s	sys 0.007s
I20260709 12:20:20.343568 28336 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:20.344784 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:20.346534 27736 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:20.348479 27734 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:20.349231   568 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:20:20.247254   794 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:20.325150   825 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:20.384523 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:20.386204 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:20.306794   880 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.644s	user 0.010s	sys 0.000s
W20260709 12:20:20.398286 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:20.400388 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:20.365105   829 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.402707   541 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:20:20.447866 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46158.8us
I20260709 12:20:20.443883   880 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.450613   880 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:20.453028 32744 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.453624 32744 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:20.454437   894 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:20:20.463296   828 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:20.470549 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:20.402486   868 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:20.435251   825 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:20.401901 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.018s	sys 0.022s
W20260709 12:20:20.362989   764 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.532s	user 0.005s	sys 0.005s
I20260709 12:20:20.388903 27581 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:20:20.404106 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:20.404994 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 293225us
I20260709 12:20:20.384773   925 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:20.426496 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 176852us
I20260709 12:20:20.428324 27589 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:20.341995 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:20.382964 32644 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:20.447440   829 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:20.479758   825 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:20.489564 27657 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:20.387777 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:20.490839 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86846us
I20260709 12:20:20.492903 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:20.497731 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 155875us
I20260709 12:20:20.525118 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:20.480942 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.018s	sys 0.022s
W20260709 12:20:20.526016 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 831500us
I20260709 12:20:20.531980 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:20.534865 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.539557 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:20.544413 27737 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:20.545500 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 331024us
W20260709 12:20:20.546831 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54024.9us
I20260709 12:20:20.545569 27738 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:20.547894 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:20.554371 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:20.555673 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:20.500511 32644 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:20.520033   770 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:20.562873 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:20.478129   868 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.553697   828 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:20.498479   764 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.495908   763 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:20:20.552484   514 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:20.554402   845 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:20.554193   935 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:20.564435   866 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:20.561298   793 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:20.582624 32644 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:20.581784   472 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:20:20.587914 32708 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.591392   894 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:20.609668 28334 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:20.572465   897 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:20:20.597054 32744 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:20.618343   880 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:20:20.591347   948 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:20:20.628736 27585 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:20.614185   894 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:20.635437 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:20.618270   866 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:20.604759   935 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.617170 32708 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.609728   952 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:20:20.632607   958 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:20.604931   953 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:20.641425 32744 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 12:20:20.671965   894 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:20.679268   951 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:20:20.743822 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 179700us
I20260709 12:20:20.680862   883 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:20.683979 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:20.739233 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 183613us
W20260709 12:20:20.636678   311 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.014s	sys 0.032s
I20260709 12:20:20.675972   828 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:20.685070   959 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:20.784332 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 100451us
I20260709 12:20:20.790058   764 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:20.793987   960 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:20:20.795040   868 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:20.796753   935 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:20.798321 32644 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.637441   514 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:20.803747   925 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:20.804265   549 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:20.777596   793 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.779537   897 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.805863 32708 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:20.741693   958 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.778754   951 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:20.806145   866 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.807134   952 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:20.780535   883 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:20.812594 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:20.780817 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.816088 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:20.819603   943 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.014s	sys 0.022s
I20260709 12:20:20.833091 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:20.856339 27587 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:20.845005 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:20.827277 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:20.851290 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70590.6us
I20260709 12:20:20.784438   311 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:20.823956   957 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:20:20.871029 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:20.849866 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 313456us
I20260709 12:20:20.864121 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:20.860100 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:20.878080 27584 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:20.869958 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:20.880968 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:20.824410   959 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:20.835129   864 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.734s	user 0.012s	sys 0.000s
I20260709 12:20:20.890187 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:20.888695 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:20.860317   657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s	user 0.022s	sys 0.005s
I20260709 12:20:20.885964 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:20.854815   943 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.014s	sys 0.022s
W20260709 12:20:20.833137   891 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.055s	user 0.013s	sys 0.013s
I20260709 12:20:20.832206   754 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:20.903334 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:20.972975   896 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:20.973659 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70409.4us
I20260709 12:20:20.973795   896 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.903515 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:20.976231 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72774.5us
I20260709 12:20:20.977561   968 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:20.978135 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:20.978331   968 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:20.979542   964 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:20.855661   935 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:20.915169 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:20.850988   958 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:20.981226   382 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:20.982229   825 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.925510 28336 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:20.927722   957 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.828143 32744 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 12:20:20.937693   770 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.867990   953 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:20.940091   969 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:20:20.941347   961 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:20:20.837363   514 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.942523 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2632) took 1227 ms. Trace:
I20260709 12:20:20.945703   763 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:20.872234   925 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:20.954365   959 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:20.955062   951 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:20.962038 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47309.2us
I20260709 12:20:21.011651 27584 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
W20260709 12:20:20.962172 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102147us
W20260709 12:20:20.909675   945 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.531s	user 0.004s	sys 0.007s
I20260709 12:20:20.962803   880 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:20.963449   845 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:20.963899 32644 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.039475   825 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.041079   896 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:20.901901   879 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:20:20.965073   883 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.042758   883 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:20.846634   897 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:20.829061   829 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.963937   952 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.923156   868 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:20.967424   971 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.908119   864 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.979864   968 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.903561   541 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:20.998525 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:20.926012   657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.022s	sys 0.006s
W20260709 12:20:20.984892   891 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.207s	user 0.014s	sys 0.013s
I20260709 12:20:21.004593 27583 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 12:20:21.080372 27583 pending_rounds.cc:85] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 6
I20260709 12:20:21.080767 27583 pending_rounds.cc:107] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:20:21.081027 27583 raft_consensus.cc:2889] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:21.062484   880 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:21.008975 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 326500us
I20260709 12:20:21.036036   945 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.012959 27587 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:21.029599 27585 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:21.029857 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:21.034029 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:21.038538   845 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:20.882437 32708 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:20:21.048707 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:20.993500   950 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:21.094867 27730 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:21.035245   794 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.584s	user 0.013s	sys 0.000s
I20260709 12:20:21.003201   770 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:21.068027 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 207229us
I20260709 12:20:21.030426   382 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:20:21.000919   957 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.006803 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:21.022681 27738 rpcz_store.cc:276] 0709 12:20:19.714834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:19.716773 (+  1939us) service_pool.cc:224] Handling call
0709 12:20:20.942287 (+1225514us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:20.942456 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1701086592}
W20260709 12:20:21.093830 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64051.8us
I20260709 12:20:21.098589 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:21.105598 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 98914.9us
I20260709 12:20:21.054415   311 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.113700   935 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:21.087904   964 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:21.117769 27587 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [15.9-17.10]
I20260709 12:20:21.064035   952 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.097739   948 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.097962   872 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:21.101338   868 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:21.069258   971 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.130525 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:21.132046 27664 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:21.105625 32644 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:21.094695   894 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.133490 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:21.142863   829 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.137822   770 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:21.101228   794 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.148983 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:21.149314   964 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.100967   950 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.157644 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:21.115165   865 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:20:21.157059   441 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:20:21.160071   957 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.169375   616 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:21.170871 27582 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:21.179832 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:21.180732 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:21.125849   872 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.184723   764 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:21.191421   951 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:21.234215 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:21.186581   957 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:20:21.192909 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:21.193537 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:21.181779   974 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.237005   974 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:21.194808   982 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:20:21.166558   988 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:21.197502   514 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.198071   897 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:21.187733   976 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:20:21.199141   845 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.200543   894 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:21.182993   981 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:20:21.202262   452 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:21.147071   880 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.203269   950 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.205804   657 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:21.206430   945 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:21.231882 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:21.128248   948 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.232957 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:21.245435 27587 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
W20260709 12:20:21.250922   994 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:20:21.261106   896 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:21.280918 27581 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:21.206475   925 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 12:20:21.246169   829 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:21.278167 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:21.185591   616 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:20:21.259258   452 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:21.271303 27730 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:21.286968 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:21.243538   764 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.311810 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:21.316898 27589 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:21.306367 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:21.318174 27587 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:21.323149 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:21.325191 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:21.320789 32644 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:21.297194   382 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:21.327347 32644 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:21.330348 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:21.325501   958 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.336836   958 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.347438 27587 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:21.319733   897 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:21.353484 32644 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.350764 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:21.298719   896 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.310472   829 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.328378   844 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:20:21.365347   897 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:21.394558 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:21.395256 27587 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
W20260709 12:20:21.400287   997 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:20:21.400761   971 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.387759   883 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.407840   883 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.408468 28332 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:20:21.410679 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:21.349202   382 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.413012 28328 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:21.422551 27660 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:21.428555 27662 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:21.433390 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:21.412258 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:21.426461 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:21.452236 27664 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
W20260709 12:20:21.455217 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 182653us
I20260709 12:20:21.337792   452 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.465715 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:21.469074 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:21.467845 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:21.443676 27483 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 9 to 13, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 13 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:21.466554   657 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.471587   872 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:21.472071   657 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.473235   896 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.474220   657 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:21.475507 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:21.478457 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:21.411276   883 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.439937   794 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.465180  1005 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:21.433120   974 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:21.468161 32744 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 12:20:21.491853   948 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:21.505857 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:21.533761 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:21.536860 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:21.540170 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:21.542377 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:21.425974   897 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:21.529388   452 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:21.574810 27655 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:20:21.545387 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:21.607414 27660 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:20:21.509142   974 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:21.607858 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:21.530391   883 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:21.629998 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95641us
I20260709 12:20:21.617400   999 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:21.545513 32644 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:20:21.542452   896 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:20:21.625061 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:21.647543 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:21.991020 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 359401us
I20260709 12:20:21.991026 32644 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:21.545282   472 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:20:21.719043  1015 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:20:21.506317  1005 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:21.778712   541 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:21.648542   883 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.001159   950 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:22.001605   950 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:22.002503   950 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:21.749863   988 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.505s	user 0.013s	sys 0.000s
I20260709 12:20:21.730667   952 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:21.695791   829 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:21.888576   501 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:21.972663 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:21.956826   954 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:21.904654 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:21.719432 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.079349 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:21.814941 27733 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:20:21.776449 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111541us
I20260709 12:20:21.997448   472 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:22.090569 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:21.888599 27731 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:20:21.717294 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:22.094249 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:21.542380   794 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.110s
I20260709 12:20:21.982053 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.035806   541 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:21.793371 27660 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:21.750584 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 841) took 1432 ms. Trace:
I20260709 12:20:21.986660   828 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:21.816777   996 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.859577 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:21.826660 27729 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:20:21.825397 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:21.815567   944 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:21.819772 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:21.544817   764 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:21.545153   881 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:21.631258   974 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:21.680696   937 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.408s	user 0.018s	sys 0.015s
I20260709 12:20:21.994976 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:22.095643   472 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:22.020020   864 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.031669 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:22.134910 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103337us
I20260709 12:20:22.032685 32644 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:21.964452   975 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.909s	user 0.002s	sys 0.000s
I20260709 12:20:22.042783 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2366) took 1229 ms. Trace:
I20260709 12:20:22.046967 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:22.077559 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105375us
I20260709 12:20:22.137118 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.100275   311 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:22.084709 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.141959 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.088878 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:21.777966 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:22.143088 28330 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:22.143126 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.096468 28334 rpcz_store.cc:276] 0709 12:20:20.318487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:20.320353 (+  1866us) service_pool.cc:224] Handling call
0709 12:20:20.364318 (+ 43965us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:20.615373 (+251055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:21.750324 (+1134951us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:21.750521 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560488192}
I20260709 12:20:22.097440 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.050038   896 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.051878   879 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:22.076601   950 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:22.050319   932 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.104681 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.093493   501 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.099797 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:22.079694   988 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:22.083390   931 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:22.173158 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73455.6us
I20260709 12:20:22.116168 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.107468 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:22.175882 27657 raft_consensus.cc:2376] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14. Re-sending same reply.
I20260709 12:20:22.110742 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:22.141083   948 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:22.123473 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:22.150400   311 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.041898  1018 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:20:22.092192   452 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.089183   969 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:22.128943 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:22.210604 27662 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:22.096474   616 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:22.130470 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.131510 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:22.137887 27659 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:21.790565 27732 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:22.135665 32644 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.099515   828 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:22.151279 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:22.220490 27736 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:22.152664 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.221117 27662 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:20:22.137171 27734 rpcz_store.cc:276] 0709 12:20:20.813703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:20.813932 (+   229us) service_pool.cc:224] Handling call
0709 12:20:20.814207 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:20.814808 (+   601us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:20.991894 (+177086us) log.cc:844] Serialized 14 byte log entry
0709 12:20:22.015372 (+1023478us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b1800391000. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287
0709 12:20:22.015711 (+   339us) log.cc:844] Serialized 14 byte log entry
0709 12:20:22.042278 (+ 26567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:22.042360 (+    82us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:22.042425 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.042530 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:22.042704 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66710,"prepare.queue_time_us":284743,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":2724,"raft.queue_time_us":106353,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":39311,"spinlock_wait_cycles":11414272,"thread_start_us":527140,"threads_started":3,"wal-append.queue_time_us":295384}
I20260709 12:20:22.120180 32744 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.114300   881 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:22.105312  1013 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:22.161000 27664 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:22.103515 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:22.221096   937 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:22.178581 27733 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:21.988442  1017 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.201937 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.122085   944 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.212361 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.216078 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:22.285804 27732 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:22.286327 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.235827   881 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.224480 27589 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:22.207199   547 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:22.157945   864 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:20:22.122242   996 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.319970 27664 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:22.176656   822 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:22.315105 27655 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:20:22.218335   964 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:20:22.206804   311 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.264966  1008 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:22.281601   562 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:22.211789   975 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:20:22.266542 32644 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.236757 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.094094   954 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.206514 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:22.352509 27657 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:22.166272   932 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.219578   616 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:22.218736   794 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.271811 27731 raft_consensus.cc:2368] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22.
I20260709 12:20:22.233867   948 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:22.161037   879 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.298811 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.384119 27589 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:22.219276   452 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.263674  1013 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.264024   969 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:22.305295   944 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.208027   931 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:22.307300  1024 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:20:22.235152 32744 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.412649 27730 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:22.413887   541 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.310393 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.258736   925 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:22.313073   382 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.313743   864 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:22.293910  1017 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.326567 28330 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:22.330569 27736 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:22.158667   896 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.453856 27733 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:22.336227 27664 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:22.344579 27732 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:22.387399   794 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.370142 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2302) took 1380 ms. Trace:
I20260709 12:20:22.377589 27731 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:20:22.334657  1035 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:22.333993   770 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:22.375519   616 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.474310   896 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.475008 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:22.475435   763 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:22.475886   763 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:22.476866   763 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.478000   763 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:22.438853   925 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.481526   925 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.438066   864 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:22.486651 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.400389 27735 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:22.381337   514 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:22.416046  1030 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:20:22.416776   472 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:22.216311 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 91418.4us
W20260709 12:20:22.418167 27627 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.014s	sys 0.006s
I20260709 12:20:22.419878 27660 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:20:22.345551   822 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.425269 27588 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:22.520397  1026 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20260709 12:20:22.352274   562 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.432173 27657 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:20:22.438004 27737 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:20:22.524974 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.348935 27734 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:22.457475 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.458372 27581 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:22.526397 27734 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:22.352342  1008 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.463599 27731 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:22.465483 27589 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [15.9-17.10]
I20260709 12:20:22.469105  1005 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.472750   931 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.473214  1017 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:22.473493  1028 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:20:22.463917  1040 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.473742   770 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.459704 27656 rpcz_store.cc:276] 0709 12:20:20.989581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:20.989728 (+   147us) service_pool.cc:224] Handling call
0709 12:20:20.990075 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:21.106445 (+116370us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:20:21.121872 (+ 15427us) log.cc:844] Serialized 14 byte log entry
0709 12:20:21.235193 (+113321us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:21.235264 (+    71us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:21.235308 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.369835 (+1134527us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54000e56d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:22.369889 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:22.370043 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16309,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":406,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":97,"spinlock_wait_cycles":267141376,"thread_start_us":112300,"threads_started":1,"wal-append.queue_time_us":112626,"wal-append.run_cpu_time_us":2400,"wal-append.run_wall_time_us":1015069}
W20260709 12:20:22.499523 27627 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.014s	sys 0.006s
I20260709 12:20:22.544263 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:22.544486 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 813984us
W20260709 12:20:22.514977 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:22.519204 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:22.520828  1031 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:20:22.520941 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.388242   883 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.497324   969 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.539093   950 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:22.554291   879 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.555341 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.555886 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2309) took 1123 ms. Trace:
I20260709 12:20:22.556069  1013 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.556319 27663 rpcz_store.cc:276] 0709 12:20:21.432177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:21.433770 (+  1593us) service_pool.cc:224] Handling call
0709 12:20:21.434232 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:21.434766 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:21.434812 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:21.783785 (+348973us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:21.784297 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:20:21.972724 (+188427us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:20:21.972770 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:21.972849 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:22.555572 (+582723us) raft_consensus.cc:1685] finished
0709 12:20:22.555594 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:22.555822 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":978,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10530,"spinlock_wait_cycles":13993728,"thread_start_us":187954,"threads_started":1,"wal-append.queue_time_us":315138}
I20260709 12:20:22.558683 27663 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
W20260709 12:20:22.523545 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.064s (SENT) [suppressed 4 similar messages]
I20260709 12:20:22.525303 27737 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 13
I20260709 12:20:22.560177 27737 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:20:22.560524 27737 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:22.529785 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.523209 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:22.482743   864 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.535156 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.537281 27586 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:22.500165   964 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:22.545935 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.524860   989 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.557690 27729 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:22.314260 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.556494 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.532413 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:22.570168 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 255979us
I20260709 12:20:22.570353 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.546669  1053 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:22.571979 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:22.571125 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:22.572876 27589 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:22.571092 27729 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
W20260709 12:20:22.575299 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52242.2us
I20260709 12:20:22.579783 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.582325   541 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.541478   932 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.584574 27664 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:22.499943   514 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.585132 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2754) took 1055 ms. Trace:
I20260709 12:20:22.605146 27659 rpcz_store.cc:276] 0709 12:20:21.529773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.226145 (+696372us) service_pool.cc:224] Handling call
0709 12:20:22.269608 (+ 43463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:22.438362 (+168754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:22.438399 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:22.454147 (+ 15748us) log.cc:844] Serialized 26 byte log entry
0709 12:20:22.530562 (+ 76415us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:22.530604 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.530670 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:22.584845 (+ 54175us) raft_consensus.cc:1685] finished
0709 12:20:22.584875 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:22.585056 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13157,"raft.queue_time_us":149044,"thread_start_us":172819,"threads_started":2,"wal-append.queue_time_us":123662}
I20260709 12:20:22.591897 27729 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:22.594677   541 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:22.494036   948 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.613298 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.572582   883 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.559866   954 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.595597 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:22.576472   879 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:22.576923   865 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:22.572814 27657 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:22.492077  1035 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.654763  1035 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.647914  1039 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.655798  1039 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.658696  1039 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.659848 27657 raft_consensus.cc:606] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:22.660336 27657 raft_consensus.cc:614] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:22.649811 27588 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:20:22.660982   770 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.662007 27588 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:22.661949   881 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.650434 27732 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:22.663784 27732 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:22.664657 27732 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.665663   822 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.666241 27732 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:22.666766 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:22.667121   562 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.671942  1008 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.672600 27588 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:22.673350   472 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.576052  1053 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.676380  1053 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.677349 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2308) took 1248 ms. Trace:
I20260709 12:20:22.677747 27658 rpcz_store.cc:276] 0709 12:20:21.429217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:21.429896 (+   679us) service_pool.cc:224] Handling call
0709 12:20:21.430341 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:21.430893 (+   552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:21.430938 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:22.025118 (+594180us) spinlock_profiling.cc:243] Waited 9.98 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:22.025568 (+   450us) log.cc:844] Serialized 26 byte log entry
0709 12:20:22.144202 (+118634us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:22.144242 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.144321 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:22.677035 (+532714us) raft_consensus.cc:1685] finished
0709 12:20:22.677057 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:22.677288 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":733,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":104,"spinlock_wait_cycles":22958464,"thread_start_us":118247,"threads_started":1,"wal-append.queue_time_us":239521}
I20260709 12:20:22.681398 27732 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:20:22.681818 27732 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 9
I20260709 12:20:22.683138 27732 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:20:22.684139 27732 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:22.615579   864 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:22.704366 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 180828us
I20260709 12:20:22.547986  1040 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.642412   541 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:22.646442   452 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.664435   881 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.712234 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:22.713418 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 117754us
I20260709 12:20:22.584964   989 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.703377  1008 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:22.715484 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:22.715948 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:22.706766 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.718309 27656 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:22.719911 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:22.709802  1053 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.723858 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.722632 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:22.680368   883 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:22.739176 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 159228us
I20260709 12:20:22.740029 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.740541 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.740033 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.730548 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:22.743166 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.744705 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:22.737262  1039 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:20:22.746311   970 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:22.746670   931 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:22.747180   931 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:22.748075   931 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.748636  1040 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.723578   547 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:20:22.756700 27659 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:22.756699 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.758687 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.726298   562 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:20:22.764484 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:22.760270  1035 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:22.768028 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.723618   452 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.034s
W20260709 12:20:22.752513 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.736955   932 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:22.723969  1008 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.738008   950 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:22.751466  1013 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:22.743851   794 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:22.801399 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57595.8us
I20260709 12:20:22.757061   845 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:22.781446 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:22.751711 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:22.756326   970 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:22.824410   970 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.776934   541 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:22.830672  1042 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.521s	user 0.000s	sys 0.003s
W20260709 12:20:22.792712   681 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 12:20:22.803151   845 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.833304   845 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.837126 27585 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:22.838315   825 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:22.838740   825 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:22.839576 27735 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:22.785534 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:22.790021 27656 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:22.726847  1053 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.795316   989 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.797011   514 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.771250   969 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:22.797837  1013 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:22.799832  1008 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.782337  1035 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:22.770789  1040 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:20:22.863029  1040 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:22.863425  1040 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:22.864202  1040 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.865424  1040 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.804347 27658 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:22.807657   616 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.867857   845 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.869370   616 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.815690   829 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:20:22.816445 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64847.9us
I20260709 12:20:22.869994   829 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.804672 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:22.870371   829 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:22.870702 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66445.8us
I20260709 12:20:22.870862   452 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:22.871485   452 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:22.839576   825 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.871965   452 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:22.872443   825 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.874897   895 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:22.847186 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2747) took 1340 ms. Trace:
I20260709 12:20:22.875412  1013 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.875556   895 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.875854 27661 rpcz_store.cc:276] 0709 12:20:21.506795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:21.507671 (+   876us) service_pool.cc:224] Handling call
0709 12:20:21.535448 (+ 27777us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.080237 (+544789us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:22.107703 (+ 27466us) log.cc:844] Serialized 14 byte log entry
0709 12:20:22.462168 (+354465us) log.cc:844] Serialized 14 byte log entry
0709 12:20:22.462315 (+   147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:22.462356 (+    41us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:22.462398 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.846882 (+384484us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b540022b354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:22.846947 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:22.847113 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34862,"prepare.queue_time_us":27774,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":169315,"spinlock_wait_cycles":1397738496,"thread_start_us":318985,"threads_started":1,"wal-append.queue_time_us":364994}
I20260709 12:20:22.877050   944 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:20:22.847746 27732 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:20:22.877907 27732 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:20:22.878156 27732 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:20:22.878412 27732 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:20:22.848852 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63419.7us
I20260709 12:20:22.879298 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.886014 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.803369   883 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.887427 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.892220 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.893296 27585 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:22.875512 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:22.854583   865 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:22.878250   895 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.898989 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.899197 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:22.894407 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:22.908906 27585 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:22.908219 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.907199 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.911363 27731 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:22.883064   794 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.803042   950 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.869831   845 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:22.883160   562 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:20:22.892822   616 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:22.919556 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:22.874950  1008 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:20:22.850185  1061 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:20:22.942032 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.942646 27735 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:22.892035  1035 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:20:22.855083   969 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:22.913473   932 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:22.944262 27587 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:22.956982 27731 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:22.936115   681 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:20:22.960624 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:22.938596   822 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:22.936734  1050 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.394s	user 0.000s	sys 0.003s
I20260709 12:20:22.949891 27585 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:22.954682   932 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.932015  1005 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:20:22.970183 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:22.971661  1057 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
W20260709 12:20:22.972316  1062 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:20:22.974890  1053 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:22.944993   616 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:22.976320   996 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:22.983565 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:23.024401 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:22.929188  1063 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:20:23.026453   955 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:22.992750 27585 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:20:23.010150 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 89296.5us
I20260709 12:20:22.929266   794 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.014434   989 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:23.017086   969 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:22.969419   822 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:23.022928 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 143994us
I20260709 12:20:22.992139   932 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:23.052593   616 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.051577 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.056102 28330 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:23.056962   974 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.057166 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2752) took 1547 ms. Trace:
I20260709 12:20:23.057803 27662 rpcz_store.cc:276] 0709 12:20:21.510070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.223013 (+712943us) service_pool.cc:224] Handling call
0709 12:20:22.223340 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:22.455192 (+231852us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:22.455225 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:22.494267 (+ 39042us) log.cc:844] Serialized 26 byte log entry
0709 12:20:22.593874 (+ 99607us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:22.593912 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.593972 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:23.056899 (+462927us) raft_consensus.cc:1685] finished
0709 12:20:23.056929 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:23.057094 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":160416,"thread_start_us":251885,"threads_started":2,"wal-append.queue_time_us":185313}
I20260709 12:20:23.061259   794 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.057340 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:23.065845 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.068167 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.069228 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2650) took 1262 ms. Trace:
I20260709 12:20:23.069656 27733 rpcz_store.cc:276] 0709 12:20:21.796954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.296811 (+499857us) service_pool.cc:224] Handling call
0709 12:20:22.453656 (+156845us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.454255 (+   599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.059168 (+604913us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54001d15d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:23.059296 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352701696}
I20260709 12:20:23.023079 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:23.026860   955 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:23.020418   865 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.039819   950 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.043529   989 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.045924   822 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:23.027531   311 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.672s	user 0.005s	sys 0.010s
I20260709 12:20:23.048393  1063 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:23.036875 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:23.082279 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45518.5us
I20260709 12:20:23.084770 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.085660 27585 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:23.083093 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:23.062902   932 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.090626 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.092833 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2373) took 1015 ms. Trace:
I20260709 12:20:23.079871   881 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:23.118994 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.132282 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.132304 27733 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:23.135102 27736 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:23.118111   763 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.638s	user 0.013s	sys 0.000s
I20260709 12:20:23.148126 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2814) took 1102 ms. Trace:
I20260709 12:20:23.148572 27580 rpcz_store.cc:276] 0709 12:20:22.046271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.046423 (+   152us) service_pool.cc:224] Handling call
0709 12:20:22.046795 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.404451 (+357656us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:22.404523 (+    72us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:22.404591 (+    68us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:22.404638 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.147874 (+743236us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:23.147922 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:23.148066 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":821836032}
I20260709 12:20:23.115890   829 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.082919   311 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.082520   452 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.153262   829 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:23.154850   311 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.156086 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2367) took 1982 ms. Trace:
W20260709 12:20:23.156440 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 2761) took 1620 ms (client timeout 2080 ms). Trace:
I20260709 12:20:23.156905   452 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.081399   865 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:23.156988 27660 rpcz_store.cc:269] 0709 12:20:21.536010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.103207 (+567197us) service_pool.cc:224] Handling call
0709 12:20:23.156205 (+1052998us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:23.156373 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":348631808}
I20260709 12:20:23.159269   865 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.088676   955 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.160502   955 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:23.161152 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.132553   881 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:23.163691   881 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:23.157006 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2819) took 1108 ms. Trace:
I20260709 12:20:23.156445 27738 rpcz_store.cc:276] 0709 12:20:21.173942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:21.174567 (+   625us) service_pool.cc:224] Handling call
0709 12:20:21.194193 (+ 19626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:21.260062 (+ 65869us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:21.260337 (+   275us) log.cc:844] Serialized 14 byte log entry
0709 12:20:21.346851 (+ 86514us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:21.346903 (+    52us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:21.346945 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.155879 (+1808934us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:23.155921 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:23.156030 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":798,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":47075,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":42816,"spinlock_wait_cycles":150927104,"thread_start_us":86167,"threads_started":1,"wal-append.queue_time_us":102031,"wal-append.run_cpu_time_us":1119,"wal-append.run_wall_time_us":1006627}
I20260709 12:20:23.164866 27586 rpcz_store.cc:276] 0709 12:20:22.048638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.377762 (+329124us) service_pool.cc:224] Handling call
0709 12:20:22.378007 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.576823 (+198816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.156803 (+579980us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54001883d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:23.156953 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365899136}
I20260709 12:20:23.158906 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:23.166944 27660 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:23.117378 27730 rpcz_store.cc:276] 0709 12:20:22.077633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.112392 (+ 34759us) service_pool.cc:224] Handling call
0709 12:20:23.092633 (+980241us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54001d3dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:23.092776 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455138688}
I20260709 12:20:23.123600 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:23.170497 27662 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:23.123860 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.171005 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:23.116290  1029 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.784s	user 0.004s	sys 0.000s
I20260709 12:20:23.093847   989 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.136050   932 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:23.178300 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:23.179224   828 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.180932   828 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:23.210412  1039 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:23.210875  1039 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:23.211884  1039 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.213160 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.144467   897 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:23.214002   897 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:23.214936   897 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.216274   897 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:23.217715 27483 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 13, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 13 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:23.119416  1005 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:23.220988  1005 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.221451  1005 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:23.131763   950 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:23.179986   764 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:23.223625   764 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.224076   764 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:23.224821  1013 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:23.225387  1013 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.223654   950 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:23.089646  1047 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.609s	user 0.000s	sys 0.003s
I20260709 12:20:23.229734   950 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.183941 32744 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:23.232519   501 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.234097   935 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:23.234586   935 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:23.119192 27584 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:23.190425 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:23.237248 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46930.5us
I20260709 12:20:23.238123 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:23.239051   996 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:23.240468 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:23.152853 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2813) took 1108 ms. Trace:
I20260709 12:20:23.241187 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.191560   989 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:23.241636  1008 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:23.241853   681 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.219738 27483 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 10 to 16. New cstate: current_term: 16 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:23.242551   989 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.243074   681 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.245071   989 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.245149  1013 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.199640 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:23.245798 27661 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:23.246696  1038 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.758s	user 0.003s	sys 0.000s
I20260709 12:20:23.246789   681 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.241264 27581 rpcz_store.cc:276] 0709 12:20:22.044307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.045348 (+  1041us) service_pool.cc:224] Handling call
0709 12:20:22.045643 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.530171 (+484528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.152627 (+622456us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:23.152798 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":948724608}
I20260709 12:20:23.247982 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2816) took 1200 ms. Trace:
I20260709 12:20:23.248921 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:23.250135 27730 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:23.131206 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.250795 27658 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:23.251416 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.203167   969 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:23.252156 27736 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:23.248327 27582 rpcz_store.cc:276] 0709 12:20:22.047724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.047869 (+   145us) service_pool.cc:224] Handling call
0709 12:20:22.048235 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:22.084469 (+ 36234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:22.084504 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:22.089230 (+  4726us) log.cc:844] Serialized 26 byte log entry
0709 12:20:22.279244 (+190014us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:20:22.279305 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:22.279425 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:23.246171 (+966746us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:23.247636 (+  1465us) raft_consensus.cc:1685] finished
0709 12:20:23.247660 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:23.247912 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35752,"raft.queue_time_us":40870,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":4955,"spinlock_wait_cycles":123735808,"thread_start_us":146866,"threads_started":1,"wal-append.queue_time_us":316549}
I20260709 12:20:23.252988 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.254422 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.255049 27730 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:23.256616  1077 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:23.257360   969 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:23.257699 27661 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:23.087498  1018 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:23.258441  1018 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:23.258584 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:23.259454  1018 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.260183 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.260753 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
W20260709 12:20:23.261132 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.213330  1039 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:23.240799   996 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.201769   968 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:23.257647 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:23.271449 27582 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:23.263054 28335 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:23.235592   935 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:23.278573 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 255548us
I20260709 12:20:23.279161 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.195278 27627 leader_election.cc:400] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22.
I20260709 12:20:23.280526 27627 leader_election.cc:403] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:23.241957 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:23.187106   881 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.272086 27738 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:23.250383 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:23.277734 27736 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:23.261965   996 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.285873   996 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.287034 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:23.287829   881 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:23.293825 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:23.273844 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.279907 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:23.283118 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.306280 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:23.309440 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.310842 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.312660 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.295677 28336 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
W20260709 12:20:23.312254 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 117161us
I20260709 12:20:23.318980 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:23.309507 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59209.7us
I20260709 12:20:23.323438 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:23.325443 27729 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:23.325335 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.303176 27660 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:23.338171 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.332063   681 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:23.341172   943 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:23.306123   883 raft_consensus.cc:2764] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:20:23.234076   764 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.558723   763 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:23.324821  1020 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22.
I20260709 12:20:23.737181 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.737686   763 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.738359 28330 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:23.738336   935 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:23.738989 28334 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:20:23.558794 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.457026 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.740442 28337 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
W20260709 12:20:23.694772 27731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.032s	sys 0.044s
I20260709 12:20:23.741711 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:23.676889 27663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.079s	sys 0.130s
I20260709 12:20:23.346318 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:23.744800   880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.327s	user 0.021s	sys 0.055s
I20260709 12:20:23.736294 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:23.746409 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:23.757527   880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.340s	user 0.021s	sys 0.056s
W20260709 12:20:23.744220 27663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.081s	sys 0.136s
I20260709 12:20:23.384351 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:23.580085 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 260888us
I20260709 12:20:23.760102 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2323) took 1372 ms. Trace:
I20260709 12:20:23.760973 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:23.325783   964 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.437s
I20260709 12:20:23.760875 27663 rpcz_store.cc:276] 0709 12:20:22.387425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.605651 (+218226us) service_pool.cc:224] Handling call
0709 12:20:22.629067 (+ 23416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.629243 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:22.932416 (+303173us) log.cc:844] Serialized 14 byte log entry
0709 12:20:23.759422 (+827006us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:23.759533 (+   111us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:23.759612 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.759815 (+   203us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:23.760010 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":234,"prepare.queue_time_us":75367,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"thread_start_us":1056880,"threads_started":2}
W20260709 12:20:23.675678  1025 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.692s	user 0.002s	sys 0.002s
I20260709 12:20:23.763918  1027 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:23.759479 27736 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 19 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:20:23.764382  1027 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:23.584970 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.764881 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.765496  1027 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.766386 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.767019 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.767498   943 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:23.767974 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.327862   968 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:23.768293 27663 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
W20260709 12:20:23.496260 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.490957 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.770354 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.577535   844 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:20:23.742005 27731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.032s	sys 0.044s
I20260709 12:20:23.772823 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:23.579560 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 299784us
I20260709 12:20:23.774633 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:23.736725 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.775983   883 raft_consensus.cc:2764] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:20:23.775300 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.776786 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.347043 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.757417 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.739660   764 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:23.788483   883 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:23.789237   883 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.768409   943 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:23.792251   883 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.782984 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2653) took 1984 ms. Trace:
I20260709 12:20:23.793524 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:23.682955   959 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.022s	sys 0.025s
I20260709 12:20:23.768776 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:23.777966 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:23.772280   844 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:23.335870   829 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.500s
I20260709 12:20:23.797487 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.805629 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.812934 27580 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [13.8-14.9]
I20260709 12:20:23.849159 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:23.813696 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.814654 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.814992 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.815316 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.816792 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.793939 27737 rpcz_store.cc:276] 0709 12:20:21.797874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.312291 (+514417us) service_pool.cc:224] Handling call
0709 12:20:22.312646 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.561004 (+248358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.632919 (+1071915us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:23.782543 (+149624us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272167168}
W20260709 12:20:23.822288 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61457.6us
I20260709 12:20:23.831247 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.832547 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:23.826531 27552 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.005s	sys 0.012s
W20260709 12:20:23.857779 27552 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.005s	sys 0.012s
W20260709 12:20:23.858137 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 771134us
I20260709 12:20:23.858247 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:23.858997 27582 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:20:23.839653 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:23.861271 27734 raft_consensus.cc:606] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:23.861315 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.861768 27734 raft_consensus.cc:614] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:20:23.796890 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.597000  1005 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.857168 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.863962 27589 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:23.845530 27735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.014s	sys 0.005s
W20260709 12:20:23.864729 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.014s	sys 0.005s
I20260709 12:20:23.864776 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.828455 28328 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:23.866536 27589 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:23.854900 27663 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:23.878353 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.007s	sys 0.034s
I20260709 12:20:23.865321 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:23.867578 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.805649 27729 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 19 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:20:23.880020 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:23.878839 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.007s	sys 0.034s
I20260709 12:20:23.881879 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:23.783404 27662 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:23.879812 27580 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:23.830842  1072 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:20:23.840520   829 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:23.887183   866 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:23.887687   829 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:23.887739   866 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:23.888644   866 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.888849   829 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.889883   866 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:23.867486 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:23.890858 27584 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:23.834349 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:23.895239 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61008.7us
I20260709 12:20:23.896241 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:23.895845 27483 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 10 to 18. New cstate: current_term: 18 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:23.898243 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:23.899022 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:23.820638   996 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:23.803399   959 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.024s	sys 0.025s
I20260709 12:20:23.899796 27733 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
W20260709 12:20:23.843060  1039 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.581s	user 0.011s	sys 0.000s
I20260709 12:20:23.809971   880 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:23.902379 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.828281  1020 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:23.912338 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.894397 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:23.901424 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.911242 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:23.905684 27585 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:23.819711   883 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:23.841388  1037 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:23.908565 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.907315 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:23.909793 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.000s	sys 0.016s
I20260709 12:20:23.889940   829 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:23.894361   845 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:23.912864 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:23.922657 27663 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:23.922894 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:23.923137 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:23.923799 27738 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:23.913803 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:23.908178 27730 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:23.914937 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:23.915501 27737 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:23.925062 27732 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:23.917200 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:23.926900 27658 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 15: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:23.903997   828 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.700s	user 0.012s	sys 0.000s
I20260709 12:20:23.884296   971 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:23.894018   897 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:23.921046 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.000s	sys 0.017s
I20260709 12:20:23.854943  1044 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:23.921056   681 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:23.932881 27659 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:23.807087   989 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.104s
I20260709 12:20:23.941956 32644 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:23.952188 32644 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.952615 32644 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:23.908257   950 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:23.899978  1040 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:20:23.943547 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:23.946110 27585 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:20:23.950677  1039 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:23.915015   883 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.912271  1020 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:23.916579   959 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:23.935500  1005 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:23.933398   845 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.933396   501 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:23.961748 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:23.907742   996 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:23.951494  1018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.027s	sys 0.018s
I20260709 12:20:23.917037   880 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.968521  1039 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.038703 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:23.933596  1037 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:24.040354 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53125.4us
I20260709 12:20:24.041569 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2670) took 1149 ms. Trace:
W20260709 12:20:23.990574  1018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.027s	sys 0.018s
I20260709 12:20:23.910298 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:23.978613   883 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:24.077961   996 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:24.072382 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:24.037463 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.042044 27731 rpcz_store.cc:276] 0709 12:20:22.892468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.958089 (+ 65621us) service_pool.cc:224] Handling call
0709 12:20:22.958434 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:22.967150 (+  8716us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:22.967181 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:22.979974 (+ 12793us) log.cc:844] Serialized 26 byte log entry
0709 12:20:23.773116 (+793142us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:20:23.773192 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.773315 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.041357 (+268042us) raft_consensus.cc:1685] finished
0709 12:20:24.041378 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.041508 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67,"raft.queue_time_us":3766,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":3655,"thread_start_us":752842,"threads_started":1,"wal-append.queue_time_us":930531}
I20260709 12:20:24.045783  1077 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:23.968807   950 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:23.939873   844 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:23.990724   562 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:24.083184 27589 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:24.082639 27483 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 6 to 8, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 8 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:23.951870  1044 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.087832 27585 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:24.040823  1039 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:24.088455 27483 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 6 to 11, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 11 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:24.047309   828 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:23.981231   763 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:23.983585   845 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:23.982551  1020 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:23.991140   959 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:24.065107 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 232) took 37149 ms (client timeout 15000 ms). Trace:
I20260709 12:20:24.066000 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:24.068102 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:24.069018  1033 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:24.071067 27584 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:24.072001 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:24.092470 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.033331 32644 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.176762 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:24.140184 27483 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 15, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.092265   865 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.264277 27483 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.265990 27483 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 19, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 19 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.273340 27483 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.275923   959 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.275333 27483 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 17. New cstate: current_term: 17 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.277192 27483 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 6 to 11, leader changed from <none> to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:24.278888 27483 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 14, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 14 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:24.082154  1077 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:24.078306  1051 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 12:20:24.149520   562 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.093995   828 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:24.149281   970 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:24.290108 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 197732us
W20260709 12:20:24.291244 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.258690   865 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.292351 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.269986   943 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:24.295532  1080 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:20:24.296743   943 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.300130  1044 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.300941  1044 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.301648  1044 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:24.311991  1084 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:20:24.313402 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 931) took 1278 ms. Trace:
I20260709 12:20:24.313784 27586 rpcz_store.cc:276] 0709 12:20:23.034808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.165774 (+130966us) service_pool.cc:224] Handling call
0709 12:20:23.166090 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:23.166426 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:23.166455 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:23.166914 (+   459us) log.cc:844] Serialized 26 byte log entry
0709 12:20:23.882070 (+715156us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:20:23.882105 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:23.882163 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.313169 (+431006us) raft_consensus.cc:1685] finished
0709 12:20:24.313192 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.313339 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1002,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":714840,"threads_started":1,"wal-append.queue_time_us":1008667}
I20260709 12:20:24.329617   865 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.329763   943 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.330330   943 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.290669   562 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.332659   943 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.333134   562 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.146842 27732 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:24.157903   763 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.334524   763 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.335747  1018 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:24.336321  1018 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:24.175901   968 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.337208  1018 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.337985   968 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.185950 28332 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:20:24.090062   935 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.340363   935 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.330346   950 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.178514   880 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:24.243909 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.037s	sys 0.081s
W20260709 12:20:24.342971 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.004s	user 0.038s	sys 0.082s
W20260709 12:20:24.245803 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.013s	sys 0.106s
W20260709 12:20:24.248258 27583 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.011s	user 0.000s	sys 0.021s
W20260709 12:20:24.343861 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.014s	sys 0.106s
W20260709 12:20:24.116989 27482 rpcz_store.cc:269] 0709 12:19:46.920194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:46.930656 (+ 10462us) service_pool.cc:224] Handling call
0709 12:20:24.064822 (+37134166us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:13.560068 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:13.561029 (+   961us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:13.561090 (+    61us) write_op.cc:435] Acquired schema lock
0709 12:20:13.561108 (+    18us) tablet.cc:662] Decoding operations
0709 12:20:13.563354 (+  2246us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:13.563506 (+   152us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:13.563550 (+    44us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:20:13.565703 (+  2153us) tablet.cc:701] Row locks acquired
0709 12:20:13.565770 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:20:13.565981 (+   211us) write_op.cc:270] Start()
0709 12:20:13.566152 (+   171us) write_op.cc:276] Timestamp: P: 1783599613565903 usec, L: 0
0709 12:20:13.566179 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:20:13.567029 (+   850us) log.cc:844] Serialized 5139 byte log entry
0709 12:20:14.136654 (+569625us) op_driver.cc:464] REPLICATION: finished
0709 12:20:14.766946 (+630292us) write_op.cc:301] APPLY: starting
0709 12:20:14.767048 (+   102us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:15.034256 (+267208us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:15.080369 (+ 46113us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:15.921178 (+840809us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:15.921544 (+   366us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 12:20:15.981350 (+ 59806us) write_op.cc:312] APPLY: finished
0709 12:20:16.027884 (+ 46534us) log.cc:844] Serialized 121 byte log entry
0709 12:20:16.081481 (+ 53597us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:16.188174 (+106693us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:16.268461 (+ 80287us) write_op.cc:454] Released schema lock
0709 12:20:16.268616 (+   155us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:16.269135 (+   519us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:16.269163 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":327305,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":630026,"apply.run_cpu_time_us":54940,"apply.run_wall_time_us":1502120,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58467,"num_ops":13,"prepare.queue_time_us":279330,"prepare.run_cpu_time_us":32686,"prepare.run_wall_time_us":82417,"raft.queue_time_us":466221,"raft.run_cpu_time_us":50585,"raft.run_wall_time_us":482208,"replication_time_us":570263,"spinlock_wait_cycles":15710976,"thread_start_us":1096574,"threads_started":5,"wal-append.queue_time_us":162736,"wal-append.run_cpu_time_us":30218,"wal-append.run_wall_time_us":2390243}]]}
I20260709 12:20:24.285844 32644 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:24.189304 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.287711 27483 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 4 to 9, term changed from 11 to 13, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 13 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.330404   970 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.290462   828 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:20:24.254083 27664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.033s	sys 0.025s
W20260709 12:20:24.354046 27664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.011s	user 0.033s	sys 0.025s
W20260709 12:20:24.344307 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 986292us
W20260709 12:20:24.344157 27583 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.107s	user 0.000s	sys 0.021s
I20260709 12:20:24.293977 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:24.360942 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67410us
I20260709 12:20:24.164902 27661 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
W20260709 12:20:24.347303 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 279296us
I20260709 12:20:24.347980  1005 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:24.366716 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.359288  1072 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:24.377460 27661 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:20:24.372362 27483 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.347988   763 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:24.378999 27483 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 12 to 15, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:24.380724 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:24.381222 27661 pending_rounds.cc:85] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 9
I20260709 12:20:24.369860   501 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:24.364315 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.376031 27663 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:20:24.381150 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:24.383819 27661 pending_rounds.cc:107] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:20:24.384163 27661 raft_consensus.cc:2889] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:24.293090   996 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:24.349390   562 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.359654   866 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.363760  1033 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.396148 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.396839 27729 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:24.396287  1033 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.397500   825 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:24.385461   501 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.398792   501 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:24.399155  1033 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.386309   829 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:24.377084 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:24.400259 27659 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:20:24.401310   829 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:24.388239 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:24.389863 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:24.403442 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.396065 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.403542   829 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.404491 27730 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:24.410105   501 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.394701 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:24.374601  1005 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.396075   866 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.413715  1005 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:24.415192   866 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.415439   970 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.386123   968 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.359452   452 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:24.386168 27483 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 10 to 12, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:24.417228   968 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:24.421753  1005 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.392336 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:24.424731  1013 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:24.425136  1013 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:24.425496  1013 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:24.428428   897 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:24.381677   935 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.396353 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:24.439818   935 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.377090   681 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.442137   935 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:24.446349 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.010s	sys 0.009s
W20260709 12:20:24.447142 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.450883 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:24.451946 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.010s	sys 0.009s
W20260709 12:20:24.453117 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.007s	sys 0.007s
W20260709 12:20:24.457388 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.007s	sys 0.007s
W20260709 12:20:24.458236 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 562041us
W20260709 12:20:24.463219 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.005s	sys 0.015s
I20260709 12:20:24.360628  1039 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.441272   562 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.444734  1044 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.475378 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.476043 27586 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:24.502188 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:24.502990 27582 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:24.505993 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.506713 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:20:24.508416 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.514048 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.509222 27580 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:24.471508   970 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.439141  1077 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.524255  1095 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.524549   970 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.525039  1095 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.526703   970 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.527485  1090 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:24.530834   382 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.531404   382 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.533648   382 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:24.523458   971 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.581s	user 0.011s	sys 0.000s
I20260709 12:20:24.539459   971 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:24.539511   828 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.446542   681 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.469293 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:24.472187 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:24.589975 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2329) took 1006 ms. Trace:
I20260709 12:20:24.542538 27582 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
W20260709 12:20:24.592391 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 123207us
I20260709 12:20:24.593061 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:20:24.469173 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.005s	sys 0.015s
W20260709 12:20:24.593940 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.590351 27655 rpcz_store.cc:276] 0709 12:20:23.592372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.593769 (+  1397us) service_pool.cc:224] Handling call
0709 12:20:24.472042 (+878273us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:20:24.589911 (+117869us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45365504}
W20260709 12:20:24.471184 27587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.007s	sys 0.015s
W20260709 12:20:24.599148 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.007s	sys 0.015s
I20260709 12:20:24.607440 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.595351 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:24.609966  1039 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.611030   968 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:24.611531   968 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.468184  1005 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:24.611917   968 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:24.612336  1005 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:24.613296   955 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:24.616691  1039 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.617250  1039 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.619207  1039 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.620204 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:24.620965 27655 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:24.622153 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:24.624899 27655 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:24.626538 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:24.536720  1095 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.627703 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:24.628410 27659 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:24.631392 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:24.631738  1072 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:24.632139  1072 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.632385  1039 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:24.632475  1072 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:24.632573 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2784) took 1508 ms. Trace:
I20260709 12:20:24.527271   970 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.633194 27656 rpcz_store.cc:276] 0709 12:20:23.124175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.124294 (+   119us) service_pool.cc:224] Handling call
0709 12:20:23.124708 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:23.279168 (+154460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:23.279197 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:23.324947 (+ 45750us) log.cc:844] Serialized 52 byte log entry
0709 12:20:24.343520 (+1018573us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:24.352494 (+  8974us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.352907 (+   413us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.353114 (+   207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.353218 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.632331 (+279113us) raft_consensus.cc:1685] finished
0709 12:20:24.632356 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.632513 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153992,"prepare.queue_time_us":50563,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":7439,"raft.queue_time_us":182910,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":17946,"spinlock_wait_cycles":17440640,"thread_start_us":1004750,"threads_started":1,"wal-append.queue_time_us":1289019}
I20260709 12:20:24.633678 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:24.641431 27735 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.009s	sys 0.015s
W20260709 12:20:24.641825 27735 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.009s	sys 0.015s
I20260709 12:20:24.645579 27656 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:24.654906 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:24.655704 27656 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:24.655417  1044 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:24.658694   955 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:24.661633 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:24.659469 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2672) took 1754 ms. Trace:
W20260709 12:20:24.708395 28335 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.008s	sys 0.019s
W20260709 12:20:24.709043 28335 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.008s	sys 0.019s
I20260709 12:20:24.706673 27735 rpcz_store.cc:276] 0709 12:20:22.905268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.967569 (+ 62301us) service_pool.cc:224] Handling call
0709 12:20:23.012798 (+ 45229us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:23.013189 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:23.013218 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:23.028588 (+ 15370us) log.cc:844] Serialized 52 byte log entry
0709 12:20:23.865194 (+836606us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:24.642194 (+777000us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.642528 (+   334us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.659084 (+ 16556us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.659164 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.659210 (+    46us) raft_consensus.cc:1685] finished
0709 12:20:24.659224 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.659397 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15740,"raft.queue_time_us":7577,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":8677,"thread_start_us":1488324,"threads_started":2,"wal-append.queue_time_us":1113639}
I20260709 12:20:24.636216   971 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:24.707508   883 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.002s
I20260709 12:20:24.669932   828 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.710175   763 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:24.731444   971 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.673746   764 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:20:24.727761   382 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.714116  1040 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.721709  1095 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.709299  1044 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.713147  1039 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.748311   935 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.749002   971 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.749378   562 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.660621   970 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.741102   763 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.749891   764 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:24.738288   883 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:24.777994 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.009s	sys 0.009s
W20260709 12:20:24.778404 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.009s	sys 0.010s
I20260709 12:20:24.782354 27663 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:24.772351   763 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:24.739116  1072 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:24.783610 27657 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.003s	sys 0.016s
I20260709 12:20:24.770273   764 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:24.786181 27657 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.003s	sys 0.016s
I20260709 12:20:24.755924  1040 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.753055   382 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:24.763211  1039 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.795492 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:24.796360 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:24.796682 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:24.798226  1070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.004s	sys 0.015s
I20260709 12:20:24.800292 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:24.799710 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 368314us
I20260709 12:20:24.801165 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:24.762367  1095 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.762598  1044 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:24.807498 27736 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.001s	sys 0.019s
I20260709 12:20:24.808651 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:24.809676 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.012s	sys 0.005s
W20260709 12:20:24.811067  1070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.005s	sys 0.015s
I20260709 12:20:24.819073 28332 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:24.813370   883 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.814222  1072 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:24.827682 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:24.814605 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:24.810671 28328 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:24.818320 27656 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:20:24.818889 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:24.820619 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.012s	sys 0.005s
I20260709 12:20:24.820864 27733 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:20:24.821031 28336 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:20:24.871687 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 278320us
W20260709 12:20:24.792752 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 181837us
I20260709 12:20:24.824474 27662 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:20:24.825040 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:24.828301 28334 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:24.887887 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.882754 27730 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:24.884106 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.890429 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:24.821480 27736 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.001s	sys 0.019s
I20260709 12:20:24.885182 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:24.878003 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:24.892445 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77974.4us
I20260709 12:20:24.889611 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.817446  1018 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:24.890587 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
W20260709 12:20:24.894012 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 300138us
I20260709 12:20:24.894558 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.895282 27587 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:24.833246 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:24.896509 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 883) took 1113 ms. Trace:
I20260709 12:20:24.897867 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.898285 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:24.903663 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:24.903733 27734 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:24.906261 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.906776 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:24.906855 27733 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:24.907716 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.911983 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.912694 27738 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:24.915242  1070 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:24.915700  1070 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:24.916101  1070 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:24.918167 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:24.918486   844 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:20:24.919925  1093 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:20:24.920370  1033 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:24.920913  1033 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.921382  1033 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:24.922109 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.922691 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2785) took 1798 ms. Trace:
I20260709 12:20:24.923148 27657 rpcz_store.cc:276] 0709 12:20:23.124520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.124987 (+   467us) service_pool.cc:224] Handling call
0709 12:20:23.125392 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:23.484149 (+358757us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:20:23.801664 (+317515us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.787947 (+986283us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:24.790783 (+  2836us) log.cc:844] Serialized 26 byte log entry
0709 12:20:24.790997 (+   214us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:20:24.791046 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.791139 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.922438 (+131299us) raft_consensus.cc:1685] finished
0709 12:20:24.922464 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.922623 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":344763,"prepare.queue_time_us":316878,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":392277,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":134357,"thread_start_us":985290,"threads_started":2,"wal-append.queue_time_us":905191}
I20260709 12:20:24.925618  1051 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.926172  1051 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.928624  1051 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:24.822896   681 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:24.955987 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
W20260709 12:20:24.957469 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54027.6us
I20260709 12:20:24.959893 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:24.963809 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52468.4us
I20260709 12:20:24.969506 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.970202 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.970392 27732 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:24.972100 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.902297 27736 rpcz_store.cc:276] 0709 12:20:23.783135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.783415 (+   280us) service_pool.cc:224] Handling call
0709 12:20:23.783843 (+   428us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:23.784644 (+   801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:23.784679 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:23.785452 (+   773us) log.cc:844] Serialized 52 byte log entry
0709 12:20:24.280425 (+494973us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:20:24.891836 (+611411us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.896146 (+  4310us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.896257 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.896316 (+    59us) raft_consensus.cc:1685] finished
0709 12:20:24.896328 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.896457 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7231,"prepare.queue_time_us":527032,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":10228,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":1094925,"threads_started":2,"wal-append.queue_time_us":880936}
I20260709 12:20:24.977367 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:24.980230   828 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:24.982898   681 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:24.985019   681 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:24.981609   828 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:24.988741   881 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:24.989547 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.989481   881 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:24.990252 27729 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:24.990967 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2817) took 2943 ms. Trace:
I20260709 12:20:24.991302 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:24.992043 27735 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:24.991461 27583 rpcz_store.cc:276] 0709 12:20:22.048056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.048199 (+   143us) service_pool.cc:224] Handling call
0709 12:20:22.092570 (+ 44371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:22.092839 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:22.117087 (+ 24248us) log.cc:844] Serialized 14 byte log entry
0709 12:20:22.834102 (+717015us) spinlock_profiling.cc:243] Waited 9.76 ms on lock 0x7b180016e4c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:23.236937 (+402835us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b180016e4c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:20:24.797622 (+1560685us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.988501 (+190879us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:24.988556 (+    55us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:24.988609 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.990794 (+  2185us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:24.990914 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9762,"prepare.queue_time_us":282066,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":32929,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":4603,"spinlock_wait_cycles":35694720,"thread_start_us":1744259,"threads_started":4}
I20260709 12:20:24.992101 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.900367   883 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:24.997565 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:24.826001  1111 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.178s
I20260709 12:20:24.999609   562 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:25.031296 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2844) took 1060 ms. Trace:
I20260709 12:20:25.031734 27585 rpcz_store.cc:276] 0709 12:20:23.971287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.973525 (+  2238us) service_pool.cc:224] Handling call
0709 12:20:23.973951 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:24.088216 (+114265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:25.031032 (+942816us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:25.031241 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261128576}
I20260709 12:20:24.990474  1109 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:25.032874 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.904527  1036 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.117s
W20260709 12:20:25.034408 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74677.4us
W20260709 12:20:25.025293  1084 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:24.999526   897 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:20:25.030530  1104 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:20:25.034099   845 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.885s	user 0.011s	sys 0.001s
I20260709 12:20:25.006592   971 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:20:25.027796  1092 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:20:25.053479 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:24.995343   844 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:25.028985   562 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.057315 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 895) took 1022 ms. Trace:
I20260709 12:20:25.036553  1102 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.063704  1115 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.064284  1115 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.059494   844 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.041610   897 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.067626   897 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:25.068622   844 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:25.051106 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84199.2us
I20260709 12:20:25.042330 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:24.948031  1018 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.071559   944 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.901806   764 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.080756 28334 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:25.058470 28335 rpcz_store.cc:276] 0709 12:20:24.035098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:24.035279 (+   181us) service_pool.cc:224] Handling call
0709 12:20:24.047663 (+ 12384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:24.047881 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:20:24.709544 (+661663us) log.cc:844] Serialized 14 byte log entry
0709 12:20:25.056937 (+347393us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:25.056987 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:20:25.057044 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:25.057111 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:25.057250 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":764718,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":71663,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":5169,"thread_start_us":1008170,"threads_started":3}
I20260709 12:20:25.043794  1109 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:25.053567   971 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.062027   562 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:25.067184  1115 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:25.088774 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51269.5us
I20260709 12:20:25.060078   970 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.090286 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:25.061575  1122 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:25.080263  1051 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:25.056454   845 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.096959 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:25.109120  1111 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:25.110113  1111 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:25.110158  1095 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.110615  1111 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:25.102994  1120 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:25.111758  1120 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:25.112147  1120 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:25.113669  1037 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.115314 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:25.091549   971 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:25.121071 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:25.085160   944 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.098414   764 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:25.101833  1122 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:25.127224 27552 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:25.128573 27552 leader_election.cc:403] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:25.118743  1039 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.117259  1051 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:25.117398   382 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.121591  1101 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:25.081002  1102 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.127839  1117 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:20:25.155598  1119 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:20:25.154960 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:25.162768 27589 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:25.187698 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72492.3us
I20260709 12:20:25.188411 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:25.193347  1027 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:25.211526  1013 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.000s
I20260709 12:20:25.211719  1117 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:25.211611 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:25.212883 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:25.213308 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:25.219774 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:25.182042  1115 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:25.225664 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:25.222448 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:25.227154  1018 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.212133  1013 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.229375 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:25.218014  1098 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:25.231839 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:25.233588 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:25.236076 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.243432 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:25.230137 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.244097 27738 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:25.212138   897 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.222101  1125 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:25.248370  1143 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:25.249433 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.241307 27732 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:25.248893 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:25.241253  1107 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.180s	user 0.001s	sys 0.001s
I20260709 12:20:25.255366 27586 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:25.242178  1105 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:25.235117  1036 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:20:25.256434 27553 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:25.303428 27553 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
W20260709 12:20:25.295655 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62127.5us
I20260709 12:20:25.243945  1018 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:25.221120  1130 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:20:25.299713 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:25.304548 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:25.307117 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58345.8us
I20260709 12:20:25.320374 27586 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:25.327301 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:25.309198  1036 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:25.284252  1105 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.232461  1051 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.306079   935 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:25.339511  1115 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:20:25.325487  1020 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.176s	user 0.014s	sys 0.000s
I20260709 12:20:25.340710  1020 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.342978  1020 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.323937  1138 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.338521   897 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.339584   935 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.326635  1125 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:25.313778   382 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:25.328785  1140 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.126s	user 0.000s	sys 0.003s
W20260709 12:20:25.358913 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 169317us
I20260709 12:20:25.358940 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:25.329892  1132 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:20:25.324339   844 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:25.324532   955 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:25.345964  1101 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:25.350907  1105 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:25.149657  1097 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:25.333113  1044 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.189827  1122 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:25.375571 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 119271us
I20260709 12:20:25.405112 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.362252  1136 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.362504   935 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:25.417484   968 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.805s	user 0.007s	sys 0.006s
W20260709 12:20:25.417649 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:25.424099  1147 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:25.460620 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:25.461318 27734 raft_consensus.cc:606] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:25.128145  1111 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.372808   382 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.430702  1125 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.431250  1037 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:25.368382   944 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.432411 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.306433  1117 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.451922 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:25.371620   844 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.471293   844 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:25.472280  1020 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:25.472741  1020 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.473106  1020 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:25.475477  1044 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.475718   880 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:25.476099  1044 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.478768  1044 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.480193  1147 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:25.480618  1147 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:25.481520   971 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.481467  1147 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.482959  1147 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:25.483809 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51520.2us
I20260709 12:20:25.484659 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.486021 27552 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:25.486364  1020 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.486891 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.351517  1131 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:20:25.492228  1111 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:25.493485  1131 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:25.487654  1101 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:25.495952  1101 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:25.136245  1109 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.497310 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:25.498168  1101 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:25.498566  1101 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:25.498958  1101 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:25.504590 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.416109 27589 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:25.488731  1020 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.506431 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.512248 27589 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:25.513902 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:25.514926  1013 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.487898   970 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:25.529384 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:25.530068 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.530859 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:25.530438 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.530900 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.532023 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:25.515264   382 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.533080 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.507634 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.517050   501 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:25.517808  1070 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.601s	user 0.002s	sys 0.008s
I20260709 12:20:25.362179  1138 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.518551  1115 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:25.546444  1115 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:25.546929  1115 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:25.548147  1138 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.509770  1125 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:25.520292   763 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.730s	user 0.010s	sys 0.001s
I20260709 12:20:25.522183  1040 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.505901  1102 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.551492  1125 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.551620   763 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.458856  1136 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.462138  1122 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:25.375000  1033 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.409126   955 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.464972   968 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.570782  1143 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.571367  1143 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.572610 28334 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:25.569664  1136 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.573823  1143 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.574501  1115 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:25.539681 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:25.540234 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.544307 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.541524  1097 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:25.545678 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2794) took 1172 ms. Trace:
I20260709 12:20:25.584705 27737 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:25.553340 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:25.584637 27662 rpcz_store.cc:276] 0709 12:20:24.372791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:24.372899 (+   108us) service_pool.cc:224] Handling call
0709 12:20:24.373206 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:24.891093 (+517887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:25.545319 (+654226us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54000e42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:25.545586 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1033998080}
I20260709 12:20:25.585353 27663 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:25.556722 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:25.557085 28333 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:25.558507 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:25.513525   950 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:25.559597 27661 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:25.556471   382 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.568706 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:25.569623   955 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.571723   968 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.531510 28337 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:25.577625 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:25.595430 27736 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:25.595855 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:25.580933 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.583386 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.598738 27730 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:25.599756  1136 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.601693  1136 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.603904 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.605149 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.605363  1152 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:25.605880  1152 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.606770 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.608107  1152 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.609331 27730 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:25.611939 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:25.594873   382 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.618168  1115 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.618537  1077 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:25.618906   382 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:25.620213   562 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.522s	user 0.009s	sys 0.002s
I20260709 12:20:25.620723   562 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:25.632043   883 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.611s	user 0.009s	sys 0.004s
I20260709 12:20:25.552736  1070 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.623040   562 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.600309   971 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.611240   944 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:25.632593   883 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:25.614804   968 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:25.633236   764 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:20:25.638656  1085 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.368s	user 0.005s	sys 0.010s
I20260709 12:20:25.635129  1033 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.637035  1027 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 12:20:25.646278   881 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.766314 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.515311 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.561368 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:25.767937 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 151122us
I20260709 12:20:25.586112 27655 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:25.768998 27589 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 19 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:20:25.526216 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2322) took 3148 ms. Trace:
I20260709 12:20:25.769438 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:25.770596 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.587157 27657 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:25.581131 27659 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:25.741453 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:25.744220 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.747135 28329 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:25.747565 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:25.753645 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.756084 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.639860   681 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.639556   763 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.761668   764 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.766464   996 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:25.770082 27664 rpcz_store.cc:276] 0709 12:20:22.377858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:22.593079 (+215221us) service_pool.cc:224] Handling call
0709 12:20:23.159941 (+566862us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:23.341682 (+181741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:23.341722 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:23.342322 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 12:20:24.354503 (+1012181us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:24.354541 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.354634 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:24.633809 (+279175us) raft_consensus.cc:1685] finished
0709 12:20:24.633832 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:25.506642 (+872810us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54000e6fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:25.526112 (+ 19470us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":181273,"raft.queue_time_us":231785,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":17562,"spinlock_wait_cycles":1302927104,"thread_start_us":1011789,"threads_started":1,"wal-append.queue_time_us":1288108,"wal-append.run_cpu_time_us":1746,"wal-append.run_wall_time_us":1001755}
W20260709 12:20:25.769026 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 207574us
I20260709 12:20:25.719012 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2812) took 2694 ms. Trace:
I20260709 12:20:25.535827 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.661543  1072 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:20:25.657150  1120 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.545s	user 0.006s	sys 0.005s
I20260709 12:20:25.813305  1120 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.674537  1105 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.663807  1136 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.820124 27732 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:20:25.820715 27732 pending_rounds.cc:85] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:20:25.821010 27732 pending_rounds.cc:107] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:20:25.821470 27732 raft_consensus.cc:2889] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:25.821049  1101 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.734469 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:25.789472 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:25.824115 27738 raft_consensus.cc:606] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:25.824900 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.638010   935 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.825376  1147 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:25.826318 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:25.826453  1152 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:25.827723  1111 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
W20260709 12:20:25.807160 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 271269us
I20260709 12:20:25.831579   944 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:25.832130   944 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:25.832599   944 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:25.832896  1105 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.805708 27734 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:25.807394 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69928us
I20260709 12:20:25.808211  1102 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:25.810034 27733 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:25.835953  1102 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.836463  1102 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:25.837499  1070 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.838584 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:25.828092   935 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.795253 27588 rpcz_store.cc:276] 0709 12:20:23.024315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:23.154260 (+129945us) service_pool.cc:224] Handling call
0709 12:20:23.154626 (+   366us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:23.155004 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:23.155034 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:23.182333 (+ 27299us) log.cc:844] Serialized 52 byte log entry
0709 12:20:23.936033 (+753700us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:24.255263 (+319230us) log.cc:844] Serialized 14 byte log entry
0709 12:20:24.259387 (+  4124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:24.259461 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:25.636792 (+1377331us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:25.718108 (+ 81316us) raft_consensus.cc:1685] finished
0709 12:20:25.718325 (+   217us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:25.718729 (+   404us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":691492,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":161098,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"spinlock_wait_cycles":12931328,"thread_start_us":1049309,"threads_started":2,"wal-append.queue_time_us":1061739}
I20260709 12:20:25.806604   764 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.809899  1072 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:25.840166 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43875.3us
I20260709 12:20:25.842921 27586 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:25.825392  1077 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:25.844173 27580 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:25.844712  1077 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.845489   764 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:25.847548  1136 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:25.847357  1072 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.849483  1072 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:25.854844  1036 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.855399  1120 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.855633  1036 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.858593  1036 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.858603 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:25.861250  1133 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.862162  1133 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:25.863579  1156 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.055s	user 0.000s	sys 0.002s
I20260709 12:20:25.865342 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:25.865497 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:25.872471 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:25.874380 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.854233  1077 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.875214  1040 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:25.875727  1101 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:25.875897 27582 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:20:25.876309  1040 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:25.877898  1040 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.879065  1152 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:20:25.879940 27583 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:25.880291  1115 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:25.881162  1115 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:25.881728 27583 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:25.882591  1115 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.886803  1115 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:25.871392 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.891937 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:25.899427  1101 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:25.899855 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:25.901077  1105 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:25.901309  1039 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:25.902006  1039 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:25.905124   845 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:25.901047  1070 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:25.907234 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:25.907457  1070 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:25.907927  1070 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:25.909055  1070 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:25.910346 27587 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:20:25.912302  1039 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:25.912786 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.865314  1133 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:25.912972 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72346.5us
I20260709 12:20:25.914297 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.912357 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.915982 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.916656  1039 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:25.920653  1105 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:25.917378 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:25.926854 28329 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:25.916018 27738 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:25.916858 27735 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:25.928011 27583 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:25.943676 27734 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:25.945237 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:25.946313 27737 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:20:25.949177 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.958134 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:25.961134 27580 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:25.962502 27580 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:25.910385  1070 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:25.970345 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 901) took 1174 ms. Trace:
I20260709 12:20:25.978766 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:25.970930 28329 rpcz_store.cc:276] 0709 12:20:24.796336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:24.796455 (+   119us) service_pool.cc:224] Handling call
0709 12:20:25.970015 (+1173560us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:25.970221 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2157137280}
I20260709 12:20:25.975075   681 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:25.988927 27586 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:25.989859 27663 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:25.995719 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63860.6us
I20260709 12:20:26.011523 27737 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:26.009819  1101 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:26.013863 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.018352 27738 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:26.018378 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:26.021449 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:26.026911 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.026615 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.031571 27586 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:26.042778 27583 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:26.044961 27584 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:26.045881 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2828) took 1154 ms. Trace:
I20260709 12:20:26.046263 27584 rpcz_store.cc:276] 0709 12:20:24.892652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.027386 (+1134734us) service_pool.cc:224] Handling call
0709 12:20:26.035243 (+  7857us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:26.045546 (+ 10303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:26.045807 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:26.046931 27585 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:26.050146 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.051585 27738 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:26.051923 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 134770us
I20260709 12:20:26.042507  1072 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:26.057066 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.059629 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:26.056156 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:26.064754 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.071105 27737 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:26.072649  1133 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:26.073355  1133 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.075834 28335 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:26.076248 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.076792 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.077584 27733 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:26.079607 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.077615 27588 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:26.082630 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.041942  1102 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.080907 27553 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:26.086930 27553 leader_election.cc:403] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:26.069576 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.066864  1117 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:26.092909 27584 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:26.093418 27738 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:26.093922 27735 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:26.095428 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:26.095067 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:26.099393 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:26.099407 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.102175 27730 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:26.101871 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:26.105435 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.113066 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.113269 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:26.112244 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96147us
I20260709 12:20:26.131563 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:26.156329 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:26.168273 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:26.172825 27664 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:26.172879 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:26.124166   828 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:20:26.175731 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:26.176381 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.105523  1072 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.184697 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:26.192288 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:26.177170 27663 raft_consensus.cc:606] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:26.196578 27663 raft_consensus.cc:614] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:26.195474 27733 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:26.198346  1143 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:26.196189 27731 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:26.203725 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.194200  1136 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:20:26.193867   844 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.722s	user 0.017s	sys 0.010s
I20260709 12:20:26.213852  1005 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:26.213979 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:26.216831 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 133423us
I20260709 12:20:26.214919 27655 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:26.183903   944 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.220520 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:26.223230 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:26.221992 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.227577  1040 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:26.230765 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:26.221998  1105 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:26.235392  1105 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.236495 27660 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:20:26.237262 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132363us
I20260709 12:20:26.237936  1105 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:26.238142 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.239059  1136 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:26.239526  1136 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.239971  1136 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:26.240221  1163 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:20:26.215163  1044 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.242081  1044 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.244160 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:26.244679  1044 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:26.245478  1044 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:26.245623  1040 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:26.246110  1044 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.212569  1143 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:26.228837 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:26.235850  1072 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:26.248147 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.206912   764 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 12:20:26.259646 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.225135   844 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.264436 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.265350 27660 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:26.265816   844 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:26.266420 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:26.266499 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.267572 27660 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:26.267982 27664 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:26.268486 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.269348 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.270213 27664 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:26.257880 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:26.276609  1164 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:20:26.276449 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.250878  1133 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:26.281049 27731 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:26.260254  1102 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:26.285656 27657 raft_consensus.cc:1813] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.282286 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:26.283804 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.292810   959 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:20:26.293071  1143 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:26.293416   959 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:26.293970   959 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:26.283032  1072 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:26.295960   844 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:26.296619   844 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:26.297201   844 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:26.294932 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.300805 27657 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:26.256311  1105 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:26.307272 27583 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:26.311390 27661 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 23 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:20:26.308387  1147 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
W20260709 12:20:26.313215 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46847.9us
I20260709 12:20:26.316712  1142 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:20:26.319916 27659 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:26.321563 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.323597  1105 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:26.319053 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67113.6us
I20260709 12:20:26.326046  1105 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:26.326644 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65451.4us
I20260709 12:20:26.287358  1115 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:20:26.328007 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.329484  1101 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:26.330286 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:26.331457  1101 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:26.331296  1036 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:26.332227  1101 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:26.332422  1036 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:26.332808  1036 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:26.333868  1036 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.334661 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.334743 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.335796 27734 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:26.333555  1101 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.339461 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.341198 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.342654 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.347884 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:26.355741 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.357368 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.357697 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:26.360085 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.360013 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.363706 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.364532 27737 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:26.365541 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.366529 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.366559 27735 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:20:26.368986 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.363945 27733 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 12:20:26.372233   845 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:26.372846   845 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.373278 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.373937   845 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.374616 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.375820 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.376668 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.377166  1102 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:26.377764  1102 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.309548  1072 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.379878 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.381408  1033 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:26.382117  1033 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.382493 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:26.382913 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67809.5us
I20260709 12:20:26.383622  1039 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:26.383912   935 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:26.384421  1039 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.384555   935 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.385566  1039 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.387081 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.387715 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:26.389081 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.375906 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.369060 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:26.390247 27584 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:26.377717 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.390995 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.392405 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.393064 27580 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:26.379034  1102 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.390429 28337 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:26.380574  1105 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:26.396790 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.385675   935 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.400393 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.403041 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.408672 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.383260  1033 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.335314  1036 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:26.413235 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49881.2us
I20260709 12:20:26.419209 27580 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:26.380502  1072 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:26.420300 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92436.7us
I20260709 12:20:26.420485 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.420692 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:26.421429 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 126661us
I20260709 12:20:26.422070 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.422492 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.422885 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.426759 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.416071  1152 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:26.430413 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:26.419879 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 136175us
I20260709 12:20:26.421330  1122 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:26.261780  1044 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.428040 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.420365   764 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:26.434541 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:26.456912 28335 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:26.452291   828 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:26.433151  1122 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.461848 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.463691 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:26.464634 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.464956 27663 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:26.466953 27737 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:26.437178  1152 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:26.470414 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.452533  1133 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 12:20:26.472867 27583 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:26.461826  1040 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:20:26.479598 27733 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:20:26.484761 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.476119 27661 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:26.464617  1122 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:26.470099   828 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:26.489881 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.498517 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:26.482821  1070 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:20:26.515668 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.521806  1040 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:26.522318  1040 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:26.495247  1168 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:20:26.500830   828 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.478317  1136 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.482600   681 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:20:26.542228 27661 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:26.550272 27704 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:26.552160 27663 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:20:26.532933   944 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:26.529845  1070 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:26.512511  1142 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:20:26.565023  1111 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:26.569707 27583 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:26.570381 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.571789 27737 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53816
I20260709 12:20:26.576309 27731 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:26.549451  1077 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:26.575225 27737 raft_consensus.cc:606] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Received request to transfer leadership
I20260709 12:20:26.587304 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:26.587325 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:26.574625  1117 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:26.647807 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53505.4us
I20260709 12:20:26.596246 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:26.646940 27661 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:26.593850 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:26.650416 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 162704us
I20260709 12:20:26.585841  1077 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:26.528043  1040 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.663230 27656 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:26.656556   764 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:26.679378 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 215894us
I20260709 12:20:26.690380 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.580422  1115 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:20:26.593552   996 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
I20260709 12:20:26.666051  1077 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:26.698366 27661 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:20:26.590337  1152 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.696781 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:20:26.723712 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 125) took 19162 ms (client timeout 15000 ms). Trace:
W20260709 12:20:26.733017 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 271743us
I20260709 12:20:26.734877 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.740571 27661 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:26.720150  1027 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:26.722329  1040 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:26.758292 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 98381.4us
I20260709 12:20:26.700021  1039 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:20:26.740195 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 81e6b2dabf8f4e6cb549ca250430e568 with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53816
I20260709 12:20:26.718254   959 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:26.766198 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.747989  1027 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.769701 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.775671 27485 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 6 to 12, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:26.778386  1188 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:20:26.780010  1170 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:20:26.781252 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:26.782526 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:26.771992  1027 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:26.784961   681 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.785465   681 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.788475  1143 raft_consensus.cc:993] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:26.789050  1143 raft_consensus.cc:1081] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:26.779206 27485 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 14, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 14 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:26.794791   681 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:26.797220  1101 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:26.797689  1101 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:26.721877  1117 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.720554  1070 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.801433 27485 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 5 to 8, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:26.762938   845 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:26.814951 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65360.9us
I20260709 12:20:26.793668  1136 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:26.823057 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88306.1us
I20260709 12:20:26.822198 27660 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:26.768219  1142 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:26.827118 27581 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:20:26.876688 27738 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:26.878453 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.880765 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.875919 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.885434 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.886199 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.798812  1101 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.893028 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:26.893651 27661 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:26.829355 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.828900 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.826892  1005 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.906095 27736 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:20:26.907584 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.913089 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:26.919620 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:26.919713 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:26.922513 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 232303us
I20260709 12:20:26.800664   764 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.925392 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:26.930081 27552 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:26.807915  1122 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.827095   845 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.899417  1136 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.910959  1105 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:20:26.939900  1175 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
W20260709 12:20:26.732035 27483 rpcz_store.cc:269] 0709 12:20:07.488634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:07.488888 (+   254us) service_pool.cc:224] Handling call
0709 12:20:26.597546 (+19108658us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:24.795639 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:24.796281 (+   642us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:24.796351 (+    70us) write_op.cc:435] Acquired schema lock
0709 12:20:24.796371 (+    20us) tablet.cc:662] Decoding operations
0709 12:20:24.797970 (+  1599us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:24.798099 (+   129us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:24.798126 (+    27us) tablet.cc:685] Acquiring locks for 18 operations
0709 12:20:24.943726 (+145600us) tablet.cc:701] Row locks acquired
0709 12:20:24.943800 (+    74us) write_op.cc:260] PREPARE: finished
0709 12:20:24.944003 (+   203us) write_op.cc:270] Start()
0709 12:20:24.944191 (+   188us) write_op.cc:276] Timestamp: P: 1783599624943922 usec, L: 0
0709 12:20:24.944219 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:20:24.945152 (+   933us) log.cc:844] Serialized 7056 byte log entry
0709 12:20:25.191575 (+246423us) op_driver.cc:464] REPLICATION: finished
0709 12:20:25.366921 (+175346us) write_op.cc:301] APPLY: starting
0709 12:20:25.406512 (+ 39591us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:25.656873 (+250361us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:25.659931 (+  3058us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:25.674434 (+ 14503us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:25.674630 (+   196us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=36,mrs_lookups=0
0709 12:20:25.674661 (+    31us) write_op.cc:312] APPLY: finished
0709 12:20:25.674973 (+   312us) log.cc:844] Serialized 164 byte log entry
0709 12:20:25.675167 (+   194us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:25.675339 (+   172us) write_op.cc:454] Released schema lock
0709 12:20:25.675411 (+    72us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:25.675816 (+   405us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:25.675869 (+    53us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5099676,"reported_tablets":26,"spinlock_wait_cycles":8576,"child_traces":[["op",{"apply.queue_time_us":134852,"apply.run_cpu_time_us":24582,"apply.run_wall_time_us":316456,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79367,"num_ops":18,"prepare.queue_time_us":371565,"prepare.run_cpu_time_us":19708,"prepare.run_wall_time_us":248587,"raft.queue_time_us":140193,"raft.run_cpu_time_us":10918,"raft.run_wall_time_us":45062,"replication_time_us":247116,"thread_start_us":632556,"threads_started":4,"wal-append.queue_time_us":102298,"wal-append.run_cpu_time_us":44220,"wal-append.run_wall_time_us":1676602}]]}
I20260709 12:20:26.946561 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:26.951365 27737 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:26.926819   944 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:26.938282  1101 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:26.953666 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:26.960911 27485 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:26.963126 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 137336us
W20260709 12:20:26.959714   844 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.662s	user 0.015s	sys 0.000s
I20260709 12:20:26.913976  1005 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:26.969816 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51675.1us
I20260709 12:20:26.971277  1183 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:20:26.944818  1136 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:26.977408   944 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:26.978129   845 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:26.977855   844 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.993186 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:26.982909 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:26.992537  1111 raft_consensus.cc:993] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:27.000392 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:27.032843 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.037406 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.038973 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:27.048921 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 316909us
W20260709 12:20:27.053614  1174 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:20:27.067168  1005 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:27.056102 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46451.7us
I20260709 12:20:27.068619  1136 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:27.060817 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81354.2us
I20260709 12:20:27.070233 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.070139  1027 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:26.993188   944 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:27.071558   944 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:27.072070   844 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.072430  1040 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:27.072702 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:27.073747  1198 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:20:27.073987 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.074990  1133 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:27.075471  1039 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:27.075260 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.077981 27663 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:20:27.078461  1077 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.061379  1197 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:27.063567 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72079us
I20260709 12:20:27.106827 27656 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:27.107394 27552 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:27.064234  1187 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:26.989462  1180 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:27.079350 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.079808   959 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.080405 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:27.081612 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:27.081997 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 91976us
I20260709 12:20:27.080401 27485 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 9 to 11, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 11 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:27.084479  1184 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.193s	user 0.000s	sys 0.000s
I20260709 12:20:27.061019  1117 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.057376 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:27.112202 28337 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:27.116864 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:27.117681  1117 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:27.117553 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.118177  1117 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:27.118641  1117 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:27.118793 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:27.128898 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48618.5us
I20260709 12:20:27.127810 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.130671 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:27.132704 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 119520us
I20260709 12:20:27.134011 28331 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
W20260709 12:20:27.134341 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 214719us
I20260709 12:20:27.135270 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:27.135380 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.139595 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:27.141988 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 136577us
I20260709 12:20:27.142618 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.129598  1111 raft_consensus.cc:1081] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:27.131381   845 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:27.151058 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.132565  1040 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:20:27.156172 27729 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:20:27.133015  1033 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:27.161808 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:27.163961 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:27.164120 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:27.146597  1039 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:27.169694  1186 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:27.147527   764 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:27.173799  1105 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:27.174062  1077 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.175311 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:27.153348   935 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:27.171129 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:27.178997 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.151242  1133 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:20:27.190757 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63055.7us
I20260709 12:20:27.180341 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.191855 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:27.194391 27733 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
W20260709 12:20:27.202972 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52011.6us
I20260709 12:20:27.211663  1208 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:27.212077  1196 tablet_replica.cc:333] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:20:27.212836  1200 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:27.206936   996 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:27.190680  1036 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.779s	user 0.006s	sys 0.006s
I20260709 12:20:27.157836  1177 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.100s
W20260709 12:20:27.226212 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55198.8us
I20260709 12:20:27.190639  1105 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:27.209007  1122 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.231258 27582 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:27.233965  1040 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:27.222047   996 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:27.239761 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59547.8us
I20260709 12:20:27.242494 27658 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:27.240412 27485 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 8 to 13, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 13 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.243129 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.244704 27485 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 5 to 8, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:27.245283  1213 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
W20260709 12:20:27.246971  1203 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:20:27.231009  1039 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:27.231495  1105 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:27.250691 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 83456.4us
I20260709 12:20:27.251643 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:27.253602  1177 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.254101  1177 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.256444  1177 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.235749   845 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.257504   845 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.259837   845 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.235941  1133 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:20:27.223253   959 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 12:20:27.248736   996 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:27.279462  1193 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.226s	user 0.000s	sys 0.003s
I20260709 12:20:27.280036 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.284819 27583 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:27.281132 27736 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 7. (term mismatch)
I20260709 12:20:27.281680 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:27.239166  1187 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:20:27.288297 27735 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:27.285741 27736 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 6
I20260709 12:20:27.289975 27736 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 12:20:27.290376 27736 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:27.264194  1027 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.267673   845 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:27.291827   845 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.292198 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.294961 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:27.297329 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.320976  1187 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:27.322211  1187 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:27.322579 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:27.323381 27737 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:27.323549  1189 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:27.295758 27729 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:27.292209   845 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:27.325506  1189 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:27.332774 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 128833us
W20260709 12:20:27.334512 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 119758us
I20260709 12:20:27.335191 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:27.281653   944 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:27.340543 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 148749us
I20260709 12:20:27.322608  1187 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:27.330933  1208 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:27.342272 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:27.347134 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.347795   959 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.348241 27581 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:27.348719 27581 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:27.348819  1224 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:27.349212 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.347277  1142 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.349835 27584 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:27.351032 27585 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:27.348289   959 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.351934 27587 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:27.351982 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.352432 27587 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:27.352710 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:27.353116   959 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.353307 27587 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.354254 28331 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:20:27.354664 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:27.354904 27587 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:27.349543 27581 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.351410 27585 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:27.356415  1224 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:27.356801  1224 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.357340  1224 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:27.356781 27585 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.358285 27585 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:27.359549   959 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:27.365496 27587 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:20:27.324517  1039 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:20:27.336426  1120 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:27.384021 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:27.388104 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.367115  1027 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:27.401917 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:27.386835  1117 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.350422  1142 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.411577  1187 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.412323  1142 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.413043  1187 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:27.417253  1197 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:27.403652 27585 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:20:27.408346 28337 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:27.409711 27587 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:20:27.396276  1027 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.383035  1105 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:27.439834 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.431806  1117 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.329234  1189 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.448498  1189 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:27.441825  1223 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.107s
W20260709 12:20:27.452024  1195 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.477s	user 0.002s	sys 0.001s
I20260709 12:20:27.452636  1200 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:27.450397 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 215250us
I20260709 12:20:27.441797  1152 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:20:27.457266 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.397630  1072 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:20:27.470427   996 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.473497   996 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.474756 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.453013   501 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:27.477425 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 142308us
I20260709 12:20:27.440246  1027 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:27.483511 27661 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:27.481678 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:27.478708 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:27.487516 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 150120us
I20260709 12:20:27.455158  1044 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:27.490573 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.498863 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:27.447238  1224 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.502775 27662 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:20:27.503217  1214 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:27.503005   935 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 12:20:27.503805  1227 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:27.505015  1220 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
W20260709 12:20:27.505841  1226 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:20:27.505939  1177 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:20:27.507357 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53252.9us
I20260709 12:20:27.508250  1152 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.513214  1105 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.499904  1223 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:27.551903  1152 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.554075  1152 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.515290 27734 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:27.516265 27737 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:27.519878   944 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.492686  1180 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:27.525609  1208 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:20:27.555790  1180 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:27.556888  1180 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.557533 27581 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:27.557750  1227 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.510376   996 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:27.560240  1227 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.560443   996 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:27.453194  1200 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.487056   501 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:27.561851 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:27.562335 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.493924  1186 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:27.531868 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.535504  1214 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.542414  1224 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.526053  1115 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:27.524034  1040 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:20:27.562827  1200 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.546681  1027 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.565392 27736 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:27.566797 27552 leader_election.cc:400] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
W20260709 12:20:27.572717 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48792.2us
I20260709 12:20:27.536749  1117 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.577921 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53816
W20260709 12:20:27.562942 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.585482 27735 raft_consensus.cc:606] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Received request to transfer leadership
I20260709 12:20:27.581346 27552 leader_election.cc:403] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:20:27.562850   501 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.582899  1232 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.583390  1229 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:27.602298 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110725us
I20260709 12:20:27.644296 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.650625 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.651185 27664 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:27.575438  1214 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:27.650432 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.643898  1117 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.655640 27662 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:27.655709 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 133685us
I20260709 12:20:27.656834 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.657608 27586 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
W20260709 12:20:27.653546 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87119.2us
W20260709 12:20:27.660964 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.654366 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.662868 27657 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:27.574682   845 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.673153 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.685911  1240 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.644148  1213 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.689801  1240 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.689917  1213 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.691731  1240 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.687084 28337 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:27.687520 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:27.692473 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:27.693146 27659 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:27.685966  1183 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:20:27.703576 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:27.706629 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.660230  1189 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:27.706638 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.708083 27588 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:27.712714  1242 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:27.715276  1117 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.715790 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.716197  1241 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.716784  1241 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.717374 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.718307 27586 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:27.722035 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.664100  1115 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:27.707425 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.732096 27663 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:27.674579  1177 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:27.692111  1213 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.738929 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.740772 27581 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 12:20:27.744140 27581 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 9
I20260709 12:20:27.744277 27656 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:27.744416 27581 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:20:27.719751 27587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:20:27.670677  1229 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.744721 27581 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:27.745110 27587 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:27.754745 27587 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:27.732460  1240 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:27.760589 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.761135 28335 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:27.756901 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.763183 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.749663 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:27.751224 27471 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 7274fd8a2115405c9b78c61214bff736 already in progress [suppressed 2 similar messages]
I20260709 12:20:27.774148 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.664567  1232 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.780061 27656 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:20:27.751574  1179 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:20:27.780102 27659 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:27.782840 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:27.745249  1222 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:27.753788  1229 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.772958  1240 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:27.792647  1234 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:20:27.810286 27662 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:27.755523  1177 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:27.807802  1211 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.564s	user 0.002s	sys 0.002s
I20260709 12:20:27.779367  1120 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.795771  1179 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:27.720913  1186 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.826257 27588 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:27.846336 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.846072 27587 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:27.846378 27471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 141547us
I20260709 12:20:27.850335 27588 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:27.851724 27581 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:27.855952 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:27.856884 28335 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:27.859311 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:27.861316 27587 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.850332  1189 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16.
I20260709 12:20:27.821352  1222 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.833323  1115 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.844321  1120 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:27.889989 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 2844) took 1518 ms (client timeout 1564 ms). Trace:
W20260709 12:20:27.890411 27582 rpcz_store.cc:269] 0709 12:20:26.371710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.376430 (+  4720us) service_pool.cc:224] Handling call
0709 12:20:27.889774 (+1513344us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b540011ba54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:27.889910 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1962703616}
I20260709 12:20:27.865612   845 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.869251 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.864394  1240 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.871744   996 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.860613   681 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:27.864802  1179 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.876904  1233 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.876761 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:27.878400  1194 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.824s	user 0.002s	sys 0.002s
I20260709 12:20:27.879527  1213 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:27.879168  1133 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:20:27.886020 27732 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:20:27.895602  1189 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:27.935061  1189 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.935942 27662 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:27.937098  1189 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.901800 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:27.902441 27589 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:27.904099 27585 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:20:27.938802 27585 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:27.860260  1027 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:27.908116  1233 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.939057 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:27.939723 27589 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:27.940524 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.909366  1187 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.941040 27589 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:27.941242  1187 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:27.911881 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.913609  1241 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.913995  1039 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.915125  1044 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:27.917762  1036 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:20:27.918632  1222 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.918807 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:27.919657  1177 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:27.920481  1232 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.905901   681 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.923295   996 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:27.923360 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:27.934450  1077 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:20:27.936885 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:27.941620  1120 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:27.906352 27588 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:27.942642 27586 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:27.912516 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53816
I20260709 12:20:27.939148 27585 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:28.046531 27731 raft_consensus.cc:606] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Received request to transfer leadership
I20260709 12:20:27.893610 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:28.048035 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 154494us
I20260709 12:20:28.048772 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.050597 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:27.954105  1245 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.051656  1245 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:27.964895 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:28.053369  1251 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:28.053752 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 88935.4us
I20260709 12:20:28.053624 27585 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.054487  1251 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:28.054967  1251 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:27.964883  1117 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:27.965498  1229 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:27.967985   845 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:28.055989  1117 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:28.055799   845 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.056819  1117 consensus_queue.cc:1243] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:20:28.057199  1228 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:28.057586  1228 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.057916 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:28.058575   845 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:27.968603  1133 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:28.059021  1222 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:27.971993  1227 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:27.973078   828 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.976516 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:27.986001 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:27.922873  1214 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.005316 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.058439  1228 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.057591  1101 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:28.059490  1222 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:28.085644 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2730) took 1176 ms. Trace:
I20260709 12:20:28.091035  1101 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.048058   959 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:28.091460  1133 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:28.092128  1133 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.092677  1133 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.092217  1101 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.093727  1101 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.094462  1245 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:28.053397  1036 raft_consensus.cc:2764] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:20:28.069082  1219 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:28.075963 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.085825 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.096999  1228 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:28.097576 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.055896  1251 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.095623  1222 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.095512 27730 rpcz_store.cc:276] 0709 12:20:26.909299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.927238 (+ 17939us) service_pool.cc:224] Handling call
0709 12:20:26.948274 (+ 21036us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:27.097176 (+148902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:27.097223 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:27.111659 (+ 14436us) log.cc:844] Serialized 26 byte log entry
0709 12:20:27.263702 (+152043us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b18002d88c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:27.284581 (+ 20879us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:27.284620 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:28.020666 (+736046us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54002e77d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:28.053810 (+ 33144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:28.053848 (+    38us) raft_consensus.cc:1685] finished
0709 12:20:28.053861 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:28.085519 (+ 31658us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106689,"raft.queue_time_us":112256,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":12712,"spinlock_wait_cycles":77560320,"thread_start_us":20697,"threads_started":1,"wal-append.queue_time_us":190545}
I20260709 12:20:28.101756  1219 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.102710   828 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.091058 27656 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:28.103729 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.104338 27663 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:28.104919  1219 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.105518 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:28.106165 28335 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
W20260709 12:20:28.095546 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 119140us
I20260709 12:20:28.107374 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:28.090006 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.108991   845 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:28.107372  1245 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:28.109710   845 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.090672 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:28.111765 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:28.112634  1186 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:28.112838   845 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.113049  1186 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:28.110401  1245 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.113477  1186 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.113678  1245 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:28.115041  1189 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:28.115865  1189 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:28.116288  1189 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:28.116693  1189 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:28.116871  1036 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:28.118206   959 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:28.118600   959 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.119760   959 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.124527 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.125512  1264 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:28.115828 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:27.905475  1181 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 12:20:28.004078  1044 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:28.148530 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.139671 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.148754 28329 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:28.173514 27587 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:28.016378   681 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.159307 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:28.016451  1240 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:28.137306  1219 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:28.164722 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.167845 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.124554  1186 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.017112  1077 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:28.169497 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45088.7us
I20260709 12:20:28.145776 27585 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.158167 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.176410 28334 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:28.171197 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:28.105769   828 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.185312 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.004220  1039 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:28.186043 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.178815 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.216055 27582 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:28.218926 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.219801 27587 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:28.217662 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2359) took 1506 ms. Trace:
I20260709 12:20:28.222324 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.223011 27588 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
W20260709 12:20:28.225835 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:28.227501 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56375.6us
W20260709 12:20:28.233103 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47863.1us
I20260709 12:20:28.233896 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.182242  1219 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.184319  1227 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:28.220809 27655 rpcz_store.cc:276] 0709 12:20:26.678352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.688027 (+  9675us) service_pool.cc:224] Handling call
0709 12:20:26.718615 (+ 30588us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:26.747235 (+ 28620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:26.747280 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:26.765320 (+ 18040us) log.cc:844] Serialized 26 byte log entry
0709 12:20:26.891669 (+126349us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:26.891704 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:26.891769 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:27.877733 (+985964us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:28.172058 (+294325us) raft_consensus.cc:1685] finished
0709 12:20:28.172080 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:28.184331 (+ 12251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6375680,"thread_start_us":83951,"threads_started":1,"wal-append.queue_time_us":197680}
I20260709 12:20:28.186626  1233 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.249840 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.225963  1264 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:28.238374  1027 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:28.253710 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.219314  1070 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:20:28.241206  1229 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:20:28.262686 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.160279  1189 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.243281  1219 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:28.260722 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.215538  1133 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:28.272212  1265 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:20:28.274873  1179 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:28.244043  1251 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:20:28.282745  1186 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.283866  1224 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:28.287959 27656 raft_consensus.cc:2368] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16.
I20260709 12:20:28.288208 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.288604 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.289122 27583 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.289300 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.289824 27586 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:28.290997 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:28.303555  1250 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:20:28.304724  1187 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:28.304996 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44362.5us
I20260709 12:20:28.306295  1214 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.304728  1189 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:28.267815  1120 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:20:28.308724  1120 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:20:28.309222  1189 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:28.309607  1189 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.310451  1189 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.295826 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.312698  1196 raft_consensus.cc:2243] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:28.313825 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.248919  1227 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.316015  1196 raft_consensus.cc:2272] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:28.316299  1039 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:28.316481  1177 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:28.317175  1227 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.317317  1177 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:28.316322 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.319464  1196 ts_tablet_manager.cc:1916] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:28.320436  1027 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:28.322912 27586 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:28.324457  1245 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.325692 28335 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:28.329535 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:28.331415  1214 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:28.291062  1251 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:28.336827 27582 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:28.337198 27582 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.337340  1147 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:28.337718  1147 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.331194 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:28.343879 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:28.337949 27582 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.344805 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:28.348361 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:28.348856 27582 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.353466 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:28.354308 27655 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.357826 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.390413 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.258363  1077 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.391660 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:28.348124   996 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:28.401969  1245 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:28.403553 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44546.9us
I20260709 12:20:28.405637 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:28.408715 27660 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:28.406726 27584 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:28.410836 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.407819 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:28.415411 27584 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 12:20:28.437050 28334 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:28.388062  1233 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:20:28.406229  1115 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:28.442592 27589 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.000s
I20260709 12:20:28.443003 27589 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:28.444214 27589 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:28.445520 27584 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:28.414130   996 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:28.399673  1216 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:20:28.449852 27585 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 10. Current term is 11. Ops: []
I20260709 12:20:28.451810 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:28.435171  1188 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:28.454407 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.457804 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:28.460695   996 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.463624 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.464099  1197 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:28.464573  1197 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.464610  1039 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:28.465052  1197 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:28.465730  1218 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:28.465842  1216 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:28.466436  1216 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.468066  1233 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:28.464385 28333 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:28.468686  1233 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.469655   996 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:28.473035  1117 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:28.475175  1117 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.476107  1117 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.474677 27580 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:20:28.406227  1207 consensus_queue.cc:1175] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 18 status { last_received { term: 18 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 18 index: 15 } }
I20260709 12:20:28.477288 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:28.476804 28334 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:20:28.486299 27655 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:28.478889 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.480274 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.481171 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:28.491441 27583 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.490417 27582 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260709 12:20:28.493328 27582 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 9
I20260709 12:20:28.493717 27582 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 12:20:28.493526 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:28.494251 27582 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:28.498868 27660 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:28.507901 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:28.519846 27585 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:20:28.520567 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.521126 27589 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:28.470911  1233 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.527400 27583 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
W20260709 12:20:28.527928 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 113717us
I20260709 12:20:28.477265  1117 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.557312 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:28.468530  1216 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.477643  1039 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:28.495620  1033 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:28.477325   935 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.482290   959 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:28.557540 27583 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:28.528900 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:28.558135 27732 raft_consensus.cc:606] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:28.572739 27732 raft_consensus.cc:614] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:28.528926 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:28.519083  1188 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:28.450847  1115 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.533787 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:28.566730  1039 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:28.479290  1213 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.523728  1206 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:28.574375 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45549.3us
I20260709 12:20:28.529004  1240 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:20:28.565864 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 316756us
I20260709 12:20:28.563177   845 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:28.578836 27580 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:28.492049  1219 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.474663  1197 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.479035   996 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:28.583820 27701 leader_election.cc:400] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16.
I20260709 12:20:28.592259   845 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:28.593215   845 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.593829 27732 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:28.595707  1188 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.596936 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.598140 27656 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:28.599746 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.600389 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:28.600988 27662 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:28.602519   996 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.603727   996 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:28.589715  1228 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 12:20:28.607717 28334 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:28.608307  1228 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.608803  1228 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.609498  1142 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:28.609941  1142 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:28.610903  1228 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.610764  1142 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.611603 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:28.577706  1216 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:28.612048  1142 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.612334  1216 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.612774  1216 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:28.612969  1039 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.614181  1039 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:28.584692  1227 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:28.616832  1044 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.576817   681 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.618233  1209 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:28.618827  1209 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.619872 27730 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:28.621321 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.617192  1115 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.591830 27701 leader_election.cc:403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:28.632977 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:28.634982 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.633606 27659 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
W20260709 12:20:28.635915 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.636138 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.626896 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:28.638018 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54375.7us
I20260709 12:20:28.638298 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.638759 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:28.645421 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.646128 27587 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.570360  1027 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 12:20:28.649688 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:28.652441 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.651005 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.652590 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:28.652807 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.653425 27656 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:28.653841 27655 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:28.655014 27662 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:28.668233 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:28.668767 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:28.669198 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:28.669899 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.671342 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.578179   935 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.675153 27657 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:28.675664 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.678793 27588 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:28.677278 27733 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:28.592300  1180 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:28.681547 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:28.641623  1228 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.652485  1186 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:28.690604 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.704268  1264 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.668315  1233 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:28.651775  1143 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:28.710033 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:28.710649 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 181768us
I20260709 12:20:28.586966  1213 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.714926 27660 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:28.714927 27737 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:28.725911 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57971us
I20260709 12:20:28.692265  1190 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.701593  1187 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:28.738525 27588 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
W20260709 12:20:28.720011  1222 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.621s	user 0.008s	sys 0.002s
I20260709 12:20:28.746996 27587 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:28.714135  1251 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:20:28.714385  1143 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.724393  1240 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:28.707669  1197 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.782012 27656 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:28.714215  1115 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:28.785105 27734 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:28.715116  1180 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:28.794417  1213 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:28.795300  1224 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.000s
I20260709 12:20:28.795746  1224 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:20:28.796219 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.787084 27729 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260709 12:20:28.792665  1277 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:20:28.741077   681 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.800740 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:28.712198  1228 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:28.802215 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 238705us
W20260709 12:20:28.790947 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 109514us
I20260709 12:20:28.796641 27587 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:28.740600  1209 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:28.717471  1264 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.819361 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.797282 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.824023 27588 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:28.786836  1143 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:28.823526 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 289831us
I20260709 12:20:28.821049 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:28.792308  1179 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
I20260709 12:20:28.834738 27662 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:28.837471  1251 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:28.838536  1264 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:28.808755  1281 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
W20260709 12:20:28.835614  1273 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:20:28.753062  1190 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.844235 27583 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:28.855077 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:28.787012  1222 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.864691 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.801661  1115 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.833455  1227 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:28.794427  1180 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.880893 27588 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:28.883098 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:28.856177  1179 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.854863  1233 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:28.893482 27732 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
W20260709 12:20:28.894376 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87363.2us
I20260709 12:20:28.898835 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:28.849002  1209 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:20:28.855996  1197 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:20:28.879604  1281 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:28.907271 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.907668 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:28.915895 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:28.870774  1224 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:20:28.888195  1122 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:28.923573 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.924345 27581 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
I20260709 12:20:28.931896 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:28.932786 27733 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:28.935446 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:28.939069 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.907226  1288 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:28.921579  1213 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:20:28.955466 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 223747us
I20260709 12:20:28.920979  1291 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:28.902354  1179 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:28.968031 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.969056  1133 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:28.970062   996 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:28.970495  1291 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:28.971464  1291 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.974545  1190 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:28.974677 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:28.975872  1299 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:28.975728  1190 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:28.976436  1299 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.043462 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:29.043967 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.938699  1122 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:28.942204  1177 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:29.051043 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:29.051828 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.054054  1299 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.053819  1233 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.934027  1285 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.134s
I20260709 12:20:29.056151  1285 consensus_queue.cc:1243] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:20:29.064884  1287 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.065068  1196 ts_tablet_manager.cc:1929] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 11.8
I20260709 12:20:29.065642  1196 log.cc:1199] T 81e6b2dabf8f4e6cb549ca250430e568 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/81e6b2dabf8f4e6cb549ca250430e568
W20260709 12:20:29.068588 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:28.945962   935 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:29.078810 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 121765us
I20260709 12:20:28.948873  1213 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:29.099045 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 142208us
I20260709 12:20:28.920542  1240 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.049551  1187 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.970505   996 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:28.936753  1281 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.065562  1287 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.108448 28334 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:20:29.118187 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.118942 28337 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:29.112730 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.115013 27589 pending_rounds.cc:85] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 10
I20260709 12:20:29.119633 27589 pending_rounds.cc:107] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:20:29.120015 27589 raft_consensus.cc:2889] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:29.067862  1142 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:29.078117  1289 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:20:29.102772  1213 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:29.123431 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 212267us
I20260709 12:20:29.137462 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.115309   828 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.138609 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.140825 27586 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 12:20:29.135779  1299 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:29.143167  1299 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.145316  1299 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:29.148985  1222 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:29.083926  1177 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.151548   996 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.152747   996 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:29.153064  1177 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.125120  1264 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:20:29.115927  1187 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.121194  1188 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:20:29.160387  1196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53816 (request call id 1) took 2595 ms. Trace:
I20260709 12:20:29.160931  1196 rpcz_store.cc:276] 0709 12:20:26.595168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.691524 (+ 96356us) service_pool.cc:224] Handling call
0709 12:20:27.036705 (+345181us) ts_tablet_manager.cc:1151] Deleting tablet 81e6b2dabf8f4e6cb549ca250430e568
0709 12:20:27.036810 (+   105us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:20:28.335275 (+1298465us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:29.064928 (+729653us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:29.160305 (+ 95377us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1092156,"tablet-delete.queue_time_us":225835,"thread_start_us":76168,"threads_started":1}
I20260709 12:20:29.164531   828 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.979349  1227 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:28.956861  1288 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.173908 27731 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:29.176716 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:29.177670 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.134661  1142 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.181553 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.178527 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:29.183666 27662 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:29.137290  1072 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.203450 27733 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:29.214056 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.172680  1197 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:29.233069 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.180662  1224 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.186638  1228 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.234959 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.244446 27734 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:29.195492  1142 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:29.249879  1288 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.209990  1187 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.255206 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:29.186228  1209 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:20:29.213629   935 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.172684  1192 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:20:29.223862  1197 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.196415  1143 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:29.267479 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90844.8us
I20260709 12:20:29.267949 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.216961  1072 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.269325 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:29.270263   959 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.694s	user 0.013s	sys 0.000s
I20260709 12:20:29.270787  1298 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:29.263998  1197 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:29.246106  1224 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.283034  1300 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:29.292650  1309 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.309365  1309 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.310315 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:29.294477 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170304us
W20260709 12:20:29.300559  1304 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:20:29.300980  1294 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:29.302268   828 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:29.302906  1039 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.688s	user 0.011s	sys 0.000s
I20260709 12:20:29.300608  1143 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:29.303508  1253 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.056s	user 0.010s	sys 0.017s
I20260709 12:20:29.303938  1298 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:29.305183  1300 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:29.263803   935 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.277076  1218 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:29.308089  1186 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.595s	user 0.007s	sys 0.003s
I20260709 12:20:29.308301 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:29.315121 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.342669 27662 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:29.261404  1222 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:20:29.311712  1072 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.308873  1281 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.306741  1287 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:29.347057 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.332645  1209 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.337322  1224 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.347934  1186 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16.
I20260709 12:20:29.349768  1309 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:29.363762  1300 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:29.366050  1216 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.753s	user 0.009s	sys 0.005s
I20260709 12:20:29.391539  1216 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.393036 27735 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:29.394060  1216 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.356140  1209 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.367640 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 927) took 1710 ms. Trace:
I20260709 12:20:29.394840 27661 rpcz_store.cc:276] 0709 12:20:27.657643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:27.657887 (+   244us) service_pool.cc:224] Handling call
0709 12:20:27.658251 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:27.799244 (+140993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:27.799273 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:27.799865 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 12:20:27.931617 (+131752us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:27.931678 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:28.405799 (+474121us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:28.405844 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:29.175423 (+769579us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:29.367408 (+191985us) raft_consensus.cc:1685] finished
0709 12:20:29.367438 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:29.367581 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":343,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":13938,"spinlock_wait_cycles":334465024,"thread_start_us":87870,"threads_started":1,"wal-append.queue_time_us":212485}
I20260709 12:20:29.370457  1298 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.372042 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.373262 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:29.397123 28337 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:29.373970 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.398325 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.387235 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.374397  1316 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:29.375450  1294 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.400833 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:29.375392 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.381565   935 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.396298  1209 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.408203  1189 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:29.382058  1213 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.382895  1275 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:29.408979  1275 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.409397  1275 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:29.385294  1117 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.812s	user 0.007s	sys 0.003s
I20260709 12:20:29.346843  1218 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.419948  1218 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.402967 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:29.420926 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 165848us
I20260709 12:20:29.403297 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:29.401990  1306 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:20:29.401701  1101 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:20:29.403570  1315 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:29.403853   959 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:29.404469  1316 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:29.402328  1177 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:29.443310 27586 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:29.405687 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 919) took 1388 ms. Trace:
I20260709 12:20:29.405438  1294 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.412231 28337 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:20:29.444346 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.417488  1033 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:29.421344 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.424260  1117 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:29.438570   845 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:29.439390  1213 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.444221 28335 rpcz_store.cc:276] 0709 12:20:28.016881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.115568 (+ 98687us) service_pool.cc:224] Handling call
0709 12:20:29.405497 (+1289929us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:29.405629 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444138240}
I20260709 12:20:29.446658 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.461218 27586 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:20:29.462188 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.462949  1288 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.463522  1288 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.464612 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.465160 27735 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:29.465770  1288 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:29.465999 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.466684 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:29.467268 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.468039 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.468442 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.469079 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:29.469863 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.470402  1316 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:29.470767  1316 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:29.471617  1316 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.472867  1316 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:29.470786 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:29.449043 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.445753  1177 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.474937 27581 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:29.475113  1177 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:29.475446  1315 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:29.475842 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.476563 27581 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:29.476491  1315 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.477949 27581 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:29.448123  1033 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:29.480154 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:29.479954  1033 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.481208  1033 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:29.481236  1288 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:29.481863  1288 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.482275  1288 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:29.420852  1299 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:29.483486  1299 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:29.484257 27730 raft_consensus.cc:606] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:29.452750  1294 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.456357 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.484349  1299 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.485494  1299 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:29.485900 27730 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: []
I20260709 12:20:29.457446 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.470971 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:29.487658 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:29.487679 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:29.489259 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67984.5us
I20260709 12:20:29.445367 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:29.492271 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46981.1us
I20260709 12:20:29.494683 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.495539 27661 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
W20260709 12:20:29.500309 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.502661 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.503374 27735 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:29.504674 27730 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:29.492612 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:29.505723 27585 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:29.508634 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:29.513833 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:29.484336  1101 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:29.522336  1077 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:29.523085  1101 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.504010   845 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:29.527734   845 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:29.528856   845 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.506067  1294 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:29.532621  1294 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.533125  1294 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:29.534056 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.509996   935 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:29.536008   935 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:29.536448   935 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:29.520953  1142 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.511840 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.514189   959 raft_consensus.cc:2764] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:20:29.510787  1187 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:29.517042 27730 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:29.544775 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:29.542893 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.545949 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.546149  1179 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.547035 27585 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:29.540583 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.552101 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.554042 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.554901 28335 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:29.556465   959 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:29.556538 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.557382 27586 raft_consensus.cc:2368] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16.
I20260709 12:20:29.557672  1142 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.554056  1206 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.559752  1206 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.560341 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.560940 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:29.561789 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.561904  1206 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.562511  1206 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:29.562861  1206 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.514808  1227 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.565811 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:29.562875 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.563210  1206 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:29.586905 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.588441 27730 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:20:29.582147  1291 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.590145  1291 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.595048 28331 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:29.572248  1227 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.595388  1115 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.597106 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 885) took 1198 ms. Trace:
I20260709 12:20:29.599222 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.597453 28331 rpcz_store.cc:276] 0709 12:20:28.398993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.399573 (+   580us) service_pool.cc:224] Handling call
0709 12:20:29.596946 (+1197373us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400153454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:29.597050 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2721887488}
I20260709 12:20:29.571899  1187 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:29.611047 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.612011 27580 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:29.614039 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:29.617172  1115 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:29.617648  1115 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:29.618048  1115 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:29.597453  1227 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.597340  1291 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.609898  1187 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.644548 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:29.649104  1197 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:29.652688 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.654126 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.585695  1288 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:29.652123 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.665067 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.666833 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:29.669734 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.650437 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:29.660634 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.673928  1197 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.644730  1187 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:29.697306 27659 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:20:29.658701   959 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:29.702756 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:29.702915 27732 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:29.705086 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.706928 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.707724 27729 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:29.710908 27730 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:29.712962 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:29.715076 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.715889 27580 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:29.716678 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.717221 27730 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:29.708652  1317 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
W20260709 12:20:29.710330  1312 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260709 12:20:29.718295  1142 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:29.679976  1177 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.757550  1142 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:29.758458  1142 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.759186  1177 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.727962 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.759753 27657 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:29.760696 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.708182   959 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.730680 27582 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:20:29.716115  1315 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:20:29.740890 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.744637  1288 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:29.747948 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.753294 27659 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:29.765967 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.767375 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.767542 27580 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:29.768591 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.769244 27659 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:29.770089 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:29.753521 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:29.708936  1197 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:20:29.727993  1327 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:20:29.782589 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:29.721571 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.783321  1291 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.786752 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.783787  1033 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.803145 27730 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:29.788000 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.772033   959 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:29.784219  1315 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:29.813161 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.789674  1288 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:29.791581 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.791796 27663 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:29.754437  1213 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.771847  1177 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.771203 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:29.795312 27657 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:29.795413 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:29.772470  1253 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:20:29.761312  1101 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.764983  1189 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.800176 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2907) took 1150 ms. Trace:
I20260709 12:20:29.805814 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.844844 27655 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:29.810061  1179 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:29.844370 27583 rpcz_store.cc:276] 0709 12:20:28.649917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.650798 (+   881us) service_pool.cc:224] Handling call
0709 12:20:29.799980 (+1149182us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:29.800109 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438873984}
I20260709 12:20:29.810081 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.814189 27585 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:29.849074 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.849772 27583 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:29.850574 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.851724 27729 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:20:29.852761 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:29.856158 27580 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:29.800812  1197 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.841135  1245 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.863651  1197 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:29.863960  1245 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.866320  1245 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.814906  1033 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:29.868203  1333 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:20:29.869688  1033 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.853482  1189 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.873733 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.862440  1115 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.877372  1115 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.859361 27730 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:20:29.844308  1241 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:20:29.860572 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.884743 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:29.861910  1336 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:20:29.861294  1294 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:29.862635  1329 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:20:29.854583 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:29.803156   935 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.883466 27585 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:29.860103 27580 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:29.887992 27657 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:29.888409 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:29.807893  1209 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.843634  1180 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.897379 27655 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:29.900053 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.902273 27585 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:29.904904 27587 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:29.803545  1181 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.812808  1291 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.854592  1213 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.847576  1177 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.857895  1101 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.912827 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.913394 27662 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:29.898391  1229 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.926071  1315 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.897810  1245 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.929381 27659 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
W20260709 12:20:29.933414 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:29.935631 27658 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:29.938118 27662 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:29.925642  1181 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.941082  1315 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.939685  1245 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.946770  1315 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.948475   935 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.946731  1245 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.954778 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:29.956895  1213 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.957902 27664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:29.961856 27664 raft_consensus.cc:606] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:29.962693 27664 raft_consensus.cc:614] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:29.959452 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:29.964085 27580 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:29.965317  1177 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.968665  1101 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:29.969579  1101 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:29.970199  1101 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.911948  1209 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.974303 27664 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
W20260709 12:20:29.975304  1195 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.506s	user 0.001s	sys 0.004s
I20260709 12:20:29.907301  1180 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:29.980762 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:29.981422 27659 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:29.983754 27580 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:20:29.987604 27664 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:29.988710  1101 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.989346  1181 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.989560  1294 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.990100 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.990710 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:29.990895 28331 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:20:29.991613 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:29.993146 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:29.993772 28337 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:29.996099  1180 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:29.997030 27589 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:29.997267 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2880) took 1825 ms. Trace:
I20260709 12:20:29.997634 27582 rpcz_store.cc:276] 0709 12:20:28.172260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.517965 (+345705us) service_pool.cc:224] Handling call
0709 12:20:28.518235 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:29.764376 (+1246141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:29.997012 (+232636us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:29.997179 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2786924416}
I20260709 12:20:29.998528 27664 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:29.998766 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2896) took 1889 ms. Trace:
I20260709 12:20:29.992839  1197 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.006099 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.006942 27582 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:20:30.003676 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.008364 27729 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:20:30.006902 28335 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:30.009126 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.008399 27659 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:29.939459  1229 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.001343 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:30.010815 27630 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:30.010725 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.012593 27630 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:20:30.012816 27735 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:29.999778 27584 rpcz_store.cc:276] 0709 12:20:28.109392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.465238 (+355846us) service_pool.cc:224] Handling call
0709 12:20:28.465507 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:29.723417 (+1257910us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:20:29.723738 (+   321us) log.cc:844] Serialized 14 byte log entry
0709 12:20:29.766566 (+ 42828us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:29.766624 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:29.766667 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:29.998522 (+231855us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:29.998573 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:29.998706 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":146,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":81,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":2892231040,"thread_start_us":42177,"threads_started":1,"wal-append.queue_time_us":62762}
I20260709 12:20:30.014689 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.014979 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.015295 28335 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:30.016539 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.017071 28337 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:30.015959  1070 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:30.019984 27582 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:30.020895  1229 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.020987  1213 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15.
I20260709 12:20:30.021487 27661 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:20:30.021847 27661 pending_rounds.cc:85] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 13
I20260709 12:20:30.022099  1264 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.022118  1120 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:30.021785 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.022140 27661 pending_rounds.cc:107] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:20:30.013622 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:30.024089 27661 raft_consensus.cc:2889] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:30.026786 28337 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:29.980268  1187 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.030906 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:30.028925  1343 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:30.033432  1343 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:30.015509  1181 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:30.034883  1343 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:30.035930  1223 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:20:30.037393  1197 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.029579  1224 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 12:20:30.024564  1245 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:30.022737  1120 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:30.042625  1181 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:30.044822  1303 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.957s	user 0.001s	sys 0.001s
I20260709 12:20:30.029464  1070 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:30.022703  1264 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.046995 27734 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:30.060010 27734 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:30.057204  1120 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:30.060554  1229 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.060878 27734 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.061300  1229 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.062026 27734 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:30.053148  1147 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.056607 27735 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
W20260709 12:20:30.034240  1275 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.001s
I20260709 12:20:30.009776  1206 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.075670  1315 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:30.089234  1315 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.089681  1315 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.089567  1343 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.064064  1223 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:20:30.071410  1224 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:30.092772  1224 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:30.093719  1224 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:30.095373 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2750) took 1485 ms (client timeout 1582 ms). Trace:
I20260709 12:20:30.095661  1223 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:30.095881  1219 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:30.095825 27731 rpcz_store.cc:269] 0709 12:20:28.610634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.611383 (+   749us) service_pool.cc:224] Handling call
0709 12:20:30.095165 (+1483782us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54002e86d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:30.095273 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589025664}
I20260709 12:20:30.096306  1223 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.097512  1044 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.098814  1223 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.101204  1120 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.105054 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2439) took 1149 ms. Trace:
W20260709 12:20:30.107170 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 334151us
W20260709 12:20:30.109364 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:30.110718 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.111279  1120 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.059914  1181 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.081177  1147 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.081826  1229 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.077876  1070 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:29.498718 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 81e6b2dabf8f4e6cb549ca250430e568 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:20:30.111572  1005 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:30.112902  1187 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.095105  1224 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:30.113348 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:30.128262  1206 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.087312  1275 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.113963  1209 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.078480 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2374) took 1483 ms. Trace:
I20260709 12:20:30.105897  1315 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.080307  1264 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.119786 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.105527 27738 rpcz_store.cc:276] 0709 12:20:28.955603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:29.485661 (+530058us) service_pool.cc:224] Handling call
0709 12:20:29.485924 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:29.486138 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:29.650358 (+164220us) log.cc:844] Serialized 14 byte log entry
0709 12:20:29.761720 (+111362us) log.cc:844] Serialized 14 byte log entry
0709 12:20:30.104533 (+342813us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b180007b100. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:20:30.104690 (+   157us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:30.104736 (+    46us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:30.104780 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:30.104861 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:30.104968 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":323311,"prepare.queue_time_us":266058,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":348501,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":10596,"spinlock_wait_cycles":57676416,"thread_start_us":274383,"threads_started":2,"wal-append.queue_time_us":147153}
I20260709 12:20:30.120003 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:30.134122 27586 raft_consensus.cc:2368] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:30.133966 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.134647 28337 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.064700  1245 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:30.123898 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:30.125160 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.136554 28333 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:20:30.126057 27471 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 627301us
I20260709 12:20:30.130309 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.131348 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:30.131892 27660 rpcz_store.cc:276] 0709 12:20:28.595247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.596675 (+  1428us) service_pool.cc:224] Handling call
0709 12:20:30.078246 (+1481571us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:30.078375 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269801856}
I20260709 12:20:30.142431 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.143630 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.144249  1122 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.144956 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.135593  1192 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:30.145550 27734 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:30.147121 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.148368 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.148568 27731 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:20:30.149569 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.151616 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:30.130661  1115 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.154670 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.148985 28331 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:30.157359 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.158833  1005 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:20:30.140377  1281 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:20:30.153167  1233 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:20:30.135136  1291 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.159565 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.167574 28334 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
I20260709 12:20:30.168382  1240 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:20:30.159291  1005 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.170146  1189 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.170153  1005 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:20:30.170468  1343 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.170039   959 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.176662 28329 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:20:30.160593  1245 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.177671 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.178401  1264 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:30.179109 28334 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:30.162606  1288 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:30.159968 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.181097 28337 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:30.181097 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:30.184134 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.182624 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:30.193495  1264 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:30.195775  1264 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:30.196748  1264 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.192025  1233 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
W20260709 12:20:30.201051 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:30.182240  1077 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:30.181227   959 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.220614 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.221647  1111 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:30.222424  1111 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:30.222635   996 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:30.222971  1111 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.223402   996 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:20:30.221714 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74672.8us
I20260709 12:20:30.192632  1291 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.164757  1115 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:30.229233 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45172.9us
I20260709 12:20:30.230273 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.233284  1297 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 12:20:30.217082  1206 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.236901 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:30.237726 28330 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:30.235868  1299 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 12:20:30.241467  1115 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.236341  1291 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.248193  1181 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.249748  1291 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.250401  1181 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.232795  1216 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:30.274868 27656 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:30.277675 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.273521  1291 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.258689  1311 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:20:30.273860  1122 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.281662 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:30.277235  1214 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:30.289278  1143 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:30.291050 27583 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:20:30.258777  1206 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.276369   996 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:30.294384   996 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:30.299105 27704 leader_election.cc:400] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16.
I20260709 12:20:30.299548 27704 leader_election.cc:403] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:30.292133  1316 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.276427  1005 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.296499  1281 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:20:30.302690  1315 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.304734  1316 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.278447  1298 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:20:30.298877  1275 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.301294   681 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 15. Current term is 16.
I20260709 12:20:30.298338   996 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.314560  1206 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.314752   996 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:30.292936  1143 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:30.293443  1188 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:30.311682  1219 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:30.279732  1291 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.311200 27704 leader_election.cc:400] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:30.322417 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:30.315286 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.322955 27580 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:30.323362 28334 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:30.316258 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 158998us
I20260709 12:20:30.322746 27704 leader_election.cc:403] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:20:30.317937 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:30.318876 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.321765 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.332110  1209 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:30.332506  1209 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.332759  1291 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.324741 27581 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:30.333523  1316 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.333986 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.335014 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.328348 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:30.332847  1209 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:30.335117 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:30.357849 27581 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:30.358781  1353 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:20:30.361905  1297 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.316682  1188 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:30.357625  1280 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.363555  1188 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:30.315119  1206 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:20:30.325208  1143 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.368778  1143 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:30.369309  1279 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.370054  1279 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.371519 28330 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:30.371959  1186 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.320451  1219 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:30.372555  1186 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.373646  1219 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.316735  1315 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:30.383561 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.400142 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:30.404322 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.375061  1186 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.343840 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:30.345605 27580 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:30.407938 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.351243 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.408998 27587 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:30.409139 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:30.410105 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66361.2us
I20260709 12:20:30.410742 28329 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:20:30.354579 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53286us
I20260709 12:20:30.356659 27585 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:30.411916 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.356741 27582 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:30.407995 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:30.375830 27535 auto_rebalancer.cc:893] Removing replace marker failed: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:20:30.418273 27535 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 7274fd8a2115405c9b78c61214bff736 already in progress
I20260709 12:20:30.419272 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.419829 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.418180 27733 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
W20260709 12:20:30.422089 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:30.422276 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.424221 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.425706 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.362592  1297 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.428460 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.429239 27580 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:30.429878 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.433017 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:30.433658 28331 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:20:30.434938 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:30.418834 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7274fd8a2115405c9b78c61214bff736 already in progress
I20260709 12:20:30.436251 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.445176 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.446617 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.376075  1322 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.453900 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:30.448226 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.448478 27587 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:30.449126 27580 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:30.459136 27630 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:30.449270 27581 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:30.460553 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.461107 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.461344 27661 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:30.461164 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.397899  1219 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:30.397917  1279 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:30.464010 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.443027 28334 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:30.464624 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:30.464855 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.482055 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:30.465957 27587 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:20:30.467159 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.459573 27630 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:30.473634 27581 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:30.483156 27580 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:30.473453 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.462703 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:30.484651 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.485260 27733 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:30.488576 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:20:30.490072 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:30.486027 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.448818  1286 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:30.408088  1315 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.499676 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.500283 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:30.494344 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.505003 27662 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:30.500582 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:30.506780 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:30.507030 27656 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.510177 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.500762 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.502522 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.515203 28328 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:30.460798  1122 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:20:30.520484 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55903.4us
I20260709 12:20:30.521432 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.513089 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:30.516460 27663 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:20:30.473975  1147 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:30.378739   959 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 12:20:30.494659  1229 raft_consensus.cc:2764] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:20:30.459424  1115 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.498864  1120 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:30.485164  1197 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:30.474367  1216 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:30.527791 27660 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:30.536528 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.539629 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:30.501099  1286 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.542997 27656 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:30.545845 27733 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:30.571136 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.571868 27733 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:30.572863 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.498916  1044 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.558456 27581 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:20:30.511783  1033 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:30.585587 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.485646  1299 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:30.510313  1343 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:20:30.562651 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.595063 27661 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:30.511149 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.542105 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:30.566977 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:30.498811  1298 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.574039 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:30.599514 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.496894  1354 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:30.586257 27663 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:30.464046  1322 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.486241  1264 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
W20260709 12:20:30.588529 27485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 223) took 40486 ms (client timeout 15000 ms). Trace:
I20260709 12:20:30.596271 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:30.558722   681 raft_consensus.cc:2764] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
W20260709 12:20:30.597889 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86803.9us
I20260709 12:20:30.605899 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:30.593065  1332 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:20:30.564896  1147 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:30.586299  1187 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.613221 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.582139  1286 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:30.618265 27628 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:30.593750  1033 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.627897  1311 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:30.628635  1207 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.631932 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:30.617583  1229 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:30.626361 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:30.634302 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 168455us
I20260709 12:20:30.591281  1197 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:30.640566 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 186707us
I20260709 12:20:30.609817  1005 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:30.625609  1189 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:30.598765  1120 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:30.612540  1291 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:30.574807  1115 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.608493  1354 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:30.609794 27485 rpcz_store.cc:269] 0709 12:19:50.083636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:50.083876 (+   240us) service_pool.cc:224] Handling call
0709 12:20:30.569432 (+40485556us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:27.659997 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:27.811644 (+151647us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:27.811701 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:20:27.811714 (+    13us) tablet.cc:662] Decoding operations
0709 12:20:27.812620 (+   906us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:27.812709 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:27.812734 (+    25us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:20:27.813379 (+   645us) tablet.cc:701] Row locks acquired
0709 12:20:27.813457 (+    78us) write_op.cc:260] PREPARE: finished
0709 12:20:27.813623 (+   166us) write_op.cc:270] Start()
0709 12:20:27.813759 (+   136us) write_op.cc:276] Timestamp: P: 1783599627813565 usec, L: 0
0709 12:20:27.813775 (+    16us) op_driver.cc:348] REPLICATION: starting
0709 12:20:27.814640 (+   865us) log.cc:844] Serialized 2844 byte log entry
0709 12:20:28.120804 (+306164us) op_driver.cc:464] REPLICATION: finished
0709 12:20:28.161510 (+ 40706us) write_op.cc:301] APPLY: starting
0709 12:20:28.161652 (+   142us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:28.286502 (+124850us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:28.286612 (+   110us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:28.686214 (+399602us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:28.686563 (+   349us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=14,mrs_lookups=0
0709 12:20:28.686635 (+    72us) write_op.cc:312] APPLY: finished
0709 12:20:28.687060 (+   425us) log.cc:844] Serialized 73 byte log entry
0709 12:20:28.687261 (+   201us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:28.687429 (+   168us) write_op.cc:454] Released schema lock
0709 12:20:28.687512 (+    83us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:28.707104 (+ 19592us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:28.711513 (+  4409us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3842,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":40358,"apply.run_cpu_time_us":14797,"apply.run_wall_time_us":526510,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33387,"num_ops":7,"prepare.queue_time_us":76891,"prepare.run_cpu_time_us":46528,"prepare.run_wall_time_us":467180,"raft.queue_time_us":71967,"raft.run_cpu_time_us":14125,"raft.run_wall_time_us":40569,"replication_time_us":306849,"thread_start_us":283025,"threads_started":4,"wal-append.queue_time_us":231467,"wal-append.run_cpu_time_us":13733,"wal-append.run_wall_time_us":1740010}]]}
I20260709 12:20:30.596271  1241 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
W20260709 12:20:30.676072 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49782.3us
I20260709 12:20:30.675187 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.599938  1299 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.655843  1033 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:30.683281 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68818.6us
I20260709 12:20:30.683844 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:30.701814 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.702956 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.612339  1298 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.683091  1224 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:30.710405  1363 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:30.711127  1101 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:30.723644 27582 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 11: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:30.679157 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 117830us
I20260709 12:20:30.675184  1291 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.673673  1189 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:30.729885 28331 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:30.722824 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.680842  1229 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:30.651702  1187 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.685631   681 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:30.748926 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.749953 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:30.752981  1189 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:30.525358 27481 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 17, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 17 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:30.758301 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:30.685143  1354 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:30.763775 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:30.670635  1207 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.752820  1291 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.748792  1297 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:30.815970 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:30.747689  1224 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:30.736966  1279 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.066s
W20260709 12:20:30.807049 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 112502us
I20260709 12:20:30.737478  1101 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:30.765858  1241 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:30.758409  1316 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:30.840119 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:30.859112  1316 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.873679 28331 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:20:30.875267 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 108471us
I20260709 12:20:30.752470  1264 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
I20260709 12:20:30.736330  1147 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.816249  1233 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:30.816237  1315 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:30.767084   681 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.853471  1101 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:30.770246  1190 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.111s
I20260709 12:20:30.762547  1372 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.126s
I20260709 12:20:30.922031 27732 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:30.860277   959 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:30.908118  1361 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
W20260709 12:20:30.909816  1209 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.502s	user 0.008s	sys 0.000s
I20260709 12:20:30.802403  1362 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.127s
I20260709 12:20:30.879843  1316 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:30.890908  1363 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:30.801334  1120 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:30.860715  1297 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:20:30.941742 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.942448 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:30.910248  1111 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.687s	user 0.007s	sys 0.002s
I20260709 12:20:30.931885  1373 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:30.924515  1143 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.000s
I20260709 12:20:30.853827  1197 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.936300  1322 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:30.934610   996 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:30.954421 27580 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:30.974020 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:30.934108   959 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:30.975132 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:30.979461 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:30.969686  1364 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:20:30.890375  1229 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.001760   681 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.004745 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:30.974377 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 134325us
W20260709 12:20:31.218081 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 469194us
I20260709 12:20:31.034968  1311 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:30.972285  1373 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:30.951804  1209 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.057845 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43842.2us
I20260709 12:20:30.952348  1224 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.059715  1280 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.696s	user 0.009s	sys 0.003s
I20260709 12:20:30.971704  1111 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.209921  1216 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:20:31.073983 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 173902us
I20260709 12:20:30.993999  1264 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260709 12:20:31.007627  1033 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.094426 27738 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:31.063669  1187 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.076642  1115 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:31.182842 27585 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:20:31.085168  1376 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:30.923959 27481 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 16, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 16 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:31.073621  1279 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.102s
I20260709 12:20:31.148298  1315 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.275317  1315 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.276140  1382 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:31.278710  1315 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.148594  1241 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:31.279857  1291 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.281724  1219 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.003s
I20260709 12:20:31.199862 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:31.213311 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:31.283428 27732 raft_consensus.cc:606] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:31.283946 27732 raft_consensus.cc:614] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:30.994527  1190 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:31.284724  1190 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:31.285987  1190 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.287962  1366 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20260709 12:20:31.218968 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:31.289502 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70616.1us
I20260709 12:20:31.290339 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.235517 27662 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:20:31.222107  1044 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.630s	user 0.008s	sys 0.005s
I20260709 12:20:31.249325 27655 raft_consensus.cc:2368] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:20:31.294152  1044 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:31.016456  1299 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.295725 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:31.298509 27657 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:31.302505  1070 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:20:31.304011  1378 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.155s	user 0.001s	sys 0.001s
I20260709 12:20:31.306711 27730 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:20:31.308199 27730 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:20:31.310882  1293 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:31.311671  1293 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.314483  1293 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.268393 27737 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:20:31.281000  1241 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:31.345844 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91576.4us
I20260709 12:20:31.303020  1070 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:31.347182 27656 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:20:31.229557  1373 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:31.244740  1280 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
W20260709 12:20:31.359102 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.022s	sys 0.021s
W20260709 12:20:31.359493 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.022s	sys 0.021s
W20260709 12:20:31.359870 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 514684us
I20260709 12:20:31.361279 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:31.363569 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:31.369287 27736 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:31.269622  1187 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:31.355989  1188 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.992s	user 0.012s	sys 0.000s
I20260709 12:20:31.395430 27581 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
W20260709 12:20:31.359512  1370 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.365s	user 0.001s	sys 0.000s
I20260709 12:20:31.333758  1293 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:31.403443 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:31.254668  1297 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.123s
I20260709 12:20:31.403470 27581 pending_rounds.cc:85] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:20:31.356983  1190 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:31.348629  1315 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:31.407800 27585 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:20:31.463011 27581 pending_rounds.cc:107] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:20:31.470217 27581 raft_consensus.cc:2889] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:31.356376  1289 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
W20260709 12:20:31.471527 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 252608us
I20260709 12:20:31.449720   959 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:31.448760 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 115941us
I20260709 12:20:31.450284  1207 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:31.475508 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 234157us
I20260709 12:20:31.406772  1187 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.380240  1381 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.127s
W20260709 12:20:31.494890  1189 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.725s	user 0.014s	sys 0.000s
I20260709 12:20:31.494943 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.408658  1188 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.473078  1380 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:31.501526  1286 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.840s	user 0.010s	sys 0.005s
I20260709 12:20:31.426644 27733 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:31.495831  1122 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:31.520015 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.521143 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.526868 27586 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:31.530297 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:31.533049 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:31.534399 27731 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:31.516520  1315 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:31.536504  1315 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.529405  1122 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.542377  1291 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.543784  1033 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.474468  1070 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.548070  1387 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 12:20:31.511828  1392 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.549196 28335 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:31.549255  1392 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.550642 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 926) took 1137 ms. Trace:
I20260709 12:20:31.519419  1189 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.551070 27729 rpcz_store.cc:276] 0709 12:20:30.413206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.413344 (+   138us) service_pool.cc:224] Handling call
0709 12:20:30.448078 (+ 34734us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:30.897945 (+449867us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:30.897983 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:30.898525 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:20:31.207801 (+309276us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:31.207864 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:31.207987 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:31.550371 (+342384us) raft_consensus.cc:1685] finished
0709 12:20:31.550395 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:31.550575 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28804,"raft.queue_time_us":53621,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":10168,"thread_start_us":713825,"threads_started":2,"wal-append.queue_time_us":485953}
I20260709 12:20:31.553664  1189 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.521545  1381 consensus_queue.cc:1243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:20:31.554920  1355 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.878s	user 0.006s	sys 0.004s
I20260709 12:20:31.555647  1384 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.556473 27582 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:31.556236   996 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:31.519160  1383 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:31.559127  1187 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.521200  1375 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:20:31.562863  1188 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.522287 27481 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:31.519991  1286 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.568820 28332 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:31.570192  1286 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:31.574743  1316 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.614s	user 0.010s	sys 0.001s
W20260709 12:20:31.575350  1370 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:20:31.556268  1384 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.576313  1187 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.576915 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 928) took 1162 ms. Trace:
I20260709 12:20:31.576817  1187 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.577947  1384 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.578910  1187 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.520155  1380 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.581596  1380 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:31.582459 27663 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:31.584520 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:20:31.586197 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:31.586299 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 942) took 1149 ms. Trace:
I20260709 12:20:31.539005  1315 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:31.587021 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56821.2us
I20260709 12:20:31.592348 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:31.592476 27481 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:31.577287 27734 rpcz_store.cc:276] 0709 12:20:30.414579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.414760 (+   181us) service_pool.cc:224] Handling call
0709 12:20:30.415187 (+   427us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:30.462247 (+ 47060us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:30.462282 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:30.509682 (+ 47400us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:30.519452 (+  9770us) log.cc:844] Serialized 52 byte log entry
0709 12:20:30.748065 (+228613us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:30.757612 (+  9547us) log.cc:844] Serialized 14 byte log entry
0709 12:20:30.764079 (+  6467us) log.cc:844] Serialized 14 byte log entry
0709 12:20:30.933910 (+169831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:30.934000 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:31.559836 (+625836us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b180037d200. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:31.576705 (+ 16869us) raft_consensus.cc:1685] finished
0709 12:20:31.576725 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:31.576856 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87991,"prepare.queue_time_us":84862,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":31298,"raft.queue_time_us":222735,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":50974,"spinlock_wait_cycles":18608000,"thread_start_us":282434,"threads_started":2,"wal-append.queue_time_us":357430}
I20260709 12:20:31.586694 28332 rpcz_store.cc:276] 0709 12:20:30.421595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.519122 (+ 97527us) service_pool.cc:224] Handling call
0709 12:20:31.571115 (+1051993us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:31.571264 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2088276352}
I20260709 12:20:31.590685 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:31.604146 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:31.594913  1209 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:31.609256 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 318964us
I20260709 12:20:31.613776 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.610606 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:31.620457 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:31.633955 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:31.634611 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:31.635187 27587 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:31.636615 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.594971  1392 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.606945  1033 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:31.575214  1316 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.645767 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:31.648082 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.608688  1383 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:31.632097  1389 raft_consensus.cc:2764] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:20:31.655242 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.664191 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.666396 27587 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:31.668435 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:31.637789  1189 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:31.670037 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.669693 27552 leader_election.cc:400] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:20:31.670773 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.672784 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.673533 27735 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:31.673683 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.677371 27730 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:31.678625 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:31.680312 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.681348 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.683218 27735 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:31.683864 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.663585  1281 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:31.652087  1397 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:31.643702  1209 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:20:31.643888  1315 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:20:31.594074 27481 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:31.668579  1142 raft_consensus.cc:2764] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:20:31.668634  1372 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:31.691377 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.644134   959 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:20:31.682108  1033 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:31.683267  1384 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:31.687075  1354 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.883s	user 0.013s	sys 0.000s
I20260709 12:20:31.681453  1392 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:31.677469  1400 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:31.666771  1383 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.696504  1275 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:20:31.689822  1281 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:31.703541 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.711035 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:31.713977  1316 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.715251  1383 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.716486 27481 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 10 to 16, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 16 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:31.706115  1177 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.696518  1181 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:31.721827 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 925) took 1310 ms. Trace:
I20260709 12:20:31.709303  1400 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:31.709059  1101 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.778s	user 0.008s	sys 0.004s
I20260709 12:20:31.685567  1375 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:20:31.708287  1397 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:31.697593  1372 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.736793 27738 rpcz_store.cc:276] 0709 12:20:30.411565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.411711 (+   146us) service_pool.cc:224] Handling call
0709 12:20:31.721592 (+1309881us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:31.721730 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1338338048}
W20260709 12:20:31.745647 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49353.6us
I20260709 12:20:31.746165 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:31.746371 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:31.746737 27738 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:31.720253 27730 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:31.718693  1275 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.748070 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:31.748234 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:31.755270 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.720149  1291 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:20:31.780184  1281 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.783807 27738 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:31.785775  1281 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:31.786892 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:31.779520  1033 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:31.787739  1033 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.787873  1293 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:31.745168  1101 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.788223  1033 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:31.788471  1293 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:31.789395  1293 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.778337 27730 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:31.707763  1354 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.791468  1374 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:31.792268 28331 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:31.793596 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:31.744077  1400 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:31.748981  1383 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:31.777563  1316 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:31.754307  1275 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:31.735069  1402 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:20:31.738626  1177 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.804490  1375 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:31.809854  1375 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:31.810314  1408 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:20:31.810753  1375 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.812006  1375 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:31.835563  1298 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:31.803134  1189 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:31.838326  1189 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.838781  1189 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:31.838858   959 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:31.839520   959 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.840971  1362 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:31.841354  1362 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:31.841751   959 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.842176  1362 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.835711  1177 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.803858  1316 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.848781 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:31.858515 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:31.853372 27661 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:31.854305  1181 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:31.856006 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:31.861552 27663 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:20:31.856753  1354 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.864619 27658 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:31.864866 27659 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:20:31.865249 27659 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 9
I20260709 12:20:31.865481 27659 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:20:31.865367 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:31.865795 27659 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:31.865644 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.861012  1316 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:31.860067 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.806978 27481 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 18, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:20:31.867830 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:31.852191 27660 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:31.843257  1362 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:31.871873 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.872574 27663 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 11.
I20260709 12:20:31.873489  1354 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:31.873672 27481 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 19 to 22, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 22 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:31.878772  1322 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:31.879738  1120 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:31.879951 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.870016 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:31.882330 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.884377 27660 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:20:31.887874 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.857203  1307 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:31.891853 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:31.917804 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.919729 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:31.920948 27660 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:31.888720  1364 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:31.936375  1364 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.939366  1364 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:31.882339  1375 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:31.940076  1307 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.946341  1307 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.939577 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:31.939297 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:31.948282 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:31.948478 27664 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:31.949738 27659 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:31.888839  1322 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:31.961558  1322 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.963619  1280 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:31.964193  1368 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.104s	user 0.009s	sys 0.007s
I20260709 12:20:31.964448 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:31.964490  1280 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:31.966118 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:31.966282  1214 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:31.967060  1367 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:31.967064 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2937) took 1514 ms. Trace:
I20260709 12:20:31.968720 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2942) took 1028 ms. Trace:
I20260709 12:20:31.968359 27589 rpcz_store.cc:276] 0709 12:20:30.452877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.501948 (+ 49071us) service_pool.cc:224] Handling call
0709 12:20:30.507831 (+  5883us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:30.508972 (+  1141us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:30.509011 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:30.509534 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 12:20:30.579554 (+ 70020us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b180039f220. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:30.644702 (+ 65148us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:30.644759 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:30.644849 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:31.963569 (+1318720us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:31.966869 (+  3300us) raft_consensus.cc:1685] finished
0709 12:20:31.966888 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:31.967006 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":697,"raft.queue_time_us":806,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":10836,"spinlock_wait_cycles":38173312,"thread_start_us":64921,"threads_started":1,"wal-append.queue_time_us":252828}
I20260709 12:20:31.969331 27582 rpcz_store.cc:276] 0709 12:20:30.939935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.942428 (+  2493us) service_pool.cc:224] Handling call
0709 12:20:30.942809 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:31.556882 (+614073us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:31.968492 (+411610us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b54001248d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:31.968662 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1410781056}
I20260709 12:20:31.969626  1033 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.922358  1181 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:31.941457  1291 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.976083  1291 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.977111  1373 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.583s	user 0.009s	sys 0.001s
I20260709 12:20:31.977550  1373 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:31.980693  1407 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:20:31.954742  1120 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.983485  1372 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:31.985038  1406 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:20:31.956362  1298 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:20:31.988535 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:31.990409  1275 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.889906  1354 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:31.992516 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2790) took 1568 ms. Trace:
I20260709 12:20:31.981879 27481 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 15, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 15 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:31.993858  1354 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:31.995458  1181 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:31.996342 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 927) took 1582 ms. Trace:
I20260709 12:20:31.996555  1181 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:31.996784 27731 rpcz_store.cc:276] 0709 12:20:30.413974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.414098 (+   124us) service_pool.cc:224] Handling call
0709 12:20:30.414360 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:31.534776 (+1120416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:31.996104 (+461328us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:31.996284 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2575279872}
I20260709 12:20:31.967258 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:32.000512 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:32.001437 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.003368 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.005376 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.000172 27737 rpcz_store.cc:276] 0709 12:20:30.423821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.425427 (+  1606us) service_pool.cc:224] Handling call
0709 12:20:31.992335 (+1566908us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:31.992461 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1935242624}
I20260709 12:20:32.016553 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:31.963116  1322 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:32.022535 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:31.994606  1291 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.029181  1219 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:31.966132  1101 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:31.985071  1120 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:32.016508  1372 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:32.022112  1192 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:20:32.084162 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48954.6us
I20260709 12:20:32.022401  1364 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:31.958582  1190 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:20:32.092589  1363 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:32.093060  1363 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:32.093987  1363 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.097469 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.098186 27659 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:20:32.098695 27656 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:32.099452 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.100301 27658 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:32.100878 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.101513 27659 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 11.
I20260709 12:20:32.103222 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:32.105549  1416 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20260709 12:20:32.026163  1281 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:32.088246 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:32.089188  1122 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:32.108055 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.089625 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.095074  1363 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:32.110776 27737 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:32.110706 27659 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:32.032058  1033 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.130741 27663 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:20:32.138463 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.087612  1390 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.024099  1298 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.085s
I20260709 12:20:32.109616  1373 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.150444 27729 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:20:32.152874 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.127646  1192 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:32.140723  1122 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.139515  1219 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:32.193728  1219 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:32.194190  1219 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:32.195737  1298 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:32.196221  1298 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:32.197130  1298 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.198418  1298 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:32.199935 28330 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:20:32.201469 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.203245 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:32.203935 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.204615 27586 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:32.204664 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.205646 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:32.209724 27583 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:20:32.163686 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.165203 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.170145 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2463) took 1711 ms. Trace:
I20260709 12:20:32.172415 27734 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:20:32.174252  1414 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.348s	user 0.000s	sys 0.003s
I20260709 12:20:32.212347 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.154307  1390 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.212409 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:32.185173  1275 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.185765  1070 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:32.085098 27481 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 19, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 19 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.188194  1111 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:32.188792  1420 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 12:20:32.189777  1419 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:20:32.110033  1372 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.190399  1122 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.211809 27733 rpcz_store.cc:276] 0709 12:20:30.458240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.769807 (+311567us) service_pool.cc:224] Handling call
0709 12:20:32.169969 (+1400162us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:32.170088 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298733312}
I20260709 12:20:32.139844  1189 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.137981  1316 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:32.205648  1405 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.396s	user 0.001s	sys 0.002s
I20260709 12:20:32.161267  1315 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:20:32.213522  1367 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:32.222177  1279 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:32.228603  1224 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.020s	sys 0.064s
I20260709 12:20:32.228200  1364 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:32.228817  1179 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.209553  1383 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:20:32.213363  1233 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:20:32.264909 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:32.265975  1307 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:32.267832  1147 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:32.269315 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.272029   959 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:32.275425 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:32.220618  1264 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
W20260709 12:20:32.259605  1224 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.021s	sys 0.064s
W20260709 12:20:32.315068 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45894.1us
I20260709 12:20:32.325138 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:32.331571 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:32.259840  1367 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.261236  1279 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.343456 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.355780 27655 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:32.348609 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 2951) took 1064 ms. Trace:
I20260709 12:20:32.327828 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:32.289041  1192 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.357708 27580 rpcz_store.cc:276] 0709 12:20:31.284195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:31.324989 (+ 40794us) service_pool.cc:224] Handling call
0709 12:20:31.325359 (+   370us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:20:31.325717 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:31.325748 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:20:31.326345 (+   597us) log.cc:844] Serialized 52 byte log entry
0709 12:20:31.652966 (+326621us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:20:31.653025 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:31.653118 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:32.205053 (+551935us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:32.348301 (+143248us) raft_consensus.cc:1685] finished
0709 12:20:32.348321 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:32.348541 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44447,"raft.queue_time_us":56414,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":18807,"spinlock_wait_cycles":44332288,"thread_start_us":326072,"threads_started":1,"wal-append.queue_time_us":454059}
I20260709 12:20:32.366065 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.367012 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.366259 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.369657 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:32.370577 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:32.371523 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.372110 27737 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:32.376015 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.376415 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.287941  1179 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.377964 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.378860 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
W20260709 12:20:32.379019 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51294.5us
I20260709 12:20:32.381012 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:32.382650 27738 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:32.383260 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:32.328235  1147 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:32.383693 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.384287 28337 raft_consensus.cc:2368] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:20:32.388451 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.392174 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.342971  1181 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:32.393020 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.394366 28328 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:32.352723  1316 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.388731 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.400063 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:32.312927   959 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.391659 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.403637 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.404351 27660 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:32.390625 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.396862 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.390218 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:32.399360 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.407492 27655 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:32.389102 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.408653 27664 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:32.410300 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:32.385161  1400 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.000s
I20260709 12:20:32.411917 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.414196 27737 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:32.416337 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.388307  1120 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:32.417274 27582 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:32.388880  1315 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:32.418263 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 93212.9us
I20260709 12:20:32.428874 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.437752 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:32.429396  1362 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.002s
I20260709 12:20:32.420482  1400 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.440840 27661 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:32.440214 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.442060  1400 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.427699 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:32.438589 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.399189  1275 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:32.413215  1417 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.616s	user 0.003s	sys 0.000s
I20260709 12:20:32.445374 27655 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:32.449905 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.431952 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:32.413378  1177 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:32.450940 27660 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:32.456046 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:32.422564  1316 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:32.457702  1122 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:32.458662   959 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:32.462924 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.469214 27587 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:32.463280 27660 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:32.464135 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.467053 27730 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 12:20:32.476588 27730 pending_rounds.cc:85] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 10
I20260709 12:20:32.476835 27730 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:20:32.477066 27730 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:20:32.473011 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76414.6us
I20260709 12:20:32.445267  1343 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:20:32.459906  1440 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:32.496701 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.465617  1362 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:32.504642 27664 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:32.500577 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.501484 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.465132  1316 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:32.474720  1374 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:32.448311  1315 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:32.503507 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65028.9us
I20260709 12:20:32.474997  1177 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:32.519958 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.495182  1390 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:32.498323  1186 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:32.509328  1386 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:32.395396 27481 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 21. New cstate: current_term: 21 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:32.562440 27481 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 18, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 18 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:32.572124 27481 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 18. New cstate: current_term: 18 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:32.550868 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:32.573700 28328 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:32.505874   959 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:32.509593  1372 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.553831 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:32.575752  1429 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:20:32.576193  1372 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:20:32.577311  1431 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:20:32.578367 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:32.579092  1433 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:20:32.579878 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.581466 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.505714  1362 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:32.555809  1343 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:32.556587  1275 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:32.560195  1177 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.547799 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:32.551048  1245 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:32.548909 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87160.8us
I20260709 12:20:32.526015 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:32.586634 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 130735us
I20260709 12:20:32.516070  1316 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:32.514940  1441 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:32.613338 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.513537  1120 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:32.620725 27660 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:32.620512 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:32.622174 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96293.1us
W20260709 12:20:32.572153  1427 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 12:20:32.629817 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.626385 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:32.630146 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:32.630450 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:32.552366  1386 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:32.631743 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:32.631815 27589 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:32.590730  1362 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:32.606174  1372 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:32.633836 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.616528  1264 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:20:32.653187 27702 leader_election.cc:400] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:20:32.653661 27702 leader_election.cc:403] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:20:32.618240  1343 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.655741 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 975) took 3756 ms. Trace:
I20260709 12:20:32.656751  1343 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.656158 27588 rpcz_store.cc:276] 0709 12:20:28.899708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:28.900152 (+   444us) service_pool.cc:224] Handling call
0709 12:20:29.800129 (+899977us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:29.800553 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:29.800594 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:29.810192 (+  9598us) log.cc:844] Serialized 26 byte log entry
0709 12:20:30.325509 (+515317us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b18004235e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:30.557663 (+232154us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:30.557703 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:30.564699 (+  6996us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:31.381886 (+817187us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:31.557459 (+175573us) raft_consensus.cc:1685] finished
0709 12:20:31.557488 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:32.655541 (+1098053us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b540008a2d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:32.655674 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":540,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":4555,"spinlock_wait_cycles":2088824704,"thread_start_us":231929,"threads_started":1,"wal-append.queue_time_us":798880,"wal-append.run_cpu_time_us":11716,"wal-append.run_wall_time_us":1968649}
I20260709 12:20:32.661374   959 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:32.664085  1439 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:32.647392  1219 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.668820 27735 raft_consensus.cc:2368] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21.
W20260709 12:20:32.669797  1436 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:20:32.670452  1443 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:20:32.671461  1279 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:32.636552  1233 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:20:32.637320  1190 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:32.626327  1245 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.633232  1390 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:32.645053  1402 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:32.648535  1367 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:32.637928  1445 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:20:32.680791 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2471) took 1034 ms. Trace:
I20260709 12:20:32.681181  1233 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:32.681736  1233 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:32.675788  1402 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:32.684631  1402 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:32.685195  1438 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.174s	user 0.002s	sys 0.000s
I20260709 12:20:32.681279 27734 rpcz_store.cc:276] 0709 12:20:31.646791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:31.647641 (+   850us) service_pool.cc:224] Handling call
0709 12:20:32.172016 (+524375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:32.212484 (+ 40468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:32.680514 (+468030us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54000a73d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:32.680717 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1205186432}
I20260709 12:20:32.696535 27734 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: [16.10-18.11]
I20260709 12:20:32.700399 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.702137 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:32.703668 28334 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:20:32.711895 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:32.712451 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63276.4us
I20260709 12:20:32.718394 27734 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:20:32.718816 27734 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 8
I20260709 12:20:32.719504 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:32.721701 27734 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:20:32.721967 27734 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:32.721915 27657 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:32.721100 27661 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:20:32.652786  1192 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:32.723788 27664 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:32.725808 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:32.730209 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.730844 27737 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:32.682638  1233 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.732055 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:32.734684  1233 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:32.742758 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:32.711517  1315 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:32.755589 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 2791) took 2307 ms (client timeout 2161 ms). Trace:
W20260709 12:20:32.756201 27736 rpcz_store.cc:269] 0709 12:20:30.434620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:30.434738 (+   118us) service_pool.cc:224] Handling call
0709 12:20:32.741769 (+2307031us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b540017f2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:32.741908 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2051374976}
I20260709 12:20:32.741194  1120 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.774966 27734 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
W20260709 12:20:32.763391  1298 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.565s	user 0.014s	sys 0.000s
I20260709 12:20:32.732834  1397 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:20:32.787276 27738 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:20:32.785303  1192 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:32.798781  1070 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.002s
I20260709 12:20:32.732432  1386 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.823840 27663 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:32.796905  1279 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.804461  1377 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
W20260709 12:20:32.833743 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 193330us
W20260709 12:20:32.834250 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 139412us
I20260709 12:20:32.824059  1363 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:32.829149  1245 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:32.804435  1192 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:32.840948 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:32.848979 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.849932 27736 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:32.851261 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:32.853421 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:32.834884  1147 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:32.856065 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.868055 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.831511  1279 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:32.876768  1447 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:20:32.881419  1279 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:32.856640  1446 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:32.839720  1363 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.874863  1400 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:32.865362  1397 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:20:32.891898  1443 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:32.892445  1241 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
W20260709 12:20:32.919623  1453 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:20:32.920261  1241 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:32.920650  1241 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:32.921448  1241 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.923286  1241 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:32.886746  1206 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
W20260709 12:20:32.959098 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91142.5us
I20260709 12:20:32.959820  1397 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:32.959985 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:32.960474 27627 leader_election.cc:400] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21.
I20260709 12:20:32.960863 27627 leader_election.cc:403] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:20:32.960757  1400 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:32.961526  1374 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:32.962009 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:32.962939  1400 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:32.964881  1400 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:32.966145  1400 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:32.968567 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:32.969303 27737 raft_consensus.cc:2376] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21. Re-sending same reply.
I20260709 12:20:32.970552  1374 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21.
I20260709 12:20:32.975787 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:32.976452 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:32.977066 27663 raft_consensus.cc:606] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:32.996551 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:33.004215 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 149656us
I20260709 12:20:32.995644  1465 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:33.021669  1456 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:33.022243  1456 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.022739  1456 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:33.027537  1457 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:33.028195  1457 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.028687  1457 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:33.032807 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:33.033468 27663 raft_consensus.cc:606] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:32.987869  1190 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:33.004488  1223 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:20:33.050417 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.060787 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "723b2024e4414796ba50ece654596156"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:33.062039 27663 raft_consensus.cc:606] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.062520 27663 raft_consensus.cc:614] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:33.049319  1223 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:33.047137  1190 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:33.076814  1470 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:20:33.078233  1460 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:20:33.076932  1190 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.080840  1189 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.083021  1192 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.085691  1363 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:33.086808 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.088132 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:33.094488 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90637.7us
I20260709 12:20:33.096891 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:33.097560 27734 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:33.097638  1362 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.103660  1223 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.116065 27734 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:33.118021 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.122977 27734 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:20:33.124725 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:33.128548  1362 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:33.130277  1457 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.134287 27659 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:33.114848  1367 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.135697  1367 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.137845  1367 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.138453  1316 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.140563  1316 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.143031 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.143245 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.143776 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.145151 27664 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:20:33.146955 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.147737 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.147835  1354 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:33.147778 27730 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:20:33.155984  1456 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.158468 28332 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:33.162158 27734 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:33.164498 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.166438 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:33.167070 27730 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:33.168056  1281 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:33.168454  1281 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:33.169193  1219 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:33.169253  1281 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.170621  1281 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:33.148346  1354 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:33.171967  1402 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:33.172533  1354 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:33.173393  1486 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:20:33.173638  1354 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:33.174034  1396 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:20:33.160984  1457 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.180557 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:33.181030 27730 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:33.182710 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:33.186408 27659 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:33.174403  1396 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:33.189837  1443 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:20:33.201337 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.213965  1456 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.214807  1396 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.215961  1396 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:33.217110 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.217645  1441 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:33.218228  1441 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.206550  1443 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:33.218969 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:33.219627 27734 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:33.221078  1443 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.223950  1452 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:33.224087 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 492105us
I20260709 12:20:33.224561  1452 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.226636  1452 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.227335  1147 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.227830  1147 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.229485  1315 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:33.229817  1147 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.230135  1315 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:33.230311  1147 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.230785  1147 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.219520  1264 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:33.232518 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:33.244342 27587 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:33.244498 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.234619 27734 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:33.240139 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:33.227182  1452 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:33.249812 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:33.252383 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.202214  1457 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:20:33.231252  1315 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.285630 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:33.266914 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.278165 27737 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
W20260709 12:20:33.316907 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64637.8us
I20260709 12:20:33.300885 27587 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:20:33.320173 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.322407 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:33.277370 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.323290 27664 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:33.323604 27663 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:33.291009 27655 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
W20260709 12:20:33.260386  1474 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.217s	user 0.004s	sys 0.000s
W20260709 12:20:33.327647 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81347.3us
I20260709 12:20:33.329702 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.329612 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.333976  1147 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.335642  1404 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:33.252810  1452 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.348580 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.276924 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:33.343041 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:33.344264 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.349391 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.261008  1311 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:33.352842 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.314878  1111 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:33.364848 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:33.277432  1466 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.368868 27730 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:33.354389 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.275638  1186 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.276110  1441 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.367100 27584 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:33.248404  1223 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.104s
I20260709 12:20:33.372303 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.373154 27733 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:33.377077 27584 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
W20260709 12:20:33.350849  1433 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:20:33.377887 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:33.378374 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:33.352910  1452 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:33.381533 27582 raft_consensus.cc:2368] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:33.380930 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:33.380790 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:33.383656 27663 raft_consensus.cc:606] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:33.383737 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.385782 27587 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:33.388751 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:33.326625  1315 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.371338  1404 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.371177  1147 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.404786 27656 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:33.406679 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:33.347666  1120 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:20:33.402021 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.412025 27580 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:33.379866  1241 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:33.379473  1311 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:33.412531  1385 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.419167  1385 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.421600  1385 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.380051  1186 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.424114  1315 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.425206  1186 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:33.426195  1473 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.426743  1473 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.278774  1264 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.430948 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.409420  1404 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:33.408809  1367 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:33.375931  1111 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.397996  1363 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:20:33.374719  1466 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.444120 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.448655 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:33.439246   996 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:33.415302  1147 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.453199  1241 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.457746  1363 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:20:33.449385  1376 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.441095  1120 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:20:33.459825  1311 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.449581  1367 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:33.461423  1475 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.417s	user 0.002s	sys 0.000s
I20260709 12:20:33.467664 27585 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:20:33.467559  1463 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:33.468384  1179 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.471292   996 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:33.481859 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.495697  1435 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:33.500289 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:33.500643 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.500947 27580 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:20:33.482870 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:33.501845 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.502919 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:33.489921 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.505707 27586 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:33.492266 27656 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:20:33.492743 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44172us
I20260709 12:20:33.508710 27552 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:20:33.509193 27552 leader_election.cc:403] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:33.510576 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.494299 27582 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 11. Current term is 12. Ops: []
W20260709 12:20:33.497411 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53354.3us
I20260709 12:20:33.444865  1375 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:33.512184 27586 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:33.488901  1363 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:33.496822  1373 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:33.516511  1197 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:33.518072  1316 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:33.520498 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:33.519549 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.515964 27630 leader_election.cc:400] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:33.499862  1397 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:33.513505 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.546657 27586 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:33.546746 27580 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:33.549402 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.508566  1390 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:33.492080  1241 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.511866  1186 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:33.551663 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:33.551970 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:33.532840  1147 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.559552 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.548504  1264 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:20:33.561224 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.537475  1111 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.558691 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:33.563764 27729 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:33.522884  1322 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:20:33.563812 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.548719  1373 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:33.555013  1219 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:33.555935  1186 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.556074  1316 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:33.504645  1435 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.496332  1376 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.574397 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:33.513700  1463 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.560895  1197 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:33.553853  1070 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:20:33.559665  1364 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:33.564716   996 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.557813  1293 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.017s
W20260709 12:20:33.610044  1493 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:20:33.566423  1224 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:20:33.559552  1396 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:20:33.603669 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.608954 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:33.623144 27586 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:20:33.628684 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.630729 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:33.576488  1496 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:20:33.582445  1322 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:33.607805  1111 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:33.648504 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.562338  1367 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.586133  1435 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.651899  1241 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.656286 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.647948  1111 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.657747  1111 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:33.642304  1214 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:33.661279  1214 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.661628  1214 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:33.671273  1400 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:33.672441  1376 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.673462  1373 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.625159  1224 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.674836 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2906) took 1551 ms. Trace:
I20260709 12:20:33.612197  1463 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.679298 27658 rpcz_store.cc:276] 0709 12:20:32.123569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:32.123694 (+   125us) service_pool.cc:224] Handling call
0709 12:20:32.123998 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:33.149259 (+1025261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:33.149303 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:33.149353 (+    50us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:33.149402 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:33.674625 (+525223us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:33.674663 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:33.674787 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2357302656}
I20260709 12:20:33.607492  1390 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:20:33.681690 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.577379  1186 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:33.691254 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:33.696419 27731 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 9
I20260709 12:20:33.697052 27731 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 12:20:33.697376 27731 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:33.705133 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:33.705914 27663 raft_consensus.cc:606] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:33.706318 27663 raft_consensus.cc:614] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:33.702256 27658 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:33.711525  1452 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:33.713701  1499 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
W20260709 12:20:33.721737  1467 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.681s	user 0.003s	sys 0.000s
I20260709 12:20:33.725158  1363 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.726814  1363 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:20:33.730229  1501 raft_consensus.cc:993] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:33.730183 27587 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:33.730854  1070 raft_consensus.cc:1081] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:33.732878  1386 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:33.741106 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:33.746678  1505 raft_consensus.cc:993] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:33.747504  1505 raft_consensus.cc:1081] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:33.690960  1466 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:33.685338  1224 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.752518 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.753165 27729 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:33.754374  1224 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.755481  1452 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.756183  1111 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.761094 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.761716 28334 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:33.751089 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.764775 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
W20260709 12:20:33.766120 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47817.7us
I20260709 12:20:33.770586  1375 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:33.771347 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:33.771212  1375 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.772161  1224 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:33.772825 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:33.773072  1224 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:33.775787 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.778072 28331 raft_consensus.cc:2410] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:20:33.773736 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:33.779789  1435 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:33.798933 27663 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:20:33.808665 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:33.824707 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:33.786545  1382 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:33.812899 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:33.832840 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59179.1us
I20260709 12:20:33.815429 27663 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 9
I20260709 12:20:33.833840 27663 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 12:20:33.834120 27663 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:33.816697 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.836441 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:33.837267 27738 raft_consensus.cc:2368] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:33.796432  1375 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.821832  1111 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:33.822525 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:33.783648  1435 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.812556 27661 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:20:33.832209 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.838582 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.813823  1444 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:33.840304 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.846155 27627 leader_election.cc:400] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:33.868582 27627 leader_election.cc:403] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:33.820314  1186 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.858562 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:33.797884  1281 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:33.811797  1224 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:33.840148 27732 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:33.866740 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:33.876487 27580 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:33.876853 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:33.813123  1392 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:33.878937  1392 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:33.877985 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:33.840590  1501 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:33.838976  1382 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:33.866894  1435 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:33.873678  1508 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 18
W20260709 12:20:33.900655 27628 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b423a3b3ee1452fb419df4c9c0a8d67. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:33.874452  1224 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:33.889213  1501 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.885028  1281 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.922869  1501 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:33.904260  1375 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:20:33.959967 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.964113 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:33.965677 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:33.966475 27585 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:33.936792  1322 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.967494 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:33.938899  1216 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.938134  1362 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:33.971714  1214 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.003545 27732 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:33.993764  1142 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:34.005254  1142 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.005575  1392 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.013904  1186 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.003188  1214 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.017841 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:34.019927 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.036199  1281 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.016902  1216 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.043089  1216 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.044698 27587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:20:34.045295 27587 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:34.045660 27587 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:20:34.048159  1513 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:20:34.049733 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.057175 27588 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:34.058055 27588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:20:34.058595 27588 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:34.058904 27588 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:34.059305 28328 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:34.016749  1224 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.060707  1224 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:34.059353 27582 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:34.068567 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:34.058046 27587 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:33.937176  1382 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.068768 27588 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.073621 27582 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:34.024857  1142 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.080681 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:34.081368 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:33.996798  1322 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:34.067263  1512 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:20:34.088452 27588 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.082891 27587 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.093017 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:34.094910 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:34.097827  1070 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:34.106647  1434 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:20:34.103232  1505 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:34.149210  1376 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:20:34.152014  1404 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.709s	user 0.005s	sys 0.006s
I20260709 12:20:34.152453  1404 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.152881  1435 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.153565 27589 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:34.153990 27658 raft_consensus.cc:2368] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21.
I20260709 12:20:34.155006  1435 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.155450  1404 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.128146  1392 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:20:34.094105  1206 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:34.157958 27583 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:20:34.160208 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:34.163348 27581 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 21
W20260709 12:20:34.167963  1516 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 12:20:34.178237 27581 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:34.181003 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:34.126637  1033 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:34.259346  1364 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:34.260171  1364 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:34.224293  1435 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:34.179733  1209 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:20:34.223984  1033 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:34.272394  1367 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.280236  1501 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.283277  1501 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.296872 27587 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:34.276404  1392 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:34.303851 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:34.304461 28337 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:34.295960  1435 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:34.304329  1505 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:20:34.314226  1505 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:34.320828 28337 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:34.283464  1367 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.322576 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:34.311499  1400 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:34.314554  1505 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:34.308926  1033 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.314615  1364 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.303539  1392 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.339246  1115 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:34.330816  1524 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:34.338140  1443 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
W20260709 12:20:34.345325  1468 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.302s	user 0.003s	sys 0.008s
I20260709 12:20:34.330236  1400 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.351874  1524 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:34.355479  1115 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.356514  1524 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.358068  1115 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.354068 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2433) took 1513 ms. Trace:
I20260709 12:20:34.355273  1400 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.351292  1508 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:34.359290 27662 rpcz_store.cc:276] 0709 12:20:32.840774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:32.840955 (+   181us) service_pool.cc:224] Handling call
0709 12:20:32.841274 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:32.841673 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:20:32.841705 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:32.842191 (+   486us) log.cc:844] Serialized 26 byte log entry
0709 12:20:32.891624 (+ 49433us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:32.891686 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:32.891780 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:34.308378 (+1416598us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:34.353718 (+ 45340us) raft_consensus.cc:1685] finished
0709 12:20:34.353748 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:34.353995 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2121,"raft.queue_time_us":1428,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":10612,"spinlock_wait_cycles":4327808,"thread_start_us":49089,"threads_started":1,"wal-append.queue_time_us":200862}
I20260709 12:20:34.361368  1322 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.362401  1179 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:34.338680  1364 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:34.342104  1033 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:34.362816  1179 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:34.426409 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.362756  1362 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:34.331600  1435 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.337486  1122 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.470137 27729 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:34.471612 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:34.515285 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.531963 27732 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:20:34.538733 27657 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:34.541815 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:34.542428 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:34.543138 28331 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:34.548384 27737 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:20:34.555272 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:34.340437  1505 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.561321 27737 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:34.563424 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:34.563475 27657 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:20:34.490554  1362 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:34.474244  1264 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:34.589802 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:34.602087 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:34.604658 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:34.606777 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 259375us
I20260709 12:20:34.607301 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:34.571962  1216 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.616230 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.617300 27581 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:34.544010  1122 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:34.646080  1506 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:20:34.653837  1367 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.655176  1537 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:34.656242 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:34.656508  1537 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:34.656814 27661 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:34.608110  1505 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:34.657153 27661 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:34.657950 27661 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.658834 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:34.661679 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:34.662388 27733 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:34.591032  1264 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.627545  1216 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.603008  1473 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:34.651062  1224 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.590s	user 0.007s	sys 0.003s
I20260709 12:20:34.596428  1179 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.726080 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:34.729146  1264 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.729959  1216 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.730170  1374 raft_consensus.cc:993] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:34.730759  1374 raft_consensus.cc:1081] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:34.732920 27738 raft_consensus.cc:2368] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:34.734476  1179 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:34.738234 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2964) took 1371 ms. Trace:
I20260709 12:20:34.738636 27589 rpcz_store.cc:276] 0709 12:20:33.366612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:33.366956 (+   344us) service_pool.cc:224] Handling call
0709 12:20:33.369184 (+  2228us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:34.153984 (+784800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:34.737999 (+584015us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54002ba7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:34.738178 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1803367808}
I20260709 12:20:34.703459 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:34.743391  1518 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.649s	user 0.007s	sys 0.003s
I20260709 12:20:34.746472 27588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:20:34.747229 27588 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:34.747678 27588 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:34.751766 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:34.711301 27730 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:20:34.617957  1142 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.717208  1122 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:34.719666  1392 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.756117 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:34.722704 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:34.757619 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2973) took 1193 ms. Trace:
I20260709 12:20:34.763012 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:34.763840 27733 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:34.758334 27586 rpcz_store.cc:276] 0709 12:20:33.564669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:33.565189 (+   520us) service_pool.cc:224] Handling call
0709 12:20:33.565475 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:33.623866 (+ 58391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:34.757408 (+1133542us) spinlock_profiling.cc:243] Waited 57.2 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:34.757569 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131599104}
I20260709 12:20:34.768673 27658 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260709 12:20:34.757937  1142 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.777446  1142 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.778472  1483 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:34.779075  1483 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.755353  1229 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
W20260709 12:20:34.781772 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.783859  1122 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:34.784363  1245 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:34.784852  1245 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:34.784535  1122 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.785778  1245 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.790058  1389 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:34.790786  1307 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:34.791347  1307 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.793418  1501 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:34.793975  1307 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.794109  1501 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.794965  1382 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:34.796932  1501 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.796985  1374 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:34.816257 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.812839 27588 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.809429 27583 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Committing config change with OpId 16.10: config changed from index -1 to 10, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:34.772092 27736 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:34.830333 27658 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 10
I20260709 12:20:34.830679 27658 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:20:34.830971 27658 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:34.787241  1245 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:34.833186 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.767830  1434 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:34.848048 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:34.798453  1224 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Committing config change with OpId 16.10: config changed from index -1 to 10, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:34.833434  1142 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:34.825966  1101 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:34.873512 27588 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.874328 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:34.881992 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:34.882711 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:34.883869 27657 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:34.789538  1122 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.844923  1264 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:34.887019 27583 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:34.886941 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.889323 27586 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:34.870870  1505 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:34.826900  1374 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.873593  1367 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:20:34.892158 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.876509  1316 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.846190  1362 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.884835  1404 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:34.871574  1434 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.903836  1209 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.923694 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.908728  1501 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.925961 27588 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
W20260709 12:20:34.924538 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 345609us
I20260709 12:20:34.929359 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.933235 27589 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:34.913252  1367 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:34.915006  1404 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:34.898315  1264 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.937115 27589 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:34.903663  1505 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.943465 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:34.948369 27589 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:34.914450  1316 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.952051 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.939003  1404 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:34.936039  1443 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.953634 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.944253  1298 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.953408  1534 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:34.957613  1298 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.957891  1534 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:34.959652 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:34.960700  1534 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.961962  1362 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.964731  1434 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.930155  1209 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.965610  1483 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:34.970810 27736 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:34.950614 27583 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:34.972332 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.934401  1501 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:34.980270 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:34.985440 27662 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:34.941220  1392 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:20:34.986682 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:34.983227 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.948545  1532 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:20:34.992908 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:34.992249 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:34.996666 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:34.997103 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.998073 27588 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:34.998935 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:35.004314  1540 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:20:35.005762  1216 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:35.006238  1216 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:35.006628  1216 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:35.008170  1505 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:35.008946  1538 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:20:35.009841  1364 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:35.010252  1364 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.010586  1364 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:35.011641  1501 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.012413  1374 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.976831  1443 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.017217  1443 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:34.993440  1483 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:35.018033  1483 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:35.018427  1483 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:35.020422 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.021158 27662 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:20:34.992473 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:35.023389 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.027422 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:34.986788  1307 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.983887  1534 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:35.029763 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 419174us
I20260709 12:20:35.034013  1122 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:35.033898  1534 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.034569  1122 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:35.035027  1316 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.035473  1122 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.036182  1534 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:35.039206 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:34.992120  1245 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:20:35.028118  1362 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:20:35.029491  1444 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:20:35.033910  1298 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:35.077373 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:34.994167  1367 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.081284  1264 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.084611 27655 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:20:35.032284  1307 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.099380  1209 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.100638 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:35.104775 28328 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:35.118256 27660 raft_consensus.cc:2410] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:20:35.142580  1547 ts_tablet_manager.cc:933] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.095607  1364 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:35.166961 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 239019us
I20260709 12:20:35.169849 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.170928 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.172122 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.174592 27656 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:35.174620 27657 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:35.191728 27657 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:35.159734  1384 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:35.193852 27664 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:35.194693 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.197655 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.195487   996 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.198860 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.199110   996 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.199568 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
W20260709 12:20:35.204222 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 438958us
I20260709 12:20:35.204604 27664 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:35.175658  1298 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.208808 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.195745  1384 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:35.200453  1362 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.209120 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:35.177654  1216 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.202194  1280 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.229728 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.230371 27664 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:35.211952  1179 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.228951  1542 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:35.248459 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.212553  1298 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.248553 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:35.232548  1280 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.277204 27552 leader_election.cc:400] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21.
I20260709 12:20:35.259106 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:35.290431 27552 leader_election.cc:403] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:35.229557  1362 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.291690 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.293443 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:35.294059 28328 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:35.292330 27664 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:35.233613  1483 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.252833  1367 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.239460  1179 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.304458 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:35.255287  1544 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:20:35.308282  1367 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.305612  1542 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:35.288033  1551 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:20:35.256347  1522 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.460s	user 0.003s	sys 0.003s
I20260709 12:20:35.304299 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:35.292611  1307 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:35.315006 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55999.6us
I20260709 12:20:35.256073  1298 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.286628   996 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.283448  1142 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:35.329641 28328 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:35.250144  1187 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:20:35.318980  1542 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:35.311678 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.315351  1364 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.334901 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:35.336488 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 2979) took 1619 ms. Trace:
I20260709 12:20:35.251935  1384 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.341674 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.342903 27584 rpcz_store.cc:276] 0709 12:20:33.717344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:34.024184 (+306840us) service_pool.cc:224] Handling call
0709 12:20:34.024463 (+   279us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:34.028404 (+  3941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:34.028432 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:34.028883 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:20:34.186516 (+157633us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:34.246127 (+ 59611us) log.cc:844] Serialized 14 byte log entry
0709 12:20:34.246620 (+   493us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:34.246767 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:34.795977 (+549210us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b18001d7580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:35.256001 (+460024us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b240013aca8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:20:35.336288 (+ 80287us) raft_consensus.cc:1685] finished
0709 12:20:35.336313 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:35.336434 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":60746,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":3830,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":38465,"spinlock_wait_cycles":37497984,"thread_start_us":96293,"threads_started":1,"wal-append.queue_time_us":169249}
I20260709 12:20:35.326200  1562 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21.
I20260709 12:20:35.344115  1298 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:35.353463 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:35.342047  1552 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:20:35.357003  1560 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.357549  1298 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.360605  1216 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.356657  1280 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.358299  1264 raft_consensus.cc:2764] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:20:35.363018 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:35.328469  1547 tablet_copy_client.cc:323] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:20:35.342814  1542 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:35.368772 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.371477  1483 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.372864  1362 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.365517  1554 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:35.375043  1554 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.377278  1554 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.378087  1264 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:35.378377 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:35.149987 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:35.368165  1400 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:35.379825  1400 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.380604 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:35.357611  1179 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.381505  1264 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:35.382141  1264 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:35.385123 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.385485  1400 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.385886 27656 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:35.386035  1400 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.386538  1400 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.357599  1560 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.387849 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.389729 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.389750  1560 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.348989  1364 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.395107  1364 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.395942  1187 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:35.397379  1364 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.398519 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.399149 27655 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:35.399684 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:35.400395  1187 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:35.402463 27552 leader_election.cc:400] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:35.404662  1187 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:35.405102  1404 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.406172 27589 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:35.407188 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:35.416342  1367 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:35.387933  1549 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.156s	user 0.002s	sys 0.000s
I20260709 12:20:35.416397 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:35.418577 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.419503  1564 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:35.420140 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2441) took 1605 ms. Trace:
I20260709 12:20:35.407202  1404 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.420686  1400 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.420527 27663 rpcz_store.cc:276] 0709 12:20:33.814500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:33.814649 (+   149us) service_pool.cc:224] Handling call
0709 12:20:33.815091 (+   442us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:20:33.859359 (+ 44268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:33.859398 (+    39us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:20:34.734509 (+875111us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:34.735160 (+   651us) log.cc:844] Serialized 78 byte log entry
0709 12:20:35.120313 (+385153us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:35.137882 (+ 17569us) log.cc:844] Serialized 14 byte log entry
0709 12:20:35.157554 (+ 19672us) log.cc:844] Serialized 14 byte log entry
0709 12:20:35.157825 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:35.157974 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:35.387379 (+229405us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:35.419917 (+ 32538us) raft_consensus.cc:1685] finished
0709 12:20:35.419937 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:35.420084 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16272,"prepare.queue_time_us":24814,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":111,"raft.queue_time_us":21510,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":7026,"spinlock_wait_cycles":49290368,"thread_start_us":384551,"threads_started":1,"wal-append.queue_time_us":465593}
I20260709 12:20:35.424045 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.425040 27663 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:35.430583 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:20:35.433548 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:35.432413 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:35.434294 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.437664 27659 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:35.437350 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:35.444614  1316 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:35.434258  1404 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:35.451261 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.451960 27735 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:35.445154  1316 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:35.435205  1367 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.457751 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.458249 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.462082 27736 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:35.458163 27735 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:35.459579 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.463773 27731 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:20:35.447754  1216 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.468549 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.447927  1560 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.473701 27736 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:35.479835 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:35.459518  1559 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.480444 27736 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:35.462668  1444 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:35.486474 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:35.486956 27737 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:35.464111  1570 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.466811  1224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60452 (request call id 0) took 1756 ms. Trace:
W20260709 12:20:35.499665 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:35.464879  1367 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.509028 27736 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:35.494076  1298 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:35.514369 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49953.8us
I20260709 12:20:35.475601  1216 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.515311 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.518173 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.519137 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:35.480855  1560 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.519819 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:35.520893  1367 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.521273  1404 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:35.521414  1571 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:35.522970 27658 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:20:35.526382 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:20:35.526582 27734 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:35.527499 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.529352 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.538077 27737 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:35.538301 27655 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:35.486162  1559 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.532725 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.543115 27729 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:35.543471 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:35.543905 27656 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:35.544080 27656 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:35.544647 27729 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:20:35.545071 27729 pending_rounds.cc:85] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 7
I20260709 12:20:35.545338 27729 pending_rounds.cc:107] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:20:35.545625 27729 raft_consensus.cc:2889] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:35.534343 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.538949 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:35.547240 27737 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:35.541383 27659 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:35.539552 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:35.551959 27663 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:35.551811 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.553714 27664 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:20:35.554361 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.554972 27663 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:35.544596 27656 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.502770 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:20:35.559026 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:35.498039  1570 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.496114  1224 rpcz_store.cc:276] 0709 12:20:33.700221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:33.700367 (+   146us) service_pool.cc:224] Handling call
0709 12:20:33.744020 (+ 43653us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b4c003e2fc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98614a20 00007f0e98625a29 00007f0e985eb644 00007f0e98612f2d 00007f0e98612b93 00007f0e9a2a11af 00007f0e9a2a0f3e 00007f0e9a326888 00007f0e9a34375c 00007f0e9a34476a 00007f0e9a34391b
0709 12:20:33.745875 (+  1855us) log.cc:844] Serialized 499 byte log entry
0709 12:20:35.447965 (+1702090us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":394,"raft.run_cpu_time_us":1499,"raft.run_wall_time_us":1497,"spinlock_wait_cycles":7777152}
I20260709 12:20:35.578287 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.513094  1316 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.589726 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:35.594240 28334 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:20:35.605255 27661 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:20:35.608597 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:35.552531  1571 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.557626  1573 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:20:35.704226 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 105307us
I20260709 12:20:35.711021 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:35.713840 27734 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:35.704743  1537 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:35.723963  1537 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.724412  1537 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:35.703743  1404 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.573741 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 6b423a3b3ee1452fb419df4c9c0a8d67 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-6b423a3b3ee1452fb419df4c9c0a8d67
W20260709 12:20:35.738356 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 274735us
I20260709 12:20:35.742103 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.744187 27584 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:35.672614  1367 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.755342 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:35.755733 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.759033 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.762382 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:35.763129 27661 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:35.712523  1573 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:35.766791 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:35.767705 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.768405 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2455) took 1159 ms. Trace:
I20260709 12:20:35.770035 27661 rpcz_store.cc:276] 0709 12:20:34.609384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:34.656002 (+ 46618us) service_pool.cc:224] Handling call
0709 12:20:34.799141 (+143139us) spinlock_profiling.cc:243] Waited 46.3 ms on lock 0x7b18001263c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a383646 00007f0e9a2bd58e 00007f0e9a33ef9e 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c
0709 12:20:35.145344 (+346203us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b58004571d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c
0709 12:20:35.605046 (+459702us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5800456ed0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34c27c 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c
0709 12:20:35.768349 (+163303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1631181696}
I20260709 12:20:35.777163 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.777778 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.778523 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:35.779451 27661 raft_consensus.cc:606] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:35.781911 27661 raft_consensus.cc:614] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:35.738526  1546 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:35.738106  1264 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.780594 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:35.786237 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.792114 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:35.797307  1575 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:20:35.801000  1541 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:35.812630 27734 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:35.713256  1524 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:20:35.808368  1298 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:35.848994 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2452) took 1270 ms. Trace:
I20260709 12:20:35.860116 27660 rpcz_store.cc:276] 0709 12:20:34.578128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:34.582495 (+  4367us) service_pool.cc:224] Handling call
0709 12:20:35.848748 (+1266253us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:35.848924 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1184732032}
I20260709 12:20:35.738098  1307 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 12:20:35.774667  1573 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:35.783190  1364 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:35.790464  1400 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.765952  1444 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:35.811254  1571 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.787762  1546 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.889263  1546 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:35.896039  1392 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.896590  1392 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.897689 27674 tablet_copy_source_session.cc:215] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:35.898779  1392 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.905314 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.906953 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.908846 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:35.808102  1578 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:20:35.770673  1367 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.768929  1566 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.086s
I20260709 12:20:35.769678  1542 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.860563  1070 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:35.881752  1229 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:35.888650  1197 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.875353 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:35.803270  1264 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:35.943207 27734 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
W20260709 12:20:35.928068 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 206921us
I20260709 12:20:35.847532  1541 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.945358 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:35.929605  1560 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.887504  1400 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:35.963989  1576 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:20:35.965490  1570 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.953403  1307 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:35.966790  1307 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:35.940183  1229 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.967617  1542 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.969815  1229 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.947284  1559 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.937651  1070 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.972952  1541 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:35.974255  1070 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.954123  1537 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.978269  1537 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:35.979488  1574 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:20:35.981884  1400 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:35.943424  1197 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:35.969880  1367 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:36.008911  1579 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:20:36.070513 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:36.072914 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 306479us
W20260709 12:20:36.076246 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.077281 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:36.077690 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.079586 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.081790 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.081979 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.082954 27733 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:36.083863 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.084175 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.085282 27733 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:20:36.072817 27584 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:36.087401 27733 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:36.077762 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:36.088192 27660 raft_consensus.cc:606] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:36.089165 27733 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
W20260709 12:20:36.072813 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 292144us
I20260709 12:20:36.092211 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:36.101824 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.103986 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.106603  1367 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:36.107594  1122 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.002s	sys 0.010s
I20260709 12:20:36.108732  1400 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:36.109839  1197 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.112474  1264 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:35.984728  1307 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.115842  1070 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:36.103977 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.103986 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:36.125891  1122 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.914s	user 0.002s	sys 0.010s
I20260709 12:20:36.131507  1400 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.133302  1367 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.135658  1070 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.176765  1216 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:36.140389 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48519.2us
I20260709 12:20:36.185009 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.186455  1101 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:36.141757  1362 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:36.187580  1362 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.188030  1362 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:36.188548  1101 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:36.146293 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 2999) took 1193 ms. Trace:
I20260709 12:20:36.193529 27589 rpcz_store.cc:276] 0709 12:20:34.953245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:34.953417 (+   172us) service_pool.cc:224] Handling call
0709 12:20:36.145758 (+1192341us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:36.146015 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1032274816}
W20260709 12:20:36.151551 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50034.9us
I20260709 12:20:36.152436  1597 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:36.161276 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:20:36.161403  1524 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:36.162648  1589 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:36.164059  1400 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:36.152411 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 174001us
I20260709 12:20:36.174851  1367 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:36.178399 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.177803  1070 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.180495 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.137475 27660 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:36.190671 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.191849  1280 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:36.187136 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.144932 27588 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:36.217399 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.217594 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.218626 27582 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:20:36.218750 27587 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:36.219760  1589 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.227710  1280 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:36.224087 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.219547 27732 raft_consensus.cc:606] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:36.223145 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:36.231001 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.220618 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.239532 28331 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:36.222198 27583 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:36.241357  1547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b423a3b3ee1452fb419df4c9c0a8d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:36.242233 27732 raft_consensus.cc:614] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:36.243906 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:36.223634 27584 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:36.252812 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44857.3us
I20260709 12:20:36.263916  1411 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:36.264628  1411 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.265146  1411 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.278558  1245 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:36.293717 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.295667  1585 raft_consensus.cc:2764] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:20:36.260188  1101 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:36.296878  1101 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.265228  1384 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:36.308032 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:36.308880 27655 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:20:36.309736 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:36.310431  1280 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:36.311494  1548 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.080s	user 0.009s	sys 0.007s
I20260709 12:20:36.311270  1280 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.311954  1362 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.299921  1245 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.323629 28328 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:36.323413 27659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:36.324324 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:36.325695 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:36.327173 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.328709 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:36.334973  1435 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:36.315397 27656 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:36.348925  1435 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.356945  1578 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:36.357522  1578 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.358071  1578 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:36.359480  1245 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.340994 27659 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.361898  1435 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.357509 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:36.365643 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:36.366979  1435 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:36.367465  1435 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:36.368054  1435 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:36.369876 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:36.371888  1070 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:36.372313  1070 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.372706  1070 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:36.373109 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.344774  1101 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.376469 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:36.376704 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.377529 27581 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:36.378661  1280 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:36.380988 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:36.381438 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 113810us
I20260709 12:20:36.382473 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.383492  1101 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:36.384469 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.385021 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.386119  1589 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.390239 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.390555 28337 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:36.391113 27656 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:36.393841 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.394388 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:36.394758  1546 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.505s	user 0.016s	sys 0.004s
I20260709 12:20:36.367218 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:36.397806  1573 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.512s	user 0.007s	sys 0.016s
I20260709 12:20:36.405081 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.406502 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.406883 27655 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:20:36.407811 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.411605  1587 raft_consensus.cc:993] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:36.412179  1224 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:36.418418  1070 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.421272 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.423807  1362 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.365916 27481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 126) took 9697 ms. Trace:
I20260709 12:20:36.424726  1389 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:20:36.425195 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48828us
I20260709 12:20:36.425889  1142 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:36.426113  1571 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:36.430313 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.431737 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.433393  1584 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:20:36.434975 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.439388 27581 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:36.443789 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:36.444518 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100139us
I20260709 12:20:36.455274 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.461100 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.450515  1400 raft_consensus.cc:1081] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:20:36.468711 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.471611 27589 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:36.471463 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:36.457432  1245 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:36.477753 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:36.481839 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.485510 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:36.487134  1223 raft_consensus.cc:2764] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:20:36.487771  1562 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:36.488276  1562 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:36.489774  1562 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.493039  1597 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:36.493502  1597 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.493885  1597 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.495585 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.496341 27584 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:36.506191 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.473376  1589 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:20:36.513012 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 145873us
I20260709 12:20:36.534950  1435 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.535393  1456 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:20:36.452662  1546 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:36.520388 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 126733us
I20260709 12:20:36.520946  1224 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:36.512400  1142 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.543244 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.544446 27731 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:36.546593 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.549682  1142 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:36.523110 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73764.5us
I20260709 12:20:36.555887 28334 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:36.556201 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.556972  1546 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:36.525945 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48711.9us
I20260709 12:20:36.528563 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.451782  1573 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.519027 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:36.478462  1280 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:36.560918 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:36.565438 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98656.8us
I20260709 12:20:36.566536 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:36.567557 27657 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:36.567684  1362 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:36.568555  1362 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:36.569988  1362 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.572964 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.573097  1280 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.573818  1280 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:36.574625 27582 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:20:36.559784  1224 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:36.575734 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.578276 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.578604  1573 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.580255  1216 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:36.580868 27585 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:36.582075 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.558166  1589 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:36.583963 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.585041 27581 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:36.542085 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.525982  1578 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.561959 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:36.537731 27583 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:36.583904 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:36.587092 27661 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:36.588207  1578 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.590901 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.591514 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:36.536767 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.592211 27581 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:20:36.621207 27581 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:36.621706 27581 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:36.625155 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:36.627313  1573 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:36.628355  1573 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:36.629665  1573 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.631438 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:36.631696  1216 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:20:36.633105 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51197.1us
I20260709 12:20:36.634786 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:36.592808 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.635776 27580 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.595567 27656 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:36.502300  1562 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:36.644722  1400 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:20:36.597522 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62892.9us
I20260709 12:20:36.646312 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.561272 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:36.648805 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:36.601254 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.603849 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.651834 27585 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:36.541113  1435 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.652662  1245 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:36.558223 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:36.654095 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96045.3us
I20260709 12:20:36.588270  1384 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:36.652617 27581 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.655753 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.658061 27583 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:36.682725 27581 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.695667 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.689067 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.699703 27553 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:36.700814 27553 leader_election.cc:403] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:20:36.701647 27630 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:36.702077 27630 leader_election.cc:403] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:20:36.676491  1578 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:36.709077 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74400.9us
I20260709 12:20:36.707305 28334 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:36.725349 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:36.733716 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78033us
I20260709 12:20:36.734834 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.735486 27583 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:36.739208 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:36.728354  1435 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:36.749799 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:36.749987 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:36.750466 27655 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.733556  1587 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:36.733943  1223 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:36.759460  1179 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:36.740729  1070 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.765404 28331 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:36.762181 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.763268  1179 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.769253 27580 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:36.770845 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:36.772349  1456 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:36.773077  1456 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:36.774456  1456 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.741658  1597 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.746568  1224 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:20:36.792989  1224 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:36.793382  1224 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:36.791648  1411 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.525s	user 0.005s	sys 0.010s
I20260709 12:20:36.795535  1411 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:36.796809 27662 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20.
I20260709 12:20:36.801966  1280 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.806952  1280 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:36.808614  1584 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:36.818500 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:36.813246  1584 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:36.820983  1584 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.779976  1142 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:20:36.759851  1435 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.743536 27484 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 7 to 14, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 14 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.826371 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:36.827454 28337 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
W20260709 12:20:36.836876  1580 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:20:36.794313  1224 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.849691 27484 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 13 to 18, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 18 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:36.864835 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 149105us
W20260709 12:20:36.865170 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55824.4us
I20260709 12:20:36.852306  1524 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:36.855075  1589 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:36.872293 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:36.874624  1382 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:36.874944  1584 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:36.889843  1587 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:36.890457  1587 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.903855 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.904592 27580 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.905051 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:36.908072 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:36.909667  1607 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:20:36.910007 27587 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:20:36.910924 27628 consensus_peers.cc:597] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8355fe4a548a43f8b5215ea9ae7ba8b2. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:36.912189 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:36.912863 28331 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.915877  1547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58246 (request call id 975) took 2240 ms. Trace:
I20260709 12:20:36.916477  1547 rpcz_store.cc:276] 0709 12:20:34.704625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:34.719919 (+ 15294us) service_pool.cc:224] Handling call
0709 12:20:35.143110 (+423191us) ts_tablet_manager.cc:934] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
0709 12:20:36.239494 (+1096384us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:20:36.439781 (+200287us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:36.448919 (+  9138us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:20:36.848038 (+399119us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:20:36.848514 (+   476us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:36.848966 (+   452us) ts_tablet_manager.cc:989] Replica registered.
0709 12:20:36.871681 (+ 22715us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":2463,"tablet-copy.queue_time_us":421943,"thread_start_us":302129,"threads_started":2}
I20260709 12:20:36.918426  1547 tablet_copy_client.cc:806] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:36.921587  1587 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:36.924788 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:36.884924 27484 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.928009 27655 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:36.884496  1411 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:36.941421 27484 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 14 to 15, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:36.919221  1547 tablet_copy_client.cc:670] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:36.956028 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:36.957024 27582 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:36.924198  1224 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:36.966086 27583 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
W20260709 12:20:36.976194 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 258731us
I20260709 12:20:36.972468 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:36.887033  1589 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:36.982467 27732 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:36.953812  1070 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:20:36.987048 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79051.6us
I20260709 12:20:36.945088  1591 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:20:36.979009  1384 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:36.975023  1578 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:36.974970  1216 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:37.009737 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.010233 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.009876  1216 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.010802 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:37.011170 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:37.011699   996 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:37.012351   996 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.013095  1216 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:37.014825   996 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:37.015743 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:37.016762  1608 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:20:37.016803  1216 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:37.017855  1610 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:20:37.018064  1612 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:20:37.025993 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:37.027451 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46946.3us
I20260709 12:20:37.027805 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:37.028100 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:37.028463 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:37.032114 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.033111  1483 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:37.033610 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:36.450636 27481 rpcz_store.cc:276] 0709 12:20:26.668671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.722573 (+ 53902us) service_pool.cc:224] Handling call
0709 12:20:36.365654 (+9643081us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:32.765084 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:32.765635 (+   551us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:32.765674 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:20:32.765691 (+    17us) tablet.cc:662] Decoding operations
0709 12:20:32.766838 (+  1147us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:32.766911 (+    73us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:32.766939 (+    28us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:20:32.897992 (+131053us) tablet.cc:701] Row locks acquired
0709 12:20:32.900017 (+  2025us) write_op.cc:260] PREPARE: finished
0709 12:20:32.921216 (+ 21199us) write_op.cc:270] Start()
0709 12:20:32.921330 (+   114us) write_op.cc:276] Timestamp: P: 1783599632921165 usec, L: 0
0709 12:20:32.921352 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:20:32.921993 (+   641us) log.cc:844] Serialized 5090 byte log entry
0709 12:20:33.098575 (+176582us) op_driver.cc:464] REPLICATION: finished
0709 12:20:33.247255 (+148680us) write_op.cc:301] APPLY: starting
0709 12:20:33.252198 (+  4943us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:33.472732 (+220534us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:33.472831 (+    99us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:34.045996 (+573165us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:34.046266 (+   270us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 12:20:34.046325 (+    59us) write_op.cc:312] APPLY: finished
0709 12:20:34.046645 (+   320us) log.cc:844] Serialized 121 byte log entry
0709 12:20:34.046819 (+   174us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:34.046963 (+   144us) write_op.cc:454] Released schema lock
0709 12:20:34.047014 (+    51us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:34.083418 (+ 36404us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:34.089165 (+  5747us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":107684,"apply.run_cpu_time_us":24867,"apply.run_wall_time_us":842558,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58467,"num_ops":13,"prepare.queue_time_us":65458,"prepare.run_cpu_time_us":15382,"prepare.run_wall_time_us":254902,"raft.queue_time_us":16142,"raft.run_cpu_time_us":23154,"raft.run_wall_time_us":87089,"replication_time_us":177057,"thread_start_us":221493,"threads_started":4,"wal-append.queue_time_us":118254,"wal-append.run_cpu_time_us":4239,"wal-append.run_wall_time_us":2006568}]]}
I20260709 12:20:37.028124 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:37.135593 28337 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:37.036589  1384 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:37.037292 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.137969 27734 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:20:36.987267  1597 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.039093  1435 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.040194  1553 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:37.041061 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.042379  1589 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:36.987190  1367 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:37.000838 27484 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.984386  1179 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.052860 27580 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:37.065045 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:36.987807  1070 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:37.117686  1216 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:37.125810  1613 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 12:20:37.126742 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:37.128226 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 466540us
I20260709 12:20:37.128494 28331 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:20:37.139573 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 113659us
I20260709 12:20:37.166815  1367 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.167759  1589 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:37.168677  1589 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:20:37.169185  1589 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:37.169535  1589 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:37.169828  1367 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:37.170523  1101 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:37.127485  1578 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:37.172897 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 144894us
I20260709 12:20:37.138995  1382 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.139634  1384 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:37.175236 27583 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
W20260709 12:20:37.150800 27704 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.738s (SENT) [suppressed 2 similar messages]
I20260709 12:20:37.184161 27582 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:37.184257 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:37.176987 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:37.146946   996 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:37.188881  1618 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 12:20:37.189929   996 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:37.190447   996 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:37.147419  1483 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:37.159000  1070 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:37.195869 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.158604  1435 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:37.198282 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.198261 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:37.200165 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:37.200137 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 247230us
I20260709 12:20:37.198741 27484 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:37.205590 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:37.179831  1553 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:37.228223 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:37.216533 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:37.217303  1307 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:20:37.217667 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:37.219897  1367 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:37.219637 27484 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 14 to 17, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 17 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.221993 27735 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:37.221638  1483 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.179630  1179 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:37.170400  1589 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.223987  1591 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:20:37.224803  1280 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:37.224989 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.253566  1280 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.254668 27729 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:37.226015  1620 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.255748  1620 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.256752  1280 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:37.249742  1179 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.259524  1624 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.259990  1179 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.226584  1224 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:37.209021 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.238193 28292 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20.
I20260709 12:20:37.265260 28292 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:37.265286 27664 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:37.232342 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:37.241245 28337 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:37.195365  1389 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:37.222620 27734 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:37.245543 27738 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:20:37.223453  1142 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:20:37.245910 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 73642.3us
I20260709 12:20:37.242210 27731 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:37.242952 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:37.252202  1435 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:37.268759 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:37.287983 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:37.285544 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46548.6us
I20260709 12:20:37.298105  1620 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:37.298686 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86112.8us
W20260709 12:20:37.303926 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 153174us
I20260709 12:20:37.260314  1624 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.306721 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:37.321425 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:37.291365  1223 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.535s	user 0.005s	sys 0.007s
I20260709 12:20:37.209542  1382 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.283985  1435 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:37.294744  1546 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20.
I20260709 12:20:37.273772  1142 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:37.320864 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44113.3us
I20260709 12:20:37.275343  1179 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:37.346941 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:37.342064 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:37.360988 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:37.317813  1362 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:37.261744  1367 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.309989 27484 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.323136  1589 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:37.424573 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.356173  1142 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.448493  1142 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:37.359442 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 143051us
I20260709 12:20:37.458372 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:37.327876  1585 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:37.363277 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:37.459276 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117314us
I20260709 12:20:37.366178 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:37.359375  1389 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:37.461385  1614 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:20:37.461472  1389 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.424226 27584 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:20:37.464802  1585 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:37.340768  1223 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:37.433400 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.434497  1367 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:37.434794  1578 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:37.436738  1216 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:37.440346 27484 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 6 to 12. New cstate: current_term: 12 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:37.441218  1382 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.360682  1179 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:37.446241 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.341331  1280 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:37.450889 27583 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:37.347797  1435 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:37.501964 27586 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:37.470212 27731 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
W20260709 12:20:37.505100 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46824.6us
I20260709 12:20:37.470629 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.487984 27735 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:37.424384  1553 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:37.500376 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 209774us
I20260709 12:20:37.512105 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:37.514703 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:37.516131 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56020.1us
I20260709 12:20:37.495623  1587 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:37.529731 27484 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:37.506145  1179 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:37.550833 27484 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 15, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.553253 27584 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:20:37.553808 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:37.508061  1620 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:37.497906  1384 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.564896 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:37.565932  1223 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:37.571826  1367 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:37.573283  1367 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:37.571867 27484 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 12 to 15, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 15 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:37.578297 27662 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:37.585510  1639 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:37.588032  1585 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.554658  1562 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:37.605201  1638 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:20:37.606166 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:37.608397 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:37.571887  1625 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 12:20:37.569221  1620 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:37.627595  1562 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:37.629071  1562 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.634289  1070 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:37.641255 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 150957us
I20260709 12:20:37.639906 27484 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 13 to 15, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.657902 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:37.637387  1620 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:37.607125   996 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:37.666851 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:37.643889  1639 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:37.680707 27484 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 15, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.674757 27584 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:37.667065  1587 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:37.671203  1362 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:37.672215  1384 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:37.694021 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 348143us
I20260709 12:20:37.696743 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:37.697145 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.697922 27659 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:37.698012 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:37.679363  1639 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:37.703441 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.707155 27659 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:37.703452 27662 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:20:37.720954 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.722681 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:37.723697 27731 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:37.722779 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:37.724200 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.724563 27729 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:20:37.724853 27735 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:37.695981  1624 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.701246  1101 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:37.729351 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.714108  1571 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:20:37.744626 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 176178us
I20260709 12:20:37.745833 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.740226 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.747510 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.746744 27484 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:37.713683  1280 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.714085  1639 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:37.759359 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.757714 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:37.773514 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68088.7us
I20260709 12:20:37.777231 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.778121 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.712020  1587 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.779232 27659 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:37.781180 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:37.786106 27735 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:37.792826 27738 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:37.746116 27730 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Committing config change with OpId 16.10: config changed from index -1 to 10, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:37.793031 27484 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.748767  1101 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:37.803612 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:37.797901 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.765048  1571 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:37.806599 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.807657 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:37.817611 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:37.816576  1316 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:37.813304  1571 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:37.805701  1101 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:37.781992  1280 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.826543 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:37.835261 27582 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14. Re-sending same reply.
I20260709 12:20:37.832767 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.835712 27484 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 10 to 13, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 13 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:37.861935 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:37.812070  1216 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.022s
W20260709 12:20:37.873636 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56211.8us
I20260709 12:20:37.878046 27661 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:20:37.882908 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:37.839448  1573 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:37.881935 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:37.877246   996 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:37.905395  1652 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:37.905887  1652 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:37.906425  1652 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:37.908274  1619 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20260709 12:20:37.910624  1435 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.913725  1435 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:37.924304 27659 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:37.905928  1649 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:37.830873  1362 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:37.944515 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56384.2us
I20260709 12:20:37.946218 27729 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:37.947500  1316 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:37.949466  1643 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.009521 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:38.023479  1644 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:38.024217  1644 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:37.939401  1483 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.024549 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.026157 27657 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:38.036180 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 969) took 1056 ms. Trace:
I20260709 12:20:38.037644  1644 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:38.041832 27738 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:20:38.036755 27737 rpcz_store.cc:276] 0709 12:20:36.980165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:36.980689 (+   524us) service_pool.cc:224] Handling call
0709 12:20:36.999002 (+ 18313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:37.048531 (+ 49529us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:20:37.139314 (+ 90783us) log.cc:844] Serialized 14 byte log entry
0709 12:20:37.672184 (+532870us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:37.672242 (+    58us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:20:37.672295 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:38.035838 (+363543us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:38.035908 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:38.036074 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6206,"prepare.queue_time_us":83749,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":44,"raft.queue_time_us":8522,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":112940416,"thread_start_us":508005,"threads_started":2,"wal-append.queue_time_us":335957}
I20260709 12:20:38.044775 27484 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 15, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:38.054136 27733 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:38.070706 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:38.033445  1589 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.592s	user 0.019s	sys 0.007s
I20260709 12:20:38.102164  1658 ts_tablet_manager.cc:933] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:38.102365  1638 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:38.113368  1659 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 12:20:38.071135  1435 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:38.114387  1435 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:38.116425  1435 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:38.116720 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:38.055509  1653 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.192s	user 0.002s	sys 0.000s
I20260709 12:20:38.028568  1524 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:38.061009  1646 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 12:20:38.122023 27484 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:38.125654 27664 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:38.128998 27732 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:20:38.042333  1216 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:38.059764  1587 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.145475 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.146411 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.154945 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:38.154920 27484 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 13 to 15, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 15 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:38.175717 27663 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:20:38.027272  1316 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.041785  1483 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:38.177528  1631 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:20:38.249338 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 43535.7us
I20260709 12:20:38.150470  1367 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:38.440135  1668 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260709 12:20:38.426211 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 356034us
I20260709 12:20:38.318423 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.398378  1658 tablet_copy_client.cc:323] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:20:38.315414 27484 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 8 to 12, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 12 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:38.411391  1280 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:38.377856 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:38.445218 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:38.439286  1665 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:20:38.427081 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.413940 27738 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:38.145573  1524 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.309661 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.164111  1216 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.248765  1578 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:38.419593  1644 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.477756 27657 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:38.123692  1652 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.039852  1643 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:38.484586 27484 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:38.103225  1638 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:38.520962  1671 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:20:38.522495  1638 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:38.513309  1571 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.676s	user 0.008s	sys 0.002s
W20260709 12:20:38.496963  1670 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:20:38.502223  1578 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.486083  1389 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:38.509819 27660 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:20:38.515211  1483 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.515635  1223 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60452 (request call id 1) took 2425 ms. Trace:
W20260709 12:20:38.492465  1179 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.899s	user 0.013s	sys 0.001s
I20260709 12:20:38.516952  1644 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:38.549032 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.518469  1316 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:38.519661  1142 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.071s	user 0.010s	sys 0.003s
I20260709 12:20:38.479832  1362 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:38.492056  1591 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:38.525200  1643 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.495515 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:38.481900  1620 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.802s	user 0.011s	sys 0.000s
I20260709 12:20:38.524744 28330 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:20:38.529803 27484 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 9 to 15, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:38.488807 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:38.552450  1101 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.687s	user 0.007s	sys 0.004s
I20260709 12:20:38.493119  1367 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:38.573031 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.540877  1669 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
W20260709 12:20:38.578004 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98874.5us
I20260709 12:20:38.570784  1591 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:38.567227  1639 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.772s	user 0.009s	sys 0.002s
I20260709 12:20:38.578912 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:38.543694  1571 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.585379 27656 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:38.586577 27589 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:38.589694  1578 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:38.590922  1216 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:38.591220  1635 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:38.591846  1635 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.593228  1367 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:38.540091  1676 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:20:38.593966  1635 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.593587  1070 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.574404  1620 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.548573  1223 rpcz_store.cc:276] 0709 12:20:36.128142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:36.128352 (+   210us) service_pool.cc:224] Handling call
0709 12:20:36.314407 (+186055us) log.cc:844] Serialized 499 byte log entry
0709 12:20:36.887954 (+573547us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b1800216d60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:36.973884 (+ 85930us) spinlock_profiling.cc:243] Waited 9.35 ms on lock 0x7b1800216d60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:37.223739 (+249855us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b58001094d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a2a243c 00007f0e9a2a8d53 00007f0e9a2a8cd9 00007f0e9a2a8c69 00007f0e9a2a8c31 00007f0e9a2a7dfd 0000000000388634 0000000000388479 00007f0e9725b475 00007f0e9725d7c1
0709 12:20:38.493246 (+1269507us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51133,"raft.queue_time_us":70192,"raft.run_cpu_time_us":3372,"raft.run_wall_time_us":346041,"spinlock_wait_cycles":592558976,"thread_start_us":362096,"threads_started":1,"wal-append.queue_time_us":435246,"wal-append.run_cpu_time_us":2119,"wal-append.run_wall_time_us":1043114}
I20260709 12:20:38.597590  1620 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:38.594509  1635 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:20:38.600689  1571 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.602959 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.608841 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.610546 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:38.591928 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 973) took 1264 ms (client timeout 1513 ms). Trace:
I20260709 12:20:38.610126 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:38.617612 27732 rpcz_store.cc:269] 0709 12:20:37.327363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:37.327535 (+   172us) service_pool.cc:224] Handling call
0709 12:20:38.591721 (+1264186us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:38.591852 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1765643648}
I20260709 12:20:38.619949 27584 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:38.611953 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.553112  1179 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.566603  1524 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.581601 27484 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:38.552253  1389 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.615224 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.618621  1669 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:38.621802 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:38.622396 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:38.622972 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:38.628832 27659 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:38.633283 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:20:38.633875 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-8355fe4a548a43f8b5215ea9ae7ba8b2
I20260709 12:20:38.635684 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.636034 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.636799 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.637574 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.638366 27663 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:38.637791 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.638978  1676 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:38.639963 27657 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:20:38.632488  1638 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:20:38.641291  1179 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.630677  1573 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:20:38.642712 27661 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:38.566804  1362 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:38.647608  1683 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:20:38.649528  1652 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.575064  1101 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.651724  1101 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.652750 27580 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:20:38.628721 27737 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:38.663095 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:38.664955 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:38.665664 27656 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:38.617784  1681 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:20:38.676646 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46864.7us
I20260709 12:20:38.680812 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.681830 27662 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:38.682037 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.679241 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:38.687894 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.688987 27662 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:38.689088 27484 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 14, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 14 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:38.611354  1639 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.674535  1367 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:38.696177 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.687467  1578 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:20:38.688683  1686 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:38.687376  1280 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:38.688314  1669 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:38.712483 27655 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:38.722285 27584 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:38.724123 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.692567  1681 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:38.730647 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.733533 27484 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 15, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:38.737025 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.737743 27656 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:38.691159  1546 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:20:38.748473 27660 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:38.745968 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.749183 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.756781 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.744421  1547 tablet_copy_client.cc:538] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.764767  1639 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.765867  1681 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:38.769235  1687 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:20:38.770942  1591 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.772385  1389 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.774469 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.774017  1669 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.775290 27658 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:38.779119  1678 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.779843  1678 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.735343  1578 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:38.781500  1578 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:38.711838  1676 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:20:38.782985  1676 consensus_queue.cc:1243] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 18.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:20:38.783672  1676 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:38.784078  1676 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:38.785055  1676 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:38.732240  1245 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:20:38.787142  1546 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:38.787649  1661 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.491s	user 0.003s	sys 0.009s
I20260709 12:20:38.787744  1546 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:38.795989  1546 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.758487 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.798249 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:38.806897 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 121305us
I20260709 12:20:38.799186 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.808471 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:38.812098 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.809172 27659 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:38.812322 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.814853 27663 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:38.843571 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:38.852564 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119007us
I20260709 12:20:38.853267 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:38.854049 27658 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:38.858263 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:38.856571  1584 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.868582  1584 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:38.869869  1362 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:38.870818  1584 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.862834 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:38.872334 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.873006 27656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:38.873657 27656 raft_consensus.cc:606] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:38.874035 27656 raft_consensus.cc:614] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:38.875018 27674 tablet_copy_source_session.cc:215] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.855558  1678 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:38.877089 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.878356 27657 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:38.882081 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:38.883352 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.883900 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.891168 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:38.901340 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:38.894696 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.856737  1142 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.907709 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.907879 27584 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:20:38.908779 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:38.909821 27660 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:38.915299 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:38.916080 27583 raft_consensus.cc:2368] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:38.902525   996 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:38.905189  1679 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:38.927886 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68483us
I20260709 12:20:38.929298 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:38.996244 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:38.914669  1669 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:38.928452  1587 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.021598 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.023173 27627 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16.
I20260709 12:20:39.027737 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:39.029910  1591 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:39.030398  1591 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.030862  1591 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:39.036975 27588 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
W20260709 12:20:39.010856  1690 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:20:38.920091  1142 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.042528 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.050525  1382 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:39.051748 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.052250  1142 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.052592  1382 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:39.053547  1179 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:39.003114  1691 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.058725  1691 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.059562 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:38.998553  1619 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:39.041255 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 149412us
I20260709 12:20:39.012002  1704 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.088529   996 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:39.000804  1679 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.098057 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.103217 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:39.096813  1704 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:39.105666  1699 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:20:39.040467  1587 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:39.107424  1702 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.164s	user 0.001s	sys 0.001s
I20260709 12:20:39.049460  1669 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.116528 27584 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:20:39.120209 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.113497 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.129506 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:39.130836  1400 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:39.131918  1704 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.130787 27630 leader_election.cc:400] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:39.132874 27630 leader_election.cc:403] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:20:39.086776  1619 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.128407  1652 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.181818  1435 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20.
I20260709 12:20:39.219595  1573 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:39.219817  1692 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:20:39.220374  1573 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.223032  1573 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.227728  1547 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:20:39.219682  1696 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:20:39.233369  1679 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.234849  1691 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.240114  1591 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.205641  1713 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:20:39.248101  1619 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.247648   996 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.250526  1382 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:39.266016 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 346908us
I20260709 12:20:39.282145 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.284900 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:39.275765  1656 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.091s	user 0.024s	sys 0.009s
I20260709 12:20:39.298235 27731 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:39.253191  1704 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.308845 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.321635 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.323089 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.330772 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.333791 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:39.337714 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.342320 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:39.319631  1656 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.143s	user 0.024s	sys 0.010s
I20260709 12:20:39.287770  1587 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.345515 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.332302  1362 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:39.350514 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95305.9us
I20260709 12:20:39.370129 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.355937 27731 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:20:39.358790 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.326510  1483 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:39.336079  1696 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:39.324227  1553 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:39.318197  1645 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:39.318746  1400 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:39.298087  1585 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.062s
I20260709 12:20:39.345500  1620 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:20:39.349167  1709 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:39.381485 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:39.335196  1704 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:39.395413 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.392385  1713 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:39.401068  1711 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:39.401968  1715 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:39.405066  1591 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.407783 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.380234  1717 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:39.411052 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.411933 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:39.409368 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.385875  1645 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.406490  1696 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.426918 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.430418 27580 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:39.386629  1400 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.410506  1573 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:39.416992  1709 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:39.438323 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.439951  1710 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:39.440742  1400 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.442538  1645 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.399045  1553 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.434859  1656 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.438665  1573 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.437525  1717 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:39.438910  1591 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:20:39.464568  1709 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:20:39.451144  1655 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.618s	user 0.005s	sys 0.005s
I20260709 12:20:39.456243  1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:39.464520 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.446949  1710 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.474385  1709 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:39.476768  1573 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.473501  1710 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.469295  1656 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:39.471138  1704 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:39.468941  1553 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.494323 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2970) took 1608 ms. Trace:
I20260709 12:20:39.499009 27661 rpcz_store.cc:276] 0709 12:20:37.886136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:37.887310 (+  1174us) service_pool.cc:224] Handling call
0709 12:20:37.903385 (+ 16075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:38.643115 (+739730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:39.493902 (+850787us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b5400291654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:39.494222 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1699822464}
I20260709 12:20:39.503253  1382 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:39.509661 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.511011 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:39.493280  1639 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:39.485323  1656 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.513936  1553 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:39.514621  1711 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:39.511621  1704 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:20:39.515180 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:39.499661  1731 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.511809  1411 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:39.525803  1711 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.536315 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.503862  1382 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.541507 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.538362 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.543416 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.543609 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.525316  1639 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.538115  1734 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:39.527868  1553 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.549285 27663 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:20:39.542774  1645 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:39.553396  1710 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.539083  1711 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:39.568470 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.567066  1639 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.540624  1411 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.562537  1715 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:39.569928  1571 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:39.540289   996 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:39.557235  1734 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.570796 27587 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:39.556747  1382 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:39.577713 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 180378us
I20260709 12:20:39.579633 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:39.580418  1584 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:39.580857  1584 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.580932  1411 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.581406  1584 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:39.588604 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.594419 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.595065 27737 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:39.577865  1734 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:39.595718 27729 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:39.574646  1571 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:39.597349  1571 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.598546  1571 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.601161  1644 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.602073  1553 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.606837 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:39.608130 27580 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:39.536888  1731 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.609324 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.612174 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.609952 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.607901  1733 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:39.613425  1733 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:39.613709 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.616312  1280 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:39.616746  1280 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:39.617602  1280 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.570415  1645 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.618898  1280 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:39.621163  1645 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.622272 27737 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:39.623042 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3044) took 1831 ms. Trace:
I20260709 12:20:39.623490  1546 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:39.623718 27589 rpcz_store.cc:276] 0709 12:20:37.791214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:37.791434 (+   220us) service_pool.cc:224] Handling call
0709 12:20:37.791848 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:38.587038 (+795190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:39.622681 (+1035643us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:39.622926 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1827079040}
I20260709 12:20:39.626272 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 2965) took 1882 ms. Trace:
W20260709 12:20:39.626906  1736 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.132s	user 0.003s	sys 0.000s
I20260709 12:20:39.626868 27655 rpcz_store.cc:276] 0709 12:20:37.744136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:37.744318 (+   182us) service_pool.cc:224] Handling call
0709 12:20:37.773404 (+ 29086us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:38.713107 (+939703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:39.625937 (+912830us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:39.626194 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2159033600}
I20260709 12:20:39.613767  1733 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.629839 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.630554 27731 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:39.624208  1546 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:39.611032 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72735.6us
I20260709 12:20:39.642885 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.642907 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.645385 27737 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:39.658022  1731 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.660175 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.661688 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.662765  1641 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:39.663218  1641 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.663556 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1000) took 1514 ms. Trace:
I20260709 12:20:39.664086  1643 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.637844  1715 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:39.666025  1715 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.638725  1656 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:39.668838  1715 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:39.664075 28331 rpcz_store.cc:276] 0709 12:20:38.149189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:38.149360 (+   171us) service_pool.cc:224] Handling call
0709 12:20:38.181406 (+ 32046us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:38.526037 (+344631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:38.526068 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:38.526113 (+    45us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:20:38.526149 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:39.663272 (+1137123us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:39.663345 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:39.663473 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":792109824}
I20260709 12:20:39.671125 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.681865  1740 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:39.663643  1641 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:39.642140  1710 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:39.691382  1546 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.691140  1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58246 (request call id 998) took 1983 ms. Trace:
W20260709 12:20:39.693547 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50699.1us
I20260709 12:20:39.694188 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:39.694921 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.699787 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.593349  1713 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:20:39.705113 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:39.705391 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.706247 27737 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:39.710907 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.713398  1734 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.714809 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.711906  1733 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.721931 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.722200 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.723120 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.723802 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.716616 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.727515 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.728204 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.723802 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.729393  1710 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.719161  1733 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:39.729416  1678 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:20:39.729985 27732 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:20:39.732252 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:39.730695 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:39.732784 27656 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:39.734033 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.734263 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.740679 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.755211 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.742882 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.755894 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.756498 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.732705 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.751129 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:39.759040 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64196.2us
I20260709 12:20:39.759670 27734 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:20:39.698446  1578 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:39.698257  1664 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.760119 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.761565 27737 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 13: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:39.757121 27658 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:39.770954 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.756870 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:39.764783 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.765309 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.704845  1656 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:39.767343 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.767578 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.775214  1280 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.775174  1641 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.778247 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.780114 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.781855 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.783496 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.784756 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.785967 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.787478 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.791698  1733 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:39.792485 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.792799  1740 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:39.793035  1641 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.793303  1740 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.793776  1740 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:39.793853 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.794536 27736 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:39.757786  1731 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:39.797003 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.798005 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.798666 27583 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:39.800599 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.797657 27731 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:39.799386 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.796223 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:39.806840 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:39.772356  1717 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:39.819600 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.820998 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:39.813082 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48378.3us
I20260709 12:20:39.814240 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.775681  1645 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:39.824048 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:39.824569 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.825624 27658 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:20:39.827641 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:39.830534 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.830857 27580 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.795346  1656 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:39.833314 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.848129 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.799993  1731 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.849459 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.850204 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:39.845268 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:39.852101 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:39.848760 27662 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 12.
I20260709 12:20:39.852923  1483 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:39.864852  1742 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:20:39.853147  1710 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.012s
I20260709 12:20:39.865849 27664 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:39.866489 28335 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:20:39.866887 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.868247 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.869517  1620 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:39.869845 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63131.3us
I20260709 12:20:39.870095  1620 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.819043  1553 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:39.871745 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:39.872829  1571 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:39.873217  1571 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.873462 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:39.840148  1645 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.776248  1578 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:39.875761  1719 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:39.876206  1719 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:39.876402  1620 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:39.813748  1142 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:20:39.874250  1718 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:39.866959 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.897557 27664 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:39.871639  1553 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.818953  1524 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:39.857303 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:39.853178 27733 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:20:39.891808 27737 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:20:39.863875 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:39.877246 27656 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:39.873610  1571 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:39.887653 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.826458  1734 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:39.903829 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.819640   996 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:20:39.907276 27738 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:20:39.911158 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:39.925930 27655 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.920037 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:39.920743 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.876638  1719 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:39.692687  1658 rpcz_store.cc:276] 0709 12:20:37.708162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:37.708283 (+   121us) service_pool.cc:224] Handling call
0709 12:20:38.102600 (+394317us) ts_tablet_manager.cc:934] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
0709 12:20:39.454839 (+1352239us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:20:39.600849 (+146010us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:39.600973 (+   124us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:20:39.688044 (+ 87071us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:20:39.688384 (+   340us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:39.690948 (+  2564us) ts_tablet_manager.cc:989] Replica registered.
0709 12:20:39.691032 (+    84us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":70024,"tablet-copy.queue_time_us":215240,"thread_start_us":311882,"threads_started":2}
I20260709 12:20:39.929232 27702 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:39.956274  1720 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:39.919154  1524 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.956877  1720 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.957505  1389 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:39.958577  1524 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.952194  1710 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:39.959476  1739 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:39.959959  1734 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:39.960503  1734 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:39.961468  1734 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:39.962777  1734 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.853547  1731 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:39.895083  1578 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:39.848259  1589 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:20:39.960016  1739 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:39.968168 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:39.929160 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.955406 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.970046 27736 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:39.974839 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45760us
I20260709 12:20:39.985184 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:39.988096 27580 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:39.989152 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:39.981711 27655 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:39.931077  1587 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:39.997073 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.010813 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.022598 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:39.959039  1720 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:39.870114  1711 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.031422 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.026481 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:40.022405 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93321.3us
I20260709 12:20:40.032714 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:39.997174  1658 tablet_copy_client.cc:806] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:40.039001 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:40.028719 27730 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:39.924511  1620 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:20:40.078555 27731 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
W20260709 12:20:40.083093 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44216.2us
I20260709 12:20:40.025883  1524 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:40.023156  1692 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:20:40.084074 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:40.024376  1645 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:40.098932 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 113814us
I20260709 12:20:40.037079  1400 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:40.078819  1411 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:40.021396  1382 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:20:40.027215  1639 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:40.100168  1101 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:40.116503  1101 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:40.116804  1101 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:40.106099 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.091360 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:40.053908 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.043613  1591 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:40.034705  1715 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:40.048789  1389 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:40.034650  1713 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:40.110160 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:40.129053 27731 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
W20260709 12:20:40.125299 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71468.7us
I20260709 12:20:40.108290  1658 tablet_copy_client.cc:670] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:40.080561  1619 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:20:40.108285  1704 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:40.109793  1411 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:40.113030  1721 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:20:40.180089  1411 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:40.180871  1721 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:40.181303  1721 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:40.182252  1721 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.163708  1652 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:40.184062  1652 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:40.184507  1652 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:40.184681 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:40.132507 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:40.187394  1721 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:40.188226 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55796us
I20260709 12:20:40.189067 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:40.189821  1589 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:40.190200 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:40.190554  1589 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:39.996129  1656 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.191278  1733 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:40.193719 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:40.169492  1711 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:40.170861  1584 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.000s
I20260709 12:20:40.163772  1709 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
W20260709 12:20:40.175864  1748 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:20:40.209703 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:40.133442 27731 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:40.211793 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.111029  1573 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:40.110899  1637 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:20:40.213397  1382 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:40.213963  1382 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:40.212566 27731 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:40.214861  1382 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.217238  1739 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.128571  1689 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:20:40.218870  1689 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:40.219550  1689 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:40.220824  1411 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.205816  1101 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:40.223886  1553 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.128548  1645 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:40.224886  1101 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:40.217038  1584 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.201432 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.227474  1669 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.233804 27738 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:20:40.235975 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.240150 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.240473 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:40.228075  1584 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.232266  1709 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:20:40.258908 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 153923us
I20260709 12:20:40.267493 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.281103 27660 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:40.258463  1645 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:20:40.293301 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 162592us
I20260709 12:20:40.295969 27731 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:20:40.297274 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.295575  1645 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:40.289194  1709 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:40.289650  1715 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:20:40.343969  1645 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:20:40.358755 27535 auto_rebalancer.cc:629] Failed to schedule move for tablet 37a0fdc57f6240869f0977e359f258c0: Illegal state: Replica 3f90188458da46f6ba138d5f82c55b69 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:40.343616  1411 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.371248  1652 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:40.373754  1698 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.438s	user 0.007s	sys 0.000s
I20260709 12:20:40.373805  1652 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:40.383623  1756 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:20:40.382083  1678 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 12:20:40.389758  1678 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:40.390321  1678 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:40.390225  1709 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.392961  1571 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:40.396066 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.398339 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.398903  1571 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.399261 27735 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:40.400132  1762 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:40.401454  1762 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:40.407444 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.402058  1762 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:40.414871 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.417377  1656 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:40.417610  1762 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.422633 27735 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:20:40.425649 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.424595 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:40.435388  1762 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:40.391876  1678 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.442984  1411 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:40.445749  1731 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:40.455000  1740 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.002s
I20260709 12:20:40.416501  1652 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:20:40.459239  1645 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.411947  1571 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:20:40.466475 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.472018  1411 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:40.476496  1731 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.474185  1740 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:40.476886  1717 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:40.477352  1717 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:40.478248  1717 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.478766  1740 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.479480 27583 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:40.438732  1719 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.471678  1689 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:40.485719  1689 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:40.520566 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.516407  1678 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:40.517127  1717 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:40.501828 27730 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:40.527580 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.523733  1411 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.585821 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.588847  1571 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:20:40.592154 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.591929 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:40.600144 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.596108 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:40.597260  1771 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:20:40.605018 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.605726 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:40.609865 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.610947 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.611508 27589 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:40.611979 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.597397  1765 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:40.613837 27659 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:40.614789 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.615101 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.616070 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:40.618878 27581 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:40.619415 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:40.619907 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.621145 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.626025 27581 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:40.627379 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.631299 27586 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:40.631623 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:40.634438 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:40.635601 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:40.609166  1739 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:40.644421  1645 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.653155 27730 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
W20260709 12:20:40.655504 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52300us
W20260709 12:20:40.662592 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62499us
I20260709 12:20:40.620900  1524 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:40.632681  1587 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:40.678088 27661 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:40.640475  1765 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:40.684548  1777 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:20:40.700717  1772 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:20:40.740764 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2572) took 1135 ms. Trace:
W20260709 12:20:40.741441 27702 consensus_peers.cc:597] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6d40162b2bb4b61adc69c9fa3f03326. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:40.741235 27733 rpcz_store.cc:276] 0709 12:20:39.605771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:39.606404 (+   633us) service_pool.cc:224] Handling call
0709 12:20:39.623267 (+ 16863us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:39.910727 (+287460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:40.740485 (+829758us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:40.740691 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":528042880}
W20260709 12:20:40.701761  1774 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
W20260709 12:20:40.702926  1775 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
W20260709 12:20:40.692062  1778 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:20:40.704658  1719 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.717144 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.722914  1696 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:40.746032  1696 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:40.747110  1696 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.750625 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:40.753505  1732 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:40.754130  1732 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.756886  1732 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:40.756767  1731 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:40.729790  1720 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:40.757606  1732 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:40.758085  1720 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:40.757815  1719 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:40.759244  1720 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.759986  1719 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:40.760883  1720 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:40.761005  1719 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.762434  1719 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:40.736429  1657 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:40.765743 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:40.767150 27581 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:40.767083 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:40.726881  1676 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:40.773775 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:40.776345 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:40.784789 27582 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 12:20:40.748633  1773 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.180s	user 0.003s	sys 0.000s
I20260709 12:20:40.748629  1696 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:40.778844 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 145190us
I20260709 12:20:40.787087 27585 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:40.788545 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:40.792071 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:40.776583  1620 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:40.799822 28330 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:40.776985  1657 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:40.804884 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 161801us
I20260709 12:20:40.787480  1709 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:40.804329 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:40.822748 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2511) took 1063 ms. Trace:
I20260709 12:20:40.825063 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.823266 27656 rpcz_store.cc:276] 0709 12:20:39.764034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:39.770329 (+  6295us) service_pool.cc:224] Handling call
0709 12:20:39.794782 (+ 24453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:39.913359 (+118577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:40.822448 (+909089us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b540022b354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:40.822670 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188905856}
I20260709 12:20:40.807607  1620 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:40.857658  1620 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:40.827721  1740 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:40.860801  1740 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:40.833364  1546 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:20:40.835800  1784 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:40.864048  1546 consensus_queue.cc:1243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:20:40.828317  1280 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:40.869282  1587 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:40.870851  1786 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:40.873458 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:40.876247  1623 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:40.877238  1623 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.883996  1657 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:40.886256  1657 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:20:40.886710 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:40.890486 27730 pending_rounds.cc:85] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 12
I20260709 12:20:40.890955 27730 pending_rounds.cc:107] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:20:40.891353 27730 raft_consensus.cc:2889] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:40.893018  1553 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:40.893563  1689 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:20:40.893168 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62055.6us
I20260709 12:20:40.894776  1553 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:20:40.893323 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66730.4us
I20260709 12:20:40.894845 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:40.897435 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 91846us
W20260709 12:20:40.887286 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:40.894904  1389 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:40.907588 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:40.910117 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:40.912561 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:40.918957 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:40.839160  1524 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:40.921622 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:40.935314 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 271781us
I20260709 12:20:40.906505  1553 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:40.925311  1400 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:40.909556  1389 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:40.925293  1280 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:40.931481  1591 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:40.927366  1639 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:40.929108  1524 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:40.942747  1760 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:40.944159  1382 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 12:20:40.959743  1781 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:20:40.957790  1524 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:41.002375 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211028us
I20260709 12:20:41.008517 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.024295 27581 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:40.959309  1785 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:41.032581 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:41.039469  1710 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:41.046458  1553 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.048542  1389 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.036600  1620 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:20:41.054044 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:41.057974  1553 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:41.058827  1620 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:41.059437  1620 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:41.058851  1664 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:20:41.063931 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 155193us
W20260709 12:20:41.066759 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 119580us
I20260709 12:20:41.067869 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.036952  1740 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:41.071553 27582 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:20:41.074443 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:41.076009 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 150001us
I20260709 12:20:41.063400  1664 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:41.076613  1179 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:41.080862 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.056458  1639 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:41.090911 27658 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:20:41.084255  1689 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.091356  1639 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.091545  1689 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.072674  1637 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:41.063336  1623 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.088502 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:41.088883 27583 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:20:41.089403  1400 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:41.090062 27588 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:41.090528  1664 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:41.060377  1620 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:41.075512  1739 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:41.101828 27660 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:20:41.085323 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.122617 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.101673  1637 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:41.126152  1637 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.127427  1637 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:41.092895  1280 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:20:41.132401  1389 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:41.110719  1739 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:41.122018  1179 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:41.134243  1739 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:41.174288  1547 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.147938  1585 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:41.196305 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47430us
W20260709 12:20:41.199502  1788 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:20:41.171371  1620 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:41.201885 27584 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 15. (term mismatch)
I20260709 12:20:41.210126 27584 pending_rounds.cc:85] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 14
I20260709 12:20:41.210397 27584 pending_rounds.cc:107] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260709 12:20:41.210625 27584 raft_consensus.cc:2889] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:41.168525  1641 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.207134 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.214973 27588 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:41.172371  1280 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:20:41.222600 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.211403  1652 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.266819 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.242280  1179 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:41.247735  1676 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.246289  1781 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:41.240543  1798 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:20:41.206419  1786 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:20:41.278353 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:41.272221  1739 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:41.252228  1639 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.240123  1641 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.210399  1585 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:41.317746 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 197813us
W20260709 12:20:41.310137  1756 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:20:41.284235  1689 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.276532  1645 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:41.269035  1652 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.313288  1676 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.321390  1547 tablet_bootstrap.cc:492] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:20:41.374704  1547 ts_tablet_manager.cc:1403] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 2.147s	user 0.055s	sys 0.023s
I20260709 12:20:41.375473  1734 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.374704 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.376842 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3062) took 2253 ms. Trace:
I20260709 12:20:41.376039  1734 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.377262 27584 rpcz_store.cc:276] 0709 12:20:39.123749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:39.127930 (+  4181us) service_pool.cc:224] Handling call
0709 12:20:39.128174 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:41.211057 (+2082883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:41.376566 (+165509us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:41.376762 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4768181120}
I20260709 12:20:41.378334  1547 raft_consensus.cc:359] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:41.379305  1734 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.380055  1411 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:41.380606  1411 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:41.380714  1719 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.002s
I20260709 12:20:41.381351 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.382000 27586 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:41.383805 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.380038  1734 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:20:41.385558  1734 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:41.354198 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.320765  1796 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:41.408510  1796 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:41.408921  1657 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.003s
I20260709 12:20:41.409024  1796 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:41.306378  1179 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:41.335424 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:41.397794 27587 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
W20260709 12:20:41.412479 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58383.5us
I20260709 12:20:41.318286  1732 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.407995  1142 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:41.404546 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:41.339968  1794 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:20:41.336407  1797 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:20:41.428117  1585 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:41.361717  1793 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:20:41.421923 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:41.342095  1696 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.549s	user 0.008s	sys 0.002s
I20260709 12:20:41.426379  1805 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:41.426846  1689 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:41.426509  1411 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.434118 27736 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:41.373700  1641 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.403887  1547 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:20:41.406375  1734 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:41.439913 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2588) took 1140 ms. Trace:
I20260709 12:20:41.411677  1179 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:41.315430  1781 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.340047  1524 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.439499 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:41.441486 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.445757 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.442005 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.453390 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.433805  1787 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.445627 27731 rpcz_store.cc:276] 0709 12:20:40.299252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:40.304295 (+  5043us) service_pool.cc:224] Handling call
0709 12:20:40.305410 (+  1115us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:40.306761 (+  1351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:40.306789 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:40.308128 (+  1339us) log.cc:844] Serialized 26 byte log entry
0709 12:20:40.888405 (+580277us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b18002bfd20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:41.118438 (+230033us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:41.118497 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:41.118586 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:41.435827 (+317241us) raft_consensus.cc:1685] finished
0709 12:20:41.435860 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:41.439847 (+  3987us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1570,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":2829,"spinlock_wait_cycles":75013504,"thread_start_us":229802,"threads_started":1,"wal-append.queue_time_us":867298}
I20260709 12:20:41.424268  1732 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.455091  1726 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:41.459512  1787 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.460821 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.461727  1787 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.449914  1734 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:41.456624  1547 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:41.468736 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:41.449581  1658 tablet_copy_client.cc:538] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.458272  1781 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:41.472280 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.475247 27587 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:20:41.475510 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.471259 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.445351  1780 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.473816 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.446437  1806 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:41.478181 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:41.479126 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.479554 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.481452 27656 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 24. Current term is 25. Ops: []
I20260709 12:20:41.483481  1676 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.485103  1652 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:41.471086  1720 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:41.481143  1780 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.488798  1780 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.488900  1524 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:41.474396  1717 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:41.489781  1717 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:41.479238  1740 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:41.491014  1717 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.456856  1732 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.501732 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.501789 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.443980  1689 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.509498 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:41.513288  1732 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:20:41.514300  1547 ts_tablet_manager.cc:1434] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.139s	user 0.009s	sys 0.001s
W20260709 12:20:41.546682 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68567.8us
I20260709 12:20:41.567039 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.446980  1805 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.582301 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.476141  1664 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.504081  1720 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:41.516755  1696 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.585260 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.589872 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.584880  1657 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 12:20:41.605008  1696 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.607568 27734 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:41.573930  1571 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:41.611387  1571 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.613759  1571 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.616593  1711 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.617364  1711 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.621023  1711 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:41.623790  1807 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:20:41.625174 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.626129 27587 raft_consensus.cc:2410] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:20:41.646185 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.646952 27583 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:41.652248 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.652616 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.653223 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-6b423a3b3ee1452fb419df4c9c0a8d67 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:20:41.653738 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-6b423a3b3ee1452fb419df4c9c0a8d67 on tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:41.654345 27664 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:41.655932 28330 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:41.661455 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:41.662688 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.666128  1546 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:41.667789  1546 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:20:41.668179  1546 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:41.669163  1546 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.670626  1546 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:41.679837 27731 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23.
I20260709 12:20:41.680925  1664 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:41.684273  1805 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.686223  1689 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.569180  1652 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:41.496259  1806 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.689370  1806 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:41.578369  1676 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 12:20:41.697212  1731 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.697901  1731 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.701408 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:41.707504 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:41.710070 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:41.712086 28292 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23.
I20260709 12:20:41.713047  1483 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:41.716521 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.717495  1571 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.717862  1696 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.718102  1571 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.718811 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:41.720330  1731 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.574847  1780 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:41.721048  1720 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.721393  1780 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:41.721930  1780 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:41.722443  1280 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:41.721598  1720 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.722838  1280 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:41.723239  1280 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:41.724669  1720 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:41.726063  1571 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:41.733691  1620 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.520s	user 0.010s	sys 0.000s
I20260709 12:20:41.735833  1280 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.739413 27587 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:41.740267  1711 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.740909  1711 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.742646 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:41.743307 27583 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:41.745208 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:41.750573 27581 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:41.752287 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.753300  1664 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:41.753736  1664 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:41.754110  1664 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:41.760284  1808 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:20:41.762393  1664 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.766738 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:41.767272 27581 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:41.770306 27587 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:41.771641 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.771906  1740 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:41.772607  1704 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:41.773102  1704 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:41.774145  1704 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:41.781847 27581 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:41.783179 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.745398  1280 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.786671 27660 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:41.789572 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.815706 27664 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
W20260709 12:20:41.787307 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75357.9us
I20260709 12:20:41.872723 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:41.881747 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:41.883260 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.883862 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:41.870589  1709 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:41.898195  1709 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:41.899223  1709 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.904451  1664 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.925876 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:41.937690 27583 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:41.884704  1658 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
W20260709 12:20:41.946640 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60674.8us
I20260709 12:20:41.949283 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:41.950613 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.857380  1711 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:41.905759  1792 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.925202  1689 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:41.928002  1637 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:41.932523  1781 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:41.941193  1591 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.962688  1641 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:41.911484  1734 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.982149 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:41.956753  1766 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.022s
I20260709 12:20:41.987586 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.974462  1731 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:41.990509 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.990621 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.991277 27664 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:41.991852 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.965988  1389 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:20:41.993637 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.997532 27660 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:41.998103 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:41.993815 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:41.994614 27662 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:20:41.980710  1620 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:41.971256  1792 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.973775  1689 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.012131 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.023147 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.024988 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:41.987821  1740 raft_consensus.cc:2764] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:20:41.975800  1591 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.093847  1726 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:42.101697  1726 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:42.102370  1726 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.102852  1726 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.108311  1726 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.096570  1676 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:42.115895 27583 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 13. (term mismatch)
I20260709 12:20:42.119928 27583 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 12
I20260709 12:20:42.119980 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.116963 27581 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:20:42.118942 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.121209 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.121546 27580 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:42.121819 27586 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:42.120294 27583 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:20:42.122471 27583 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:42.122694 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:42.124058  1812 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:20:42.124512 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:42.125370  1400 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:42.125986  1739 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:42.127002  1400 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:42.127357  1400 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:42.127705 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:42.128422  1739 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:42.128206  1400 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.128846  1739 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:42.129252  1739 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:42.129417  1400 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:42.131117  1689 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.133244  1734 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.135669  1689 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:42.138763 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.139482 27583 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:42.141309  1739 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.142704  1591 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:42.144284  1796 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.735s	user 0.004s	sys 0.007s
I20260709 12:20:42.145049  1739 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.144872  1796 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:41.997080  1731 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.147523  1796 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.152627  1411 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:42.159144  1483 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:42.159633  1483 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:42.160439  1483 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.161670  1483 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:42.162765 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.090925  1620 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:42.163627 27660 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:20:42.164465 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:42.164157  1620 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.165001 27655 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:42.168713 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.163036 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.169973 27656 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:42.170872 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.173722 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.116972 27585 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:42.174607 27736 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:42.175938 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:42.178764 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:42.181806  1760 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:42.182348  1760 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.184351  1760 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:42.185976 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:42.186774 27662 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:20:42.091545  1798 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.196024  1798 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.198096  1798 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.190418 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:42.202625 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:42.204923 27659 raft_consensus.cc:2410] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:20:42.193305 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:42.213001 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:42.222709 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:42.223577 27729 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19.
I20260709 12:20:42.225268 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:42.231824 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.231810 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.233019 27581 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:42.193554 27655 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:42.195137  1664 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.209558  1798 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:42.235973  1664 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.236595  1798 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.198613  1780 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.238845  1664 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.240208 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:42.192822  1710 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.252457 27580 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:42.254300 27587 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:42.329638 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:42.336206 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.336432 28334 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:42.337566   996 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.338092   996 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.330703 28330 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:42.238829  1798 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:42.344807 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:42.357787 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:42.358726 27733 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:42.358711 27730 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:20:42.344303  1689 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:20:42.361868  1689 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:42.362793  1689 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:42.363152  1689 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:42.369904 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:42.374632   996 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.331766  1818 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.339303  1785 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.408722 27730 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:20:42.354079  1809 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.398795  1623 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:42.417215  1623 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.420898  1780 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:42.428494  1792 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.411994  1818 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.429983  1524 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:42.430411  1524 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:42.431288  1524 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.339514  1664 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.432435  1664 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.332803  1710 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.364032  1689 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:42.435459 27733 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:20:41.911726 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:42.403752  1788 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:42.412482  1785 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.403024   996 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:42.439914   996 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.440729 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:42.404793  1717 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:42.413089 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 161282us
I20260709 12:20:42.442301  1717 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.415392  1731 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.444515  1809 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.441735 27660 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:20:42.464409  1591 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:42.487740  1623 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.496506 28334 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:42.506152   996 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:42.514343 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 489394us
W20260709 12:20:42.516088 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.008s	sys 0.002s
W20260709 12:20:42.516594 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.008s	sys 0.002s
I20260709 12:20:42.516460 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.524132  1623 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:42.524904  1623 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:42.525661 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.527042  1814 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:42.526247  1623 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.529212 27660 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:42.524052   996 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.532766   996 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:42.534060 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:42.535101  1814 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:42.535663  1814 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:42.539135 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:42.517144 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 553529us
I20260709 12:20:42.553444 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:42.554132 27587 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:42.567504   996 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.536140  1814 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:42.563890  1676 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:42.575883  1710 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:42.576875  1546 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.906s	user 0.008s	sys 0.006s
I20260709 12:20:42.517323  1689 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:42.583684 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:42.584755  1657 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:42.585270  1657 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:42.586401  1657 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.575914  1676 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:42.593200  1676 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:42.593314  1711 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.622058   996 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.622845 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:42.623711 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:42.592308  1731 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.589241  1785 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.591111  1816 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:20:42.631088 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.626031 27732 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:42.615931  1795 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:42.621228  1620 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.620971  1796 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.621398  1382 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:42.624917  1734 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:42.645949  1643 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.016s	sys 0.048s
I20260709 12:20:42.632647  1664 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.640902  1818 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.641287  1809 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.633168  1731 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.654417  1280 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:42.651273  1643 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.017s	sys 0.048s
I20260709 12:20:42.633883  1711 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:42.678390 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 342235us
I20260709 12:20:42.654773 27587 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
W20260709 12:20:42.655622 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 413094us
I20260709 12:20:42.682696 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:42.626892   996 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:42.683322 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.650574  1620 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.652570  1796 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.719340 27627 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19.
W20260709 12:20:42.681015  1411 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.004s
W20260709 12:20:42.752074 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.050s	sys 0.058s
I20260709 12:20:42.707207 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:42.756810 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.050s	sys 0.059s
I20260709 12:20:42.652719  1734 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.753756 27627 leader_election.cc:403] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:42.662170  1825 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:42.647249  1795 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:42.755303 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 79701.5us
W20260709 12:20:42.756317 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 121090us
I20260709 12:20:42.761540 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:42.762065 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:42.653685  1382 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.758491   996 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.759424 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:42.768781 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61695.7us
I20260709 12:20:42.768829 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.770742 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.677747  1280 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:42.773646 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206145us
I20260709 12:20:42.751683  1837 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:42.763801 27729 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:42.769798 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:42.822471 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:42.822847 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53140.4us
I20260709 12:20:42.791790 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:42.823455 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59694.6us
I20260709 12:20:42.823673 27660 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:42.825021 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.825803 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:42.826857 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.793571 27664 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:42.797721 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.830746 27655 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:42.772277  1587 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:42.812227 27658 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 12:20:42.845131 27658 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 11
I20260709 12:20:42.828935 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:42.814030  1828 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:20:42.815281 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.786705  1719 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:42.787145  1834 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19.
I20260709 12:20:42.753206  1785 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:20:42.821604 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56557.6us
I20260709 12:20:42.765616  1658 tablet_bootstrap.cc:700] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent opening tablet: real 0.170s	user 0.002s	sys 0.000s
I20260709 12:20:42.860069 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.832154 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.861992 27655 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:42.832866 27660 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:20:42.834510  1827 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 12:20:42.834288 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:42.864266 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 103039us
I20260709 12:20:42.864548 27663 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:42.835248  1829 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:20:42.836881  1280 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:42.845552 27658 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 12:20:42.877580 27658 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:20:42.813237  1820 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 12:20:42.765136  1825 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.866182   996 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:42.845042  1587 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:42.850230  1835 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:42.879966 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.883900   996 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:42.851626  1836 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:42.900118  1818 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:42.900767  1818 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:42.901315  1818 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:42.914531  1816 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:42.929085 28334 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:42.899434 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.924968  1851 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:42.935863  1838 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:20:42.936410  1382 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.899595  1280 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.937668  1280 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.939833  1280 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:42.940299  1795 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:42.925370  1836 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:42.945140  1836 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.876912  1717 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:42.929514   996 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.915226  1816 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:42.950408  1816 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:42.857596  1719 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:42.954496 27664 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:20:42.888167  1587 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:20:42.958168 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58867.6us
W20260709 12:20:42.958393 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:42.956847 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.960963 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.961829 27587 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:42.961795 27586 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:42.959090 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.968750 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.969605 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:42.894631  1835 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:42.975873 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 153472us
W20260709 12:20:42.954025  1831 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.162s	user 0.001s	sys 0.001s
I20260709 12:20:42.970391 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.977643 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.983166 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.985356 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.986889 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.987982 27587 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:42.987723 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.989041 27660 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:42.990525 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:42.981740 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:42.993662 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 128793us
I20260709 12:20:42.986769 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:42.999105 27656 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:42.991291 27660 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:42.999838 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:42.998718 27586 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:42.958199  1780 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.021099 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.022471 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.023518 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:42.978025  1717 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:43.029773 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:42.980934  1792 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:20:43.026185 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60173.8us
I20260709 12:20:43.023171 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:42.971120  1715 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:42.974825  1791 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:42.975544  1731 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.037173 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.039623 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:42.997400  1711 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:42.998159  1837 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:43.049274 27586 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:20:43.049650 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.018966  1795 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.019131  1620 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.060911 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.031333  1851 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:43.027585  1689 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:43.067690 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.025388  1823 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:43.074965 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.075788 27585 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:43.076963 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.033334  1717 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:43.077766 27586 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.001915  1814 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.039217  1791 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.034237  1844 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:43.081099  1844 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.081468  1844 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:43.040855  1792 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:43.081794 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.082094  1792 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:43.081468 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.082974 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1031) took 1609 ms. Trace:
I20260709 12:20:43.083642 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.084343  1719 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.083398 27657 rpcz_store.cc:276] 0709 12:20:41.473991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:41.474101 (+   110us) service_pool.cc:224] Handling call
0709 12:20:41.474377 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:41.688350 (+213973us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:20:41.970563 (+282213us) log.cc:844] Serialized 14 byte log entry
0709 12:20:42.459095 (+488532us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:42.459175 (+    80us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:20:42.459236 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:43.082757 (+623521us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:43.082802 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:43.082919 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":327983,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":445,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":29395,"spinlock_wait_cycles":491393280,"thread_start_us":756631,"threads_started":2,"wal-append.queue_time_us":654409}
I20260709 12:20:43.062883  1837 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.084524 27583 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:43.088999 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118776us
I20260709 12:20:43.089999 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.091071 27589 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:20:43.092114 27585 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.053400  1850 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:43.059576  1734 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.060132  1711 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:43.086539  1280 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:43.095726 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.137142 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.097529 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.138142 27589 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:43.097558 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.138509   996 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:43.058297 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54936.3us
I20260709 12:20:43.137306 27581 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:43.097124 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.077839  1851 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:43.137794 27736 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 13.
I20260709 12:20:43.145303 27589 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:20:43.032698  1780 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.086498  1717 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:43.078987  1652 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:43.179168  1652 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:43.179747  1652 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:43.139540 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 170046us
I20260709 12:20:43.103780 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:43.139978 27734 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:43.183671 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.183954 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:43.186657 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:43.086306  1791 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:43.188462 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.190538 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.191116  1657 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:43.192162 27586 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:43.192747  1587 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.050632  1715 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.197095  1715 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.197389  1739 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:43.198731  1739 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:43.199101 27659 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.107268  1859 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:43.200254  1859 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:43.136976  1711 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.200711  1859 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:43.200889 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.148500 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.139067   996 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.202463  1711 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.204087   996 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.063212  1795 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.091588  1835 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.204859  1780 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.206285  1795 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.207033  1389 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.207526  1389 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.210426 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.198729  1715 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:43.211315 28330 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:43.146643 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:43.211920 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65365.7us
I20260709 12:20:43.212082  1591 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:43.212538  1591 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.212905  1591 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:43.213564 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.215268  1796 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.216320  1796 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:43.216647  1796 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.216967  1796 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:43.211316 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:43.220566 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:43.221848 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.222468 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:43.225355 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.225714 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.226433 27582 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:43.224763 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.105201  1620 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:43.229277 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:43.218093 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.229277 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.140426  1280 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.240803  1389 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.146291  1851 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:43.238034 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.143776  1664 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.096s
I20260709 12:20:43.225958 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.245510 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:43.200260  1739 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.259436 27582 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23.
I20260709 12:20:43.231123  1709 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:43.229925  1715 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:43.263132 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.235962  1620 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:43.242938  1546 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:43.249419  1280 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:43.283118 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.295972 27586 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
W20260709 12:20:43.286944 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 259884us
I20260709 12:20:43.258279  1704 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:43.259752  1587 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:43.289995 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:43.307783 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 204072us
I20260709 12:20:43.305166 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.312117 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:43.288602  1857 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
W20260709 12:20:43.278719  1846 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.345s	user 0.001s	sys 0.000s
I20260709 12:20:43.317020 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.269979  1854 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:20:43.282702  1620 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:43.282791  1825 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.305925  1546 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.325289 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:43.293787  1483 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.132s	user 0.009s	sys 0.009s
I20260709 12:20:43.274784  1739 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.328088  1856 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
W20260709 12:20:43.356877  1847 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:20:43.357889  1814 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.358644  1591 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:43.359145 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49877.3us
I20260709 12:20:43.360064 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.361028  1591 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.361357 28295 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23.
I20260709 12:20:43.361759 28295 leader_election.cc:403] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:20:43.330766  1546 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:43.371351 27664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:43.371939 27664 raft_consensus.cc:606] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:43.372272 27664 raft_consensus.cc:614] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:43.374145 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.374763 27738 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:43.305577  1858 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:43.379215 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.379525 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.387398 27738 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:43.387682 27734 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:20:43.380136 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.380831 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.390038 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.380834 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.381477 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.381471 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.381855 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.385113 27656 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:43.385949 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.394209 27661 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.398561 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.398595 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.399773 27660 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.347303  1709 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:43.403054 27663 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.400713 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.404554 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.406369 27586 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:43.330245  1652 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.413225  1652 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.412900  1620 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.414281 27582 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:43.416249 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.370765  1809 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23.
W20260709 12:20:43.421494 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 398343us
I20260709 12:20:43.363106  1861 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:43.421150 27661 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.457015 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.458041 27735 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:43.459157 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.460103 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:43.460886 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:43.462863 27657 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:20:43.465250 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 202195us
I20260709 12:20:43.416551  1709 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.491453  1709 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:43.492065  1863 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.450991 27738 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 14.
W20260709 12:20:43.543478 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 137951us
I20260709 12:20:43.394858  1791 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:43.466224  1861 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.452770  1837 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.463160  1835 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:43.586858 27735 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:20:43.589542  1689 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.509s	user 0.008s	sys 0.004s
I20260709 12:20:43.630738  1689 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:20:43.631494  1867 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.175s	user 0.001s	sys 0.004s
I20260709 12:20:43.631690  1858 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:43.632226  1858 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:43.633898  1853 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.375s	user 0.003s	sys 0.000s
W20260709 12:20:43.634835  1676 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.000s	user 0.002s	sys 0.013s
I20260709 12:20:43.635253  1676 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.636482 27580 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.637352  1676 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.587710 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:43.588805 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:43.591089 27589 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:43.583150 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 196730us
I20260709 12:20:43.583307  1571 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.587060 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.592329  1696 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 12:20:43.600760 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.588557  1854 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:20:43.603176  1546 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.604001  1868 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:43.668720  1868 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.671260 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:43.604888  1818 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.703s	user 0.009s	sys 0.006s
I20260709 12:20:43.598188  1637 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.612811  1791 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.594674  1280 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.656170  1840 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.682251  1840 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.682564  1280 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.614648  1715 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:43.683594  1546 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.684015  1715 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:43.684489  1840 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.615355 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.651571  1571 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:20:43.685546 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70274.7us
I20260709 12:20:43.686442 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:43.616571  1704 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:43.617612  1865 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:43.618670 27586 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:43.619987  1816 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:43.621129  1664 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:43.621569 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3099) took 2234 ms. Trace:
I20260709 12:20:43.706049 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:43.627750  1796 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.629804  1709 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:43.595819  1844 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.514s	user 0.008s	sys 0.008s
W20260709 12:20:43.654978 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67998.1us
I20260709 12:20:43.658615 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.589955  1863 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.665426  1848 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.667809  1858 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.596204  1861 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:43.610899  1717 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.677141 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:43.677614  1637 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.678597  1731 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:43.679129  1868 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.679800  1870 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:43.684512  1739 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.695212 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.688582 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.703409  1818 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.728482 27729 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:43.722411  1858 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:43.710304  1704 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.738291  1637 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.738905  1704 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.741060 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.741729 27589 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:43.741863 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.742758  1637 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.743350  1637 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.743896 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.744504 27661 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:20:43.745703  1637 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.635035  1851 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.711448 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.684530  1715 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:43.749364 27587 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:43.749307  1715 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.750751  1863 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.692320  1869 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:43.751302  1851 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.752343 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.716972 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:43.687252  1571 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.690280  1411 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:43.704658 27588 rpcz_store.cc:276] 0709 12:20:41.386987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:41.389140 (+  2153us) service_pool.cc:224] Handling call
0709 12:20:41.389443 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:41.395877 (+  6434us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:20:41.425449 (+ 29572us) log.cc:844] Serialized 14 byte log entry
0709 12:20:42.646546 (+1221097us) spinlock_profiling.cc:243] Waited 7.09 ms on lock 0x7b1800284740. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:20:42.885819 (+239273us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:42.889162 (+  3343us) log.cc:844] Serialized 26 byte log entry
0709 12:20:42.889295 (+   133us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:20:42.971907 (+ 82612us) log.cc:844] Serialized 14 byte log entry
0709 12:20:42.996864 (+ 24957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:43.621266 (+624402us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:43.621317 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:43.621365 (+    48us) raft_consensus.cc:1685] finished
0709 12:20:43.621378 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:43.621513 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36876,"prepare.queue_time_us":214716,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":96,"raft.queue_time_us":120537,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":22850,"spinlock_wait_cycles":31629824,"thread_start_us":308069,"threads_started":3,"wal-append.queue_time_us":1463690}
I20260709 12:20:43.599774  1859 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.722584 27737 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:43.734215 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.765417 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.708957 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.732138  1791 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:43.720116  1848 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.766381  1791 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:43.766527  1571 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:43.735981  1858 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.766857  1791 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:43.767860  1848 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.768504  1858 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.747440 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.769234 27735 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:43.750836  1715 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:43.693997  1841 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.653580  1835 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:43.689529  1854 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:43.770267 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.770177  1841 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.753005 27661 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:43.724475  1620 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.757316 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.760430 27589 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:43.772250 27586 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.716890  1844 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.764108 27736 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:43.770677  1854 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.704468  1865 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:43.766664 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57759.2us
I20260709 12:20:43.772518 27589 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:20:43.761312 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.773371 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.771136  1637 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:43.776126 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:43.749316  1796 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.781312 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.771757  1411 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.783058  1863 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:43.783648  1731 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:43.783758  1709 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.784636 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.785939  1818 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.788152  1411 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.779677  1637 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.788950  1637 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:43.790170 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.791107  1844 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.791585 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.791759  1844 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:43.792490 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.793011  1826 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:43.793088 27657 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:43.793540  1546 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:20:43.793939  1546 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.794124 27581 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 14: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.794318  1546 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:43.795883 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.796689 27582 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.797015 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:20:43.798308  1717 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:43.799428 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.775015  1854 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:43.802513 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.803138 27657 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:43.803287 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.771770  1739 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:43.804101  1739 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.804616  1739 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:43.779873  1620 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:43.806087  1620 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:43.807008  1620 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:43.795964  1865 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.824023 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.814523 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.818490 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.818887 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:43.819873 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.819562 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.825304 27661 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:43.819778 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:43.820418 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.826540 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:43.824844 27663 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:43.820824 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.821165 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.828403 27738 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 14: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.815852 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.797013  1689 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:20:43.823483 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:43.830397 27734 raft_consensus.cc:2368] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:20:43.825683 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.831058 27657 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:43.832998 27588 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.777109  1835 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:43.854467 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:43.855855 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.856631 27580 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:43.857410 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.857949 27580 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:43.837141 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.862761 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.827350  1851 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:43.874754 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.865252 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.835340  1848 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:43.836287  1855 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:43.878172  1855 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:43.869917  1657 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:43.880617  1657 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.881088  1657 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:43.813668  1858 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:20:43.877565  1841 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.861344  1689 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:43.886199 27582 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:20:43.887991 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:43.883567  1855 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.867812  1844 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:43.906069 27580 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:20:43.910727 27552 leader_election.cc:400] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:20:43.911105 27552 leader_election.cc:403] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:20:43.911639 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.912401 27663 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:43.908244 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:43.900167  1844 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:43.894444  1851 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:43.915023  1524 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:43.879244  1848 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:43.917516  1835 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:43.847445  1860 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:20:43.918345  1835 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:43.917577  1848 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:43.921849  1854 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.924341  1851 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:43.919303  1835 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.915277  1844 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:43.901400  1792 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.819s	user 0.011s	sys 0.013s
I20260709 12:20:43.934551 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:43.931105 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.935029 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:43.925418  1411 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:43.942490  1411 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:43.940570 27658 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:43.900197  1731 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:43.959822 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.946681  1411 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.931284  1870 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:43.963950  1411 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:43.968974  1546 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:43.971926  1861 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:43.972885  1875 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.069s	user 0.002s	sys 0.000s
I20260709 12:20:43.900245  1760 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:20:43.978324 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.978926 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.979643 27585 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:43.980489  1689 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.983677 28334 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:43.931522  1865 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:43.979579 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:43.990193 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:43.990861 27588 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:43.982734  1657 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:43.983155  1859 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:43.974565  1861 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:44.004109  1546 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.010174  1814 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:44.012230  1814 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:44.016731 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.022473 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.023551 27663 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:44.017820 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:44.022487 27658 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:44.025830 27738 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:20:44.027915 28332 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:20:44.037616 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:44.040437 28330 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:44.030725  1818 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:20:44.012727  1814 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:44.049655 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.050966 27628 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:44.051334 27628 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:20:44.068307 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:20:44.069370 27663 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:44.076829  1657 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.080354 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:44.081509 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:44.121812 27735 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:44.082584 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:44.085880 27732 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:44.086465 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:43.991390  1792 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:44.123754 27733 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:44.125975  1792 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:44.096305  1854 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:44.074215 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:44.127461  1696 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:44.103457 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:44.129037 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:44.110234 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.106818 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:44.130363 27657 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:44.109113 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:44.064221 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:44.112709 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:44.131353 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67271us
I20260709 12:20:44.115792 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.115151 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:44.132175 27661 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:44.118065 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:44.127427 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53310.5us
I20260709 12:20:44.127617 27734 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:44.134261 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:44.134918 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:44.142182 27731 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:44.146692 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.145884 27589 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:44.066094  1823 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:44.165665  1823 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:44.163468 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:44.031890  1865 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:44.161155 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:44.166132 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:44.217645 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81066.3us
I20260709 12:20:44.188531 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
W20260709 12:20:44.219830 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58764.6us
I20260709 12:20:44.128577  1389 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:44.218142 27663 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:44.135394  1870 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:20:44.087119  1844 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:20:44.223891 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52364.1us
I20260709 12:20:44.101687  1637 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:44.236636 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.226686 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.238658 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.164943  1715 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:44.119381  1848 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:44.160671  1792 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:44.200981 27588 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:44.239647 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.208954 27657 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:44.255496 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:44.247908 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:44.167095  1832 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:20:44.182250  1856 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:20:44.239733 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:44.262080 27662 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:44.222234  1823 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:44.115773  1739 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:44.296664 27659 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 12:20:44.297293 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57762.2us
I20260709 12:20:44.286298 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:44.287312 27733 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:44.226228  1809 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:44.231509  1657 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:44.291003 27663 raft_consensus.cc:606] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.303357 27663 raft_consensus.cc:614] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:20:44.224524 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61208.5us
I20260709 12:20:44.305284 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:44.288861  1791 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.521s	user 0.003s	sys 0.007s
I20260709 12:20:44.306375  1791 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:44.308593  1865 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:44.254375  1870 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:20:44.308888  1791 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:44.309389  1890 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:20:44.293354 28328 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22.
I20260709 12:20:44.235900  1869 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:44.298156 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.298463 27657 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:44.313140 27582 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:44.317358 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:44.284857  1792 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:44.323731 27588 raft_consensus.cc:2368] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:44.324672 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:44.324923 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.318163 27733 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:44.338299  1848 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:44.342115  1889 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:20:44.319047  1809 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:44.356243  1809 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:44.324358  1887 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:20:44.301873  1835 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:44.398689 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73926.8us
I20260709 12:20:44.399574 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:44.333349 27658 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:20:44.310563  1844 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.301414  1832 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:44.321064  1791 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:44.295718  1836 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:44.418749 27657 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:44.324708  1792 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:44.181632 27481 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 20. New cstate: current_term: 20 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:44.411163  1798 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:20:44.417265  1859 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:44.394304  1382 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 12:20:44.463486  1739 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.473047  1637 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.487905  1832 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:44.491843  1524 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.575s	user 0.005s	sys 0.005s
I20260709 12:20:44.492666 27655 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:44.426580  1836 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:44.495712  1836 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.496534  1760 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:44.497100  1760 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:44.447000 27584 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:44.569641 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 279958us
I20260709 12:20:44.572613 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:44.573374 27733 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:20:44.588541 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:44.590328 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:44.592275 27661 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
W20260709 12:20:44.618103 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 303061us
I20260709 12:20:44.628161 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:44.633069  1792 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:44.643450 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:44.580569  1739 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:44.774914 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 146820us
I20260709 12:20:44.776558 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:44.784508  1658 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:20:44.786533  1546 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:44.787266  1870 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:44.787962  1870 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:44.788952  1870 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:44.786702  1848 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:44.787352  1546 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:44.794806 27702 consensus_peers.cc:597] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6d40162b2bb4b61adc69c9fa3f03326. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.855966 27588 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:20:44.864445 27584 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:44.909094  1546 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:44.913625  1792 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:44.918753  1903 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:44.927492  1898 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:20:44.935128  1895 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:20:44.871726  1848 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:44.793970  1739 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:44.948590 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.002s	sys 0.010s
I20260709 12:20:44.879951  1641 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:44.940928  1859 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:44.985481 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.005s	sys 0.012s
W20260709 12:20:44.987336 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 587747us
I20260709 12:20:44.994997 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3157) took 1005 ms. Trace:
I20260709 12:20:44.994237  1641 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:44.995499 27583 rpcz_store.cc:276] 0709 12:20:43.989170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:43.999576 (+ 10406us) service_pool.cc:224] Handling call
0709 12:20:43.999935 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:44.021685 (+ 21750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:44.021723 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:44.022117 (+   394us) log.cc:844] Serialized 26 byte log entry
0709 12:20:44.399558 (+377441us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:44.399600 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:44.399688 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:44.994747 (+595059us) raft_consensus.cc:1685] finished
0709 12:20:44.994773 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:44.994934 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22511,"raft.queue_time_us":31427,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7278,"thread_start_us":351575,"threads_started":1,"wal-append.queue_time_us":584061}
I20260709 12:20:44.997916  1641 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:44.999944 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.004899 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.005750 27658 raft_consensus.cc:2368] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:45.001291 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.001480 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:45.007890 27657 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:45.008720 27658 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:20:45.004747  1791 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.580s	user 0.009s	sys 0.000s
I20260709 12:20:45.020675 27661 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:45.024312 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:45.024977 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.004s	sys 0.008s
W20260709 12:20:45.025306 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.004s	sys 0.008s
I20260709 12:20:45.025159 27733 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:45.026070 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:45.028005 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:44.961925  1903 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.019704  1791 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:44.990054  1859 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.032034 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:45.035135 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.000s	sys 0.024s
W20260709 12:20:45.039274 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.000s	sys 0.024s
I20260709 12:20:45.027408  1524 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:45.035076  1710 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:20:45.029551  1869 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:45.038750 27729 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:20:45.022684 27660 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.022s	sys 0.038s
W20260709 12:20:45.043314 27660 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.023s	sys 0.038s
I20260709 12:20:45.043876  1903 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:45.045593 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.046244 27583 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:45.046625 27729 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:45.031877  1717 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:45.047242  1717 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.048717  1859 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.049393  1717 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:45.049861  1814 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.984s	user 0.011s	sys 0.001s
I20260709 12:20:45.050252  1814 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.050621 27733 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:45.051421  1885 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.810s	user 0.010s	sys 0.042s
I20260709 12:20:45.052388  1814 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.052800 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.053138 27585 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
W20260709 12:20:45.052388  1823 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.741s	user 0.010s	sys 0.000s
I20260709 12:20:45.054378 27729 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:45.054162  1823 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.040874 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:45.055186 27732 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:45.032095  1696 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:45.056278  1823 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.031384  1848 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.060463 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.061074 27660 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:45.063619 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 286029us
W20260709 12:20:45.063968 27587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.012s	sys 0.013s
W20260709 12:20:45.064342 27587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.012s	sys 0.013s
I20260709 12:20:45.033953 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:45.065253 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.065658 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:45.068580 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3160) took 1140 ms. Trace:
I20260709 12:20:45.023732  1382 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:20:45.069088 27582 rpcz_store.cc:276] 0709 12:20:43.928123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:43.928224 (+   101us) service_pool.cc:224] Handling call
0709 12:20:43.928620 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:44.321402 (+392782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:45.068274 (+746872us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400088ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:45.068503 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880348928}
I20260709 12:20:45.075466 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:45.075932 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:45.079241 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:45.081729 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.084338 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.084882 27656 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:45.085546 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:45.068711  1913 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:45.077806  1696 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:45.097064  1914 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:45.099067  1739 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:45.102195 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:45.102931 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:20:45.103965  1908 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:45.104477  1908 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:45.105518  1908 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.112737 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:45.089566 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:45.107038  1908 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:45.120499 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2655) took 1185 ms. Trace:
I20260709 12:20:45.121129 27734 rpcz_store.cc:276] 0709 12:20:43.934638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:43.934788 (+   150us) service_pool.cc:224] Handling call
0709 12:20:45.120268 (+1185480us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:45.120416 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":427818240}
I20260709 12:20:45.121973  1382 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:45.122545  1382 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.123602  1657 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:45.089084 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:45.084932 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:45.132805 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48003.3us
I20260709 12:20:45.134819 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:45.135742 27731 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:20:45.113008  1836 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:45.138936   996 raft_consensus.cc:2764] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
I20260709 12:20:45.138867 27656 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:20:45.140170 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51181.5us
I20260709 12:20:45.134238 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.147009   996 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:45.147262 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:45.147347 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60855.1us
I20260709 12:20:45.149403 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:45.124854  1657 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:45.152055 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:45.155928 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:45.159288 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.159898 27661 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:45.160072 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.161079  1731 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:45.162083 27731 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:20:45.164052 27732 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:20:45.164170 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:45.165795 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:45.166429 27657 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:45.168604 27661 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:20:45.124738  1382 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.170899 27701 leader_election.cc:400] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:45.172065 27701 leader_election.cc:403] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:20:45.138520  1571 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:45.155357  1913 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:45.174800 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:45.175352   996 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:20:45.173163  1792 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:20:45.154909  1859 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:20:45.139053  1836 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:45.153599  1739 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:20:45.155490  1912 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:20:45.226429 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51700.6us
I20260709 12:20:45.226912   996 raft_consensus.cc:2764] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:20:45.231688 27582 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:20:45.245590 27585 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
W20260709 12:20:45.243498 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69029.9us
I20260709 12:20:45.246467 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:45.247393 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:45.173237  1907 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:45.254099 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:45.254817 27587 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:45.255013 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.256661   996 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:45.257314   996 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:45.200222  1913 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:45.263139 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.202687  1910 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:45.264261 27585 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:45.244863  1832 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:45.270645 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95978.5us
I20260709 12:20:45.202142  1382 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:20:45.248771  1848 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:45.274732 27587 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:45.259153  1731 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:45.258837  1907 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:45.279587 27583 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:45.263746   996 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:45.283203 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1050) took 1971 ms. Trace:
I20260709 12:20:45.283605 27655 rpcz_store.cc:276] 0709 12:20:43.311562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:43.311702 (+   140us) service_pool.cc:224] Handling call
0709 12:20:45.283010 (+1971308us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400229f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:45.283121 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2712370432}
I20260709 12:20:45.276371  1689 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:45.251533  1914 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:20:45.280357  1907 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:45.275240  1809 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 12:20:45.310364  1524 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:20:45.320809  1760 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:45.326666 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:45.327466  1910 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:45.327797 28330 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:45.328540  1910 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.280504  1848 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.299453  1689 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:45.277478  1832 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.306318  1844 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:45.278419  1836 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.335479  1914 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:45.350231  1904 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.304s	user 0.000s	sys 0.001s
I20260709 12:20:45.314958  1913 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.351285  1848 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:45.354921 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:45.339339  1689 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:20:45.359861 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84605.4us
W20260709 12:20:45.363526 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 92009.1us
I20260709 12:20:45.366323 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.367045 27664 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:45.359570  1907 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.370540 27655 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:45.370806  1658 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.343231  1731 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.361459  1914 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.374383 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:45.377949 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:45.364341 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:45.379801 27664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:45.380439 27664 raft_consensus.cc:606] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.380488 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:45.384326 27733 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:20:45.384775 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:45.385623 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:45.374778  1907 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:45.391194 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.362957  1836 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:45.343690  1844 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.398769 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:45.412549 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 957) took 1424 ms. Trace:
I20260709 12:20:45.417198 27660 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:20:45.412940 28328 rpcz_store.cc:276] 0709 12:20:43.978243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:43.978679 (+   436us) service_pool.cc:224] Handling call
0709 12:20:45.396043 (+1417364us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:45.402579 (+  6536us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":721399040}
I20260709 12:20:45.420415  1928 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:45.429153 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.422067 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:45.434504 27663 raft_consensus.cc:606] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.433611 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:45.440210 27663 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 12:20:45.427263  1658 tablet_bootstrap.cc:492] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:20:45.411808  1781 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:45.466449   996 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:45.468621  1914 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:45.469106  1664 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:20:45.469519  1664 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:45.470362  1664 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.471307  1664 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:45.473326  1657 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:45.485924 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:45.440003  1641 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:45.486673 27661 raft_consensus.cc:606] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:45.487264 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 2988) took 1381 ms. Trace:
I20260709 12:20:45.428115  1929 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:45.457916  1915 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:45.441445  1928 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:45.492204 27730 rpcz_store.cc:276] 0709 12:20:44.105913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:44.106091 (+   178us) service_pool.cc:224] Handling call
0709 12:20:44.132331 (+ 26240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:44.136567 (+  4236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:44.136613 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:44.136669 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:20:44.136713 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:45.486971 (+1350258us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:45.487035 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:45.487191 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9227648}
I20260709 12:20:45.494460 27704 leader_election.cc:400] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:45.499086 27660 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:20:45.502529 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:45.528323 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.530076 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:45.531688 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.531155 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.535282 27663 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:45.540724 27589 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:20:45.547941 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:45.549139 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:45.549803 27660 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:45.518170  1658 ts_tablet_manager.cc:1403] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 3.923s	user 0.065s	sys 0.043s
I20260709 12:20:45.575613 27588 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:20:45.439148  1922 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.098s
I20260709 12:20:45.579949 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:45.580152   996 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:20:45.581557 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:45.579123 27661 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:20:45.555542 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 104850us
W20260709 12:20:45.585638  1932 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:20:45.586727  1856 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:45.587356  1856 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.496917  1929 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:45.538447  1848 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:45.518209  1863 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:45.503616  1851 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:45.521907  1915 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:45.539448  1899 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:20:45.486325  1781 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:45.591181 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148712us
I20260709 12:20:45.538424  1798 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:45.596320 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.596546  1848 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.538225  1832 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.542560  1571 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:20:45.668996 27587 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
W20260709 12:20:45.677161 27702 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.4:39321 timed out after 1.902s (SENT) [suppressed 14 similar messages]
I20260709 12:20:45.597563  1704 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:45.547197  1919 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:45.673480  1898 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:20:45.603513  1851 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.682566  1942 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:45.687186  1930 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260709 12:20:45.692826 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96588.9us
I20260709 12:20:45.601524  1915 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:45.593307  1814 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:45.655709  1740 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:20:45.688087  1781 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.697732  1910 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:20:45.718142 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:45.594872  1929 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:45.739754 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 62617.8us
I20260709 12:20:45.601184  1912 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:45.745921 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:45.691565  1657 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.757200 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:45.602029  1863 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.846601 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:45.748497  1938 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:20:45.879276 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 210303us
I20260709 12:20:45.696710  1798 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.755945  1870 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.698069  1919 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.749289  1382 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:45.900826  1946 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:20:45.901378  1652 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:45.904511  1652 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.905316  1919 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:45.842195  1920 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:45.906239  1920 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.907636  1652 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:45.908442  1920 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.747265  1942 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.910692  1942 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:45.911736  1836 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.506s	user 0.009s	sys 0.000s
I20260709 12:20:45.912420  1870 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.679899  1928 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.915207  1870 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:45.920917  1643 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:45.924975  1944 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:20:45.928460  1856 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:45.930912  1937 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:45.847841  1912 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:45.697392  1704 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:45.894479  1924 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:46.021761 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82085.7us
W20260709 12:20:46.052485 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 167854us
I20260709 12:20:46.063989 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.064244  1382 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:46.068403  1382 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.751965  1814 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.086380 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:46.087489 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:46.085904 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:46.096603 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 12.
I20260709 12:20:46.106540 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.106806 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:46.081339  1652 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.052042  1835 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:20:46.069690  1863 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.111472  1710 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.113201  1720 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.110962 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:46.116261 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.083400  1919 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.123533  1851 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.123858  1658 raft_consensus.cc:359] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:46.133500  1863 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.135233  1798 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.152110 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:46.136128  1652 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.136466  1848 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.136842 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.137601  1710 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:45.986333  1643 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.139416  1908 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.068943  1382 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.071004  1924 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.083630  1942 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:20:46.116225  1740 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.156687 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:46.165563 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:46.143432  1704 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.146013  1814 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:46.203760 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47184.9us
I20260709 12:20:46.205037  1740 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.152691  1720 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.206686  1710 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.207613  1709 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:46.208722  1924 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.151268  1658 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:20:46.163885  1280 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:46.214594  1710 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:46.166940  1854 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:46.150699  1919 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.207187  1740 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.248582 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:46.249456 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.251503 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:46.154999  1863 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.198719  1908 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.322996 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:46.237197  1854 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:46.412776 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:46.445158 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122280us
I20260709 12:20:46.241626  1280 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:46.254743  1818 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:46.245164  1704 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.215s
I20260709 12:20:46.516623 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:46.264614  1689 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.891s	user 0.000s	sys 0.012s
I20260709 12:20:46.503631 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.165699  1912 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:46.458185  1814 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.440176  1854 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:46.441210  1740 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.458612  1872 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:46.547348  1689 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.519258  1643 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.460456  1720 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:46.602305  1947 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:20:46.312350  1658 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:46.607440  1740 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:46.610706 27730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.041s	sys 0.111s
W20260709 12:20:46.617488 27730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.041s	sys 0.111s
I20260709 12:20:46.610788  1664 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:46.592635  1859 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:46.480958  1940 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.221s	user 0.000s	sys 0.000s
I20260709 12:20:46.610054  1704 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.632083  1664 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.644371  1664 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.635648  1652 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.610194  1814 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.527525  1818 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.648293  1704 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:46.657692  1915 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.907s	user 0.005s	sys 0.006s
W20260709 12:20:46.664245  1929 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.907s	user 0.010s	sys 0.004s
I20260709 12:20:46.666453  1919 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.636689  1859 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:46.669204  1935 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.072s	user 0.017s	sys 0.007s
W20260709 12:20:46.684376  1948 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.443s	user 0.002s	sys 0.000s
I20260709 12:20:46.720568 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.571384 27582 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:46.732648 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3173) took 1205 ms. Trace:
I20260709 12:20:46.733279 27581 rpcz_store.cc:276] 0709 12:20:45.527261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:45.532799 (+  5538us) service_pool.cc:224] Handling call
0709 12:20:45.533153 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:45.897545 (+364392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:45.897579 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:45.897618 (+    39us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:45.897651 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:46.732045 (+834394us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54000bd6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:46.732124 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:46.732442 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":837504768}
I20260709 12:20:46.758942 27580 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:20:46.762360 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:46.763952 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:46.765305 27661 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:20:46.659785  1382 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.785353  1382 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.786860  1382 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.792654  1382 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.670245  1915 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.823848  1710 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.825249  1709 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:46.826295  1709 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.826892  1915 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.828179 28330 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:46.832072 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:46.661649  1740 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.851579 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:46.855597 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58088.9us
I20260709 12:20:46.856647 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:46.857616  1859 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.857972 27658 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:46.858222  1620 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:46.670917  1929 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.863631  1942 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.864862  1942 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.867691  1818 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.871812  1658 ts_tablet_manager.cc:1434] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 1.292s	user 0.009s	sys 0.014s
W20260709 12:20:46.872575  1658 ts_tablet_manager.cc:1467] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Tablet startup took 1297ms
I20260709 12:20:46.656617  1280 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.876550  1924 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:46.877584  1924 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.878847  1907 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.879719  1953 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.880271  1907 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.881009  1953 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.885035  1970 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.889153  1814 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.890904  1696 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.892086  1696 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.894919  1972 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.896027  1972 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.898277 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.899358 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:46.899828  1856 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:46.901244 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.903115 27589 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:20:46.901224  1856 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.678450  1908 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.831570  1710 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.886523  1970 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.909976  1858 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.881258  1863 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.898334  1689 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.914566  1869 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:46.911090  1858 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.858902  1620 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:46.865795  1929 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.936393  1942 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:46.941208 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84306us
I20260709 12:20:46.941856 27660 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:20:46.919924  1972 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.950779 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:46.951889 27580 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:46.957010  1858 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.953177  1869 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.960233  1869 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.960289  1924 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.962175  1970 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:46.962553 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:46.965687  1689 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:46.965956  1908 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.966749  1689 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.967716  1739 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.968639  1739 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.969588  1689 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.969769  1978 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.970686  1978 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.971613  1798 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:46.972177  1976 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.972507  1798 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.973397  1978 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.957835  1620 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.973047  1976 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.979866  1915 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:46.980386 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:46.980930  1915 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:46.982436 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44409.5us
I20260709 12:20:46.984236  1979 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.984606 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:46.985011  1979 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:46.987473 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:46.992534 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:46.989621 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:46.995586  1953 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:46.995771 27659 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:20:46.996380  1953 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.997094  1953 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:46.997867  1976 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:46.998519  1976 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:47.004150 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.005191 27729 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:47.005226  1859 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:47.005811  1859 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.006202  1859 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:47.007602  1844 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:47.009100  1961 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20260709 12:20:47.010613  1960 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:47.011096  1960 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:47.012411  1960 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.014300  1960 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:46.981339 27734 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:47.015297  1620 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:47.017324 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:47.014372 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.019855 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
W20260709 12:20:46.873152  1658 ts_tablet_manager.cc:1469] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Trace:
0709 12:20:37.708162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:37.708283 (+   121us) service_pool.cc:224] Handling call
0709 12:20:38.102600 (+394317us) ts_tablet_manager.cc:934] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
0709 12:20:39.454839 (+1352239us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:20:39.600849 (+146010us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:39.600973 (+   124us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:20:39.688044 (+ 87071us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:20:39.688384 (+   340us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:39.690948 (+  2564us) ts_tablet_manager.cc:989] Replica registered.
0709 12:20:39.691032 (+    84us) inbound_call.cc:177] Queueing success response
0709 12:20:41.595020 (+1903988us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:20:45.579574 (+3984554us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:20:45.747945 (+168371us) tablet_replica.cc:261] Starting instrumentation
0709 12:20:45.748637 (+   692us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":70024,"raft.queue_time_us":11781,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":6495,"tablet-copy.queue_time_us":215240,"tablet-download-pool.queue_time_us":41939,"tablet-download-pool.run_cpu_time_us":1441,"tablet-download-pool.run_wall_time_us":1237808,"thread_start_us":311882,"threads_started":2}
I20260709 12:20:47.024501  1913 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.025202  1913 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.027577  1798 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.028611  1571 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.029430  1571 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.031100  1972 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.031961  1844 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.031850  1972 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:47.032335 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53602.9us
I20260709 12:20:47.032778  1844 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.034247  1972 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.035243  1844 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.035755  1913 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.035812  1841 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:46.967180  1908 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.036763  1841 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.038419  1984 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:47.038892  1984 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.039376  1984 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:47.039669  1731 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.038247  1977 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Committing config change with OpId 14.10: config changed from index -1 to 10, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:47.040429  1731 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.043697  1731 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.045485  1856 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.046131 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.047719 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.048439 27734 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:47.048908 27729 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:20:47.050506 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.053658  1953 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.055433  1739 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.059783  1915 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:47.064085 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:47.064225  1571 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.066001 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:47.067934 27481 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 20, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 20 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.069845 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.070398 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.070837 27663 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:47.071345 27657 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:47.072973 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.073678 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:47.075356  1835 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.075960  1835 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.068646  1854 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:47.059654 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82722.3us
I20260709 12:20:47.082551 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.083056 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.084131 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.038960  1908 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.110997 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.111655 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.111974 27737 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:47.113024 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.113148  1854 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.114650 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.078516  1835 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.039219  1841 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:47.104274 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70251.8us
I20260709 12:20:47.127625 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.132231 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:47.154282 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.123145 28334 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:47.121706 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:47.153558 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57151.7us
I20260709 12:20:47.115793 27735 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:47.169425 27657 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:47.080152 27481 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 20, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 20 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.154107 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.126811 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.123108 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.140794 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:47.144079 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.115525 28329 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:47.125491   996 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.147377 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.111023 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.144768 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.149829 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.115489 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.179838 28330 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:47.148010 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.148659  1814 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.122756  1715 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:20:47.137732 27733 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:47.122366 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.183948 27581 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:47.184151 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:47.184731 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 52863.9us
W20260709 12:20:47.133105  1971 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.197s	user 0.003s	sys 0.000s
I20260709 12:20:47.154841 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.177462 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:47.189790 27731 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:47.189333 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:47.132921 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-8355fe4a548a43f8b5215ea9ae7ba8b2 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:20:47.143602  1652 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.149750  1731 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.128448  1704 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.184866 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.124437 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.121243  1979 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.115443  1907 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.151609  1720 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:47.148808  1571 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.191674  1835 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:20:47.119927  1915 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.076s
I20260709 12:20:47.150269  1953 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:47.215921 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44854.6us
I20260709 12:20:47.216465 28329 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEARNER]: Advancing to term 20
W20260709 12:20:47.216943 27484 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 224) took 33584 ms (client timeout 15000 ms). Trace:
I20260709 12:20:47.217041  1739 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:47.206270  1856 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.217630 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.217432  1814 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.218843 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.219594 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.199653  1854 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.220312 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.217491 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-8355fe4a548a43f8b5215ea9ae7ba8b2 on tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:47.221395 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.223346 27731 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:47.223891 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.224385 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.225390 27588 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:47.185079  1984 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.226138 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.227524   996 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.190155  1689 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.237407 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.259593 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.259850 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.260227  1823 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:47.260780 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.261349  1844 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:47.261646  1823 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:47.261834  1844 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.262428  1844 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:47.262807  1870 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:47.263244  1870 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:47.263581  1720 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.264273  1870 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.265700  1870 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:47.266331  1863 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:47.266822  1863 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.267261  1863 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:47.268278  1856 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:47.270884  1856 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:47.271855 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 106277us
I20260709 12:20:47.271976 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.272930 27584 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:47.273418 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.274326  1710 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.274930 27582 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:47.276275   996 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.277038 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.183940 27481 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 20, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 20 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.258535  1689 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.252198  1709 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.253329 27657 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:47.248983 28336 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:47.262369 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.276203 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.257582 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.272831 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.251295  1950 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:47.251832  1953 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.281482 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.286175 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.286650 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.288210 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.284833 27589 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:47.242924  1739 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.291126 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.278805 27481 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 24, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 24 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.253765  1854 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.309285 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.319824 27583 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:47.288813  1950 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.290911  1696 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.322851  1840 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:47.323050 28336 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Advancing to term 17
I20260709 12:20:47.326853 27481 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 15, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.334860 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.335027 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.326772 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.336982  1720 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.291049  1715 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.247260  1652 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:47.337697 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:47.331820 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.351085 27662 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:47.299955  1739 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:47.340859 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.314148 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.332646 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:47.346171 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86679.3us
I20260709 12:20:47.326656 27659 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:47.306838  1977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:53824 (request call id 1) took 6934 ms. Trace:
I20260709 12:20:47.324637 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.322868 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.285247  1731 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.284749  1571 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.358062 28329 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:20:47.360409 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:47.367595 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59761.6us
I20260709 12:20:47.362382  1908 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.361661 27481 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 18, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 18 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.347957  1950 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:47.371708 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.370002 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.374336  1731 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.370658 27663 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:20:47.378474 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:47.377628 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53085.9us
I20260709 12:20:47.350708  1840 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:47.382987 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.382531  1732 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:47.387693  1732 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.388123  1732 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:47.388429  1715 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.390753  1854 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.383085  1908 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:20:47.396629 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75957.5us
I20260709 12:20:47.397475 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.397778  1908 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.398486 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.384867 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.400839 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.385587 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.401530 27581 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:47.398918 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.399780 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.400166 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.399972 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.403995 27659 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:47.350088  1823 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:20:47.406930 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.376680  1620 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:47.412734 27583 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:47.381410  1840 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:47.413743  1814 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:47.414047  1994 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:20:47.413944 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.384347  1704 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:47.414660 27656 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:20:47.416299 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.417076  1995 ts_tablet_manager.cc:933] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.418704 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:47.420730 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.422406 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.422574 27655 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:47.421476 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:47.423833 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:47.421417 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.358853  1977 rpcz_store.cc:276] 0709 12:20:40.371972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:40.498856 (+126884us) service_pool.cc:224] Handling call
0709 12:20:40.590443 (+ 91587us) log.cc:844] Serialized 499 byte log entry
0709 12:20:40.756803 (+166360us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b5400049aa0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c1142 00007f0e9a2a1eaa 00007f0e9a2a8d53 00007f0e9a2a8cd9 00007f0e9a2a8c69 00007f0e9a2a8c31 00007f0e9a2a7dfd 0000000000388634 0000000000388479 00007f0e9725b475
0709 12:20:40.853733 (+ 96930us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b180049a860. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:47.306673 (+6452940us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5202,"raft.queue_time_us":6506,"raft.run_cpu_time_us":3751,"raft.run_wall_time_us":227010,"spinlock_wait_cycles":10530560}
I20260709 12:20:47.424336 27481 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 7 to 16. New cstate: current_term: 16 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.425990 27481 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.426608 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.420670  1704 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.427496  1704 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:47.428614  1841 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:47.429003  1841 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.429366  1841 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:47.430831  1832 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:47.431964 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.432168 27656 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Committing config change with OpId 14.10: config changed from index -1 to 10, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:47.433396  1652 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.420507  1984 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.415004  1924 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.435791  1924 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.437851  1924 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.438196  1844 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.438127  1840 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.440150 28336 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:20:47.426906 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.449895  1715 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.399680  1953 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.452682 28330 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEARNER]: Advancing to term 18
I20260709 12:20:47.440538 27659 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:47.457576 27581 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:47.419579  1620 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.468153 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.495697 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.449798 27481 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 20, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 20 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:20:47.501454 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69555.8us
I20260709 12:20:47.465346  1731 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:47.468040  1620 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:47.472426  1571 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:47.510171 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111382us
I20260709 12:20:47.516877 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.520565 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.510958 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:47.473155  1913 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
W20260709 12:20:47.238692 27484 rpcz_store.cc:269] 0709 12:20:13.576553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:13.649151 (+ 72598us) service_pool.cc:224] Handling call
0709 12:20:47.135137 (+33485986us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:39.051287 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:39.052041 (+   754us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:39.052094 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:20:39.052116 (+    22us) tablet.cc:662] Decoding operations
0709 12:20:39.054394 (+  2278us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:39.054518 (+   124us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:39.054551 (+    33us) tablet.cc:685] Acquiring locks for 26 operations
0709 12:20:39.236186 (+181635us) tablet.cc:701] Row locks acquired
0709 12:20:39.236251 (+    65us) write_op.cc:260] PREPARE: finished
0709 12:20:39.236475 (+   224us) write_op.cc:270] Start()
0709 12:20:39.236678 (+   203us) write_op.cc:276] Timestamp: P: 1783599639236387 usec, L: 0
0709 12:20:39.236726 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 12:20:39.406090 (+169364us) log.cc:844] Serialized 10150 byte log entry
0709 12:20:39.597677 (+191587us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a343e28 00007f0e9dd7e581 00007f0e9dd7dd58 00007f0e9dd829f1 00007f0e9dd829a9 00007f0e9dd82939 00007f0e9dd82901 00007f0e9dd81b2d 0000000000388634 0000000000388479
0709 12:20:39.808613 (+210936us) op_driver.cc:464] REPLICATION: finished
0709 12:20:40.176393 (+367780us) write_op.cc:301] APPLY: starting
0709 12:20:40.176492 (+    99us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:40.601741 (+425249us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:40.609877 (+  8136us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:41.132324 (+522447us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:41.132900 (+   576us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=52,mrs_lookups=0
0709 12:20:41.132947 (+    47us) write_op.cc:312] APPLY: finished
0709 12:20:41.133500 (+   553us) log.cc:844] Serialized 228 byte log entry
0709 12:20:41.245065 (+111565us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:41.269817 (+ 24752us) write_op.cc:454] Released schema lock
0709 12:20:41.312381 (+ 42564us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:41.358553 (+ 46172us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:41.358599 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":33690,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":304619,"apply.run_cpu_time_us":86760,"apply.run_wall_time_us":1182073,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112807,"num_ops":26,"prepare.queue_time_us":13400,"prepare.run_cpu_time_us":29316,"prepare.run_wall_time_us":546647,"raft.queue_time_us":118483,"raft.run_cpu_time_us":16265,"raft.run_wall_time_us":280022,"replication_time_us":535822,"spinlock_wait_cycles":35281152,"thread_start_us":249644,"threads_started":5,"wal-append.queue_time_us":202964,"wal-append.run_cpu_time_us":15000,"wal-append.run_wall_time_us":2321863}]]}
I20260709 12:20:47.476562  1859 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.525552 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:47.532095 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56802.2us
I20260709 12:20:47.519588  1814 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:47.502740  1382 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:20:47.541476 27658 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:47.530114  1929 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:20:47.518035  1953 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.021s
W20260709 12:20:47.553735 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45035.8us
I20260709 12:20:47.557363 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.559156  1912 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:47.560565  1950 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.562213  1859 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.562736  1950 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.554623  1922 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:47.544476  1913 consensus_queue.cc:1243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:20:47.572666 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.574131  1859 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:47.574220 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.575860  1859 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:47.576560  1840 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.584486 28330 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:47.585382  1844 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.586087  1844 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:20:47.585314 27481 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 12, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 12 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.586710  1844 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:47.587273 27481 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.588238 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:47.588752 27481 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 12 to 14, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 14 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.589792 27588 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Committing config change with OpId 14.10: config changed from index -1 to 10, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:47.593034 27702 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:47.595000 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 149943us
I20260709 12:20:47.595870  1715 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:47.595255 27481 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 20, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 20 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.578243  1995 tablet_copy_client.cc:323] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:20:47.540460  1863 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.601512 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.601434 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.610551 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:47.611564 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54292.1us
I20260709 12:20:47.612457  1732 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.612186 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.614550  1732 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.617780  1739 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.619642 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.618757 27589 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:47.618579  1863 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.620741 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:47.603235 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.625813  1908 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:47.635799  1411 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 12:20:47.636888  1950 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:47.637181 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:47.637416  1950 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.637938  1950 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:47.638824  1908 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.639395  1908 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.641813  1411 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:47.642238  1411 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:47.643136  1411 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.597608  1907 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.631362 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:47.653766 27661 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:47.657233 27734 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:20:47.597079  1715 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:47.672087 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.659938  1870 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:47.664384  1411 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:47.675276 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:47.680608 27657 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:47.681512 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:47.682678 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:47.659467  1620 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.699116 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.700142 27733 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:47.707659 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:47.705490  1984 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:47.707312 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet a6d40162b2bb4b61adc69c9fa3f03326 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:20:47.708626 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.708985  1984 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:47.709156 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet a6d40162b2bb4b61adc69c9fa3f03326 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-a6d40162b2bb4b61adc69c9fa3f03326
I20260709 12:20:47.709455  1984 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:47.710011 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.710682 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:47.711439 27655 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:47.707777 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.712296 27732 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:47.711539 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:47.712829  1999 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:20:47.715528  1861 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:47.715993  1861 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.716368  1861 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:47.720119  1908 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.720698 27748 tablet_copy_source_session.cc:215] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:47.724536  1984 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:47.727629 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58727.6us
I20260709 12:20:47.728650 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.669874  1869 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.679557  1870 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.732255 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.733951 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.737346 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:47.734788 27737 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:47.739517 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.733175 27738 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:47.668192  1715 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.669622  1907 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:47.749413 27729 raft_consensus.cc:2368] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23.
W20260709 12:20:47.754954 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72402.8us
I20260709 12:20:47.756368 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.758273   996 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:47.758939   996 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.761202   996 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.763634 27655 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:47.736167  1870 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:20:47.759361  2002 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:20:47.777957 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.746843  1732 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:47.783831 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:47.784907 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56368.5us
I20260709 12:20:47.789294 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.789927 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:47.775404  1994 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.226s	user 0.000s	sys 0.003s
I20260709 12:20:47.679077 27481 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 16, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 16 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.793890 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.794793  1620 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:47.802591 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 150934us
I20260709 12:20:47.805401 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.804505 27730 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:47.807968 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.741339  1869 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.783108  1732 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.802961  1696 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:47.822662  1732 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:47.832955 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:47.841238  1704 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.843057 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.843647  1280 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:47.844084  1280 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:47.844427 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.845227 27585 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:47.827114  1912 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:47.846221 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.831122  1984 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.855502 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.856348  1715 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.856674 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.852492 27585 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:47.858294 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.867694 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:47.847636  1620 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:47.849402  1739 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:47.876653 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90797.2us
W20260709 12:20:47.873601 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84766.1us
I20260709 12:20:47.878962 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:47.862462  1912 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.845142  1280 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:47.889185 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.893666 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.896324 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:47.897729  1913 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:47.898245  1913 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:47.899228  1913 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.901656 27589 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:47.903604 27584 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:47.876425 27481 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 20, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 20 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:47.906281  1869 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.907125 27481 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 21, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.858539  1841 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:47.915484  1841 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:47.917516 27582 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:47.918452 27581 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:47.922188 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:47.888082 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:47.923497 27737 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:20:47.926168 27660 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:47.931362 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.934810 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:47.936290  1928 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:47.937575  1924 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.938278  1979 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
I20260709 12:20:47.938234  1924 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.941183  1924 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.878247  1908 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:47.941787  1908 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.942214  1908 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:47.943753 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:47.922698  1912 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:47.944365 27735 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:20:47.908159  1907 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:47.907059  1704 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:47.954365 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:47.934052 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44976.3us
I20260709 12:20:47.959492 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.962916 27585 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:20:47.963486  1382 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:47.964076  1382 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:47.964596  1382 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:47.969746 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.971276 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:47.974545 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:47.984063 27627 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:47.987011 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62966.1us
I20260709 12:20:47.939042  1732 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:47.987589 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:47.989285 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.989573 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.992575 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.995705 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:47.974320  1835 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:47.997277 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.001111 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.001556 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:48.003129 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.942431  1950 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.005455 27586 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:47.990993  1957 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:48.007241 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.990829  1739 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:20:47.995548  1280 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:47.990139  1924 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:48.014194 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:48.012625 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 132986us
I20260709 12:20:48.016688 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:47.998808  1859 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:48.018872 27735 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:48.004130  1832 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:48.023744  1870 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.025979  1870 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.023792  1832 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:48.027230 27580 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:20:48.027892  1832 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.019085  1928 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:20:48.014236  1924 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.029799  1924 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:48.032383  1732 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.034723 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:48.005944  1907 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:48.020838 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.021842 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.035454  1978 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.035784  1907 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.036769  1855 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:48.036940  1907 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:48.037179  1861 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.039681 28334 raft_consensus.cc:2368] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:20:48.039918  1950 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.005920  1987 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:48.046347  1987 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:48.046886  1987 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:48.049944  1835 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.053527 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:47.993790  1929 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 12:20:48.055646  1835 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:20:48.055738  1927 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:48.056190  1835 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.056632  1995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d40162b2bb4b61adc69c9fa3f03326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:48.058269 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.066685 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.011530  1739 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.069340 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.066061  1927 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.070694  1929 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:48.083748 27627 leader_election.cc:400] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23.
I20260709 12:20:48.073426  1739 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:48.093446  1912 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.089399  1861 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.089659  1929 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.099908  1929 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:48.064630  1835 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.105594  1835 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:48.107064  1912 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.108547  1987 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.105024  1907 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.114444  1987 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.080567  1859 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.122622  1950 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:20:48.126746 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:48.126345  1382 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:48.131556 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95267.1us
I20260709 12:20:48.143008 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.143676 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:48.144774 27658 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:48.146147 27659 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:48.148360 28330 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:48.156352  1689 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.156617 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.156472 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.126818  1823 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:48.160570  1823 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:48.160919 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.161837 27582 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:48.164256  1907 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.168161 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.165836  1908 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.181442 27588 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
W20260709 12:20:48.179787 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 172760us
I20260709 12:20:48.087249 27627 leader_election.cc:403] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:20:48.169822 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.186952 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.188925 27655 raft_consensus.cc:2368] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24.
I20260709 12:20:48.191449 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.123214  1950 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:48.160972  1823 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:48.201042 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.206763 27661 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:48.227300 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.227998 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.234524 27589 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 24
W20260709 12:20:48.229489 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 207710us
I20260709 12:20:48.180528  1841 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:48.238008 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 158410us
W20260709 12:20:48.240765 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84395.7us
I20260709 12:20:48.243782 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.245184 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:48.246531 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86007.2us
W20260709 12:20:48.251827 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 276015us
I20260709 12:20:48.259918 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.203261  1859 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:48.262162 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.262429 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.263260 27731 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:48.263447 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.263667 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.264053 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.271883 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.249637  1704 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:20:48.200497  1924 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.276954 27735 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:20:48.272667 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:48.278949 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.261854  1855 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.280949 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.285589 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.286087 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.286607 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:48.243561  1869 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:48.294852 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.286201 27735 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:48.294909 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.297039 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.275699  1908 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.298897 27731 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 12:20:48.299257 27731 pending_rounds.cc:85] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 10
I20260709 12:20:48.281549  1704 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.299624   996 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:20:48.305166 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.286005  1715 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:48.273423  1870 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:20:48.299600 27731 pending_rounds.cc:107] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:20:48.312018 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.312335 27731 raft_consensus.cc:2889] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:48.316025 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.301596  1977 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:20:48.320679 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65625.4us
I20260709 12:20:48.323083 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:48.302894  1704 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:48.319465 27659 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:48.334053 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.337731  1984 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:20:48.338151  1984 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:48.295373  1855 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:48.340230 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.341281  1855 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.341852  1664 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:20:48.344056  1739 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.311802  1929 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.348742 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.349201  1929 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.349921 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.350630 27733 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:48.351341 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.351890 27736 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:48.352386 27732 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:48.338948  1984 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.352962 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.353348 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.353660  1984 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:48.354117 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.354851 27736 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:48.359822  1739 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:48.303296  1869 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.361742  1854 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:48.363108  1869 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:48.364742 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.365469 27663 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:48.365756 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.368659 27658 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:48.313876  1913 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:48.374089 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.379429 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.384187 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.389158 27581 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:48.390574 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67549.8us
I20260709 12:20:48.391605 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:48.392825 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58869us
I20260709 12:20:48.393514 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.394940 27702 leader_election.cc:400] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:20:48.395421 27702 leader_election.cc:403] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:48.396726 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.399852 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.400424 27735 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:48.402005 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:48.401753  1856 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:48.404980  2022 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:20:48.325169  1960 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 12:20:48.402228 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.411312 27584 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:48.325364  1696 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 12:20:48.413849  1620 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:48.414588  1854 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:48.414994  1854 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:48.295866  1950 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.402005 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.409451  1664 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:20:48.399339  1977 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:48.396668  1280 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.327626  1715 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.431471 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.396935  1929 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:48.448709 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.449498 27658 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:48.443622 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50193.2us
W20260709 12:20:48.444340 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:48.411478  1960 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:48.450623 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.421953  1913 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:48.452129 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.421356  1382 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.462757  1854 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.464172  1854 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:48.468309  1929 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.469170  1280 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.470649  1929 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:48.471724  1924 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.474925 27663 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:48.476147 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.479130 27589 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:20:48.480257 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 85506.5us
I20260709 12:20:48.456581  1913 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.456382  1960 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:48.481839 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.470917  1704 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.484222  1280 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:48.483940  1823 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.495141 27733 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:20:48.499221  1859 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23.
I20260709 12:20:48.499660  1924 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:20:48.505043  1715 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:48.501139 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.536901 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:48.510082  1664 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:20:48.514084 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.484824 27735 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:48.515944 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:48.538291 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.542054  1823 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.543568 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.544189 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:48.545040 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.546144 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.546629 27730 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:48.548094 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.549170 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.550086 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.550583 27734 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:48.551643  1863 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.552165  1863 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.552508 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.553624 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.554328  1863 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:48.554539 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.524879 28334 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:48.556690  1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58252 (request call id 966) took 1399 ms. Trace:
I20260709 12:20:48.556843 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:48.557114  1995 rpcz_store.cc:276] 0709 12:20:47.190177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:47.197158 (+  6981us) service_pool.cc:224] Handling call
0709 12:20:47.427393 (+230235us) ts_tablet_manager.cc:934] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
0709 12:20:47.934636 (+507243us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:20:48.398769 (+464133us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:48.403907 (+  5138us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:20:48.555943 (+152036us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:20:48.556217 (+   274us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:48.556557 (+   340us) ts_tablet_manager.cc:989] Replica registered.
0709 12:20:48.556613 (+    56us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":33374,"tablet-copy.queue_time_us":163303,"thread_start_us":190172,"threads_started":2}
I20260709 12:20:48.558755 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.559085  1620 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:48.559439 27731 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:48.559564  1620 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.560098  1620 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:48.562372  1704 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:48.563058 27737 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:20:48.565076 27586 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:48.566031 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.566807  1953 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:48.567238  1953 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:48.567677  1953 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:48.568612  1912 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:20:48.570520  1844 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:48.570915  1844 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:48.571179  1841 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.571939  1844 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.572664  1960 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.531411 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:48.532256 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.519693  1913 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:48.536283 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:48.576398 27655 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:48.533396 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51638.6us
I20260709 12:20:48.574215  1960 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:48.578085 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.591289 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.592556 27701 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:48.592810 27701 leader_election.cc:403] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:20:48.579741 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65750us
I20260709 12:20:48.580125  1859 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:48.593842  1859 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.597038  1950 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:48.551311 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.600082 27734 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:48.600710  1411 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:48.601208  1411 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.580938 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:48.581765 27657 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:48.607136  1411 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.608253 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.609922  1411 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:48.588505 27659 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:20:48.613842 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.628875 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:48.629179 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:48.629913 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:48.634395 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.637691 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.640827 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:20:48.632601  1924 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:48.642724 27655 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:48.636648  1856 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:48.642903 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50495.4us
I20260709 12:20:48.654505 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:48.659896 27737 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:48.661656 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.662608 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.664589 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.673134 27735 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:48.577298  1995 tablet_copy_client.cc:806] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:48.633939  1869 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:20:48.633770  1709 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:48.691437 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:48.663925  1739 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:48.661835  1854 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:48.664399  1957 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:48.636793  1832 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:20:48.685884  1927 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:48.697474  1382 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:48.721069 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 163263us
W20260709 12:20:48.711042 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 415801us
I20260709 12:20:48.700040  1739 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.674880  1689 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:48.671168  1620 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.701946 27736 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:48.694643 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:48.747846 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.594725  1859 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.671949  1907 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:20:48.689899  1835 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:20:48.690265  1924 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.704432  1995 tablet_copy_client.cc:670] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:48.768357 27552 leader_election.cc:400] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24.
I20260709 12:20:48.709534  1709 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:48.735785  1739 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:48.787982 27585 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:48.784200 27552 leader_election.cc:403] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:48.792316 27581 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:48.768137  2010 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:48.736291  1856 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:20:48.793290 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:48.774495  1732 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:48.767529  1978 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:48.732743  1854 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.781949  1280 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:48.800868  1709 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.801539  1620 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:48.805852 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 141318us
I20260709 12:20:48.805868 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:48.806633 27586 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:48.773943  1979 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:48.805961 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:48.808287  2034 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24.
I20260709 12:20:48.770012  1863 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:20:48.810913 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.817662 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.827258 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:48.836678 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:48.849265 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:48.850713 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.852826 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:48.854064 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:48.812996  2010 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:48.860026 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.863437  1854 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:48.864464  2036 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:20:48.824654  2042 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:48.868072  2010 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.818821  1979 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.871227  1979 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:48.875675 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:48.876832 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50462.3us
I20260709 12:20:48.812088  1280 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.867977  1912 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:48.889928 27655 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:20:48.858451  1929 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:48.858734  2034 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:48.855208  1840 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:48.875649  1987 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:48.900204 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.848431  2003 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:20:48.883967  1835 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:48.891793  1912 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.916715 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:48.920228 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:48.908666 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:48.857931  1924 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.928591  1280 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:48.929443  2030 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:48.933116  1953 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.933676  1995 tablet_copy_client.cc:538] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:48.922145  1987 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.935158  1978 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:48.908126  1957 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:20:48.936003  1869 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:48.936631  1856 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:48.920378  1835 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.901283  2034 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:48.899930  1854 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:20:48.880338  1696 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:20:48.948294 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:48.984514 27586 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:48.935851  1927 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:20:48.958480 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 104498us
I20260709 12:20:48.985735  1927 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.986191 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:48.986799  1835 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:48.988722  1823 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:20:48.924468  1912 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:48.989856  1987 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.970219  1869 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:48.952704  1978 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:48.964879 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:48.919723  2040 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:48.995797  2040 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:48.996857  2040 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:48.998025  1863 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:48.967633 27588 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:48.964330 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:48.970098 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.002996 27657 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:48.920770  1907 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:20:48.975992  1732 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:48.979573  1411 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:48.980054  2041 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:20:48.974802 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.004961 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:49.020370 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:49.020087 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:49.003993  2033 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:49.035116 27657 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:20:49.042256  1739 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:48.973898  2034 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:49.018512  1411 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.044463  1841 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:49.011915  1732 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.046718 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:48.986744  1927 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.027204 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.060017 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:49.032832  2033 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.076249 27663 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:49.076306 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:49.077631 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60534.3us
I20260709 12:20:49.079576 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:49.080335  2055 raft_consensus.cc:1064] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
W20260709 12:20:49.080873  2055 raft_consensus.cc:1068] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:20:49.081475  2055 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:49.082998 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.082046  2055 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.086649  2055 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.057574  2016 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:49.111454  1840 raft_consensus.cc:2764] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:20:49.115543 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.125131 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:49.127048 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.135017 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.139559 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.144227 27661 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:49.148110 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.153234 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.141520  1715 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:49.145020  1664 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:20:49.160989 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78076.6us
W20260709 12:20:49.151142  1950 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.554s	user 0.005s	sys 0.006s
I20260709 12:20:49.162490  2030 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.005s
I20260709 12:20:49.169090  2030 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:49.170022  2030 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:49.170384  2030 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:49.170651  1844 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:49.175515 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.151625  1854 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:49.150494  1953 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.184360 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.156653  1704 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:20:49.189337  1664 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:20:49.189563 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:49.156632  2028 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:49.188820 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.191936 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.194916  1411 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.195106  1715 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:49.195462  1620 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:49.197078  1912 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.202872  1620 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:49.203366 27582 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:49.157161  2033 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:49.194953  1913 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.607s	user 0.007s	sys 0.004s
I20260709 12:20:49.204463  2033 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:49.204764  1913 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.206840  1912 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.207224  1913 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:49.193795 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.216909 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:49.227690 27734 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:49.233697 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:49.234972 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.246555 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.235486  1913 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:20:49.256451 27657 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:49.239435  1953 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:49.242695  1844 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:49.242348  1841 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.267505 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.266095 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.245033  1924 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:49.268257 27738 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:49.161406  1732 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.269891 27658 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:49.249333  1704 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:49.251627  1854 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:49.239791  2028 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:20:49.286074 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:49.286944 27581 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:49.288983 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:49.266389  1739 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.269150  1924 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:49.185103  2030 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.258838  1978 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:20:49.321269  2051 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:20:49.296998  1924 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:49.298180  1732 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:49.329654 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 151008us
I20260709 12:20:49.291460  1841 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.276705  1620 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:49.348672 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:49.335022  2056 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:49.352221  1995 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:20:49.358618 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.363117 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:49.352223  2062 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:20:49.369057  2063 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:49.369731  2063 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.370194  2063 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:49.361049  2055 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 12:20:49.362495  2065 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:20:49.374017 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:49.383301  1823 raft_consensus.cc:1064] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:20:49.385041 28336 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:49.386106 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:49.389310  1709 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:49.392510  1908 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:49.392983  1908 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:49.393942  1908 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.396179  1908 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:49.408851 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.414602 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.411775  2065 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:49.416838  1844 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:49.417708  2065 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:49.418051  1912 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:49.418452  1912 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.419468  1912 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.420634  2045 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:20:49.420773  1835 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:49.421715 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.424813 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.432638 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.435727 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.421141  1912 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:49.425053  2065 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:49.442270 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:49.421156  2045 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.443089 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:49.433391  1953 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:20:49.423857  1840 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:20:49.444335  1870 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:49.444623  2056 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:49.455042 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.457672  1978 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.461391 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:49.461588 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:49.463419 27733 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:49.464916 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.465764 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.466570 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.462199 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:49.468223 27659 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:20:49.469673 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.470492 27663 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:20:49.470781 27738 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:49.472332 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:49.454672  1841 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:49.475170 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:49.467244  2033 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:49.469604  2056 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.492985  2056 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.474752  1978 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:49.483930  2061 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:20:49.494956  1978 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:49.495136  2061 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:20:49.495524  2061 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.495980  1978 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:49.496397  2061 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.497583  2061 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:20:49.498886  2060 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.412s	user 0.001s	sys 0.000s
I20260709 12:20:49.491647  2034 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.500995 27586 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
W20260709 12:20:49.513015  2052 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.256s	user 0.003s	sys 0.000s
I20260709 12:20:49.473778  2045 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:49.515965 27733 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:49.484687  2065 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:49.520212  2065 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.520612  2065 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:49.521471 27658 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:20:49.524380 27582 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:20:49.525393 27735 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:20:49.485445  1841 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.561764  2077 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 12:20:49.535033  1855 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:20:49.535691  1915 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:20:49.570586 27735 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:49.572363  1840 raft_consensus.cc:1064] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:20:49.572624  1870 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:49.573073  1870 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.547205  2063 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.573786  1854 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:20:49.576561  2066 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.256s	user 0.003s	sys 0.000s
I20260709 12:20:49.576249  1870 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:49.577680  2081 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:49.577890  1870 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:49.578732  2081 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:49.578455  1841 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:49.579128  2081 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.579998  2081 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.581084  2081 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:49.595403 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.575774  1835 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.604139  2063 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:49.606698  2053 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.523s	user 0.003s	sys 0.000s
I20260709 12:20:49.613867 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.640357  1856 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:49.642639 27736 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:20:49.645192 27734 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:49.646425 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.648823 27731 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:20:49.656764 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.669677  1869 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.678680 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.679345 27731 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:49.683365 27581 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:49.685923  1856 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:49.619429  1908 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:49.700979 27731 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:49.701803 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:49.707823 27586 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:20:49.707572 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.716025 27580 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:49.619730  1835 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.727056 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.693261  2034 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.699864  2071 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:20:49.625699  1924 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.707425  1869 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:49.751780 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44685.9us
I20260709 12:20:49.717401  1844 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:20:49.699420  1908 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:49.797739 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44851.8us
I20260709 12:20:49.743340  2030 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:49.722036  2028 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:20:49.798430 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.804430 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:49.714849  1709 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
W20260709 12:20:49.806224  2082 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
W20260709 12:20:49.811369 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:49.811636  1907 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:49.859946  1709 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.860731  1709 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.811854  2034 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:49.864008  2034 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:49.865366  2034 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.800498  1960 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:49.736562  2040 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:20:49.868204  1960 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:49.817266  2065 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.824050 27662 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:49.751152  1913 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:20:49.827448  2071 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:49.828898  2028 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:49.891749  2028 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:49.892341 27586 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:20:49.892815  2028 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.840710 27735 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:49.895042 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:49.852540  1835 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.857402  1908 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.863297  1620 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:49.813383  1924 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.736696  1927 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:20:49.805267 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:49.873859 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69000.7us
I20260709 12:20:49.903949 27588 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:49.876149 27585 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:49.889158  1979 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:49.890141 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:49.906461 27583 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:49.894131 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:49.860747  1840 raft_consensus.cc:1068] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Replica 3f90188458da46f6ba138d5f82c55b69 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 19 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:20:49.894357  2028 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:49.911034 27581 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:49.906989 27580 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:49.926096 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 174704us
W20260709 12:20:49.923928  2085 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260709 12:20:49.925978 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:49.928462  1979 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:49.933574 27735 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:20:49.936906  2065 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:49.937686  2065 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:49.939448 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:49.940052 27729 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:49.940459 27662 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:20:49.947574 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:49.952237 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:49.955771 27481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 127) took 10073 ms. Trace:
I20260709 12:20:49.960742 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:49.961679 27588 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:49.964322 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:49.978145 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84579.4us
I20260709 12:20:49.982820 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:49.942534  1840 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:20:49.982163  1913 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:50.029475  1913 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:50.030493  1913 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:49.956183 27481 rpcz_store.cc:276] 0709 12:20:39.866499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:39.872965 (+  6466us) service_pool.cc:224] Handling call
0709 12:20:49.939601 (+10066636us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:48.355501 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:48.356077 (+   576us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:48.356129 (+    52us) write_op.cc:435] Acquired schema lock
0709 12:20:48.356147 (+    18us) tablet.cc:662] Decoding operations
0709 12:20:48.486289 (+130142us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:48.486389 (+   100us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:48.486415 (+    26us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:20:48.487776 (+  1361us) tablet.cc:701] Row locks acquired
0709 12:20:48.487831 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:20:48.487987 (+   156us) write_op.cc:270] Start()
0709 12:20:48.488158 (+   171us) write_op.cc:276] Timestamp: P: 1783599648487928 usec, L: 0
0709 12:20:48.488196 (+    38us) op_driver.cc:348] REPLICATION: starting
0709 12:20:48.489386 (+  1190us) log.cc:844] Serialized 6676 byte log entry
0709 12:20:48.994195 (+504809us) op_driver.cc:464] REPLICATION: finished
0709 12:20:49.196539 (+202344us) write_op.cc:301] APPLY: starting
0709 12:20:49.196654 (+   115us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:49.377896 (+181242us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:49.378036 (+   140us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:49.391044 (+ 13008us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:49.391428 (+   384us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=34,mrs_lookups=0
0709 12:20:49.391490 (+    62us) write_op.cc:312] APPLY: finished
0709 12:20:49.391958 (+   468us) log.cc:844] Serialized 156 byte log entry
0709 12:20:49.392887 (+   929us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:49.393081 (+   194us) write_op.cc:454] Released schema lock
0709 12:20:49.393156 (+    75us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:49.393622 (+   466us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:49.393655 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2575453,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":202016,"apply.run_cpu_time_us":26051,"apply.run_wall_time_us":197058,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75187,"num_ops":17,"prepare.queue_time_us":199898,"prepare.run_cpu_time_us":29383,"prepare.run_wall_time_us":422926,"raft.queue_time_us":264812,"raft.run_cpu_time_us":82519,"raft.run_wall_time_us":184071,"replication_time_us":505674,"thread_start_us":681511,"threads_started":4,"wal-append.queue_time_us":197955}]]}
I20260709 12:20:50.012981  1950 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:20:50.012957  1869 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:49.961553  1844 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:20:50.035537  1869 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.035995  1869 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.019265  2094 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:50.000756  2095 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:20:50.024972  1979 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:20:50.026044  1960 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:50.039028 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53492.4us
I20260709 12:20:50.047399 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:50.046838 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62881us
I20260709 12:20:50.066900 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:50.068112 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 341136us
I20260709 12:20:50.061131  2040 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.077862  2042 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 12:20:50.088882 27657 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
W20260709 12:20:50.102308 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.106381 28336 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEARNER]: Advancing to term 19
I20260709 12:20:50.108142 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.111372 27737 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:20:50.068409  1927 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:20:50.059813  2094 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.125342 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.130558 27731 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:50.132009 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:50.133270 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.125913 27730 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:20:50.100996  1704 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:50.149036 27731 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:20:50.139526 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.107086  1977 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:50.145485 27733 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:20:50.156668 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:50.158643  2018 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:50.162873 27738 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:50.197249 28336 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:20:50.101413  2040 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.120085  2093 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:50.197952 28336 pending_rounds.cc:85] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 10
I20260709 12:20:50.146262  2042 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.208204 28336 pending_rounds.cc:107] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:20:50.208498 28336 raft_consensus.cc:2889] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:50.120581  1382 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:20:50.155361  1715 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:20:50.210616  2075 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:20:50.207056  2093 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.189581  1732 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:20:50.156957  1704 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.214233  1689 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:50.217850  1979 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.217182  2042 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:50.237066  2104 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:20:50.218250  1715 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:50.232124  2093 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:50.206223  1977 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:50.245616 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117378us
I20260709 12:20:50.242601  1927 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20260709 12:20:50.251925  2042 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.263581  1809 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.264194  1809 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.266384 27581 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:20:50.266496  1809 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.266873 27581 pending_rounds.cc:85] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 10
I20260709 12:20:50.267205 27581 pending_rounds.cc:107] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:20:50.267468 27581 raft_consensus.cc:2889] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:50.268648 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.272943  1977 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:50.269335 27588 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:50.237588  1704 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:50.286846 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:50.290719 27729 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:50.294625 27734 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:50.242236  2018 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:50.302807 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:50.308543  2102 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:50.309165  2102 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.240695  1689 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.320374  2094 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:50.322293  1823 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 11, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.240942  1979 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.301023  2096 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:20:50.325129  1979 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:50.328286 28331 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:20:50.328110  1715 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.329109  1715 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:50.329263 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.330121 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:20:50.330736  2099 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.117s	user 0.002s	sys 0.000s
I20260709 12:20:50.331697  2102 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.332949 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:50.347888 27663 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:50.319064  1689 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.356473  2040 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:50.352973  2096 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:50.359726  2096 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:50.360093  2096 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:50.362418  1709 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:50.363956  1869 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.365660 27730 raft_consensus.cc:2368] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22.
I20260709 12:20:50.366457  1869 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:50.372519 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.373286 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:20:50.376214 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.376976 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:50.378427  1924 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:50.378803  1924 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.379129  1924 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.300361  1927 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:50.386680 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.409124 27586 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:50.411044 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:50.411880 27585 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.412869 27588 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:20:50.415035 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.421911 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.415861 28332 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:50.407805  1927 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.425827  1927 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:50.429378 27660 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:20:50.429842 27660 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:50.432435 27487 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 15, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 15 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.438815 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.439646 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
W20260709 12:20:50.440007 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:50.444711  2110 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
I20260709 12:20:50.451031 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:50.452399 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.453207 28330 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:50.455770 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.430755 27660 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.465492 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.469439 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.470755 28332 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:50.466312 27586 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
W20260709 12:20:50.488773 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75040.9us
I20260709 12:20:50.491604 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:50.497148 27730 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:20:50.494589 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.493994 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:50.505144 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.505898 27588 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:50.501626 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:50.500829 27735 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:50.464289  2033 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:50.514153 27585 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:50.517340  2042 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:50.518905  2108 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:50.523078 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:50.529853 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:50.532297 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:50.530885 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.462790  2056 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260709 12:20:50.544340  1809 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:50.544744  1809 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.545089  1809 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.512125  1382 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.549206 27487 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 9 to 11, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 11 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:50.546783  1382 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.512977  1709 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.553589  1382 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:50.483579  2093 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.555642 27655 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:20:50.555860  2093 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.525897  1732 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:50.556879  1732 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:50.557843  1732 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.560305 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.560940 28332 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:50.561437  1854 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:50.561831  1854 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:50.562209  1854 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:50.540596  1929 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:50.562530 27487 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 14, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 14 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:20:50.567131 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44126.6us
I20260709 12:20:50.504673  2094 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.569607 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:50.581298 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.571156 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:50.582062 27738 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:50.585353 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.537889  1977 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:50.575517  1929 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.575870  1709 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.579461  1835 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:50.597290  2030 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:50.599083 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.536696  2033 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:50.613420 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43889.5us
I20260709 12:20:50.667280  2117 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:50.667876  2117 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.668258  2117 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.668365  1739 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:50.668897  1739 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:50.669267  2096 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.669792  1739 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.654189  2045 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:20:50.673091 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62899.2us
I20260709 12:20:50.674024 27627 leader_election.cc:400] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22.
I20260709 12:20:50.674319 27627 leader_election.cc:403] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:50.673128  2030 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:50.675859  2111 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.675987  1835 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:50.676505  2030 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.677006  1835 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.610409  2094 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.680213  2094 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:50.680953 27729 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:50.547036  1924 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:50.692807 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:50.699971 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.597972  2095 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:50.618757  2017 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:50.709687 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.710055  2017 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.710403 27585 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:50.710685  2017 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:50.598949  1977 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:50.711874  1977 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:50.680987  1908 raft_consensus.cc:1064] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:20:50.598114  2055 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.110s
I20260709 12:20:50.728848 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:50.730341 27582 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:50.729197 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:50.732460 27655 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:50.727831 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:50.707432  1382 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.718387 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.730060  1739 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:50.609663  1907 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:50.734998 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:50.734253 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:50.741798 27657 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:50.753032 27656 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:50.760377 27735 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:20:50.740258  1913 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.735515  2095 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.745827  1907 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.742478  1382 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.793177  2056 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:50.773546  1913 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.769899  2033 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.837862 27664 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:20:50.842736 27664 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 11
I20260709 12:20:50.882082  2111 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22.
I20260709 12:20:50.888240 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:50.888968 28331 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:20:50.889823  1620 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.890311  1620 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:50.892503  1620 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:50.843138 27664 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:20:50.897552 27664 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:50.900385 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:50.901055 27581 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:50.830448  2056 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:50.904163 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.905074 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:50.913532  1913 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:50.914084  2056 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.913954  1912 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:20:50.919139 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:50.924731 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52139.1us
I20260709 12:20:50.926614 27659 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:50.936700 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:50.938253 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:50.941462 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.941303 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:50.937333 27733 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:50.914358  1913 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:50.904027  2111 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:50.969090 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69821.1us
I20260709 12:20:50.913190  2077 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:50.975685 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:50.952895  1927 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.526s	user 0.012s	sys 0.000s
I20260709 12:20:50.929869  2063 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.916605  1696 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:20:50.987427 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:50.967314  1929 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:50.967798  1913 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.968779  2042 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.973733  1732 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.977488  1924 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:50.982649  2111 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:51.032879 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 111519us
I20260709 12:20:51.015947  1715 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:20:51.010823  1861 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:51.030802  1913 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.061160 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:51.057845 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 316875us
I20260709 12:20:50.990242  1927 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.073051  1927 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.078788 27658 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:20:51.091779  1908 raft_consensus.cc:1068] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:20:51.099459 27660 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:51.107646 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:51.109477  1927 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:51.110023  1927 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.111030  2040 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.112392  2093 raft_consensus.cc:1064] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:20:51.111805  2040 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:51.112888  2093 raft_consensus.cc:1068] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:20:51.118019  1709 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.120486  1927 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.121387  1715 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:51.122090  1715 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:51.127823  2040 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.129876 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:51.001297  2063 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:51.137194 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76154.3us
I20260709 12:20:51.143682 27660 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:20:50.991271  2077 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.148617  1907 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.034163  2042 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.036422  1732 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:51.148232 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 206991us
I20260709 12:20:51.149138  2117 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:51.152087  2132 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 12:20:51.153474  2127 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
W20260709 12:20:51.154381  2130 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260709 12:20:51.155043  2124 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:20:51.156471  1861 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:51.158859  1696 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.160665  2111 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.161289  1382 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.177048  2063 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.133450  2095 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.220793  1861 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.181069 28332 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:20:51.170732  2017 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.196483  2143 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.145828  2094 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:51.184299 27660 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 12
W20260709 12:20:51.146158 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 232602us
W20260709 12:20:51.200536  1809 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.655s	user 0.012s	sys 0.000s
W20260709 12:20:51.183209 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 161693us
W20260709 12:20:51.191887 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:51.209288 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 214909us
I20260709 12:20:51.204442 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:51.203687 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:51.196765  2045 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.521s	user 0.005s	sys 0.005s
I20260709 12:20:51.217152 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.246275 27660 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:20:51.264591 27660 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:51.265614 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:51.266207 27660 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
W20260709 12:20:51.250854 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 310178us
I20260709 12:20:51.252607 27586 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:20:51.257757 27655 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:51.270017 27583 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:51.271225 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.259382 27663 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:51.272339 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:51.273177 28332 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:51.240231 27662 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22.
I20260709 12:20:51.281275 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:51.219028  2042 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.255776 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:51.295921 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:51.329378 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:51.296346 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.329998 27664 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:51.330263 27733 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:20:51.331125 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.278712 27580 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 13.
I20260709 12:20:51.301949 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:51.294602 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.337563 27729 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:51.338948 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.328635 27658 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:20:51.341030 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.341758 27589 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:20:51.250427  1854 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.005s
I20260709 12:20:51.285519 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.342530 27733 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:51.288275 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.339943 27729 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:51.239197  2017 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.345742 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:51.351753 27589 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.223412  1869 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.247315  1732 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.242870  2094 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:51.234885  1977 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.004s
I20260709 12:20:51.354745 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:51.347362 27583 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:20:51.388113 27655 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:51.349898 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.390203 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.233412  2096 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.245915  2077 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.242659  2071 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:51.366364 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.399534 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:51.223297  1861 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.241214  2117 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.391259 27733 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:51.395617 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67077.8us
I20260709 12:20:51.334182  2045 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:51.400170 27583 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.247727  2003 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.128s
I20260709 12:20:51.248881  1913 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.384889  2094 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.349059  1832 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:51.235546  1696 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:51.427297 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 356889us
I20260709 12:20:51.380913  1869 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.429062 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:51.431728 27585 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
W20260709 12:20:51.408406 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58001.7us
I20260709 12:20:51.412453 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.248169  1809 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.423378 27582 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
W20260709 12:20:51.405495  2105 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:20:51.406545 27655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:51.413988  2030 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.239704  2143 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.436995 28336 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:51.437985 27738 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:51.401073 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:51.434067 27733 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:51.438747 27589 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:51.442731 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:51.436954 27655 raft_consensus.cc:606] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:51.446312 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45667.1us
I20260709 12:20:51.445819 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.447798 27732 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:20:51.445381 27729 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:51.359953  1854 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.402776  2140 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:20:51.456852 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:51.451272 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.399784  1977 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.405148  1863 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:20:51.453300 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:51.407414  2071 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.426280  1912 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:20:51.424656  2045 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.461751 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:51.442096  1809 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:51.456861 27583 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:20:51.607592 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.432685  1832 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.444144  2145 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:20:51.584662 27704 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.623s (SENT)
W20260709 12:20:51.584883 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 133127us
I20260709 12:20:51.608263 27583 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:51.612993 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.456890  1732 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:51.544448  2152 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:20:51.605764 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:51.610122 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:51.549496  2141 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:20:51.446375  2030 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.612737 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:51.613183 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 168181us
W20260709 12:20:51.675972  2151 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:20:51.543686  2063 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:20:51.552382 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88330.8us
W20260709 12:20:51.681988  2115 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.220s	user 0.003s	sys 0.014s
I20260709 12:20:51.605754 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:51.461647  2028 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.148s
W20260709 12:20:51.681082 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 219389us
I20260709 12:20:51.463096  2150 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:51.755808 27733 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:20:51.755741 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:51.683830  1835 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:20:51.683691  2117 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:51.769860 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:51.770731 27737 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.744580 27589 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:51.584717 27736 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:51.550966  1715 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.746574 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:51.676957  1732 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.553908  1620 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.129s
W20260709 12:20:51.785624 27586 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.516s	user 0.009s	sys 0.001s
I20260709 12:20:51.786428 27586 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:51.760768 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:51.789726 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:51.675647  2056 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.088s
I20260709 12:20:51.758494  2135 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:51.740042  2111 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:51.751048  2063 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:51.753247  2108 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:51.784190 28332 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
W20260709 12:20:51.743407 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 130719us
I20260709 12:20:51.800030  1382 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:20:51.806684 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 222036us
I20260709 12:20:51.766670  2040 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:20:51.812824 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:51.788480  1732 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.800021  1835 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:51.793720  2142 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:51.787459  2150 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:51.826824 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:51.811102  2063 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:51.811499  2140 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:20:51.811303  2108 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:51.814340  1382 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.814404  2040 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:51.837759  2163 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:51.842777  2040 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:51.836280  1913 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:51.842947  2163 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.844712  1863 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:51.845196  1863 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:51.816463  2135 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:51.846062  1863 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.835769  2140 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.848134  2140 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:20:51.850771  2149 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260709 12:20:51.852658 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:51.853241 27735 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:20:51.835793  2142 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:51.856070  2154 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:20:51.856236  1854 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.856895  1854 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:51.857358  1854 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.858119  1715 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:51.838200  2157 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:20:51.779052  2117 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:51.866390 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:51.814837  2077 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
W20260709 12:20:51.835171  2153 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:20:51.872581 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:51.814234  2111 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.876969 27737 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:51.880573 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2746) took 1313 ms. Trace:
I20260709 12:20:51.860064  2108 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:51.890487 27735 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:51.859751  1382 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:51.902653 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:51.902810 27735 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:20:51.905042 27735 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:20:51.909194 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:51.871757  1854 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:51.938988 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:51.880654  1840 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.875855  1977 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.881146 27734 rpcz_store.cc:276] 0709 12:20:50.560709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.561449 (+   740us) service_pool.cc:224] Handling call
0709 12:20:50.561774 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:50.716424 (+154650us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:50.716468 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:50.730334 (+ 13866us) log.cc:844] Serialized 26 byte log entry
0709 12:20:51.008737 (+278403us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:20:51.008800 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:51.869431 (+860631us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54001d10d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:51.869468 (+    37us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:51.869510 (+    42us) raft_consensus.cc:1685] finished
0709 12:20:51.869525 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:51.873801 (+  4276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33136,"raft.queue_time_us":30726,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":8012,"spinlock_wait_cycles":291844352,"thread_start_us":243478,"threads_started":1,"wal-append.queue_time_us":394042}
W20260709 12:20:51.947738 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 120967us
I20260709 12:20:51.952402 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:51.894465  2045 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:51.957417  1854 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.983714 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:51.959753  1844 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:51.987195 27735 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:20:51.964138  2030 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:51.953513  1840 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.844759  2150 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:51.991503 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 940) took 1090 ms. Trace:
I20260709 12:20:51.970459  1732 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:51.973533  1869 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:51.976641  2111 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.978967 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3175) took 1251 ms. Trace:
I20260709 12:20:51.985750 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:51.923718  1835 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:51.989339 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:51.936808  1689 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:52.007609 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.008545 27656 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:51.994697 27662 rpcz_store.cc:276] 0709 12:20:50.727732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.728142 (+   410us) service_pool.cc:224] Handling call
0709 12:20:51.978741 (+1250599us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:51.978884 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1173643520}
I20260709 12:20:52.009799  2135 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:51.992134 28331 rpcz_store.cc:276] 0709 12:20:50.897953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.898085 (+   132us) service_pool.cc:224] Handling call
0709 12:20:50.898368 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:50.962447 (+ 64079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:50.962478 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:50.962913 (+   435us) log.cc:844] Serialized 26 byte log entry
0709 12:20:51.346592 (+383679us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:20:51.346941 (+   349us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:51.347037 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:51.988171 (+641134us) raft_consensus.cc:1685] finished
0709 12:20:51.988193 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:51.988369 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63729,"raft.queue_time_us":71126,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":10064,"thread_start_us":345105,"threads_started":1,"wal-append.queue_time_us":481397}
I20260709 12:20:51.993832  1844 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:52.226135  1844 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.247910  1689 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.014652  2143 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.099361  1832 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:51.989650  1854 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:52.235222  2147 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:20:52.235066  2158 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.170s	user 0.001s	sys 0.001s
I20260709 12:20:52.234859  2164 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.248629  1689 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.251621  2169 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:52.298844  2172 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:20:52.312124  2167 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20260709 12:20:52.312466  2172 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.312899  2172 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:52.315097 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3092) took 1375 ms. Trace:
I20260709 12:20:52.315748 27730 rpcz_store.cc:276] 0709 12:20:50.939216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.953469 (+ 14253us) service_pool.cc:224] Handling call
0709 12:20:50.953873 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:50.954345 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:50.954391 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:50.954970 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 12:20:51.194972 (+240002us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:51.195082 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:51.195229 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:52.314807 (+1119578us) raft_consensus.cc:1685] finished
0709 12:20:52.314843 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:52.315023 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21678,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":8343,"thread_start_us":194663,"threads_started":1,"wal-append.queue_time_us":847365}
I20260709 12:20:52.317085  1854 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.318989  2108 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.315204  2171 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:20:52.324859 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 422255us
I20260709 12:20:52.303305  2111 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.328229  2171 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.342267  2171 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:52.322525  2108 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:52.348424  2174 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:20:52.351776  1689 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.337658  2142 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.300575  2164 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.361428  2117 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.396150  2148 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:52.260828 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 275158us
I20260709 12:20:52.307153  1835 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.408876 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.424031  2117 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:52.424615  2117 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:52.422365  1977 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.425604  1977 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.425719  2117 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.424671  2161 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.427968  1977 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.427799  2161 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.410679  2030 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.429208  2030 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.429733  2180 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:52.408063  2135 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.431879  2135 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.434098  2135 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:52.435467 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.067s	sys 0.105s
W20260709 12:20:52.435972 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.067s	sys 0.105s
I20260709 12:20:52.358405  2111 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:52.445925 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.462832  2161 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.458277  2170 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.463904 27701 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22.
I20260709 12:20:52.464313 27701 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:52.445358  1732 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.422991  2148 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.458817  2030 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:52.483604 27659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3176) took 1755 ms (client timeout 1893 ms). Trace:
I20260709 12:20:52.466121  1840 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:52.483990 27659 rpcz_store.cc:269] 0709 12:20:50.728083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.728932 (+   849us) service_pool.cc:224] Handling call
0709 12:20:52.483444 (+1754512us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b5400229054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:52.483550 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":389692160}
I20260709 12:20:52.477579  2182 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:20:52.490288  2164 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.475390  2161 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22.
W20260709 12:20:52.482652  2040 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.615s	user 0.008s	sys 0.003s
I20260709 12:20:52.481464  2142 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:52.491978  2152 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.750s	user 0.011s	sys 0.001s
W20260709 12:20:52.492931  2173 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
I20260709 12:20:52.465623  2170 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:20:52.504549 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.491627  1854 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.517944  2182 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.524039  2182 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.507597  2152 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:20:52.497200  2040 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.505494  1732 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:20:52.532506 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3282) took 1292 ms. Trace:
I20260709 12:20:52.539130  2148 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.540100  1696 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.506839  2142 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.543205  2142 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.513824  1854 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.544858 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.545490 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:20:52.545754  1854 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.546464 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.547260 27662 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:52.547624 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.547768  2092 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.548130 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.548218 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:52.548271  2092 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.548859 27662 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:52.549239 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.549728 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.550282 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.551081 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.551819 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
W20260709 12:20:52.553310  2094 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.120s	user 0.012s	sys 0.000s
I20260709 12:20:52.554216 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.553875  2094 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.555694 27731 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18.
I20260709 12:20:52.533372 27584 rpcz_store.cc:276] 0709 12:20:51.239896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.796339 (+556443us) service_pool.cc:224] Handling call
0709 12:20:51.818615 (+ 22276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:51.818997 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:51.819032 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:51.827845 (+  8813us) log.cc:844] Serialized 26 byte log entry
0709 12:20:51.942356 (+114511us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:20:51.942401 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:51.942492 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:52.532201 (+589709us) raft_consensus.cc:1685] finished
0709 12:20:52.532230 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:52.532421 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":26113,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"thread_start_us":72895,"threads_started":1,"wal-append.queue_time_us":266856}
I20260709 12:20:52.557528 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3097) took 1263 ms. Trace:
I20260709 12:20:52.557607 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.557911 27731 rpcz_store.cc:276] 0709 12:20:51.293961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.301658 (+  7697us) service_pool.cc:224] Handling call
0709 12:20:52.557313 (+1255655us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:52.557447 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2802483968}
I20260709 12:20:52.549804 27659 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
W20260709 12:20:52.561189  2140 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.712s	user 0.004s	sys 0.007s
I20260709 12:20:52.561690  2140 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.562084  2117 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.562592 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3099) took 1268 ms. Trace:
I20260709 12:20:52.564083  2182 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.564353 27738 rpcz_store.cc:276] 0709 12:20:51.294539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.295647 (+  1108us) service_pool.cc:224] Handling call
0709 12:20:52.562426 (+1266779us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:52.562539 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248673536}
I20260709 12:20:52.565876 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.566201 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:52.565740 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.566828 27659 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:52.567631 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.568533  1854 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:52.568956  1854 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.569304  1854 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:52.575522 27659 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:52.575634  1696 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:52.577759  1696 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.578231  1696 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:52.578701  1854 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.582005  1854 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:52.590194  2063 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.751s	user 0.011s	sys 0.001s
I20260709 12:20:52.591231  2063 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.562126 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3271) took 1590 ms. Trace:
I20260709 12:20:52.562634  2117 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:52.528337  1382 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.606s	user 0.010s	sys 0.001s
I20260709 12:20:52.603878  2094 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.604929 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3279) took 1586 ms. Trace:
I20260709 12:20:52.642639 27581 rpcz_store.cc:276] 0709 12:20:51.018813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.019613 (+   800us) service_pool.cc:224] Handling call
0709 12:20:52.604084 (+1584471us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:20:52.604339 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:52.604576 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:52.604605 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:52.604649 (+    44us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:20:52.604684 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:52.604740 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:52.604868 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28082560}
I20260709 12:20:52.605782 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2755) took 1245 ms. Trace:
I20260709 12:20:52.606564 27660 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:20:52.607612  2111 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.615301 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:52.644255 27733 rpcz_store.cc:276] 0709 12:20:51.360277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.451035 (+ 90758us) service_pool.cc:224] Handling call
0709 12:20:52.605622 (+1154587us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54002e86d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:52.605728 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682036096}
I20260709 12:20:52.648590 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2748) took 2082 ms. Trace:
I20260709 12:20:52.616611  2175 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:52.655460  2164 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.655905  1854 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.655984  2164 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.656495  1854 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.658109 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:52.658209  2164 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.658383 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54366 (request call id 22) took 1262 ms. Trace:
I20260709 12:20:52.658865 27583 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:52.659545 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:52.659785 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:52.660120 27589 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:20:52.660516 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:52.640532  1928 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.662109  1928 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:52.662824  2137 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.420s	user 0.010s	sys 0.004s
I20260709 12:20:52.663554  2142 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.664283  1928 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.628747 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3268) took 1700 ms. Trace:
I20260709 12:20:52.664728 27585 rpcz_store.cc:276] 0709 12:20:50.928706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.928849 (+   143us) service_pool.cc:224] Handling call
0709 12:20:50.929194 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:51.431222 (+502028us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b54000d19a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:20:51.432202 (+   980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:52.628458 (+1196256us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54000d19a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34ba02 00007f0e9a347dd6 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:52.628685 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272261120}
I20260709 12:20:52.658620  1854 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.665794  2184 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.666220 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:52.666394  2184 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.666829 27585 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:52.667703 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:52.575809 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.639931 27582 rpcz_store.cc:276] 0709 12:20:50.971545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.971707 (+   162us) service_pool.cc:224] Handling call
0709 12:20:50.986773 (+ 15066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:51.434805 (+448032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:52.561888 (+1127083us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:52.562073 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1003284736}
I20260709 12:20:52.668247 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:52.648978 27736 rpcz_store.cc:276] 0709 12:20:50.565769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.566824 (+  1055us) service_pool.cc:224] Handling call
0709 12:20:50.567084 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:51.771783 (+1204699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:52.648371 (+876588us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:52.648534 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270025600}
I20260709 12:20:52.653776 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:52.654062  2117 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.658751 27655 rpcz_store.cc:276] 0709 12:20:51.405467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.406351 (+   884us) service_pool.cc:224] Handling call
0709 12:20:52.658187 (+1251836us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b4c003e2fc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98614a20 00007f0e98625a29 00007f0e985eb644 00007f0e98612f2d 00007f0e98612b93 00007f0e9a342c92 00007f0e9a34284a 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691
0709 12:20:52.658288 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7934336}
I20260709 12:20:52.654425  2180 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:52.642247  1840 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.664050  2142 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.617532  2170 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:52.635403 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 137060us
I20260709 12:20:52.668320 27585 raft_consensus.cc:2368] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:20:52.688659 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:52.692705 27733 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:20:52.693203 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:52.693857 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:52.698698 27736 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:52.701706  2184 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.705920 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:52.706147 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 130414us
I20260709 12:20:52.709136 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:52.709830 27737 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:52.707571 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:52.700748  1995 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.066s	user 0.003s	sys 0.000s
W20260709 12:20:52.720208 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65166.1us
I20260709 12:20:52.721333 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:52.669708  1960 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.719702  2192 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.658810  2164 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.726552 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.717517  2142 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.702386  2185 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.713626  2180 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:52.711128  1854 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.669255  1382 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.718451  2140 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.717746  2161 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:52.721447  2094 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.724162  2135 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:52.720245  2152 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.778447  1854 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.717598  2143 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:20:52.707796  1840 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.745386  2184 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:52.760937  2117 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:52.791402 27664 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:20:52.792932 27655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:52.793717 27655 raft_consensus.cc:606] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:52.794054 27655 raft_consensus.cc:614] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:52.771993  2150 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:52.776705  2161 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.779448  2100 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:52.781013  2135 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.743875  2172 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.748158  2192 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.799633 27584 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:52.742132  1960 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:52.797564  2193 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:20:52.761853  2164 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.782305  2094 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.796023  2030 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:52.798362  2161 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:52.801312  2135 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:52.817600 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 149713us
I20260709 12:20:52.821314  1960 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.816121  2030 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.822610  2030 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:52.823737  1854 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.824417  1840 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.826475  2164 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.827468  2063 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.784036  2152 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.829982  1995 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=10 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:20:52.830273  2152 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.774520  2185 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.830999  1995 tablet_bootstrap.cc:492] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:20:52.833204  2185 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.834482  2192 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.795531  1696 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.835539  2094 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.807245  2179 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:52.837405  1696 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.837157  2179 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.800041  2150 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.841351 27655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:52.797189  2171 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.841982 27655 raft_consensus.cc:606] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:52.797044  2117 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:52.847888 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 121443us
I20260709 12:20:52.794685  2184 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.849073 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.851583 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.851894 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.852173 27659 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:20:52.852505 27662 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:52.853156 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:52.852240 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:52.853698 27659 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:20:52.853754 27655 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:20:52.853907 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.855616 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.862015 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:52.863515  2179 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.863612 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.802268  2180 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.864880  2150 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.865114  2185 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:20:52.865487  2185 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.865880  2185 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:20:52.863636  2172 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.871806  2171 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.855190  2094 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:52.863114  1995 ts_tablet_manager.cc:1403] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 3.511s	user 0.058s	sys 0.035s
I20260709 12:20:52.872586  2071 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.874917 27655 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:52.886680  1854 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.887401  2199 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:52.888082  2134 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.643s	user 0.002s	sys 0.014s
I20260709 12:20:52.887993  2199 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.889793 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1152) took 2199 ms. Trace:
I20260709 12:20:52.890905  2184 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.890143 27661 rpcz_store.cc:276] 0709 12:20:50.690698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.690836 (+   138us) service_pool.cc:224] Handling call
0709 12:20:50.691196 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:50.691579 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:50.691606 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:50.707714 (+ 16108us) log.cc:844] Serialized 26 byte log entry
0709 12:20:51.041314 (+333600us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:20:51.138299 (+ 96985us) log.cc:844] Serialized 14 byte log entry
0709 12:20:51.147017 (+  8718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:51.147122 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:52.887367 (+1740245us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:52.889496 (+  2129us) raft_consensus.cc:1685] finished
0709 12:20:52.889590 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:52.889718 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":97009,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1908,"raft.queue_time_us":24997,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":11825,"spinlock_wait_cycles":38252928,"thread_start_us":301732,"threads_started":1,"wal-append.queue_time_us":516324}
I20260709 12:20:52.892738  2135 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.893131  2117 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.885149  1696 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:52.893975  1696 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.894373  1696 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:52.896584  2199 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.885876 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:52.875586  1382 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.898283  2135 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.911892  1908 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:52.912539  1908 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.887156  1854 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.916178  1854 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:52.916550  2163 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:52.886111  2094 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:52.922408  2094 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:52.915239  1908 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:52.923215  1704 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:52.918445  2030 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:52.962196 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 122227us
I20260709 12:20:52.967034 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:52.967010 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:52.974731  1908 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:52.975409  1908 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:52.975967  1908 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:52.967392  2198 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:20:52.958428  1704 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:52.992782 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2759) took 1631 ms (client timeout 1555 ms). Trace:
W20260709 12:20:52.993269 27729 rpcz_store.cc:269] 0709 12:20:51.361392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.366045 (+  4653us) service_pool.cc:224] Handling call
0709 12:20:52.992524 (+1626479us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54000a64d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:52.992665 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105477888}
I20260709 12:20:52.993619  1704 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:52.966655  2028 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.164s	user 0.014s	sys 0.016s
I20260709 12:20:52.973524 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:53.001519 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.004007 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:53.004837 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:53.005079 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3277) took 1986 ms (client timeout 1693 ms). Trace:
W20260709 12:20:53.005766 27588 rpcz_store.cc:269] 0709 12:20:51.018210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.018363 (+   153us) service_pool.cc:224] Handling call
0709 12:20:52.966821 (+1948458us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:20:53.004947 (+ 38126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31353600}
I20260709 12:20:53.006304 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.007566  2109 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:53.007843 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.008186  2109 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.009086 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.010426  2109 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.011919  2180 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:53.012032  2199 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.012498  2180 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.012880  1835 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.012724  2199 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.013489  1835 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:52.998139 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:53.016991 27630 leader_election.cc:400] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:20:53.022501 27630 leader_election.cc:403] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:20:53.024354 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:53.025170 27663 raft_consensus.cc:606] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:53.029065 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:52.982492  1995 raft_consensus.cc:359] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:53.029187 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.029908 27659 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:53.031840 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.052371 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.033185 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.034003 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.052920 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.079124 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.079895 27586 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:53.080670 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.080987 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.028352  2094 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:53.083055 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.083675 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.083810 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.083277 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.083801 27655 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 12:20:53.087045 27655 pending_rounds.cc:85] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 13
I20260709 12:20:53.086915 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.087540 27655 pending_rounds.cc:107] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:20:53.087874 27588 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
W20260709 12:20:53.023561  2028 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.227s	user 0.014s	sys 0.016s
I20260709 12:20:53.093165  1908 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:53.096560  1696 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.097065  1908 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.087872 27655 raft_consensus.cc:2889] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:53.089154  2094 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:53.098583  2094 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.099604 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.102586 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.103701 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.104336 27655 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:53.105468 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.106736 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:53.108151  1696 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.111864 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.015043  2199 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.092474  1835 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.021101  2185 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.017098  1382 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:53.114974  1382 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.115569  1382 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:53.116542  2071 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.117105  2071 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.118548 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.093107  1995 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:20:53.119211 27582 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:53.119678  1995 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:53.120368  2030 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.131223 28334 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:20:53.131280 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:53.136441 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:53.136804 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:53.137121 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93513.2us
I20260709 12:20:53.137400 27588 raft_consensus.cc:2376] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17. Re-sending same reply.
I20260709 12:20:53.015107  2180 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.138944 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:53.140036  1696 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:53.140556  1696 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:53.141007  1696 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:53.148705  2071 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.149682  2161 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.152047  2161 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:53.154332  1696 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.159132  2185 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.032799  2042 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.160133  2042 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.162381  2042 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.034252  1411 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:53.131048  2030 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.164258  1696 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:53.168164  2030 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.171146  2030 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:53.173368  1863 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:53.194118  1995 ts_tablet_manager.cc:1434] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.318s	user 0.010s	sys 0.006s
W20260709 12:20:53.174455 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3096) took 1886 ms (client timeout 1995 ms). Trace:
I20260709 12:20:53.132467  2199 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:53.185146 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.185845 27582 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:53.186895 27659 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:20:53.188169 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1205) took 2269 ms. Trace:
W20260709 12:20:53.192186 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60985us
I20260709 12:20:53.192814 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:53.193315 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54435.9us
I20260709 12:20:53.199406 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.200261 27661 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:53.200768 27582 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:53.201292 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3181) took 1218 ms. Trace:
I20260709 12:20:53.201426 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.201738 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:53.202057 27661 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:20:53.202474 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.203209 27588 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:53.203084 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.203769 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.204133 27581 raft_consensus.cc:2368] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18.
I20260709 12:20:53.204355 27582 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:53.204983 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.211728 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.212249 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:53.212679 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.213969 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.214919 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.215354 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.216557 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:53.217026 28336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1069) took 2110 ms (client timeout 1697 ms). Trace:
I20260709 12:20:53.217391 27664 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:20:53.195577 27732 rpcz_store.cc:269] 0709 12:20:51.287644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.287988 (+   344us) service_pool.cc:224] Handling call
0709 12:20:53.174236 (+1886248us) spinlock_profiling.cc:243] Waited 99.2 ms on lock 0x7b54001d3dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.174380 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228221824}
I20260709 12:20:53.218840 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.201632 27657 rpcz_store.cc:276] 0709 12:20:51.983120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.983263 (+   143us) service_pool.cc:224] Handling call
0709 12:20:53.201126 (+1217863us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.201239 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1621543808}
I20260709 12:20:53.220204 27657 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:20:53.220570 27659 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:53.221482 27658 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:53.220875 27657 raft_consensus.cc:606] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:53.222385 27657 raft_consensus.cc:614] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:53.223971 27659 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:20:53.198292 27587 rpcz_store.cc:276] 0709 12:20:50.918556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.920396 (+  1840us) service_pool.cc:224] Handling call
0709 12:20:50.966977 (+ 46581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:51.207858 (+240881us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:51.207903 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:51.207956 (+    53us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:20:51.208002 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:53.187812 (+1979810us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54001fe5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:53.187874 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:53.188057 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553455872}
I20260709 12:20:53.224308 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.225796  1382 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:53.217409 28336 rpcz_store.cc:269] 0709 12:20:51.106883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.107375 (+   492us) service_pool.cc:224] Handling call
0709 12:20:53.216810 (+2109435us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b5400611054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.216926 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":755387136}
I20260709 12:20:53.194105  2117 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:53.229017 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92624us
I20260709 12:20:53.239566 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.240437 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.240773 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.242967 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.229116 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.236071 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.241071 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.244915  2042 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:53.217875  2199 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.245808  2199 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:53.246625  2005 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:53.247159  2005 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.227205  1869 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:53.249178  1869 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:53.249281  2005 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.237196  2180 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:20:53.282768  2161 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.283461  2161 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.249626  1869 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:53.285765 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.286939 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.259567  1382 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.287866 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.288038 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.240077  2178 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 12:20:53.289002 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.289714 27586 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:53.290563 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.291126 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.291834 27588 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:53.293017 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.293740 27730 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:53.293979 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.179124  1411 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:20:53.294992 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.259987 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.296711 28332 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:53.240363  1704 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:20:53.264420 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.267149 28331 raft_consensus.cc:1217] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Deduplicated request from leader. Original: 14.10->[16.11-16.11]   Dedup: 16.11->[]
I20260709 12:20:53.268230 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.271809 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.234316  1696 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:53.280411 27658 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:20:53.295754 27732 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:53.300226 27733 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
W20260709 12:20:53.301559 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101175us
I20260709 12:20:53.306767 28335 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:20:53.319396 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:53.339195 27737 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:20:53.339154 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.340962 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.341536 27658 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.342715 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.391559 27733 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:53.445247 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.445540 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.344897 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.343844  2202 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.388109 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.352672  2003 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:53.448181 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:53.351971 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.390789 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.444728 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.450325 27735 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:53.344221 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.450464 27581 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:53.344663 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.443557 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.345139 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.443151 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.346135 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.455523 27737 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:20:53.307804  2180 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.447535 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.456940 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.449921 27730 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:53.489871 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.457072 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.490991 27656 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:20:53.454154 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 144169us
I20260709 12:20:53.457937 27658 raft_consensus.cc:2368] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23.
I20260709 12:20:53.492830 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:53.457928 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.493239 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:53.493917 27733 raft_consensus.cc:2368] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:20:53.459175 27587 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:53.491955 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:53.448316  2003 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.499970 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.500906 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.503870 27734 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:20:53.505107 27737 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:53.452262  1912 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.378320  2178 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:53.523747 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
W20260709 12:20:53.510067 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3262) took 3552 ms (client timeout 1623 ms). Trace:
I20260709 12:20:53.516525 27664 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:53.530444 27735 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:20:53.506734 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.531903 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.529942 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.531293 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.532898 27729 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.533115 27664 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:53.451920  1704 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:53.455638  2071 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:53.529614 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:53.530584 27580 rpcz_store.cc:269] 0709 12:20:49.957900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:49.963947 (+  6047us) service_pool.cc:224] Handling call
0709 12:20:53.509838 (+3545891us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.509955 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3020398336}
I20260709 12:20:53.535940 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.494441  2042 raft_consensus.cc:2764] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:20:53.537654 27734 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.451706  1696 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.455905  2176 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:53.539268 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.538074 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.459884  2135 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:53.540984 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.541290 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.500821  2180 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:53.501813  2161 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.499059  2217 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:53.446700  2202 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.567629 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.568521  2202 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.568967 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.533601  2178 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:53.546994 27737 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.578605 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.579480 27734 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.579054 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.585600 27737 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.587824 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.540438 27587 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:53.548301 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.555528  2199 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.557262 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3186) took 1011 ms. Trace:
I20260709 12:20:53.558099 27589 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:20:53.560748  2135 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:53.562181  2170 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.598927  2170 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.592482 27660 rpcz_store.cc:276] 0709 12:20:52.545332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:52.553945 (+  8613us) service_pool.cc:224] Handling call
0709 12:20:53.557048 (+1003103us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.557196 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118118656}
I20260709 12:20:53.599907 27583 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:53.601082  2170 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.601519 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.602567 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.602902 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.538089  1912 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.604534 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:53.562294 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:53.605259  1912 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.563720  1620 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:53.606124 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:53.605962  1620 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:53.605261 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.608392 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.609015 27729 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:20:53.609673 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.605203 27734 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:20:53.601095 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.614642 27583 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:20:53.572901 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.576967 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.617383 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:53.617705 27664 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.577292 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:53.577193 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.623961 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.547905 27735 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.544457 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.626083 27580 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:53.641521 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.647666 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.678735 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.542379  1704 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.682248 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.683410 27737 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.543650  2071 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.581892  2176 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.583511  1696 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.607522  1620 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.686851  2178 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:53.693167  2178 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:53.697121 27729 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:53.699656 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.699590  2180 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.702175  2180 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.702409  2199 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.733384  1696 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.735883  2176 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.739552 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:53.640062  1869 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:53.741688 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3292) took 1087 ms. Trace:
I20260709 12:20:53.742087 27584 rpcz_store.cc:276] 0709 12:20:52.653979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:52.660274 (+  6295us) service_pool.cc:224] Handling call
0709 12:20:53.741487 (+1081213us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.741625 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":317989760}
I20260709 12:20:53.743168 27731 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:53.744364  2003 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.748310 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.750643 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.752667 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.753237  2071 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.755053  1704 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.756173  1869 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:53.757098  2092 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.757944 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.759068 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:53.761484  2169 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:53.761996  2169 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.762378 27736 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:53.764071 27737 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:20:53.772217 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:53.773310 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:53.773698 27580 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:53.755003 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:53.779414 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.782909 27582 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:53.784610 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:53.785198 27587 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:53.786721  2169 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:53.788160 27627 leader_election.cc:400] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:20:53.788527 27627 leader_election.cc:403] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:53.790185 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:53.790354 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.790897 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
W20260709 12:20:53.795996 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 162559us
I20260709 12:20:53.796223 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.798245 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.798683 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:53.782490  2177 raft_consensus.cc:2764] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:20:53.780360  1912 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.800971 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:53.787755  2096 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:53.802996 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:53.803913 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:53.805553 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 486208us
I20260709 12:20:53.832978 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.834870 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.844053 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.845444 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:53.799546  2092 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:53.790108  1907 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:20:53.848770  2142 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:53.849180  2142 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:53.850028  2142 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:53.837934 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:53.858946 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55133.7us
I20260709 12:20:53.859957 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:53.861212 27738 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.022s	sys 0.041s
I20260709 12:20:53.853829 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:53.862083 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 409542us
W20260709 12:20:53.862124 27738 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.022s	sys 0.041s
I20260709 12:20:53.864578 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:53.866907 27580 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:20:53.874116 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.876227 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:53.836292  2040 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:53.881397 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3300) took 1215 ms. Trace:
W20260709 12:20:53.881676 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 369154us
I20260709 12:20:53.891655 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:54.004361 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112790us
I20260709 12:20:54.004319  1382 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:54.005668 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:54.006561  1382 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:20:54.007045 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:53.897879 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:53.995297 27659 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:53.851293  2142 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:53.839036  1912 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:53.996831 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 143058us
I20260709 12:20:53.900305  2071 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:53.873281 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:53.971194 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.008s	sys 0.029s
I20260709 12:20:53.857887  2169 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:54.014576 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:54.374992 27737 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.020s	sys 0.042s
W20260709 12:20:54.375013 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 501796us
I20260709 12:20:54.020447 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:54.376662 27588 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:53.890545  2177 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:53.857295  2096 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:53.903018 27589 rpcz_store.cc:276] 0709 12:20:52.666001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:52.668049 (+  2048us) service_pool.cc:224] Handling call
0709 12:20:53.881138 (+1213089us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:53.881317 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2044367232}
I20260709 12:20:54.023097 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:54.159047  2135 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:54.403198 27586 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:20:53.997793 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:54.404265 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:54.120292 27585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.019s	sys 0.058s
I20260709 12:20:54.148636 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:54.405372 27585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.020s	sys 0.058s
W20260709 12:20:54.148715 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 115130us
I20260709 12:20:54.151289 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:54.142284 27663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.023s	sys 0.027s
W20260709 12:20:54.406917 27663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.023s	sys 0.028s
I20260709 12:20:54.407341 27733 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 17: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:54.027601 27582 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:20:54.154086 27658 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:54.157945 27584 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:20:54.160274 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:53.900568  1863 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.224s
I20260709 12:20:53.997459  1689 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:20:54.023288  2030 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:20:54.011458 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:54.374346 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-a6d40162b2bb4b61adc69c9fa3f03326 received from {username='slave'} at 127.0.0.1:47076
W20260709 12:20:54.417438 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 405991us
I20260709 12:20:54.417907 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:54.418305 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:54.418344  1689 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:54.417449 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-a6d40162b2bb4b61adc69c9fa3f03326 on tablet a6d40162b2bb4b61adc69c9fa3f03326 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:54.419250  1859 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:54.148054  1907 raft_consensus.cc:1064] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:20:54.026250  2040 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:20:54.321461 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.029s	sys 0.038s
I20260709 12:20:54.366997  1913 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:20:54.339623 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:54.367064 27729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.022s	sys 0.038s
W20260709 12:20:54.369798 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.044s	user 0.010s	sys 0.029s
W20260709 12:20:54.375765 27737 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.021s	sys 0.042s
I20260709 12:20:54.377653 27552 leader_election.cc:400] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23.
I20260709 12:20:54.015726 27587 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:54.378232  1382 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:20:54.407979 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.413491  1696 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.415479  2175 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:54.492614 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84736us
I20260709 12:20:54.493832 27581 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:54.416051  2096 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:54.421720 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:54.418671 27589 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:54.408228  2003 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:54.428273 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:54.432984 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.030s	sys 0.038s
W20260709 12:20:54.281002 28329 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.010s	sys 0.046s
W20260709 12:20:54.501153 28329 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.010s	sys 0.047s
W20260709 12:20:54.500943 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79311.4us
W20260709 12:20:54.439052 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.037s	sys 0.051s
W20260709 12:20:54.502343 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.037s	sys 0.051s
I20260709 12:20:54.502096 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:54.503307 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2775) took 1200 ms. Trace:
I20260709 12:20:54.503861 27738 rpcz_store.cc:276] 0709 12:20:53.302893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.303133 (+   240us) service_pool.cc:224] Handling call
0709 12:20:53.303529 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.303761 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:20:53.864097 (+560336us) log.cc:844] Serialized 14 byte log entry
0709 12:20:54.502791 (+638694us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:54.502854 (+    63us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:54.502905 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.503097 (+   192us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.503235 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1144145,"threads_started":2}
I20260709 12:20:54.479364 27552 leader_election.cc:403] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
W20260709 12:20:54.479725 27729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.022s	sys 0.042s
I20260709 12:20:54.419202  1689 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:54.494011  1859 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:54.511619  1689 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:54.511952  1859 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.493906 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:54.515396 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:54.432471  2030 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.321319  2229 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.284s
I20260709 12:20:54.517843  1382 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:54.494259  2109 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:54.520800 27583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.000s	sys 0.033s
I20260709 12:20:54.532358 27586 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:54.542101 27587 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:54.519255 27589 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
W20260709 12:20:54.519587 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 142060us
I20260709 12:20:54.514573 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3113) took 1014 ms. Trace:
I20260709 12:20:54.544785 27730 rpcz_store.cc:276] 0709 12:20:53.500769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.500983 (+   214us) service_pool.cc:224] Handling call
0709 12:20:53.501297 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.506285 (+  4988us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.506313 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:53.532323 (+ 26010us) log.cc:844] Serialized 26 byte log entry
0709 12:20:53.837980 (+305657us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:53.838016 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:53.838090 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:54.514333 (+676243us) raft_consensus.cc:1685] finished
0709 12:20:54.514353 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.514502 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2817,"raft.queue_time_us":2928,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":267405,"threads_started":1,"wal-append.queue_time_us":868412}
I20260709 12:20:54.407497  2169 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.560348 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:54.520272  1704 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:54.533577  2071 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:20:54.563848 28330 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.011s	sys 0.027s
W20260709 12:20:54.587790 28330 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.011s	sys 0.028s
I20260709 12:20:54.513396  1907 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 10, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:54.558043 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:54.520002  2040 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:54.477396  2135 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:20:54.533159  1709 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:54.544975  2185 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:20:54.594640 27734 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.737s	user 0.012s	sys 0.000s
I20260709 12:20:54.580915 27589 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:54.633679 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:20:54.614265 27588 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:54.558408  1859 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:54.619563 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:54.374593  2042 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:54.499855  1696 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:54.583377  2202 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:20:54.661535 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103590us
I20260709 12:20:54.593037  1913 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:54.553409  2109 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.549366  2030 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:54.685799 28331 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.038s	sys 0.051s
W20260709 12:20:54.686303 28331 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.038s	sys 0.051s
I20260709 12:20:54.687294 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 996) took 1421 ms. Trace:
I20260709 12:20:54.693778 27581 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:54.692353 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:54.695255 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.687827 28331 rpcz_store.cc:276] 0709 12:20:53.265948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.266554 (+   606us) service_pool.cc:224] Handling call
0709 12:20:53.266959 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.305832 (+ 38873us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:53.789086 (+483254us) log.cc:844] Serialized 14 byte log entry
0709 12:20:54.686871 (+897785us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:54.686946 (+    75us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:54.687002 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.687082 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.687232 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1145085,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":6663,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":12983,"thread_start_us":1380202,"threads_started":2}
I20260709 12:20:54.670065  2229 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:20:54.725888 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:54.728726  2232 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:20:54.698385  1863 raft_consensus.cc:2764] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:20:54.712917  2229 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:54.744590 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 316374us
I20260709 12:20:54.686486  2042 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:54.748569 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:54.759711 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3138) took 1004 ms. Trace:
I20260709 12:20:54.759977 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:54.763902 27737 rpcz_store.cc:276] 0709 12:20:53.755095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.764764 (+  9669us) service_pool.cc:224] Handling call
0709 12:20:53.765051 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.765249 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:20:54.478024 (+712775us) log.cc:844] Serialized 14 byte log entry
0709 12:20:54.759256 (+281232us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:54.759349 (+    93us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:54.759408 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.759500 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.759635 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":964669,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":702,"thread_start_us":992295,"threads_started":2}
I20260709 12:20:54.744920  2229 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:54.767074 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:54.745682  1863 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:54.787611 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:54.786612 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:54.798780 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2665) took 1690 ms. Trace:
I20260709 12:20:54.799381 27657 rpcz_store.cc:276] 0709 12:20:53.097243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.281694 (+184451us) service_pool.cc:224] Handling call
0709 12:20:53.282118 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.310929 (+ 28811us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.310973 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:53.740361 (+429388us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:20:53.740747 (+   386us) log.cc:844] Serialized 26 byte log entry
0709 12:20:54.502020 (+761273us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:20:54.502067 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.502144 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:54.768991 (+266847us) raft_consensus.cc:1685] finished
0709 12:20:54.769016 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.787838 (+ 18822us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28397,"raft.queue_time_us":45288,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":13781,"spinlock_wait_cycles":55961984,"thread_start_us":760967,"threads_started":1,"wal-append.queue_time_us":841116}
I20260709 12:20:54.783337  1859 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:54.819249 27587 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:20:54.828197 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.033s	user 0.008s	sys 0.007s
W20260709 12:20:54.828676 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.009s	sys 0.007s
I20260709 12:20:54.829939  1704 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:54.830530  1704 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:54.831460  1704 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.813030  2096 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:54.835142  2096 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.835867  1709 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:54.836602  1709 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.837251  2096 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:54.838615  1859 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.839152  1709 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:54.840085  1709 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:54.841100  1859 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:54.840755  1709 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.845108  1709 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:54.835843 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 320520us
I20260709 12:20:54.850265 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.786113  1869 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:54.855518  1709 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:54.856343  1869 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:54.857498  1869 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:54.860193  1696 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:54.861027 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.864928 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:54.789929  2040 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:54.813700  2202 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:54.869401 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.864364  1912 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:54.867919  2169 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:54.889807 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.875582  1696 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:54.891309  2109 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:54.900703  2175 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:54.902889  2042 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:54.903702 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:54.872082  1913 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.877547  2226 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
I20260709 12:20:54.892632  1709 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:20:54.890988  2202 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:54.880601  2040 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:54.906917 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:54.882970  2216 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.021s
I20260709 12:20:54.908783  1913 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:54.906100  2248 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:54.906989  2165 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:54.914041 28292 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.029s	user 0.062s	sys 0.115s
W20260709 12:20:54.915176 28292 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.031s	user 0.062s	sys 0.115s
W20260709 12:20:54.917452 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:54.917923 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:54.926095 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.942835 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:54.912830  2030 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:54.915563 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.03133e+06us
I20260709 12:20:54.946991 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:54.965766  2250 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:20:54.966926  2230 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:20:54.941120  2165 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:54.967540  2202 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:54.991961 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45061.4us
I20260709 12:20:54.994437 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:54.945338  2246 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:54.955367  2248 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:20:54.976574  2177 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.000s
I20260709 12:20:54.913887  2016 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:20:54.992151 28332 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEARNER]: Advancing to term 17
I20260709 12:20:54.914381  1689 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:54.973136  2040 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:54.999872  2248 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:55.005221  2005 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:55.009686  2005 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:55.008014  2177 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:54.983255  1696 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:55.011565 27656 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:55.010762  2005 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.005452  2030 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:55.007102 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2657) took 1975 ms. Trace:
W20260709 12:20:54.987821  2234 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.249s	user 0.003s	sys 0.000s
I20260709 12:20:55.004908 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.012612 27663 rpcz_store.cc:276] 0709 12:20:53.031921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.077445 (+ 45524us) service_pool.cc:224] Handling call
0709 12:20:53.081565 (+  4120us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.081992 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.082604 (+   612us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:53.083044 (+   440us) log.cc:844] Serialized 26 byte log entry
0709 12:20:53.638941 (+555897us) spinlock_profiling.cc:243] Waited 2.91 ms on lock 0x7b1800030fa0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:54.407767 (+768826us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:54.407818 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.407892 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:55.006709 (+598817us) raft_consensus.cc:1685] finished
0709 12:20:55.006738 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.007023 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":448,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":2729,"spinlock_wait_cycles":6721024,"thread_start_us":768182,"threads_started":1,"wal-append.queue_time_us":1679195}
I20260709 12:20:55.024400 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:55.026099  2030 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:55.026965 28332 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:20:55.021677 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 303630us
I20260709 12:20:55.003477  1709 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:20:55.023521  2005 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:55.014083  2248 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:55.030210 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2785) took 1517 ms. Trace:
I20260709 12:20:55.035035 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.033859 27729 rpcz_store.cc:276] 0709 12:20:53.512909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.698018 (+185109us) service_pool.cc:224] Handling call
0709 12:20:53.698498 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.698814 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.698839 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:53.769934 (+ 71095us) log.cc:844] Serialized 556 byte log entry
0709 12:20:54.505824 (+735890us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:54.505897 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.505997 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:54.982934 (+476937us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:55.026810 (+ 43876us) raft_consensus.cc:1685] finished
0709 12:20:55.026832 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.030142 (+  3310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4284288,"thread_start_us":735577,"threads_started":1,"wal-append.queue_time_us":936126}
W20260709 12:20:55.029502  2142 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.905s	user 0.012s	sys 0.000s
I20260709 12:20:55.038384 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.028714  2030 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:55.041843 27587 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
W20260709 12:20:55.042725 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.000s	sys 0.026s
W20260709 12:20:55.043063 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.000s	sys 0.026s
W20260709 12:20:55.043601 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 526067us
I20260709 12:20:55.046128 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.056169 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.044497 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.037024  2246 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:20:55.067812 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.063968  2200 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:55.068840  1844 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:55.054770  2030 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:55.073130 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 188177us
I20260709 12:20:55.064486  2142 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:55.075110 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.057502 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.068091  2040 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:55.078496 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.078922 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:55.107795 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 252071us
I20260709 12:20:55.079645 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.071504  1689 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.077271 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.083686 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.085986 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.090623 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 953) took 1470 ms. Trace:
I20260709 12:20:55.066179  2254 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:20:55.076772  2176 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.079214  1844 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:55.075954  2142 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.109241 27589 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
W20260709 12:20:55.116923 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59539.9us
I20260709 12:20:55.114812 28329 rpcz_store.cc:276] 0709 12:20:53.619968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.620163 (+   195us) service_pool.cc:224] Handling call
0709 12:20:53.641414 (+ 21251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.641649 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:20:54.501628 (+859979us) log.cc:844] Serialized 14 byte log entry
0709 12:20:55.090173 (+588545us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.090245 (+    72us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:20:55.090310 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.090395 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.090543 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":878276,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":65153,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":12369,"thread_start_us":1401005,"threads_started":3,"wal-append.queue_time_us":465727}
I20260709 12:20:55.126592 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:55.129647 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52462.1us
I20260709 12:20:55.130703 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.133268 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:55.136160 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.139158 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.138093  2253 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:55.141882  2040 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:20:55.142179  2253 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.142199  1907 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:20:55.144876 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.142616  2253 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:55.117563  2165 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.129621  2142 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:55.123369  1704 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:55.119611  1844 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.135721  2169 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:55.132476  2144 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:20:55.181304  2095 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:55.186682  2095 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:55.187439  2095 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:55.188359  2095 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:55.189330 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44539.4us
I20260709 12:20:55.156240  2226 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:20:55.191000 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.153199  1689 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:55.172258 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206398us
I20260709 12:20:55.192662  2226 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:55.197211  2226 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.193295  1689 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.162433  1704 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:55.156136  1912 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:20:55.211251 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.211486  1912 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.205230  2096 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:55.212040  1912 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:55.205819  2144 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:20:55.214712  2071 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.619s	user 0.008s	sys 0.004s
I20260709 12:20:55.215910  1620 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:55.219477  2169 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:55.124540  2229 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.221179  2260 raft_consensus.cc:1064] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
W20260709 12:20:55.221659  2260 raft_consensus.cc:1068] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Unable to promote non-voter 4aa1523b9af84c3db99ba394e2b9aa9a: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:20:55.222201  2229 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:55.129478  2176 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.226751  2176 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:55.219096  2264 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:55.227677  1704 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:55.229426 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.168455  1844 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:55.240720 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.241009 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.195459 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.242015 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:55.242358 27586 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:55.242705 27589 raft_consensus.cc:2368] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18.
W20260709 12:20:55.242427 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47070.5us
I20260709 12:20:55.168438  2256 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:20:55.166070  2246 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:20:55.156953  2242 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.047s
W20260709 12:20:55.253822 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115960us
I20260709 12:20:55.226613  2248 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.191519 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1092) took 1318 ms. Trace:
I20260709 12:20:55.227713 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.236013  2142 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.236663  2253 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.239470 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:55.273236 28329 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:20:55.239682 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.243850 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.245446 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:55.245757 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:55.192569 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117725us
I20260709 12:20:55.247910  1689 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.248399  1696 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:55.254246  2042 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:55.195171  2178 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:55.164664  2165 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.080s
I20260709 12:20:55.248762 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:55.215191  2071 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23.
I20260709 12:20:55.252570  2268 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:20:55.245391 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106295us
I20260709 12:20:55.288148 28332 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:20:55.255831 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:55.260182 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:55.288837 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.258194 27583 raft_consensus.cc:2376] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20. Re-sending same reply.
I20260709 12:20:55.259938 28330 rpcz_store.cc:276] 0709 12:20:53.873554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.875049 (+  1495us) service_pool.cc:224] Handling call
0709 12:20:53.875449 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.875702 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:54.595660 (+719958us) log.cc:844] Serialized 14 byte log entry
0709 12:20:55.190813 (+595153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.190931 (+   118us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:55.191007 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.191136 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.191386 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":683038,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":9651,"raft.queue_time_us":129885,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"thread_start_us":1185762,"threads_started":3,"wal-append.queue_time_us":566893}
I20260709 12:20:55.268720 27704 leader_election.cc:400] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18.
W20260709 12:20:55.275139 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.293648 27704 leader_election.cc:403] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
W20260709 12:20:55.259394  2243 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.351s	user 0.000s	sys 0.002s
I20260709 12:20:55.278752 28331 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:20:55.281170 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.281579 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.282075 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.282119 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.282516 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.287102 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.351470 27733 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:20:55.359479 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:55.355747 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62456.8us
I20260709 12:20:55.356045 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.365990 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
W20260709 12:20:55.366973 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 79940.8us
I20260709 12:20:55.356357 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:55.358016 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:55.358291 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:55.273084  2185 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.377912  2185 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.380662  2185 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:55.380739  2253 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.259214  2264 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260709 12:20:55.381453  2185 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.382078  2264 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:55.358872  2276 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:55.382181  2185 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.267590  2256 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:55.362382  2269 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:20:55.384430  2256 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.385125  2269 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:55.385576  2269 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:55.386353  2030 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.264925  2260 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
I20260709 12:20:55.358932  2042 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:55.279768  2271 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:20:55.359211 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70487.3us
I20260709 12:20:55.387264  2256 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:55.275655  2258 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.107s
I20260709 12:20:55.296064  2255 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.089s
I20260709 12:20:55.396556  2276 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:55.397742  2276 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.398699  2185 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:55.402745 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3131) took 1650 ms (client timeout 2139 ms). Trace:
W20260709 12:20:55.403232 27734 rpcz_store.cc:269] 0709 12:20:53.752296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.752408 (+   112us) service_pool.cc:224] Handling call
0709 12:20:55.402560 (+1650152us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002e81d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:55.402684 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238990592}
I20260709 12:20:55.404795 27736 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:20:55.405230 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3114) took 1903 ms. Trace:
I20260709 12:20:55.405606 27732 rpcz_store.cc:276] 0709 12:20:53.502236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.502381 (+   145us) service_pool.cc:224] Handling call
0709 12:20:53.535061 (+ 32680us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.605520 (+ 70459us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:53.791712 (+186192us) log.cc:844] Serialized 14 byte log entry
0709 12:20:55.042187 (+1250475us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.042240 (+    53us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:55.042285 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.404982 (+362697us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:55.405034 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.405178 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1100644,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":5190,"raft.queue_time_us":16886,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":101,"spinlock_wait_cycles":161460480,"thread_start_us":1435833,"threads_started":3,"wal-append.queue_time_us":1181961}
W20260709 12:20:55.331171 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.375913  2176 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:55.376776  2248 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:55.377478 27729 raft_consensus.cc:2368] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:20:55.382517  2264 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:55.389683 27735 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:20:55.388342  2042 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.390897 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:55.388345  2169 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:55.391721  2142 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.425271 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:55.426162 27580 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:55.388556  2229 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.388541  2144 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:20:55.389411 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:55.375226  2268 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:20:55.412524 27738 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:20:55.409682 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:55.441147 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51846.3us
I20260709 12:20:55.373677 28329 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:55.388852 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:55.442176 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.388445  2271 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.287665  1696 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.427768 27589 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:20:55.419212  2200 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.461064 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.455211 27580 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:20:55.463300  2253 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:20:55.360227  2262 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.085s
I20260709 12:20:55.414557  2176 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:55.484045 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 209612us
I20260709 12:20:55.484203  2176 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.485743  2176 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:55.384577  2246 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18.
I20260709 12:20:55.447357 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.489300 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.489908  1704 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:55.459064 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.491392  2165 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:55.492148  2165 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:55.492336  2030 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:55.429695  2258 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:55.412662  1689 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:20:55.443694 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.501669 27583 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
W20260709 12:20:55.471110 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105061us
W20260709 12:20:55.502825 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 234214us
I20260709 12:20:55.455516  2229 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.505035 27730 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:20:55.474957  2192 raft_consensus.cc:993] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:20:55.477063  2095 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.480939 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1174) took 1487 ms. Trace:
I20260709 12:20:55.481195  1696 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:55.481860  2003 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:55.507716 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.525415 27737 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 13.
I20260709 12:20:55.418753  2264 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.512060 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:55.532835 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:55.533519 27580 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:55.431402  2269 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.519091 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:55.537813 27659 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:55.519788 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:55.538386 27662 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:20:55.538733 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:55.539419 27663 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:55.540782 27732 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:55.465672  2169 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:20:55.499158  2227 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.618s	user 0.001s	sys 0.002s
I20260709 12:20:55.508862 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:55.525935 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.521576 27656 rpcz_store.cc:276] 0709 12:20:53.993843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.993988 (+   145us) service_pool.cc:224] Handling call
0709 12:20:54.146839 (+152851us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:55.011974 (+865135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.480710 (+468736us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:55.480877 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1988166528}
I20260709 12:20:55.526594 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.564204 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:55.564953 27734 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:55.526832 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.566073 27737 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:55.567489 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.527398 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.569588 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.433431  1912 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.535141 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.520671 27664 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:20:55.550783 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:55.472393  2253 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:55.560338 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.560491 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.572557 27660 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:20:55.575769 27736 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:55.504993  2258 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:55.503954  2144 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:55.596197 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:55.580096 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:55.596903 27736 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:55.570360 27733 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:55.588083 27737 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:20:55.599570 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:55.600656 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97517.9us
I20260709 12:20:55.525072  2003 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:55.604259 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.469355  2200 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.622474 27588 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:55.619849 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:55.624120 27657 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:20:55.614455 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:55.605808 27734 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
W20260709 12:20:55.613111 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52745.3us
I20260709 12:20:55.613996 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:55.605088 27655 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:55.627823 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.553752  1863 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.628393 27731 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:55.616164 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1093) took 1719 ms. Trace:
I20260709 12:20:55.524973  2270 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:20:55.618743  2253 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.638340 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:55.619809  1859 raft_consensus.cc:1081] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Signalling peer ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:20:55.619920 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.620486  2095 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:55.616051 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:55.640827 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.641635 27580 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:55.642758 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.618170 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3319) took 2022 ms. Trace:
I20260709 12:20:55.629052 28336 rpcz_store.cc:276] 0709 12:20:53.896554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.896803 (+   249us) service_pool.cc:224] Handling call
0709 12:20:53.897190 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.897453 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:20:55.492740 (+1595287us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b18003807a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:20:55.492893 (+   153us) log.cc:844] Serialized 14 byte log entry
0709 12:20:55.615732 (+122839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.615801 (+    69us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:20:55.615861 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.615951 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.616097 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115686,"prepare.queue_time_us":684711,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":8903,"raft.queue_time_us":23214,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":77,"spinlock_wait_cycles":12281088,"thread_start_us":533146,"threads_started":2}
I20260709 12:20:55.618994  2229 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.631160 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:55.562935  2178 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:55.643474 27585 rpcz_store.cc:276] 0709 12:20:53.595563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.603498 (+  7935us) service_pool.cc:224] Handling call
0709 12:20:53.603792 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.604006 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.604034 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:53.604553 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:20:54.405867 (+801314us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:54.405925 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.406032 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:55.498562 (+1092530us) spinlock_profiling.cc:243] Waited 977 us on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:55.612199 (+113637us) raft_consensus.cc:1685] finished
0709 12:20:55.612228 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.618107 (+  5879us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2246912,"thread_start_us":800888,"threads_started":1,"wal-append.queue_time_us":1193175}
I20260709 12:20:55.646140 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:55.650702 27580 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:55.595669  2258 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:55.696779 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:55.698444 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.701710 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:55.708879 27657 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:20:55.710454 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:55.629966  2169 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:20:55.635046  2264 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:55.621258  2113 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.635260  2040 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.718595 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:55.715881 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:55.722961 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:55.724295  1382 raft_consensus.cc:993] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:20:55.725759 27585 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:55.727185 27585 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:55.729269 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:55.731089 27585 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:20:55.721616  2169 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:55.695132  1620 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:55.699366  1709 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:55.701943  2229 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:20:55.796387 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.840421  2269 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.701984  2279 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:55.712121  2095 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:55.853137  2287 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:20:55.854048  1704 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:55.749598  2169 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:20:55.855212  2003 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.868057  2135 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:20:55.868199 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 222175us
I20260709 12:20:55.879606  2096 raft_consensus.cc:1081] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Signalling peer ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:20:55.660265  1863 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:55.856020  2281 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:20:55.957809 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:55.960738 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:55.720317  2071 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:55.880333  2229 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:55.970654  1912 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:55.913206  2144 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:55.973640  1844 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.716s	user 0.007s	sys 0.002s
I20260709 12:20:55.974037  2135 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:55.974814  2003 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:55.737429  2040 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.975487  1863 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:56.004614 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 268092us
I20260709 12:20:56.010089  1844 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.992491 27663 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:20:55.740600  2113 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.991714 27581 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:55.894405  1709 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:55.989050 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:55.955981  2095 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:56.016880 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 329038us
W20260709 12:20:56.006976 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 379226us
I20260709 12:20:55.992790  2253 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:55.994652  2229 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:56.020364 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:55.953032  2279 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.074643  2135 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:55.963709  2178 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:56.078809 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:56.080719 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.019855  1844 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.094252 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:56.096072 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76575.5us
I20260709 12:20:56.096822 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.025962  2288 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:20:56.106704  2096 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:56.090698  1861 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:56.129431  1861 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.112960  2040 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.114073  2095 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:56.115454  2165 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.623s	user 0.008s	sys 0.004s
I20260709 12:20:56.103840 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.138934 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.119244  2294 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.025s
I20260709 12:20:56.138017 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.119472 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1221) took 1277 ms. Trace:
I20260709 12:20:56.148309 27587 rpcz_store.cc:276] 0709 12:20:54.842114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:54.864680 (+ 22566us) service_pool.cc:224] Handling call
0709 12:20:56.119280 (+1254600us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:56.119410 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405100416}
I20260709 12:20:56.139882 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.124248 27730 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:56.106307 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:56.025044  2253 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.141095 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.159155 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:56.147233  2226 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:20:56.122323 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:56.166862 27663 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:20:56.102679  2301 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:56.154655 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.171765 27662 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:20:56.161633 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55424.5us
I20260709 12:20:56.140681  1844 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:56.172719 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:56.173296 27662 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:20:56.167135 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 175891us
I20260709 12:20:56.174429 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.124372  2202 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.167443 27737 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:56.152458 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.179863 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.180060 27657 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:20:56.180958 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.099882  2262 raft_consensus.cc:2764] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:20:56.158545 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.150674  2165 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:56.184456 27585 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:56.184365 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.177597 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.184341 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.150725  1863 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:56.187281 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.159144  2109 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.197018 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.197805 27657 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:20:56.156359  2296 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:56.199003 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.170674  2096 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:56.166868  2280 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:56.167116  2271 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:56.205181 27580 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:20:56.207145 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:56.208360 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49943.7us
I20260709 12:20:56.206892 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.214247 27657 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:56.184068  2217 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:56.167634  2242 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:20:56.177932  1844 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:56.187937  2248 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.222110 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.205842  1912 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:56.225338  2299 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:20:56.212177  2280 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.213616  2096 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:56.217427  1859 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.228780 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.229450 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.243016 27737 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:20:56.243588 27657 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:56.243479 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.226018  2016 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:56.192068  2191 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:20:56.244733 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.201556  2282 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:56.196758  2293 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:20:56.191788  2202 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.220975  2217 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:56.202419  2269 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
W20260709 12:20:56.256796 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45042.2us
I20260709 12:20:56.258051 27657 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:20:56.210520  2109 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.210104  2292 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:20:56.227645  1844 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:56.221339  2294 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:20:56.259564 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.250824  2282 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:56.230955  1912 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:56.249033  2100 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:56.265045  2165 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.249881  1859 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.297047 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:56.297396 27663 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:20:56.298256 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:56.298545  1859 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.298779  2284 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:20:56.246045  2016 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:56.241771  2280 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:56.307526 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:56.241143  2296 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:56.214344  1863 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:56.263221 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.272792 27736 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:56.266554 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.312012 27657 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:56.274075 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:56.314319 27734 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:34476
I20260709 12:20:56.314649 27734 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:56.314926 27734 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:56.277737  2258 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.004s
I20260709 12:20:56.279503 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:56.280889  2109 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.324746 27657 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:20:56.282193  1861 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
W20260709 12:20:56.285053  2264 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.548s	user 0.008s	sys 0.000s
I20260709 12:20:56.255205  2242 consensus_queue.cc:1243] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 23.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:20:56.288435  2253 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.304399  1912 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:56.257963  2217 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:56.245157  2248 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:56.289526  2282 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:56.337018  2100 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:56.295295 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.369728 27587 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:56.271987 27733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:34476
I20260709 12:20:56.299343  2271 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:20:56.248167  2096 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:56.300370 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.312587 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:56.380725 27736 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:56.324215 27663 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:20:56.299633  2185 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:20:56.334689 27734 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.348623  2071 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
W20260709 12:20:56.344128 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44791.4us
I20260709 12:20:56.390825 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:56.346796 28331 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260709 12:20:56.317158  2176 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.825s	user 0.010s	sys 0.003s
I20260709 12:20:56.300052  2279 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:56.360324  2016 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.394122  2100 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.395071  2016 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:56.292143 27662 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:56.372388 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.397926 27662 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:20:56.407480 27662 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:20:56.409078 27662 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:20:56.410040  2282 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.410777 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:56.390183  1863 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.412321  2282 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:56.412492  1869 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:56.414738 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:56.397935 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:56.377669 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.415426 27581 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:56.371508 27733 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:56.337283  2258 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.389668 27658 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:20:56.344825  1696 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:20:56.288319 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:56.387887  2185 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:56.311172  2165 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.389101  2216 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:56.311546  2169 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.416110 27733 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:56.417217 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.413234  2280 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.443452 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:56.451916 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.418462 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:56.378216  2229 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:56.475210 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56826.9us
I20260709 12:20:56.476279 27587 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:56.481142 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:56.421480  2264 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:56.483076 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:56.487039 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.432163  2269 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:56.495082 27581 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:20:56.441560  2185 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:56.441900  2292 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.442001  2216 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:20:56.513245 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 220786us
I20260709 12:20:56.494424  1696 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:56.516747 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.518766 27736 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:56.519635 27732 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:56.496429  2264 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:56.499436  2271 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:56.522498 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:56.517174 27733 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.529235 27733 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:56.389822  2296 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.483582  1859 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:20:56.494746  2191 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:20:56.513824  2135 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:56.526129  2279 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:56.525430  2277 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.525086  2095 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:56.552255  2144 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:20:56.527069  2264 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:20:56.575805 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3351) took 1151 ms (client timeout 1509 ms). Trace:
W20260709 12:20:56.576264 27589 rpcz_store.cc:269] 0709 12:20:55.424309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:55.460710 (+ 36401us) service_pool.cc:224] Handling call
0709 12:20:56.575596 (+1114886us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:56.575744 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1174451456}
I20260709 12:20:56.512639  2292 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.562696  2216 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.569657  2294 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:20:56.570498  2277 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.570101  2144 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.609341  2144 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:20:56.524893  2224 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:20:56.692806  2248 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:56.698289 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:56.708026  2226 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.000s
I20260709 12:20:56.702319  2005 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.703481 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:56.703588  2280 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:56.704927  2113 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.705580 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:56.631029  2095 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.722528  2095 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.723435  2217 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.725008  2269 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:20:56.725992  2330 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.726464  2096 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.635677  1869 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.727561  2258 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.727612  1869 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:56.729955  2267 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.369s	user 0.011s	sys 0.007s
I20260709 12:20:56.718922 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:56.721230  2005 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.721983  1620 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:56.731737  2306 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.468s	user 0.004s	sys 0.000s
I20260709 12:20:56.731668  1620 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.733652  1620 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:56.735471 28334 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:55.479982 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:56.726575  2330 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:56.698719  2301 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.513s	user 0.006s	sys 0.000s
I20260709 12:20:56.742863 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:56.714603  2333 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.764653  2005 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:56.741827  2276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.315s	user 0.010s	sys 0.010s
I20260709 12:20:56.791584  1620 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:20:56.790858 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72021.8us
I20260709 12:20:56.768755 27657 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:56.794303 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:56.792335  2276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.387s	user 0.011s	sys 0.011s
I20260709 12:20:56.744550  2165 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:56.803676  1620 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:56.805083 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:56.805752 28329 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:20:56.819914  2217 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:56.822513  2326 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:20:56.801635  2335 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.823596  2335 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.826114  2335 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:56.827903  2096 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.804098  2095 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.836838 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.837611 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:20:56.838666 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:56.839454  2321 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:20:56.799849  2169 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:56.840693  2254 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:20:56.841097  2254 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:56.841292  2150 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:56.841867  2150 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:56.842104  2254 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.843302  2254 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:20:56.850693  2277 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:56.791292  2216 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:56.851928 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1014) took 1451 ms. Trace:
W20260709 12:20:56.852800  1844 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.552s	user 0.010s	sys 0.000s
I20260709 12:20:56.853031 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:56.853274  1844 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.853713 28329 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:20:56.854372 27581 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:56.856102 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:56.857259 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.847414 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:56.847435 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:56.766134  2301 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:56.846715 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:20:56.913446 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66862.3us
W20260709 12:20:56.913515 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 391062us
I20260709 12:20:56.914698 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:56.915993 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.852427 28331 rpcz_store.cc:276] 0709 12:20:55.400528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:55.401089 (+   561us) service_pool.cc:224] Handling call
0709 12:20:55.401415 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.391747 (+990332us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:56.851584 (+459837us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54005eabd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:56.851844 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2173331200}
I20260709 12:20:56.919620 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:56.920495 27582 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
W20260709 12:20:56.921587 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.798516  2100 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.933523 28336 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:56.819732  2292 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.918744 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:56.935732 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 76233.7us
I20260709 12:20:56.934902 27583 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
W20260709 12:20:56.934870 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93809.4us
W20260709 12:20:56.938203 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.849442  2288 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:56.940366 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.957191 28330 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:20:56.845922  2200 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:56.848907  2262 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:56.958122 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:56.922547  2283 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:56.953711 27582 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:56.963917 27581 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:20:56.936715  1844 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.957201  1913 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:20:56.957795  2288 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:56.978072 27589 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:20:56.966728  2283 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.957468  1696 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:56.961333  2292 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:20:56.975719 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:56.847226  2144 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.977347  1382 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:20:56.845471  2185 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.969643 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:57.225840 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3347) took 1802 ms. Trace:
I20260709 12:20:57.225205 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3351) took 1087 ms. Trace:
W20260709 12:20:57.244882 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:57.224072 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 248229us
I20260709 12:20:57.037333  1709 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.303681  2005 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.304760  2005 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:57.305554 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 335837us
I20260709 12:20:56.965893  1869 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:56.967736  1912 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.610s	user 0.005s	sys 0.005s
I20260709 12:20:57.309049  1912 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.078447  2003 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:57.311717  2003 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:57.312518  2226 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.313557  2226 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.313611  2003 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:56.977660  1913 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.940981  2100 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.034415  2144 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.303056 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3235) took 1103 ms. Trace:
I20260709 12:20:57.029390  2185 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.017395  2202 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.067655  2229 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:56.963109  2262 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:57.319648  2202 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.996187  2264 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:56.974869  2276 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:20:56.979868  2283 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:56.999806  2288 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.329032  2005 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.329178  2288 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:57.145562  2289 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.891s	user 0.000s	sys 0.013s
W20260709 12:20:57.338914 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:57.339530 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3123) took 3792 ms (client timeout 3283 ms). Trace:
I20260709 12:20:57.339737 28332 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
W20260709 12:20:57.340087 27735 rpcz_store.cc:269] 0709 12:20:53.546681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.629753 (+ 83072us) service_pool.cc:224] Handling call
0709 12:20:57.338894 (+3709141us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000a5fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:57.339195 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3931856640}
I20260709 12:20:57.342980 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3151) took 2059 ms. Trace:
I20260709 12:20:57.343209  2262 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.344928  2262 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:57.345615  2264 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.346714  2144 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:57.347640  2144 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.351802  2202 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:57.146727  2291 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.871s	user 0.009s	sys 0.005s
W20260709 12:20:57.353613  2291 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.078s	user 0.011s	sys 0.005s
I20260709 12:20:57.000087  2292 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:57.061885  2309 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.809s	user 0.004s	sys 0.000s
I20260709 12:20:57.065626  2293 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:20:57.083729  2310 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.817s	user 0.006s	sys 0.000s
I20260709 12:20:56.966765  2329 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:57.053310  2330 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:57.134195  2322 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.633s	user 0.003s	sys 0.000s
I20260709 12:20:56.770949  2333 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:57.274554  2175 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.014s	user 0.007s	sys 0.006s
I20260709 12:20:57.285938  2297 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.298240 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1179) took 1791 ms. Trace:
I20260709 12:20:57.307948  2273 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:57.033306  2150 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.318293  1913 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:57.319831  2135 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.034368  2265 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:20:57.389964  2265 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:57.391733  2265 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.393183  2265 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:20:57.396802  2297 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:57.398813 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.399256  2333 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.400748 28336 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:20:57.269703 27580 rpcz_store.cc:276] 0709 12:20:56.137679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.137799 (+   120us) service_pool.cc:224] Handling call
0709 12:20:56.138081 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.213587 (+ 75506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.224333 (+1010746us) spinlock_profiling.cc:243] Waited 66.6 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:57.224861 (+   528us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153184128}
I20260709 12:20:57.402436 28331 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:20:57.408093 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:57.319113 27657 rpcz_store.cc:276] 0709 12:20:56.200352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.339384 (+139032us) service_pool.cc:224] Handling call
0709 12:20:56.339727 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.793694 (+453967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.302221 (+508527us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:57.302721 (+   500us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":986121088}
I20260709 12:20:57.343451 27738 rpcz_store.cc:276] 0709 12:20:55.283449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:55.366704 (+ 83255us) service_pool.cc:224] Handling call
0709 12:20:55.412103 (+ 45399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:55.440845 (+ 28742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.342419 (+1901574us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b54001d2ed0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:57.342703 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103650048}
I20260709 12:20:57.354645  2192 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.412714  2192 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.286429  2169 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:20:57.339759 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.415303  2192 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.355980 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:57.417786 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 62239.8us
I20260709 12:20:57.418468  2071 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.323936  2226 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.419118 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:57.419317  2071 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.420420  2292 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:57.420486 28334 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:57.421501 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:57.365406 27658 rpcz_store.cc:276] 0709 12:20:55.507154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:55.507406 (+   252us) service_pool.cc:224] Handling call
0709 12:20:55.507761 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.419822 (+912061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.297524 (+877702us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54000e56d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:57.297942 (+   418us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2027702528}
W20260709 12:20:57.339052  2289 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.087s	user 0.000s	sys 0.016s
W20260709 12:20:57.366088  2175 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.105s	user 0.008s	sys 0.006s
I20260709 12:20:57.355284  2202 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.357527  2293 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:57.352780 27627 leader_election.cc:400] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:20:57.369691  2273 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.382134  2100 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.428608  2273 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:57.356355  2351 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:57.385356 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3155) took 1114 ms. Trace:
I20260709 12:20:57.388815  2135 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.270182 27588 rpcz_store.cc:276] 0709 12:20:55.423065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:55.428232 (+  5167us) service_pool.cc:224] Handling call
0709 12:20:55.428555 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:55.623001 (+194446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.225242 (+1602241us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54002bacd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:57.225637 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":445523712}
I20260709 12:20:57.404970  2150 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:57.399262 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:57.410908 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.14416e+06us
I20260709 12:20:57.415784  2185 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:57.415575  2169 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:57.350996  2144 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:57.428112 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75729us
I20260709 12:20:57.438002  2169 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:57.430365  2293 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.460563  2003 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:57.462760 27661 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:57.442270  2185 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:57.467449 27657 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:20:57.468289  1907 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:57.468802  1907 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:57.470187  1907 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.474660 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3358) took 1150 ms. Trace:
I20260709 12:20:57.475190 27586 rpcz_store.cc:276] 0709 12:20:56.291311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.292512 (+  1201us) service_pool.cc:224] Handling call
0709 12:20:56.306678 (+ 14166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.307193 (+   515us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:20:56.478345 (+171152us) log.cc:844] Serialized 14 byte log entry
0709 12:20:57.284213 (+805868us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:57.333182 (+ 48969us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:57.366645 (+ 33463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.373561 (+  6916us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:57.441677 (+ 68116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":765921,"prepare.queue_time_us":46402,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1969,"raft.queue_time_us":804907,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":259,"thread_start_us":170857,"threads_started":2}
W20260709 12:20:57.487272 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.431516 27731 rpcz_store.cc:276] 0709 12:20:56.270356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.270495 (+   139us) service_pool.cc:224] Handling call
0709 12:20:56.270781 (+   286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:56.271163 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:56.271192 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:56.271643 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:20:56.437128 (+165485us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:20:56.494343 (+ 57215us) log.cc:844] Serialized 14 byte log entry
0709 12:20:56.509945 (+ 15602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:56.510027 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:57.059090 (+549063us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b180049ace0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:20:57.132239 (+ 73149us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b180049ace0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db
0709 12:20:57.133336 (+  1097us) spinlock_profiling.cc:243] Waited 4.22 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:57.383390 (+250054us) raft_consensus.cc:1685] finished
0709 12:20:57.384869 (+  1479us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:57.385144 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34203,"prepare.queue_time_us":57650,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":38185,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":24657,"spinlock_wait_cycles":29625728,"thread_start_us":95625,"threads_started":1,"wal-append.queue_time_us":189334}
I20260709 12:20:57.504575 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:57.502532  2352 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:57.507193  2352 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.440877  2333 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:20:57.517225 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:57.511978 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:20:57.519055 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.520007 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:57.520485  2185 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.485607  2144 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.522881 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:57.525727 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.512681  2192 raft_consensus.cc:1064] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:20:57.466037  2351 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.531625 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:57.533782 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.535283 27661 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:57.538863 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:57.545958 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:57.541517  2291 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:20:57.548498 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91008.8us
I20260709 12:20:57.545126  1912 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.554484  2176 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:57.555054  2346 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:20:57.555799 27734 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.556527 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:34476 (request call id 3161) took 1284 ms. Trace:
I20260709 12:20:57.556871 27734 rpcz_store.cc:276] 0709 12:20:56.272167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.314238 (+ 42071us) service_pool.cc:224] Handling call
0709 12:20:57.555398 (+1241160us) spinlock_profiling.cc:243] Waited 879 us on lock 0x7b4c003eb4c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98614a20 00007f0e98625a29 00007f0e9862613f 00007f0e985ea2ab 00007f0e98617f3c 00007f0e986183fe 00007f0e988a588b 00007f0e9a2a4495 00007f0e9a2f530f 00007f0e9a33fa55 00007f0e9f30fcd0
0709 12:20:57.556431 (+  1033us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2020608}
I20260709 12:20:57.557276  2135 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.554720 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:57.544605  2297 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.548409 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:57.547622  2291 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:57.558698  1907 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:57.591905  2346 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:20:57.597891 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75582.8us
I20260709 12:20:57.547523  2144 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.587548  2175 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:57.580119  2192 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.554456  2317 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:20:57.610954  2229 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:57.555837  2176 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.628291  2351 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.629626  2176 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.607158  2175 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:57.615939  2346 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:20:57.640949 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:57.663791  2229 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:20:57.664674  2229 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:57.639247  2175 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:57.667740  2135 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.668758  2135 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.672571  2135 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:57.679757 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.666297  2229 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:57.695303 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.693472  2291 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:57.693631 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.685214  2144 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.689859  2351 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.700541 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:57.695531  2317 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:20:57.709801  2291 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:57.717868  2351 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.725553  2351 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.726511  2071 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.712587  2144 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:20:57.730086 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44190.7us
I20260709 12:20:57.708844  2169 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.709815  1913 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.737622 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3153) took 1436 ms. Trace:
I20260709 12:20:57.738381 27737 rpcz_store.cc:276] 0709 12:20:56.269680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.269823 (+   143us) service_pool.cc:224] Handling call
0709 12:20:56.270106 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.270312 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:20:56.320525 (+ 50213us) log.cc:844] Serialized 14 byte log entry
0709 12:20:57.568604 (+1248079us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b18000e7ca0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:20:57.694901 (+126297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:57.695005 (+   104us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:20:57.695088 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.695989 (+   901us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:57.705627 (+  9638us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":45640,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":166,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":134,"spinlock_wait_cycles":11075456,"thread_start_us":114810,"threads_started":1,"wal-append.queue_time_us":1386342}
I20260709 12:20:57.748858  2169 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.751988  1913 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:57.751365 28330 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Committing config change with OpId 19.12: config changed from index 10 to 12, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:57.719251  2283 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.756026 28331 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:57.759616 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:57.760416 28335 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:57.761406 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:57.761621 27584 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:20:57.763007  2283 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.764349 28328 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:57.764492 27584 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:20:57.766812  2317 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:20:57.767889  2317 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:57.768363  2317 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:57.747974  2144 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Committing config change with OpId 19.12: config changed from index 10 to 12, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:57.761521 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:57.781427 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:57.782199 28330 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
W20260709 12:20:57.790207 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 243109us
I20260709 12:20:57.791949 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:57.797572  2352 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.797771  2192 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:20:57.793908 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:57.798646 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.798581  2042 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:57.799435 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:57.799381  2042 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.797660 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:57.803503  2297 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:57.803750 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:57.803804 28333 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:57.804716  2297 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:57.805383 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:57.805658  2178 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.806501  2178 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.807551  2262 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:57.795261  2135 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.808660 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:57.808568  2135 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.807960  2262 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.810653  2262 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:57.810881  2135 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.811704 27587 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:57.812058 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:57.801941  2042 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:57.826014 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 136708us
I20260709 12:20:57.831444 28337 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:57.832348  2175 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:57.832998 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.833257 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:57.821590  2202 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.838383 28333 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:20:57.819061  2352 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.016s
I20260709 12:20:57.839315  2289 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:20:57.844348 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:57.849040 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:57.852357  2095 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:20:57.853042  2095 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:57.854153  2095 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.855690 27585 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:20:57.856395  2178 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.840533  2030 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:57.858024  2030 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.860375  2030 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:57.860761  2095 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:57.849889 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:57.868510 28334 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:20:57.864352  2202 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.872308  2150 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:20:57.862262  2175 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.866721  2352 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 21.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:20:57.872391  2003 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:57.878419 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:57.881222 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:57.878412 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.693s (SENT) [suppressed 18 similar messages]
I20260709 12:20:57.883136 27584 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
W20260709 12:20:57.891803 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52183.6us
I20260709 12:20:57.888737 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:57.843604  2317 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:57.864298  2185 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
I20260709 12:20:57.893200  2169 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:20:57.903043 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.904448 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:57.900337 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:57.919672 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91933.8us
I20260709 12:20:57.912772 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:57.902724  2003 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:57.899901  2169 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:57.926733 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.928324 27587 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 17: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:57.907471  2283 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:57.927951  2042 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:20:57.933065  2042 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.933429  2042 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:20:57.934124 27661 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:57.932528  1689 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:20:57.937654  1689 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:57.941404  2192 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:57.941912  1689 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.942167  2192 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:57.932080  2273 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.503s	user 0.013s	sys 0.002s
I20260709 12:20:57.948428 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:57.924513  2003 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:57.952003 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:57.952550 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.931273  2283 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:57.971820 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:57.967914 27584 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:57.943280  1689 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:57.975925 27589 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:57.964677  2273 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:57.977834 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:57.980067  2273 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:57.983608  2317 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:57.991537 27588 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:57.961791  2289 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:57.964044  2003 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:20:57.995347 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116990us
I20260709 12:20:57.995996  2265 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:20:58.004921 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:58.006376 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 125305us
I20260709 12:20:58.012127 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:58.005031  2202 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.005709  1913 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:58.014415  2169 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.021329 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.022776 27589 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:20:58.022046 27580 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:58.025491 27587 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:20:58.026115 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.025816 27587 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:58.026825 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 132099us
I20260709 12:20:58.027410  1913 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.032765 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.033520 27589 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:20:58.033532  2192 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.040704 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.041397 27734 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:58.041172  2283 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.041693 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.042356 27735 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:58.042958 27729 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:58.043622 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.044308 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.045008 27733 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:58.045348 27731 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 26. Ops: []
I20260709 12:20:58.045516  2283 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:58.046636 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.047199 27732 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:58.041405 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.049378 27738 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:58.050549 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.050693 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.051187 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:58.054620 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.053145 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.052268 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.051136 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.052452 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.058250 27663 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.053757 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.059803 27662 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.064740 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.065508 27586 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:58.066418 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.066591 27587 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.067721 27587 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:58.070397  2352 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.071086  2352 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.071755 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.072387 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:58.072558 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 233) took 45483 ms (client timeout 15000 ms). Trace:
W20260709 12:20:58.054206  2362 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:20:58.073061 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.051734 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.073596 27583 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:20:58.073982 27586 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:58.072604 27734 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:20:58.077457  2292 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:58.079907  2292 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:20:58.080303  2292 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:20:58.054106  2365 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:20:58.081429  2292 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.082904  2292 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:58.031669  2291 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:20:58.077759 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
W20260709 12:20:58.100714 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67949us
I20260709 12:20:58.110296 27732 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:58.104368 27735 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:58.124001 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.127951 27660 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:58.123749 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:58.136237 27656 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:20:58.137616 27662 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.138697 27585 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:58.139731 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.140527 27737 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:58.141551 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:20:58.142691 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.083948  2169 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:20:58.147446 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.147991 27736 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:20:58.094239  2352 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.149127 27584 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:58.150928 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.151125 27738 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:58.084885  2202 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:58.154739  2202 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.155287  2202 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:58.160195 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43729.8us
I20260709 12:20:58.127883 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.169786 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.092850  1913 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.178411  2135 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:58.178591 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:58.174805  2030 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.184111 27734 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:20:58.181069 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.183341  2176 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:20:58.187186 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.183490  1689 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:20:58.195763  2176 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.196163 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.197032 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.201644 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.202175 27658 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:58.198076 27661 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:58.200968 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.202250 27655 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:58.083303  2262 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.183923  2135 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.213327 27588 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:20:58.216003 27584 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:58.217717  2030 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.219637  2283 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.221896  2176 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.224642 27663 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:20:58.225389 27581 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:20:58.100610  1704 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.094s
I20260709 12:20:58.229067 27661 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:58.224001  2095 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:58.230507 27585 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:58.209311  2289 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:20:58.224676 27583 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:58.225332 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.198993  2293 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:58.272259 27658 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:58.272548 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:58.252260 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 91189.8us
I20260709 12:20:58.255239 27586 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:20:58.179132  2178 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:20:58.259755 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.255383  2283 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.277361 27584 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:58.277743 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.278044 27580 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:58.278425 27586 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:20:58.260370 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:20:58.261003 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.279502 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
W20260709 12:20:58.263460 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49839.3us
I20260709 12:20:58.255187  2229 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.281864 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:58.284240 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66931.1us
I20260709 12:20:58.268829  2030 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.251317  2352 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:58.258400 27736 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:58.246266  2135 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:58.196725  2185 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:20:58.257779  2095 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.283545  2175 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.283758 27586 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:58.285488 27588 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:20:58.285207 27587 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:20:58.094507 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 81e6b2dabf8f4e6cb549ca250430e568 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 4 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:53816
I20260709 12:20:58.294458 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.296166 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.299484 27486 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 11 to 13, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 13 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:58.303114 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.303665 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.304840 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.302526  2283 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.303937 27586 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:58.305526 27584 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:58.306237 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.306385 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.306795 27586 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:20:58.307224 28329 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:20:58.307442  2178 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:20:58.308856 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.291464 27588 pending_rounds.cc:85] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:20:58.307257 27584 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 15: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:58.284071  2176 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:58.314105  1907 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:58.313862  2176 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
W20260709 12:20:58.073156 27487 rpcz_store.cc:269] 0709 12:20:12.603599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:12.604880 (+  1281us) service_pool.cc:224] Handling call
0709 12:20:58.072267 (+45467387us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:51.175688 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:51.176246 (+   558us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:51.176281 (+    35us) write_op.cc:435] Acquired schema lock
0709 12:20:51.176298 (+    17us) tablet.cc:662] Decoding operations
0709 12:20:51.176834 (+   536us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:51.176919 (+    85us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:51.176947 (+    28us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:20:51.177273 (+   326us) tablet.cc:701] Row locks acquired
0709 12:20:51.177292 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:20:51.177430 (+   138us) write_op.cc:270] Start()
0709 12:20:51.177530 (+   100us) write_op.cc:276] Timestamp: P: 1783599651177382 usec, L: 0
0709 12:20:51.177623 (+    93us) op_driver.cc:348] REPLICATION: starting
0709 12:20:51.178298 (+   675us) log.cc:844] Serialized 1308 byte log entry
0709 12:20:52.723777 (+1545479us) op_driver.cc:464] REPLICATION: finished
0709 12:20:52.792505 (+ 68728us) write_op.cc:301] APPLY: starting
0709 12:20:52.797290 (+  4785us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:52.869162 (+ 71872us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:52.869271 (+   109us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:52.871251 (+  1980us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:52.879028 (+  7777us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0709 12:20:52.886353 (+  7325us) write_op.cc:312] APPLY: finished
0709 12:20:52.923296 (+ 36943us) log.cc:844] Serialized 41 byte log entry
0709 12:20:52.924884 (+  1588us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:52.927591 (+  2707us) write_op.cc:454] Released schema lock
0709 12:20:52.955046 (+ 27455us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:52.972898 (+ 17852us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:52.976223 (+  3325us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1165,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":55573,"apply.run_cpu_time_us":6768,"apply.run_wall_time_us":182834,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16667,"num_ops":3,"prepare.queue_time_us":550882,"prepare.run_cpu_time_us":57470,"prepare.run_wall_time_us":271675,"raft.queue_time_us":377735,"raft.run_cpu_time_us":31482,"raft.run_wall_time_us":71804,"replication_time_us":1542151,"thread_start_us":1004282,"threads_started":4,"wal-append.queue_time_us":431688,"wal-append.run_cpu_time_us":157339,"wal-append.run_wall_time_us":2412226}]]}
I20260709 12:20:58.310930 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.322916 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:58.323763 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.318351 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.287667  2346 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.310403 27588 pending_rounds.cc:107] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:20:58.293962  2095 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:58.332352 27588 raft_consensus.cc:2889] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:58.335644 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.335383 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.336774 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.345598  2376 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:58.346319  2376 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.349041  2376 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.288375  2292 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:20:58.355496 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.356222 27583 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.324074  1913 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:58.357506 27663 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
W20260709 12:20:58.360263 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.361210 27585 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:58.368088 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.287848  2229 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.359604 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:20:58.386875 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.391710 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.391985 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.335542  1689 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:58.398734 27660 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.400740 27586 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:20:58.310146  2371 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.092s
I20260709 12:20:58.404789 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.341634  2372 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:58.405153 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.327019  2352 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.326283  2175 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.384405 27584 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Committing config change with OpId 19.12: config changed from index 10 to 12, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:58.403496 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.402513 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:20:58.410089 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.404567  2003 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.411103 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.441855 27660 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:58.443575  2376 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.419857 27659 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:20:58.335794  2042 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.353027  2346 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:58.424651 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.425987  2176 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:58.427930 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.428239  2352 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.409701  2283 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:58.409502  2372 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.431777  2185 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:20:58.436619 27585 raft_consensus.cc:2376] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21. Re-sending same reply.
W20260709 12:20:58.439154 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 94601.2us
I20260709 12:20:58.454653 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.455766 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.440325 27655 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:20:58.444756 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:20:58.456524 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48417.5us
I20260709 12:20:58.457254 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.423632 27582 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:20:58.402822  2178 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.448184 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:58.456284 27655 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:58.440124  2100 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.472574  2178 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.504370 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:20:58.473505  2265 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:20:58.474591  2229 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.475967 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:58.476038  2375 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:20:58.476903 27664 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.475670 27486 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 8 to 10, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 10 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:20:58.509954 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:20:58.511408 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55714.2us
W20260709 12:20:58.511879 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 176612us
W20260709 12:20:58.512652 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 219234us
I20260709 12:20:58.451388  2283 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:20:58.510123 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60388.8us
I20260709 12:20:58.515831 27662 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:20:58.509413 27486 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 4 to 10, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 10 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.464167  2297 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:58.524999 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.453750  2372 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:58.532754 28334 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:58.504814 27655 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:58.537410 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80207.3us
I20260709 12:20:58.542599 27660 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.542495 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:58.543179  2042 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.543596 28333 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:20:58.545643  2346 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.547360  2375 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:58.548164  2375 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.548668  2135 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.526883  2100 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.550802  2135 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.551421  2100 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.509706  2176 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:58.552052  2176 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:58.552526  2176 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:58.552966  1859 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:20:58.552798  2202 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.553813  1859 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:58.554407  1859 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:58.559464 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.560099 27662 raft_consensus.cc:2376] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23. Re-sending same reply.
I20260709 12:20:58.561427 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.562417 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.562750  1913 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:58.562983 27662 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:20:58.563512  2378 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:20:58.547601  2382 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.526383 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:58.451157  2376 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.533985 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:58.572346 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.571576 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:20:58.567515 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 244669us
I20260709 12:20:58.543702 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.569985 28329 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:58.523860  2283 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:58.521436  2003 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.524173  1689 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:58.563190  1913 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:58.578361 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.579490 28336 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:20:58.578228 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.579811 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:58.580365 28334 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:58.584009 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:58.584827 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.569387  2202 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:58.588488 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.588909 27734 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:58.589944 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:58.572865  2375 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.592428  2169 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:58.592835  2169 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:58.593267  2169 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:20:58.594003  2144 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.737s	user 0.011s	sys 0.000s
I20260709 12:20:58.595371 27730 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:58.597013  2178 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.597461  2178 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:58.597867  2178 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.577378  2229 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:58.599714  2229 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.600064  2229 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:58.573614  2382 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.601433  2371 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:58.602689  2169 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.603418  2378 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:58.607175 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:58.610932  1689 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.615692 28329 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:20:58.615646 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.616528 27658 raft_consensus.cc:2376] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24. Re-sending same reply.
I20260709 12:20:58.617400 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.634680 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.643545 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:20:58.644570 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.649152 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.633118  2178 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.663614 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.664927 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.665077 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.665489 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:58.617059  2003 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:58.654790  2383 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:58.667459 27656 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:58.667589  2383 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.668324 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.668898 27656 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:58.669788 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.670424 27730 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:20:58.633369  2169 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.670774 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.669816  2383 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.678774 27656 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
W20260709 12:20:58.680809 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64313.7us
I20260709 12:20:58.680886 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:58.650965  2150 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:58.577445  2297 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:20:58.615510  2095 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:20:58.685348 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93419.4us
I20260709 12:20:58.684406 27658 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:58.690689 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.691198 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.691381 28334 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:20:58.687639 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.681632 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.694216 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.696322 27588 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:20:58.697526 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.648442  2185 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:58.665813  2382 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.704528 27658 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:58.705631 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:58.710654 27656 raft_consensus.cc:2368] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:20:58.714720 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 81e6b2dabf8f4e6cb549ca250430e568 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:20:58.719782 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.720510 27737 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:20:58.706362  2150 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:58.721570  2394 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:20:58.721799 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.721812  2150 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.723179 27736 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:20:58.723418 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.725135 27658 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:20:58.724103 27729 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:20:58.732012 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.733639 27735 raft_consensus.cc:2368] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18.
I20260709 12:20:58.730697 27583 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:58.701326  2388 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:58.738184 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:58.738803 28329 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:58.738181 28334 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:20:58.700667  2169 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:20:58.724953  2394 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.713874  2003 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:58.741137 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.754705  2401 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:20:58.758306 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.699776  2178 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
W20260709 12:20:58.767966 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:58.768483 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.736332  2390 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:58.770684 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.771021 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.797857  2400 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:20:58.804291 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:58.804328 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63281.6us
I20260709 12:20:58.751708  2394 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:58.807719 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.743232  2376 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.764127 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:58.827188 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:58.829603 28335 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:20:58.763082  2388 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:58.740756  2383 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.065s
W20260709 12:20:58.831233 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69732.7us
I20260709 12:20:58.836395 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:58.838624 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.831384 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:58.831966 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.804474  2135 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.842542  2397 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:58.843142  2397 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:58.842715  2135 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.844129  2397 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.845108  2135 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.845872  2384 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:20:58.847347 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.848066 27736 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17. Re-sending same reply.
I20260709 12:20:58.848250 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:58.848995 27658 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:58.851358 28335 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:20:58.851855 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80965.4us
I20260709 12:20:58.852988 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:58.853564 27655 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:20:58.804749  2371 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:58.745874  2185 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.855660  2376 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:20:58.864878  2376 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:58.865214  2283 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.856254  2382 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:58.878273 27658 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:20:58.881422 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.882279 27583 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:58.882834  2176 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.884122  2095 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.885015 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.889292  2372 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.890498 27731 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 19. Ops: []
I20260709 12:20:58.890702 27655 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:58.891470  2283 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.887183 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:20:58.895035 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.885227  2176 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:58.857340  2393 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:20:58.896656 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:58.856451  2202 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:58.897140  2178 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:20:58.897729 27583 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:20:58.882969  2229 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.897624  2202 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:58.897943 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.909126 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:58.898202 28329 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:20:58.913529 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:58.921600  2372 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:58.931267 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 123615us
I20260709 12:20:58.932453  2393 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:20:58.933393 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.934051 27736 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:20:58.934943 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.935055  2388 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:58.935647 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.936456 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.936934  2229 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:58.939087 27586 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:20:58.864305  2383 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:58.940280 27730 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:58.817023  1859 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.957509  2387 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:20:58.957957  2387 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.958348  2387 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:58.959616 27663 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:58.856702  2371 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:58.961201  2383 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:58.962477  2371 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.962219  2383 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.963214  2003 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.959085  1859 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:58.944330 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 264016us
I20260709 12:20:58.945809 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.945967 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.983155 27737 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:20:58.946359  2381 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:20:58.945969 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.951689 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:58.952425 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:58.953322 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55449.4us
I20260709 12:20:58.953917 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:58.955029 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.955360  2375 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:20:58.988811 27730 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:20:58.955754  2384 raft_consensus.cc:2764] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:20:58.837741  2396 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.103s
I20260709 12:20:58.993734  1913 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.996536 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:58.998733  1913 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:58.807650  2390 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.865808  2376 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.981516 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:58.983031 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:58.991699 27729 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:20:58.986559  2135 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:59.008981  2135 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:58.996874  2396 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:20:59.011490  2396 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.011945 28328 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:20:59.012514  2396 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.013185  1913 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:20:59.013827  2396 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:20:59.014369  2375 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:58.956580  2382 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:20:59.015612  2382 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.015573  2375 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.016577  2382 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.017994  2382 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:58.998853 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:58.994634 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:59.020035 27730 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:20:59.020147  2135 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:59.002873 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:59.006654 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51730.5us
I20260709 12:20:59.007452  2385 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.008104 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.033530 27737 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:20:59.000208  2401 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:20:59.002205  2371 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:59.079808 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:59.002557  2381 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:59.034564  2135 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:20:59.084249  2399 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.285s	user 0.004s	sys 0.000s
I20260709 12:20:59.085738  2030 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:59.080964 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.088500  2100 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:20:59.093060  2390 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:59.094735 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 153139us
W20260709 12:20:59.110706 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 166487us
I20260709 12:20:59.097095 27583 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:20:59.098328 27730 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:20:59.021932  2229 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.102484 28292 leader_election.cc:400] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18.
I20260709 12:20:59.104362 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.106487 28328 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
W20260709 12:20:59.106376 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73246.4us
I20260709 12:20:59.095417  2169 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:20:59.117141 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.120885 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.077147  2401 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:59.119136 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:59.120211 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.099763  2135 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:59.121723 27730 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:20:59.127031 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:59.128082 27729 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:59.109395  2388 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:20:59.108003  2030 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:20:59.130107 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.109737  2100 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:59.137678 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1196) took 1321 ms. Trace:
W20260709 12:20:59.157567 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.138106 27659 rpcz_store.cc:276] 0709 12:20:57.807469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:57.808372 (+   903us) service_pool.cc:224] Handling call
0709 12:20:59.125949 (+1317577us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:59.128454 (+  2505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1403009920}
I20260709 12:20:59.169749 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.170271 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:59.172135 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:59.074254  2385 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:20:59.173034 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:59.173593 27729 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:20:59.074846  2202 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.121106  2169 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:59.175820 27730 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:20:59.126988  2401 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:20:59.129145  2135 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.181384 27737 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 14. Current term is 15. Ops: []
I20260709 12:20:59.131927  2388 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:59.175968 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:59.178200 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:59.176676  2403 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:20:59.183034 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:20:59.131695  2288 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:20:59.159487  2390 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:59.174144  2376 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:59.120545  1859 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:20:59.194027 28328 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:59.189806 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:59.124359  2229 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.177531  2371 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:20:59.197932 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70960.2us
I20260709 12:20:59.199682 28336 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.193447 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:20:59.201558 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.183795  2169 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:20:59.208086 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.211509 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.189499  2176 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:59.190217  2388 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:59.192657  2380 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:20:59.222083  2175 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:59.219089  2384 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:59.173998  2375 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
W20260709 12:20:59.223807 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 112943us
I20260709 12:20:59.226141  2394 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.202566  2288 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:20:59.227649  2030 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.228319  2288 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.229038  2030 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.180080  2397 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:20:59.219376  2371 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.232468  2371 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.202590  2178 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:59.194375  2283 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:59.195129  2376 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:20:59.275441 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58311.2us
W20260709 12:20:59.249287 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.324150 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47379.5us
W20260709 12:20:59.297386 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86033.2us
W20260709 12:20:59.329005 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 121029us
W20260709 12:20:59.333544 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.237324  2297 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.208732  2405 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:59.192703  2381 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.336424 28329 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:59.237708  2202 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.250615  2397 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:20:59.249238  2403 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:20:59.248878  2178 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:59.331517  2378 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.004s	sys 0.013s
I20260709 12:20:59.276052  2376 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.274816  2380 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:20:59.243464  2176 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.275609  2372 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:20:59.342617  1704 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.004s
W20260709 12:20:59.347847  2413 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
W20260709 12:20:59.348663  2419 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:20:59.349134  2262 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:59.349503  2421 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:20:59.350157  2410 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.251s	user 0.001s	sys 0.000s
I20260709 12:20:59.352046 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2696) took 1302 ms. Trace:
I20260709 12:20:59.417616 27657 rpcz_store.cc:276] 0709 12:20:58.049428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.051460 (+  2032us) service_pool.cc:224] Handling call
0709 12:20:59.351896 (+1300436us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:20:59.351989 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125805952}
I20260709 12:20:59.354234  2100 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.355013  2229 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.356381 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:59.335642  2385 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.357714 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:20:59.439750 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82085.7us
I20260709 12:20:59.341645  2297 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:59.389029 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.440816 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.406054  2289 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.395928  2176 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:59.355803  2378 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.005s	sys 0.014s
I20260709 12:20:59.402645  2393 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:59.339459  2405 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:59.452816 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.277652  2283 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.388616 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:59.356159  2178 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:59.446410 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 100712us
I20260709 12:20:59.409705  2403 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.294961  2175 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:20:59.459391 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.444550  1913 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:59.453001  2393 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.463243 27585 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:20:59.463455 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:59.472224 27582 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:20:59.467290 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.449506  2372 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:20:59.476017  2388 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.479688 28335 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:20:59.473050 27584 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:20:59.459221  2428 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:20:59.484452 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.483487 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:59.485715  2273 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.486265  2273 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.488875  2393 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.449452  2289 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.490239  2393 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.491422  1913 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.473601  2100 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:20:59.492295  2420 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:20:59.485134 27585 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:20:59.454679  2405 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:20:59.506337 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98038.5us
I20260709 12:20:59.464584  2394 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:59.508545 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.509423 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.492846  1913 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.472066  2397 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:20:59.513342 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54072us
I20260709 12:20:59.514133 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:59.516362 27701 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.106s (SENT)
I20260709 12:20:59.516850 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.483422  1704 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.524758 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:59.525025 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.470762  2378 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:20:59.526319 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.526921 27734 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:20:59.527755 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.516043  2262 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:20:59.536672 27738 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 23. Ops: []
I20260709 12:20:59.512820 27729 raft_consensus.cc:2955] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Committing config change with OpId 19.12: config changed from index 10 to 12, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:20:59.547169 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.508504  2390 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:20:59.522526  2436 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:20:59.527885  2371 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:20:59.542357  2387 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.584s	user 0.004s	sys 0.009s
I20260709 12:20:59.568192 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:59.569042 27660 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:59.566350 28336 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
W20260709 12:20:59.580215 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72791us
I20260709 12:20:59.580082 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:59.555011  2378 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:20:59.536160  1704 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.509245  2380 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.608851 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:59.592707 27580 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:20:59.607523 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:59.573318  2436 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.612156 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:20:59.557353  2403 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:59.622540 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.591852  2378 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.611960  2428 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.632776 27586 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:59.620198  2394 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:20:59.633715 27586 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:20:59.575161  2387 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.622087  2388 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.639024 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:59.576715  2289 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:59.623401  2273 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:59.648281 27586 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:20:59.650693  2289 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:59.604310  2397 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.653720  2412 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:20:59.654202  2412 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:59.654561  2412 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.660487  2442 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.661078  2442 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.661799 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.662397 27733 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:20:59.651376  2297 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:59.650918  2425 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.635131  2376 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.638424  2423 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.670634 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:20:59.683125 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49671.1us
I20260709 12:20:59.679963 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:20:59.687011 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:59.693341 27662 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:20:59.693634 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.667450  2169 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.671528  2376 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:59.696909 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.665939  2415 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.429s	user 0.000s	sys 0.002s
I20260709 12:20:59.697767  2169 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:59.668049  2392 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.020s	user 0.009s	sys 0.000s
I20260709 12:20:59.699453 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.699637 27735 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:20:59.675700  2372 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:59.701025 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:59.699640 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60743.7us
I20260709 12:20:59.701218  2372 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.702837 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:59.699489 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3399) took 1365 ms. Trace:
I20260709 12:20:59.694854 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:20:59.704787 27733 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:20:59.706338 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:20:59.707104 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.704209 27587 rpcz_store.cc:276] 0709 12:20:58.333686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.335592 (+  1906us) service_pool.cc:224] Handling call
0709 12:20:58.383109 (+ 47517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:58.399699 (+ 16590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:58.399728 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:58.411614 (+ 11886us) log.cc:844] Serialized 26 byte log entry
0709 12:20:58.609371 (+197757us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b1800482440. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:20:58.647260 (+ 37889us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:20:58.647306 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:58.647396 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:59.667362 (+1019966us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:20:59.699268 (+ 31906us) raft_consensus.cc:1685] finished
0709 12:20:59.699288 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:59.699428 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4885,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":5329,"spinlock_wait_cycles":82958848,"thread_start_us":37704,"threads_started":1,"wal-append.queue_time_us":235604}
I20260709 12:20:59.705391 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:20:59.629148  2375 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:20:59.712282 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.714648 27736 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:20:59.700618 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:20:59.720037 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88524us
W20260709 12:20:59.663730  2431 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:20:59.625717  2386 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.138s
I20260709 12:20:59.723202 27732 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:20:59.658711  2401 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.645938  2390 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:20:59.722579 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:59.682211  2283 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.729185 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:59.730000 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:59.663298  2442 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.709566  2202 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:20:59.666630  1704 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:59.736982 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:59.751453 28332 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
I20260709 12:20:59.738062 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:59.738699 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:59.738919 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.739722 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.743456 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:20:59.752978 27658 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:59.668676  2030 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 12:20:59.720065  2446 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:20:59.672510  2135 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.742913  2100 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:20:59.672806  2423 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.757315 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:59.758563 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:59.754249 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:59.760106 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.756392 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.764447  2100 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:59.766256 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1159) took 1050 ms. Trace:
I20260709 12:20:59.776741 27731 rpcz_store.cc:276] 0709 12:20:58.715813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.718930 (+  3117us) service_pool.cc:224] Handling call
0709 12:20:58.719251 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:58.890893 (+171642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:59.765978 (+875085us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:20:59.766200 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":393339776}
I20260709 12:20:59.767716  2435 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:59.768224  2437 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:20:59.760661  1913 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:20:59.768638 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:59.774775 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.701216  2412 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.745579  2289 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:20:59.782440 27655 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 25 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
W20260709 12:20:59.797821 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95055.8us
I20260709 12:20:59.768532  2402 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:20:59.793624 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:20:59.802841 27657 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:59.757462  2390 consensus_queue.cc:1243] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 17.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:20:59.804289 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:20:59.794163 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 88828.7us
I20260709 12:20:59.805220 27655 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:20:59.805467 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:59.764114  1704 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:20:59.764639  2352 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:20:59.801380 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.783293  2100 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:20:59.709468  2378 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.722278  2428 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.809291 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:20:59.821133 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:20:59.814218 27731 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:20:59.809803 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.776744  2446 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.777549  2372 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:20:59.826437 27664 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.816033 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:20:59.829703 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.830102 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.792459  2030 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:20:59.725210  2436 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.830936 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:20:59.837936 27663 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:20:59.811233  1704 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.827584  1913 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:20:59.841503 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.813621  2289 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.815369  2402 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.843935  2451 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:20:59.848173  2042 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:20:59.847949  2446 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.849861  2401 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.814028  2352 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:20:59.850219 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:20:59.850808  2333 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:20:59.851866  1704 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:20:59.853832  2442 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:20:59.861438 27657 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:20:59.863961 27731 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:20:59.956010 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:20:59.905514  2100 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:20:59.915793  2396 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.901s	user 0.007s	sys 0.006s
I20260709 12:20:59.913447  2402 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:20:59.900229  2412 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:20:59.891592 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.898937  2425 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:20:59.914002  2447 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:20:59.880496  2273 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
I20260709 12:20:59.893682  2385 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:20:59.878846  2445 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.920953 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:00.005774 27730 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:20:59.892184  2433 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:20:59.885725  2438 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.251s	user 0.002s	sys 0.000s
I20260709 12:20:59.906239 27584 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 8. Current term is 9. Ops: []
I20260709 12:20:59.901602 27582 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:20:59.907689  2144 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:20:59.922675 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:20:59.926028  2376 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:20:59.894817 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:20:59.904155  2169 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:20:59.886111 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.924402 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.911180  2003 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:00.016608  2178 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.536s	user 0.010s	sys 0.002s
I20260709 12:21:00.016737  2003 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.017207  2178 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.019280  2003 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:20:59.877480  2451 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:00.020516  2451 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.021749  2451 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:00.024394 27736 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:21:00.025864 27736 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:21:00.026228 27736 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:20:59.922962  2405 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:20:59.998544  2442 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.030812  2405 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:59.972143 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 260890us
I20260709 12:20:59.978667 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:20:59.988808 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.032179 27589 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:21:00.034608  2383 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:21:00.036865  2455 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:20:59.972445  2135 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:20:59.993597 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 225041us
W20260709 12:20:59.994493 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.027060 27736 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.040987 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:00.041772 27586 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:00.042479 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:00.044981 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194837us
I20260709 12:21:00.045734 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.047549  2435 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:00.031005  2442 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:20:59.998647 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.001504 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:20:59.999647  1704 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:20:59.997761 27583 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:00.050267 27587 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:00.055023  2405 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.055756  2405 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.058732  2423 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.053695 27586 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:00.061030 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:00.061774 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:00.062366 27586 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:00.062925 27730 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:21:00.013852 27738 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:20:59.999007  2333 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:00.040570 27736 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:00.063500 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.003684 28329 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:00.047047 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.058866  2405 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.002293  2385 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:00.065531 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:00.073765 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:00.068956 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:00.065933 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 318631us
I20260709 12:21:00.078128 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:00.105077 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:00.103591 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:00.109000 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:00.116619 27586 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:00.118407  2405 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:21:00.119418  2003 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:00.119549  2394 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:00.119809 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.120107  2457 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:21:00.121209  2100 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:00.121647  2273 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:00.122323  1859 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:00.015296  2352 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.118695 27658 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:21:00.110590  2380 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:00.062460  2376 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:21:00.065230  2178 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:00.126176 27658 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:21:00.074682  2435 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:00.114928 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.014326  2144 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:00.127450 27701 leader_election.cc:400] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:00.122028 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.126753 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.130671 27657 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:00.132087 27655 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:00.134087 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.134274 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.109314  2442 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:00.135104 27663 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:00.138803 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.136415 27660 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:21:00.106801  2383 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:00.128113  2428 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:00.136066 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:00.119917  2003 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.066172  2169 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.122339  2273 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:00.127564  2376 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:00.173413  2463 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:21:00.173692 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.174907 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.175542 27657 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:00.171502  2176 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:00.149462  2385 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.177135  2385 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:00.178140  2144 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.143335  2333 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:21:00.147464  2445 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:21:00.150302  2393 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.660s	user 0.005s	sys 0.008s
I20260709 12:21:00.138830  2442 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:00.152333 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.151437 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:00.184446 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60243us
I20260709 12:21:00.176465 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.188900 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:00.191243 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:00.199193 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.199011 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:00.200309 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:00.200851 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.157065  2428 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:00.198935 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:00.171523  2383 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:00.172205  2435 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:00.145818  2457 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:00.208055 27657 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:00.215148 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.216209 27657 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:00.216527 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.216972 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.217172 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.217787 27733 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:00.217738 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:00.218564 27738 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 12:21:00.218163 27657 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:00.218945 27738 pending_rounds.cc:85] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 12
I20260709 12:21:00.219236 27738 pending_rounds.cc:107] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:21:00.219281 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.219672 27738 raft_consensus.cc:2889] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:00.190116  2442 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:00.220350  2333 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:00.220885  2333 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.217845 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.221963 27737 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:21:00.223098  2333 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.224095 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.224831  2393 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:00.225227 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.225522  2393 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:00.146500  2003 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:00.217706  1704 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:00.210402 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:00.228441 28331 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:00.209403 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.227102 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.225728 27657 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:00.199535  2100 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:21:00.231133 28328 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:21:00.232785 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:00.232856 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.236405 27658 raft_consensus.cc:2376] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21. Re-sending same reply.
I20260709 12:21:00.244712 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.245517 27657 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:00.246630 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.247606 27657 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:00.261896 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.231611  2135 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:00.265836 27584 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:00.266548  2169 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.239069  2446 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:00.273051 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.250478  2372 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:00.264096  2273 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.328096  2185 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:00.328580  2372 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.328723  2185 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:00.329217  2372 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.279364  2449 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:00.330441  2449 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:00.330052  2185 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.331525  2372 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.331398  2449 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.261425  2444 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.333660  2444 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.228147  2376 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.335753  2444 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:00.280401  2458 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.335922  2376 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:21:00.336138 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184777us
I20260709 12:21:00.336447  2458 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.337468 27552 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:00.337754 27552 leader_election.cc:403] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:21:00.338626  2458 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.338688 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.283385 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3182) took 1335 ms. Trace:
I20260709 12:21:00.292841 27657 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:00.339672  2457 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:00.340937 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.339315  2458 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:00.340204 27735 rpcz_store.cc:276] 0709 12:20:58.948162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.948266 (+   104us) service_pool.cc:224] Handling call
0709 12:20:59.699281 (+751015us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:59.700022 (+   741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.283159 (+583137us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b54002e81d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:00.283309 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1726329728}
I20260709 12:21:00.343544 27657 raft_consensus.cc:2376] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19. Re-sending same reply.
I20260709 12:21:00.348002 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.348551 27657 raft_consensus.cc:2376] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19. Re-sending same reply.
I20260709 12:21:00.349388 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.350436 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.350961 27657 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:00.351661 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:00.353735 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.354327 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.355731 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:00.357100 27736 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:21:00.360847 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:00.355198 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:00.362054 27738 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:00.348002 28332 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:21:00.364635 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.365453 27730 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:00.294472  2435 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:00.295158 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:00.368187 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73081.4us
I20260709 12:21:00.302376  2003 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.254961  2100 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:00.311478  2428 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.319468  2383 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:00.324400  2420 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:21:00.254782  2388 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.378088 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:00.418876 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 80265.6us
I20260709 12:21:00.420073  2169 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:21:00.420850 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.422240 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.422240 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.425153 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:00.426273 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.427807 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.428500 27658 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:00.425086  2475 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:00.432148 27582 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
W20260709 12:21:00.430430 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49757.2us
I20260709 12:21:00.433763 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:00.432404 27732 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:00.431131 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.434973 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:00.421777  2459 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:00.375522  2333 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:00.441445 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.441955 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.422855  2443 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:00.457944 27731 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:21:00.428651  2384 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:21:00.381016  2185 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:00.456051 27738 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:00.425753  2283 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:00.463989 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:00.432008  2475 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.442958  2435 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:21:00.450001  1689 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:21:00.429540  1704 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:21:00.485435 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.485725 28330 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
W20260709 12:21:00.486217 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44870.7us
I20260709 12:21:00.445972  2433 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:00.450199  1859 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:00.488178 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.431443  2388 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.456406  2380 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:00.498762 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.509850 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.511256 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.512768 27658 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:00.500430 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.473016  2436 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:21:00.514820  2100 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.473587  2475 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:00.521452  2289 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.522341 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.523082  2390 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:00.523512  2476 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:00.524801 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:00.518285 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:00.526319 27661 raft_consensus.cc:606] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.526705 27661 raft_consensus.cc:614] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:00.529803  2178 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:00.440862  2229 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 12:21:00.515003 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.450923  2169 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.484850  2003 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:00.533628 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62655.9us
I20260709 12:21:00.536320 28337 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:21:00.538107 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.485697  2283 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.548924  2388 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:00.549726  2283 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.485728  2405 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:21:00.551064  2405 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:00.551443  2405 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:00.552320  2405 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.553615  2405 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:00.501443  2382 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:00.536875 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:00.531122  2427 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:21:00.474622  2396 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 12:21:00.501502  1859 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:00.485016  2442 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.563453 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.587949  2386 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:00.556906  2169 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.572341  2402 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:21:00.588521  2289 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.593551 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:00.601596 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.536444  2178 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:00.591130  2478 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:00.572942  1859 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:00.573858  2449 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:00.605353 27586 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:00.609643 27657 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:00.609696 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.612619 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.615001  2412 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:00.632285  2428 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:00.632622 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.639509  2396 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:00.642022  2169 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.600181  2386 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.605540  2402 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:00.616379  2384 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:00.643920 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.672377 27580 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:00.673321 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.642071 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.674271 27588 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:00.675069 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.675770 27588 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 9: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:00.676414 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.613052  2382 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:00.679410  2442 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.683578 27732 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 16. Current term is 17. Ops: []
W20260709 12:21:00.651571 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.603188  2289 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.604694  2192 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:00.686525  2192 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.655974  2396 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.687512  2289 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.688293  2402 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.690301 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.690901 27731 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:00.688851  2396 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.692756  2386 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.693437  2178 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.694895 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.696162 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.655004  2428 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.670054 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.697209 27731 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:00.698149  2192 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.699273 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.671262 27587 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:00.699875 27664 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:00.676915 27588 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:00.678900 27580 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:21:00.700995 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.702184  2412 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.730507  2378 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:00.733918 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.734844 27663 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:21:00.735735 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.702698  2412 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.738018  2412 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:00.738806  2185 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.739390 27656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:00.739462  2185 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.742652  2185 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:00.744441 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.745146 27655 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:00.745877  2352 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:00.748157 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.750546  2095 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.751037  2095 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.752383 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:00.753142  2095 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.753669 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.754266 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.754935 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:00.755725 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:00.760833 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 9.
I20260709 12:21:00.758247 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:00.759311 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:00.759111 27733 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:21:00.763252 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.757665 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.762814 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:00.764683 27657 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:00.755290 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:00.772400 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.773653 27580 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
W20260709 12:21:00.775518 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.776175 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:00.776582 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:00.778364 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.756254  2442 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:21:00.755908  2403 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:21:00.780390  2428 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.757440  2387 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:00.781106  2169 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:00.781805  2169 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:00.756043  2425 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:00.784860 27731 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:21:00.785079  2144 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:00.787427  2380 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:00.787971  2380 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:00.788729 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.789229 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.790349 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:00.791043 27658 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:00.773169 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.796878 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.801054 27664 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:00.801175 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:00.744258  2273 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.805284 27664 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:21:00.806644 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:00.802917 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:00.771358  2427 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.813637  2428 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:00.814013  2427 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.817209 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.818504 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:00.819443  2403 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:00.820071  2403 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:00.820418 27736 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:00.821038  2403 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.822731 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.773411  2475 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.824123 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.824635 28331 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:00.824777 27589 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:00.825695  2475 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:00.782852  2169 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.828089 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2719) took 1085 ms. Trace:
I20260709 12:21:00.828480 27659 rpcz_store.cc:276] 0709 12:20:59.742466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:59.742580 (+   114us) service_pool.cc:224] Handling call
0709 12:20:59.754365 (+ 11785us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:59.799588 (+ 45223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:59.799618 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:59.809851 (+ 10233us) log.cc:844] Serialized 26 byte log entry
0709 12:21:00.186515 (+376664us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b180038e900. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:21:00.448663 (+262148us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:00.448703 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.827809 (+379106us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:00.827856 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:00.827898 (+    42us) raft_consensus.cc:1685] finished
0709 12:21:00.827911 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:00.828034 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2487,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":3790,"spinlock_wait_cycles":110679808,"thread_start_us":261961,"threads_started":1,"wal-append.queue_time_us":706327}
I20260709 12:21:00.830775 27659 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:00.830837 27658 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:00.759508  2150 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:00.760092  2289 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:00.832079 27658 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:21:00.832098  2289 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.833599 27658 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:00.834420  2297 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:00.834703 27658 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:00.835156 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1140) took 1104 ms. Trace:
I20260709 12:21:00.835578 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:00.836024 27658 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:00.779142  2371 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:00.755755  2376 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:00.837852 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:00.838248 27658 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 18. (term mismatch)
I20260709 12:21:00.838356 27737 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:00.841699 27732 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:21:00.835520 28334 rpcz_store.cc:276] 0709 12:20:59.730503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:59.730630 (+   127us) service_pool.cc:224] Handling call
0709 12:20:59.744156 (+ 13526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:59.900743 (+156587us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:20:59.900773 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:59.900814 (+    41us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:20:59.900843 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.834909 (+934066us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:00.834951 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:00.835058 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339057920}
I20260709 12:21:00.842959 27658 pending_rounds.cc:85] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 17
I20260709 12:21:00.843453 27658 pending_rounds.cc:107] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 22.18
I20260709 12:21:00.844455 27658 raft_consensus.cc:2889] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:00.798420  2042 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:21:00.798736  2405 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:00.847435 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:00.853897 27735 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:00.848837 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.940407 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1139) took 1210 ms. Trace:
I20260709 12:21:00.811563  2442 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:21:00.812875  2386 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:00.949156 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:00.840657  2371 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.957183 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:00.957930 27587 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:00.943511 28330 rpcz_store.cc:276] 0709 12:20:59.730208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:59.730431 (+   223us) service_pool.cc:224] Handling call
0709 12:20:59.739755 (+  9324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:00.486463 (+746708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.940145 (+453682us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b54002fab54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:00.940346 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1715083136}
I20260709 12:21:00.956844 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:00.965788 27729 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:00.840495  2376 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:00.969400 27580 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:00.803750  2100 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.136s
I20260709 12:21:00.847458  2042 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:00.834678  2289 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:00.923254 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.865227 27657 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:00.947873  2397 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.837466  2273 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:00.987185 27659 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:21:00.990190 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:00.963542  2371 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:00.997288 27735 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:00.998034 27659 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:00.998817 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:01.003374  2003 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:01.007042  2273 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:01.003521  2404 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:01.008088  2404 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:01.009217  2404 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:00.963604  2405 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:01.011063  2003 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.971884  2435 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Committing config change with OpId 17.13: config changed from index 10 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:00.941179  2475 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:21:01.026341 27587 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:00.982755  2376 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:01.048470 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:01.032353 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 242033us
I20260709 12:21:01.060678 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:01.067056 27730 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:21:01.069675 27662 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:01.046648  2443 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:21:01.068135 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:01.078212 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:01.079236 27588 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:01.059791  2436 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:21:01.087531 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.088315 27588 raft_consensus.cc:2376] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20. Re-sending same reply.
I20260709 12:21:01.092075 27662 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:01.044332  2371 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.093338  1689 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:00.997804  2397 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.096899 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.020232  1859 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.011300  2405 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.053489  2396 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:21:01.101997  2436 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:21:01.011678  2003 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.170637 27662 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:21:01.106374 27663 raft_consensus.cc:2410] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:21:01.107442 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.070564  2497 raft_consensus.cc:2764] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:21:01.084685  2178 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.085093  2273 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:01.011723  2042 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.181625  2397 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:01.184872  2425 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:01.186383 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:01.166350  2185 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.177584  2396 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 19.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:21:01.178066  2273 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.225693 27586 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:01.247383 27586 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:01.221812  2412 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:01.231969  2273 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:01.195252  2178 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.210249  2185 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.282447  2185 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:01.249233  2425 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:01.284493  2425 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.274169  2384 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.312839  2384 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.251019  2380 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.259586  2412 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:01.329283 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 243992us
I20260709 12:21:01.340895 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.315045  2425 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:01.347970 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.354116 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.345275  2371 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:01.351747  2352 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.605s	user 0.005s	sys 0.008s
I20260709 12:21:01.359002 28335 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:21:01.371372  2380 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:01.372695 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:01.373661 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.374287 27583 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:21:01.383950  2504 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:21:01.355650  1859 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:01.407800 27664 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:01.398039 27657 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:01.409060 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:01.375005 27580 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Committing config change with OpId 17.13: config changed from index 10 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:01.391665 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:01.409744 27664 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:21:01.420709  2376 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.426424 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:01.405406  2384 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:01.428898 27588 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260709 12:21:01.429273 27588 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:01.429487 27588 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260709 12:21:01.429364 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:01.429857 27588 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:01.430135 27657 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 8.
I20260709 12:21:01.391566  2003 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:01.450079 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58487.9us
I20260709 12:21:01.461403  2273 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.423144  2376 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:01.469329  2386 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:01.478209  2386 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:21:01.479461 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:01.471133  2507 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:21:01.470378  2505 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:21:01.356969 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.483952 27657 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:21:01.485101  2273 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:01.496069  2178 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:01.465497  1859 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:01.507824 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:01.517513 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1037) took 1333 ms. Trace:
I20260709 12:21:01.517894 28335 rpcz_store.cc:276] 0709 12:21:00.183689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.184216 (+   527us) service_pool.cc:224] Handling call
0709 12:21:00.184589 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:01.369685 (+1185096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.517268 (+147583us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54005eabd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:01.517459 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2700047232}
I20260709 12:21:01.516924 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2734) took 1078 ms. Trace:
I20260709 12:21:01.521219 27658 rpcz_store.cc:276] 0709 12:21:00.438879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.939884 (+501005us) service_pool.cc:224] Handling call
0709 12:21:00.940277 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:00.943299 (+  3022us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:00.943368 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:00.968029 (+ 24661us) log.cc:844] Serialized 26 byte log entry
0709 12:21:01.247044 (+279015us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:01.247093 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.247170 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:01.516726 (+269556us) raft_consensus.cc:1685] finished
0709 12:21:01.516747 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:01.516867 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2628,"raft.queue_time_us":27229,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":13179,"thread_start_us":216574,"threads_started":1,"wal-append.queue_time_us":335853}
I20260709 12:21:01.509779  1859 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:01.534668  2435 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.518s	user 0.008s	sys 0.005s
I20260709 12:21:01.538647  1859 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:01.550446 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.545223 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1038) took 1361 ms. Trace:
W20260709 12:21:01.552907 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.551527 28332 rpcz_store.cc:276] 0709 12:21:00.183970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.184103 (+   133us) service_pool.cc:224] Handling call
0709 12:21:00.196211 (+ 12108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:00.363182 (+166971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.544648 (+1181466us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540024a254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:01.545136 (+   488us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":348005248}
W20260709 12:21:01.554926 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:01.682019  2516 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:21:01.701619 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.634195  2486 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.743314 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:01.793282 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:01.794251 27580 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:01.796343 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.773365  2375 raft_consensus.cc:2764] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:21:01.770581  2273 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:01.779189  2003 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:01.815047 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.851743 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3424) took 1802 ms. Trace:
I20260709 12:21:01.782948  2436 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:01.734194  2486 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.852408 27582 rpcz_store.cc:276] 0709 12:21:00.049138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.106431 (+ 57293us) service_pool.cc:224] Handling call
0709 12:21:00.122389 (+ 15958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:00.437257 (+314868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.851463 (+1414206us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001883d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:01.851672 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":708282624}
I20260709 12:21:01.773870  2371 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
W20260709 12:21:01.866032 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.866801  2405 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:01.871193  2427 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:01.872611 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.873796 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.874414  2398 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:01.910063  2398 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:01.877779 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3207) took 1195 ms. Trace:
W20260709 12:21:01.881489  2520 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:21:01.910877 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.912693 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.910928 27738 rpcz_store.cc:276] 0709 12:21:00.682190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.682694 (+   504us) service_pool.cc:224] Handling call
0709 12:21:00.682976 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:01.875838 (+1192862us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:01.875869 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:01.875910 (+    41us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:01.875942 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.877557 (+  1615us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:01.877601 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:01.877722 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2742823296}
I20260709 12:21:01.882370  2178 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:01.918358  2178 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.919740  2405 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.921924 27588 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:01.923951 27588 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:01.924243 27588 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:01.925103 27588 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.925992 27588 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:01.934273  2042 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:01.936512  2375 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.937336  2375 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.908910  2429 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:01.940126  2429 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:01.944100 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.949996 27588 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:21:01.950423 27588 pending_rounds.cc:85] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 11
I20260709 12:21:01.950659 27588 pending_rounds.cc:107] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:21:01.950879 27588 raft_consensus.cc:2889] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:01.951157 27657 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:21:01.951936 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:01.952282  2393 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:01.952790  2393 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.859758  2003 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.957468 27589 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:21:01.885133  2436 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.893021  2443 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:01.961854  2443 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:01.894085  2444 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:21:01.964231  2444 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:01.964711  2444 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:01.965554  2511 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.182s	user 0.000s	sys 0.000s
I20260709 12:21:01.965883  2444 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:01.967687  2393 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:01.970000 27584 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:01.970831 27481 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 13 to 18, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:21:01.972543  2501 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.693s	user 0.004s	sys 0.005s
I20260709 12:21:01.972981 27481 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index -1 to 10, term changed from 12 to 19, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 19 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:21:01.974236 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2846) took 1174 ms. Trace:
W20260709 12:21:01.977005  2513 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:21:01.975241 27481 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:01.980002  2515 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:21:01.980834 27481 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:01.906752 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:01.988471 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81850.4us
I20260709 12:21:01.908030 27581 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:01.993228  2382 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:01.993741  2382 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:01.982378 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1267) took 1189 ms (client timeout 1535 ms). Trace:
I20260709 12:21:01.994170  2382 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:01.994496 27586 rpcz_store.cc:269] 0709 12:21:00.792767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.806302 (+ 13535us) service_pool.cc:224] Handling call
0709 12:21:01.982109 (+1175807us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b540011c954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:01.982277 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961990528}
I20260709 12:21:01.995555  2384 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:01.997387  2384 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:01.998349  2514 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:01.999634  2514 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:02.000068  2514 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:02.001056  2514 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.002646  2514 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:02.006577 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.007763  2371 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:02.008285  2371 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:02.009608  2371 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:01.855712  2273 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.013453  2273 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:02.013679  2518 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
I20260709 12:21:02.015986 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3308) took 1237 ms. Trace:
I20260709 12:21:02.016535 27662 rpcz_store.cc:276] 0709 12:21:00.778563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.173905 (+395342us) service_pool.cc:224] Handling call
0709 12:21:02.015051 (+841146us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:21:02.015317 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:02.015566 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:02.015601 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:02.015657 (+    56us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:02.015700 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:02.015772 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:02.015913 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34334464}
I20260709 12:21:02.018633 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2746) took 1218 ms. Trace:
W20260709 12:21:02.024209 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:02.025125 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.986367 27589 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:01.784102  2030 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:21:01.974799 27734 rpcz_store.cc:276] 0709 12:21:00.799447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.801729 (+  2282us) service_pool.cc:224] Handling call
0709 12:21:00.802080 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:00.812165 (+ 10085us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:21:00.812200 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:00.812682 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:21:01.084310 (+271628us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:01.084365 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.084456 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:01.971826 (+887370us) spinlock_profiling.cc:243] Waited 966 us on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:01.973839 (+  2013us) raft_consensus.cc:1685] finished
0709 12:21:01.973869 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:01.974159 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8610,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":2222464,"thread_start_us":247188,"threads_started":1,"wal-append.queue_time_us":397633}
I20260709 12:21:02.007504 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:02.043625 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.044257 27658 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:02.044625 27733 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:21:02.019168 27655 rpcz_store.cc:276] 0709 12:21:00.800456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.337031 (+536575us) service_pool.cc:224] Handling call
0709 12:21:01.337435 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:01.367574 (+ 30139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:01.367605 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:01.390580 (+ 22975us) log.cc:844] Serialized 26 byte log entry
0709 12:21:01.518942 (+128362us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:01.518986 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.519055 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:02.012958 (+493903us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:02.018260 (+  5302us) raft_consensus.cc:1685] finished
0709 12:21:02.018292 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:02.018560 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40157,"raft.queue_time_us":30317,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":143636864,"thread_start_us":83299,"threads_started":1,"wal-append.queue_time_us":222142}
I20260709 12:21:02.019948  2371 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:02.059752 27734 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:21:02.060212 27735 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:02.067049 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:02.068693 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:02.070348 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:02.071105 27737 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:02.072253 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:02.073041 27660 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:02.074755 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.075623 27657 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:02.077335 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.067991 27731 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:02.086486  2412 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.092057 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.038841  2376 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.094326 27659 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:21:02.099949 27659 pending_rounds.cc:85] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 15
I20260709 12:21:02.100204 27659 pending_rounds.cc:107] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:21:02.100448 27659 raft_consensus.cc:2889] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:02.097154 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:02.101159 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:02.098155 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.099184 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1222) took 1487 ms. Trace:
W20260709 12:21:02.101649 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100400us
I20260709 12:21:02.102654 27655 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:02.101817 27663 rpcz_store.cc:276] 0709 12:21:00.611508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.735468 (+123960us) service_pool.cc:224] Handling call
0709 12:21:02.093568 (+1358100us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:02.099115 (+  5547us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":850186496}
W20260709 12:21:02.106093 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69281.7us
I20260709 12:21:02.051067  2095 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:02.108649  2095 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.109706  2451 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:02.111550  2384 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.112174  2384 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.061748  2192 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.110440  2451 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.062776  2444 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:01.958729  2003 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.090832  2382 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.115372 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:02.100332  2486 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:02.123219  2393 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:02.124675  2192 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.104241  2429 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:01.963088  2443 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.125872  2095 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.128525  2412 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:02.154928 27589 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:02.156216 27660 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:02.138350  2003 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.138756  2135 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:21:02.162003 27663 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:21:02.165710 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:02.166458 27663 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:02.168401 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.171444 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.162695 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.117277  2376 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.173763 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2735) took 1727 ms. Trace:
I20260709 12:21:02.175648 27659 rpcz_store.cc:276] 0709 12:21:00.446603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.028823 (+582220us) service_pool.cc:224] Handling call
0709 12:21:02.093866 (+1065043us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:02.114004 (+ 20138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:02.170345 (+ 56341us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000e51d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:02.173699 (+  3354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2448299904}
I20260709 12:21:02.150681  2435 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.139199  2030 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:21:02.186014 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.157500  2412 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:02.187443 27657 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:21:02.160676  2443 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:02.145288  2393 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.194206 27580 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
W20260709 12:21:02.197268 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.053s	sys 0.109s
W20260709 12:21:02.188702  2522 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.319s	user 0.000s	sys 0.004s
I20260709 12:21:02.173475  2202 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:21:02.174130  2003 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.225550  2393 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:02.231669  2382 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:02.233795  2451 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.234702  2412 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.236588  2412 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:02.236732  2436 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.239424  2384 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:02.189172  2435 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.156708  2429 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:21:02.218891  2486 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:02.265424 27481 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 24. New cstate: current_term: 24 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:02.262627  2486 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.252322 27585 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:21:02.268656  2346 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.263641  2427 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:02.235056  2398 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.246485  2403 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:21:02.272933  2375 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.287919  2486 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:21:02.336201 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:02.308255  2398 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:21:02.347074  2524 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:21:02.306452  2405 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.351641 27481 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index -1 to 11, term changed from 15 to 20, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1), NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 20 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:21:02.290107  2346 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:02.346156  1859 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.794s	user 0.009s	sys 0.005s
I20260709 12:21:02.356144 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:02.302104  2427 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.366277 27481 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 21, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:02.364953  2192 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.369940  1859 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.379868 27658 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:02.381801  1859 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:02.396550 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:02.415755 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.004s	sys 0.008s
W20260709 12:21:02.416270 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.004s	sys 0.008s
I20260709 12:21:02.418035 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.389806  2403 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:02.423609 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.426486 28328 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:21:02.415851  2384 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.391000  2178 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:02.432708 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.010s	sys 0.003s
W20260709 12:21:02.433266 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.010s	sys 0.003s
I20260709 12:21:02.440577 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:02.456068 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.456888 28329 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
W20260709 12:21:02.457865 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:02.460877 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.461683 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 9.
I20260709 12:21:02.422216  2003 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.467038 27481 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 16, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 16 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:02.463022  1859 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:02.438654  2376 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.471897 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.444073  2405 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.444458  2444 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.444204  2435 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.476096 27481 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:02.482465 27481 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:02.485929 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:02.491803  2427 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.434947  2403 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.480578  2429 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:02.494804  2486 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.475678  1859 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.445039  2384 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:02.501037 27730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s	user 0.169s	sys 0.284s
I20260709 12:21:02.508249  2346 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.482412  2435 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:02.516450 27730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.169s	sys 0.285s
I20260709 12:21:02.519416 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:02.527905 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 412573us
I20260709 12:21:02.481400  2405 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.482786  2444 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.530737 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:02.532008 28329 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:21:02.540786 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:02.545890 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:02.548842 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:02.555042 27735 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:02.550096 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:02.521209  2385 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.012s	sys 0.017s
I20260709 12:21:02.516736  2398 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:02.558295 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:02.557952 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:02.560269 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:02.562516 27486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 238) took 61029 ms (client timeout 15000 ms). Trace:
W20260709 12:21:02.563683 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 413632us
I20260709 12:21:02.569703 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:02.570839 28295 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:02.579428 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 430418us
W20260709 12:21:02.580312 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.010s	sys 0.014s
W20260709 12:21:02.580660 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.010s	sys 0.014s
I20260709 12:21:02.590122 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:02.591094 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:02.595803 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:02.598609 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.599102 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:02.599658 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.602895 27738 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:21:02.530948  2435 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.604863 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:02.605316  2435 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.607435  2403 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:02.608385 27738 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:21:02.612227  1859 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.612849  1859 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:02.613219  1859 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:02.613559  1859 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:02.614323  2384 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:02.615815  2273 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.553097  2429 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:02.623311 27583 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:21:02.623816 27583 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:02.624325 27583 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:21:02.624683 27583 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:02.625726 27729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.035s	sys 0.047s
W20260709 12:21:02.626071 27729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.035s	sys 0.047s
W20260709 12:21:02.561657  2385 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.013s	sys 0.018s
I20260709 12:21:02.616344  2273 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.566576  2289 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:02.618402 28328 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:02.633314 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:02.634196 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:02.628716 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57960.2us
W20260709 12:21:02.641717 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50771.9us
I20260709 12:21:02.559193  2333 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:02.635586 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:02.638922 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.665350 27580 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
W20260709 12:21:02.587960  2442 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.005s	sys 0.021s
I20260709 12:21:02.584218  2514 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:02.671455 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:02.675127 27553 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:21:02.675498 27553 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:02.666831 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:02.562968 27486 rpcz_store.cc:269] 0709 12:20:01.532859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:01.533094 (+   235us) service_pool.cc:224] Handling call
0709 12:21:02.562447 (+61029353us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:58.945529 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:58.958930 (+ 13401us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:58.958983 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:20:58.959003 (+    20us) tablet.cc:662] Decoding operations
0709 12:20:58.959634 (+   631us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:58.959744 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:58.959770 (+    26us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:20:58.960230 (+   460us) tablet.cc:701] Row locks acquired
0709 12:20:58.960278 (+    48us) write_op.cc:260] PREPARE: finished
0709 12:20:58.960429 (+   151us) write_op.cc:270] Start()
0709 12:20:58.960583 (+   154us) write_op.cc:276] Timestamp: P: 1783599658960366 usec, L: 0
0709 12:20:58.960647 (+    64us) op_driver.cc:348] REPLICATION: starting
0709 12:20:58.961371 (+   724us) log.cc:844] Serialized 1308 byte log entry
0709 12:20:59.394391 (+433020us) op_driver.cc:464] REPLICATION: finished
0709 12:20:59.507309 (+112918us) write_op.cc:301] APPLY: starting
0709 12:20:59.520719 (+ 13410us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:59.846273 (+325554us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:59.846388 (+   115us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:59.904600 (+ 58212us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:59.904836 (+   236us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0709 12:20:59.904899 (+    63us) write_op.cc:312] APPLY: finished
0709 12:20:59.905185 (+   286us) log.cc:844] Serialized 41 byte log entry
0709 12:20:59.905368 (+   183us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:59.905461 (+    93us) write_op.cc:454] Released schema lock
0709 12:20:59.905517 (+    56us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:59.905927 (+   410us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:59.905955 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":103755,"reported_tablets":51,"spinlock_wait_cycles":22272,"child_traces":[["op",{"apply.queue_time_us":61117,"apply.run_cpu_time_us":10521,"apply.run_wall_time_us":424714,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16667,"num_ops":3,"prepare.queue_time_us":141647,"prepare.run_cpu_time_us":56422,"prepare.run_wall_time_us":177375,"raft.queue_time_us":160354,"raft.run_cpu_time_us":33624,"raft.run_wall_time_us":79129,"replication_time_us":433561,"thread_start_us":457848,"threads_started":4,"wal-append.queue_time_us":235740,"wal-append.run_cpu_time_us":91131,"wal-append.run_wall_time_us":1704509}]]}
I20260709 12:21:02.632943  2003 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:21:02.677210 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.006s	sys 0.037s
W20260709 12:21:02.677527 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.006s	sys 0.037s
W20260709 12:21:02.677855 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 548644us
I20260709 12:21:02.632699  2436 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.640776  2403 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:02.641010  2443 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:02.672049  2442 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.005s	sys 0.021s
I20260709 12:21:02.663841  2289 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:02.671870  2371 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:02.622900  2178 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:21:02.709296 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:02.673736  2429 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.668839  2526 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:02.673175  2404 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.666911  2192 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.674178  2375 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:21:02.722334 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.717679 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:02.725711 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47293.2us
I20260709 12:21:02.705159  2451 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:02.708324  2289 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:02.739488 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.741814 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.691505  2514 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.708534  2371 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.691709  2333 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.709632  2398 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:02.698699  2436 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.711501  2288 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:21:02.708742  2384 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:21:02.749614  2030 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:02.769994  2529 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:02.771982  2273 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:02.773193  2521 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:21:02.737726  2192 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.749363  2135 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
W20260709 12:21:02.777137  2412 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.540s	user 0.008s	sys 0.004s
I20260709 12:21:02.781589 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.784188 27587 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:02.785938 27580 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
W20260709 12:21:02.786290 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 187290us
I20260709 12:21:02.784796 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:02.825990 27580 pending_rounds.cc:85] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 9
I20260709 12:21:02.826315 27580 pending_rounds.cc:107] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:21:02.826566 27580 raft_consensus.cc:2889] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:02.706874  2443 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.790035 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:02.793306 27586 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.001s	sys 0.021s
W20260709 12:21:02.793843 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.018s	sys 0.019s
W20260709 12:21:02.838066 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.018s	sys 0.019s
I20260709 12:21:02.794994 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.737639  2442 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:02.802203 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.751252  2536 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:02.742980  2382 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:02.757835  2537 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:21:02.808501 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:02.786376  2042 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.851s	user 0.006s	sys 0.011s
W20260709 12:21:02.793107  2380 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.004s	sys 0.018s
I20260709 12:21:02.910856 27656 raft_consensus.cc:606] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:02.911006 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.911449 27656 raft_consensus.cc:614] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:02.820757  2486 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:02.782624  2371 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:02.720218  2429 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.783749  2451 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:02.826424 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:02.729640  2003 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:02.828014 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:02.929703 27580 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:02.859333  2435 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:02.730110  2404 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.860703 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:02.807925  2376 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:02.856870  2444 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.869716 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:02.817260 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 142289us
I20260709 12:21:02.953342 28335 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
W20260709 12:21:02.953231 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126874us
I20260709 12:21:02.912276 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:02.912309  2382 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:02.896049  2405 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:02.959730 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.919458 27661 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:21:02.856623  2442 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:02.817050  2030 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:02.914589  2380 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.004s	sys 0.019s
I20260709 12:21:02.858559  2531 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:02.858788  2525 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.103s
W20260709 12:21:02.913269 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82280us
I20260709 12:21:02.959565 28331 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:21:02.953549 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:02.960456 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.906072  2536 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.790019  2398 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:02.977521 28337 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEARNER]: Advancing to term 22
W20260709 12:21:03.017280 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 290344us
I20260709 12:21:02.960938  2547 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.022400 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:02.953934 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2858) took 1851 ms. Trace:
W20260709 12:21:02.986635 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74427.6us
I20260709 12:21:02.992347 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:02.997623 28336 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:21:02.840014 27586 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.001s	sys 0.021s
I20260709 12:21:03.023680 27581 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:21:03.010528 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.014325 28328 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:21:03.013247 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59750us
I20260709 12:21:02.962565  2435 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.029874 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:03.027565 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.030691 27580 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:03.029028 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.031152 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:21:02.900926  2178 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.093s
I20260709 12:21:03.026201 27730 rpcz_store.cc:276] 0709 12:21:01.102648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.103674 (+  1026us) service_pool.cc:224] Handling call
0709 12:21:01.169585 (+ 65911us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:01.520490 (+350905us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:01.520521 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:01.521092 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:21:02.518677 (+997585us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:02.518715 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:02.532689 (+ 13974us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54001d1fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:02.532731 (+    42us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:02.925867 (+393136us) raft_consensus.cc:1685] finished
0709 12:21:02.925889 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:02.953866 (+ 27977us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5670,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":247332736,"thread_start_us":1217326,"threads_started":2,"wal-append.queue_time_us":1130270}
I20260709 12:21:02.991591  2346 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.005095  2003 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:02.998499  2405 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:02.997062  2442 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:03.036047 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.037842 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:03.040328 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 362149us
I20260709 12:21:03.007774  2385 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:21:03.044301 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:03.062297 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.062723 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2863) took 1019 ms. Trace:
I20260709 12:21:03.063488 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:03.012423  2520 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:21:03.035712 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:03.058663 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.064162 28333 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:03.017602  2541 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:03.069777 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.063700 27729 rpcz_store.cc:276] 0709 12:21:02.043226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.043346 (+   120us) service_pool.cc:224] Handling call
0709 12:21:02.080133 (+ 36787us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:02.080431 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:21:02.626393 (+545962us) log.cc:844] Serialized 14 byte log entry
0709 12:21:03.061916 (+435523us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:03.062113 (+   197us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:03.062249 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:03.062382 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:03.062638 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25977,"prepare.queue_time_us":740812,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"thread_start_us":927573,"threads_started":2}
I20260709 12:21:03.017565  2536 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:02.959738  2376 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.015769  2444 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:03.036335  2435 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:03.033950  2375 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:03.087954 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52331.7us
I20260709 12:21:03.044359  2135 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:03.012902  2386 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
W20260709 12:21:03.099020 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69380.2us
I20260709 12:21:03.102849 27580 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:03.050096  2549 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:03.071794  2541 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.016251  2531 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.058864  2346 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.072268  2451 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.078063  2288 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:21:03.067864  2543 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:21:03.035290  2547 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.112277  2549 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.111795  2530 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:03.160832 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 137306us
I20260709 12:21:03.150313  2178 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:03.152038  2554 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:21:03.101657  2533 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:21:03.113809  2514 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.133275  2436 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.110821  2135 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.125993  2541 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:03.128206  2556 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:03.106568  2444 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.109472  2375 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.176553 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:03.159168  2192 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.125604  2538 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.051s
W20260709 12:21:03.179096 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70245.1us
W20260709 12:21:03.183530 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.157497  2385 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:03.169013  2333 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:03.226119 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49643.1us
I20260709 12:21:03.159824  2549 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:03.254276 28335 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:21:03.147056  2555 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
W20260709 12:21:03.232610  2527 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.257s	user 0.001s	sys 0.001s
W20260709 12:21:03.318121 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 194073us
I20260709 12:21:03.268237 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.281632 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.262233 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:03.160197  2289 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.185762  2429 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.187067  2443 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.282400 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.175649  2178 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:03.248194  2554 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:21:03.320986  2548 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.303s	user 0.001s	sys 0.000s
I20260709 12:21:03.321885 27731 pending_rounds.cc:85] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 15
I20260709 12:21:03.336199 27731 pending_rounds.cc:107] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:21:03.336550 27731 raft_consensus.cc:2889] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:03.247910  2404 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.248404  2288 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:03.174299  2530 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:03.325582 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63437.7us
I20260709 12:21:03.327196 27738 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
W20260709 12:21:03.330755 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 350356us
W20260709 12:21:03.324222 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 356097us
I20260709 12:21:03.325327  2030 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.158631  2003 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.259852  2385 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:03.258030  2382 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.349730 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:03.325129 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:03.187527  2556 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.331902  2558 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:21:03.362264 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.011s	sys 0.007s
W20260709 12:21:03.374442 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.011s	sys 0.008s
I20260709 12:21:03.376127 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2855) took 2395 ms. Trace:
W20260709 12:21:03.368625  2559 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 12:21:03.376570 27732 rpcz_store.cc:276] 0709 12:21:00.972044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.977649 (+  5605us) service_pool.cc:224] Handling call
0709 12:21:00.978156 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:00.978647 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:00.978696 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:00.995583 (+ 16887us) log.cc:844] Serialized 26 byte log entry
0709 12:21:01.960402 (+964819us) spinlock_profiling.cc:243] Waited 875 us on lock 0x7b180039fa60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:21:02.838608 (+878206us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:02.838668 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:02.838766 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:03.361535 (+522769us) raft_consensus.cc:1685] finished
0709 12:21:03.361555 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:03.367384 (+  5829us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3557,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":2355,"spinlock_wait_cycles":2046976,"thread_start_us":877965,"threads_started":1,"wal-append.queue_time_us":1834405}
I20260709 12:21:03.340726  2429 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:21:03.385845 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.390666 27658 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
W20260709 12:21:03.388561 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63557.1us
I20260709 12:21:03.385689 27659 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:21:03.394861 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73643.1us
I20260709 12:21:03.366840  2550 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:03.397068 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:03.386242 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:03.399163 27663 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:03.399766 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:03.398128 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.381603  2555 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:03.390764  2192 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:03.413715 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:03.392190  2558 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:03.411882  2552 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 12:21:03.423192 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.424744 28337 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEARNER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:03.425397 27732 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
W20260709 12:21:03.426865 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 317555us
I20260709 12:21:03.413244  2560 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:03.430702 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.432502 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.449657 27663 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:03.451897 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.407995  2550 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.493031  2550 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.426669  2580 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:03.429934  2560 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:03.415457  2532 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260709 12:21:03.454497  2371 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.502s	user 0.008s	sys 0.004s
W20260709 12:21:03.518133 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126318us
I20260709 12:21:03.519649 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:03.422284  2555 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.427452  2385 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.454128  2135 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.524209 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:03.526903 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 127195us
I20260709 12:21:03.527988 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:03.454932  2547 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.455089  2581 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:21:03.518540  2565 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 12:21:03.449816  2538 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.080s
I20260709 12:21:03.513597  2333 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:03.518148  2560 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:03.526189  2564 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:21:03.494417  2580 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.542907  2580 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:03.543176  2585 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:03.545073  2585 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:03.545886  2585 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:03.549059 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1226) took 2153 ms. Trace:
I20260709 12:21:03.549593 27661 rpcz_store.cc:276] 0709 12:21:01.396105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.541874 (+145769us) service_pool.cc:224] Handling call
0709 12:21:01.542223 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:02.982864 (+1440641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:03.548743 (+565879us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:03.548979 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3152876416}
I20260709 12:21:03.531731  2289 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.540215  2547 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:03.590058 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 62207.3us
I20260709 12:21:03.522866  2371 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.540716  2178 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.594229 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:03.594818 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:03.600596 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76448.4us
I20260709 12:21:03.601716 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:03.603291 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.604887 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:03.608151 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.608924 27731 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
W20260709 12:21:03.612112  2575 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.184s	user 0.000s	sys 0.000s
I20260709 12:21:03.613929 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:03.593572  2030 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:03.576412  2567 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:03.623772 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.622933  2288 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:21:03.627799  2371 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.624426  2532 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:03.628786  2532 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:03.629206  2532 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:03.631433  2030 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.632983  2555 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.633486  2030 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.599937  2444 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:03.648689 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47090.1us
I20260709 12:21:03.635136  2541 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.649775  2556 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:03.652755 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 197476us
W20260709 12:21:03.635463  2403 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.932s	user 0.009s	sys 0.005s
I20260709 12:21:03.654510 27589 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
W20260709 12:21:03.628064  2570 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:21:03.659873 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:03.655966 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:03.599589  2547 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:03.665506 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58656.8us
I20260709 12:21:03.653359  2288 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:03.528429  2558 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.651542  2541 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.660633  2403 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:03.661106  2574 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:21:03.668504  1859 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.055s	user 0.005s	sys 0.009s
W20260709 12:21:03.724023 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70110.8us
W20260709 12:21:03.700186  2537 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.779s	user 0.009s	sys 0.004s
I20260709 12:21:03.698984  2451 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.733614 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:03.658805  2579 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:21:03.662581  2538 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 12:21:03.647673  2567 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.732050  2333 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:03.730124  2541 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:03.753620  2541 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:03.754012  2541 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:03.759807 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.761016 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.763480 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.764858 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.765763 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:03.766633 27662 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:03.737944  1859 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.769587  1859 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:03.772797  2536 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.668s	user 0.010s	sys 0.000s
I20260709 12:21:03.775924 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54366 (request call id 28) took 3163 ms. Trace:
I20260709 12:21:03.776387 27656 rpcz_store.cc:276] 0709 12:21:00.612401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.739227 (+126826us) service_pool.cc:224] Handling call
0709 12:21:03.775706 (+3036479us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:03.775826 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4992164224}
I20260709 12:21:03.777740 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.782116  2530 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.737839  2537 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18.
I20260709 12:21:03.665733  2598 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260709 12:21:03.786701 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.778987 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.743189 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:03.743042 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.786779 27662 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:03.788731 27736 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:03.789394 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:03.750505 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.789793 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.789417 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.750823 27655 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:03.765736 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:03.751811 27738 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:21:03.791999 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.733273  2288 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:03.753180  2547 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:03.786134 27583 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:03.745731 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.794529 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:21:03.795368 27733 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:21:03.750664 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.795094 27734 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:21:03.790439 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:03.793359 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.790904 27656 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:03.798980 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.800381 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.802073 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.799053 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.773308  2536 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.811363 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.796546 27733 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Committing config change with OpId 17.13: config changed from index 10 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:03.806612 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.812548 27736 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:03.812629 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.813731 27738 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:21:03.791496  2576 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:03.798776  2581 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:03.834074  2541 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.835309  2202 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:21:03.836575  2599 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:21:03.744352  2451 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.755121  2580 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.857916  2567 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.845244 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:03.858611 27734 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:03.859421  2580 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:03.860852  2435 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.751s	user 0.009s	sys 0.002s
I20260709 12:21:03.861287  2435 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.845928 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.847555 27584 raft_consensus.cc:2368] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10.
I20260709 12:21:03.811594  2333 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:03.837270 28331 raft_consensus.cc:2955] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Committing config change with OpId 17.13: config changed from index 10 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:03.853600 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.854175 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:21:03.849946  2442 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.774s	user 0.012s	sys 0.000s
I20260709 12:21:03.844213  2560 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.870594 27734 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:03.887275 27729 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:21:03.849054  2202 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:03.896922  2202 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:03.898030 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.583s (SENT) [suppressed 9 similar messages]
I20260709 12:21:03.898312  2202 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:03.892166  2398 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:03.892406  2536 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:03.892740  2288 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.885706 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:03.907171 27660 raft_consensus.cc:606] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:03.907998 27660 raft_consensus.cc:614] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:03.909397 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.873927  2580 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:03.921228 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 257996us
I20260709 12:21:03.925386 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.927702 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.931934 27734 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:03.936268 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.939077 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.940024 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.899202  2442 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:03.943431 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45440.4us
I20260709 12:21:03.948429 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.921152  2580 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.951694 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:03.952457 27736 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:03.952770 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:03.956461 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:03.949973  2549 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.669s	user 0.002s	sys 0.010s
I20260709 12:21:03.926956  2581 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:03.995458 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1047) took 1101 ms. Trace:
I20260709 12:21:03.984691 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.001066 28336 rpcz_store.cc:276] 0709 12:21:02.894520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.894829 (+   309us) service_pool.cc:224] Handling call
0709 12:21:02.924482 (+ 29653us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.023455 (+ 98973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:03.995181 (+971726us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:03.995392 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159527808}
I20260709 12:21:04.003312 27660 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:04.003572 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:03.947438  2333 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.005162 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.007891 27658 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:21:04.009126 27732 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:04.008620 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:04.016090 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:03.981506  2610 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.017241 27659 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:04.023366 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.026804 27659 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:04.022280  2541 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.026688  2435 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.004519  2549 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.022003  2610 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:03.984994  2532 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.083214 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:04.085410  2532 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.061234  2435 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:04.156147  2435 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.159600  2549 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:04.165084 27583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1277) took 1384 ms (client timeout 1634 ms). Trace:
W20260709 12:21:04.178943 27583 rpcz_store.cc:269] 0709 12:21:02.781172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.781310 (+   138us) service_pool.cc:224] Handling call
0709 12:21:04.164876 (+1383566us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54001fefd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:04.165006 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2217023872}
W20260709 12:21:04.206573  2619 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:21:04.206476  2436 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.207703  2376 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.207396  2436 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.174661 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.210423  2436 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.172248  2541 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:04.188542  2451 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:04.221980 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.230309 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.234565 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:04.200872  2560 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.237334 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:04.238933 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:04.240026 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:04.220295  2541 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.237843  2376 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:04.305506 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64096.2us
I20260709 12:21:04.238471  2436 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:04.301956  2560 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:04.244009  2567 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:04.334158 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.302248  2597 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:04.338912 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.318972  2537 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:04.354236  2537 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.357553 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:04.246943  2598 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.088s
I20260709 12:21:04.358181 28333 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.355304  2610 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.324716  2436 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.244192  2606 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.101s
I20260709 12:21:04.333226  2567 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.364261 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:04.367019 28332 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:04.334569  2486 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.335541  2376 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:04.365979 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.369881 27553 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.347623  2597 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.370934 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.375051 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:04.356930  2560 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:04.365267  2436 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.371847  2486 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:04.377341  2626 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:21:04.377893  2623 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:21:04.368774  2567 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:04.367127  2610 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.387373 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:04.389168  2560 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.381604  2596 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.389283 28333 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:04.375960  2597 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:04.395347 28336 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:04.408098 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:04.408492 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.410705 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3466) took 1361 ms. Trace:
I20260709 12:21:04.416435 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.411077 27589 rpcz_store.cc:276] 0709 12:21:03.049665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.049780 (+   115us) service_pool.cc:224] Handling call
0709 12:21:03.076269 (+ 26489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:03.656883 (+580614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.410464 (+753581us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:04.410652 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1323891456}
I20260709 12:21:04.419003 28332 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:21:04.419118 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.420379 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:04.420769 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:04.421763 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.424464 27662 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:04.374647  2376 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.392769  2596 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.434495  2634 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:04.435206  2596 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.435516  2537 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.436206  2537 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:04.435912  2596 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:21:04.436748  2537 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:04.437392  2596 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:04.437959  2596 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:04.438539  2596 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.418718  2598 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:04.439805 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1146) took 1623 ms. Trace:
I20260709 12:21:04.440354 28328 rpcz_store.cc:276] 0709 12:21:02.816975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.820482 (+  3507us) service_pool.cc:224] Handling call
0709 12:21:02.867093 (+ 46611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.025116 (+158023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.439492 (+1414376us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:04.439731 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337758080}
I20260709 12:21:04.441253  2598 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:04.441638  2598 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:04.442427  2598 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.435097  2634 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:04.395252  2610 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.432354  2635 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:04.483652 27660 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:04.463064  2598 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:04.464735  2635 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:04.496165 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.437791  2537 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.480345  2541 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:04.529657 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 108968us
I20260709 12:21:04.529958 27657 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:04.530714 27657 raft_consensus.cc:606] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:04.524269  2333 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.535651 27655 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:04.501000  2635 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.537366  2333 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:04.539546  2638 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:21:04.542111 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.552249 27663 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:21:04.546707 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 569188us
I20260709 12:21:04.547080 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.547878 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.553237 27733 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:04.553341 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.548127 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.557142 27738 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:04.557220 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.558455 27655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:04.563036 27655 raft_consensus.cc:606] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:04.563745 27655 raft_consensus.cc:614] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:04.559213  2486 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.565819 27663 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 10.
I20260709 12:21:04.567062  2371 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:04.567663  2371 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.567790  2333 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.569325 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.569563 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.570195  2371 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.566671 27733 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
W20260709 12:21:04.614066 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60864.3us
I20260709 12:21:04.570475 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
W20260709 12:21:04.618796 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53200.3us
I20260709 12:21:04.613862 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:04.620931 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.527505  2634 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.626065 27738 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:04.620275 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.627801 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.628657  2371 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:21:04.629411 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:04.559197  2629 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:21:04.609735  2556 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:04.609938  2541 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.631600 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.640658 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.570441 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.632568 27737 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:04.613288  2443 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:04.636083  2636 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:21:04.554585  2577 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 12:21:04.648940 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.638249  2600 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.629695  2436 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.654546 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.655577 27732 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:04.644392 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.641597 27738 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:04.640007  2642 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:04.643500  2534 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:04.657313 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.554116  2606 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.084s
I20260709 12:21:04.649992  2376 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.647521  2443 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.674860 27659 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:21:04.666229 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.662338 27737 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:04.680681 27656 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:21:04.717866 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.717056 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:04.662278 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.686117 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:04.727101 27660 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.729382 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:04.731151  2427 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.663s	user 0.020s	sys 0.005s
W20260709 12:21:04.731877  2427 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.664s	user 0.021s	sys 0.005s
I20260709 12:21:04.658535  2412 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:21:04.644114  2541 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:04.673309  2628 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.481s	user 0.006s	sys 0.003s
I20260709 12:21:04.692598  2532 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.738675  2531 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:04.672892  2436 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.671653  2556 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:04.666986  2642 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:04.745337 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:04.652958  2600 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.733636  2202 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:04.755308 27660 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:04.756438 28336 raft_consensus.cc:2368] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10.
I20260709 12:21:04.749465  2540 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.015s
I20260709 12:21:04.748594  2288 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:04.765177 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.765470 27734 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:04.765789 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3475) took 1199 ms. Trace:
I20260709 12:21:04.746929  2549 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.747160  2532 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.776414  2549 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.778556  2549 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.758281  2569 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:04.779652  2569 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:04.780308  2596 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.780511  2569 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.781711  2569 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:04.782814  2596 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.757408  2537 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:04.783742  2537 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.746974  2634 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.770901 27628 leader_election.cc:400] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10.
I20260709 12:21:04.785187 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:04.774171  2541 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.773723  2288 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.774947  2600 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.775112  2375 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:04.808899 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.809733 28333 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:04.774814  2202 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.772023 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:04.772475 27584 rpcz_store.cc:276] 0709 12:21:03.566391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.566625 (+   234us) service_pool.cc:224] Handling call
0709 12:21:04.760631 (+1194006us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54002b98d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:04.765732 (+  5101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562074368}
I20260709 12:21:04.788357 27628 leader_election.cc:403] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:21:04.757085  2531 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:21:04.823208 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55489.2us
I20260709 12:21:04.824745 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.788900 27732 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:04.795617  2532 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:04.796977  2443 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.798429  2634 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.799464  2610 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.799974  2642 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:04.801702 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.804199 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.818475  2202 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:04.750007  2567 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.825444 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:04.828426 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86366.4us
I20260709 12:21:04.858013 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.940291 27662 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:04.924571 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:04.827662 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:04.942807 28333 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.942947 28335 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:21:04.941185 28330 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:04.825402 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:04.944537 27581 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:04.944468 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:04.849735 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.945803 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:04.946069 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.946457 27583 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:04.945847 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:04.946964 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 284817us
I20260709 12:21:04.947880 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:04.948468 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:04.948822 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.949592 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:04.950925 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:04.953510 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:04.954262 27586 raft_consensus.cc:2368] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
W20260709 12:21:04.796993  2630 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.424s	user 0.003s	sys 0.000s
I20260709 12:21:04.836566 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:04.772117  2597 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:04.824162  2427 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10.
I20260709 12:21:04.957990 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.972976 27627 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:21:04.974025 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:04.966724 28333 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:04.960249 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:04.975787  2380 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:04.958447  2537 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.975222 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:04.977365 27581 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:04.983170  2288 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:04.984625  2387 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:04.985131  2387 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:04.985579  2387 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:04.986425  2531 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:04.987407  2659 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:04.988363  2659 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.988761  2375 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:04.988864  2659 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:04.989490 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1152) took 1967 ms. Trace:
I20260709 12:21:04.991670 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:04.993297 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:04.993938 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.994009 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:04.989964 28331 rpcz_store.cc:276] 0709 12:21:03.022009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.022156 (+   147us) service_pool.cc:224] Handling call
0709 12:21:03.836975 (+814819us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.837174 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:03.914663 (+ 77489us) log.cc:844] Serialized 14 byte log entry
0709 12:21:04.147645 (+232982us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:04.147711 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:04.147759 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.147828 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:04.989273 (+841445us) spinlock_profiling.cc:243] Waited 778 ms on lock 0x7b540073e950. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:04.989414 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":857,"raft.queue_time_us":2225,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":96,"spinlock_wait_cycles":1789929728,"thread_start_us":230235,"threads_started":1,"wal-append.queue_time_us":320480}
I20260709 12:21:04.997663 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:04.998345 27580 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:21:04.998786 27581 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:04.999475 27587 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:04.989317  2375 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:05.003976 27583 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:05.002355 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.005545 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:05.007236 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.010983 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.014321 27584 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:05.018921 27580 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:04.809224  2596 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:05.033926 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.035693 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.023855 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:05.007458 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.032177 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:04.936466  2333 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:04.902827  2540 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.033124 27729 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:21:04.991530  2042 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:21:05.062541  2375 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.063998  2375 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:05.064435  2042 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:05.067425  2333 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.070060  2333 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.070725  2540 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.074029  2540 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.005694  2659 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:05.075695  2659 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:05.076367  2659 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:05.076583  2346 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.076749  2659 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.077250  2659 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:05.062813  2387 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.079653 27729 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:05.081223  2387 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:05.082233 27732 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:21:05.006328  2202 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:05.046494 27581 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:05.055344  2558 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:05.058862  2635 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.029760  2380 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:05.003953  2580 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:05.061808  2651 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
W20260709 12:21:05.028504  2557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.698s	user 0.020s	sys 0.019s
I20260709 12:21:05.116359 27583 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:05.118170 27588 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
W20260709 12:21:05.113768  2557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.790s	user 0.020s	sys 0.019s
I20260709 12:21:05.127279 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2881) took 1107 ms. Trace:
I20260709 12:21:05.162895 27731 rpcz_store.cc:276] 0709 12:21:04.019827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.173953 (+154126us) service_pool.cc:224] Handling call
0709 12:21:04.204956 (+ 31003us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:04.552728 (+347772us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.552765 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:04.580587 (+ 27822us) log.cc:844] Serialized 26 byte log entry
0709 12:21:04.734083 (+153496us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:04.734117 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.734177 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.127039 (+392862us) raft_consensus.cc:1685] finished
0709 12:21:05.127057 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:05.127212 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":347376,"raft.queue_time_us":347500,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":16833,"thread_start_us":94973,"threads_started":1,"wal-append.queue_time_us":107685}
I20260709 12:21:05.165851 27731 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:21:05.095466  2558 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:05.167064  2558 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:05.132115  2635 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.133716 27658 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:21:05.134732 27657 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:21:05.136240  2567 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.113036  2572 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.136951 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:05.137666  2634 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.139314 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:05.140117 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.048576  2596 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.142011  2597 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.143217 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:05.149581 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3330) took 1366 ms. Trace:
I20260709 12:21:05.173732 28336 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:05.173668  2572 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.177213  2596 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.179641 27660 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:05.180025 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.183408 28330 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:21:05.184388 27662 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:05.184861 27664 rpcz_store.cc:276] 0709 12:21:03.783423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.793112 (+  9689us) service_pool.cc:224] Handling call
0709 12:21:05.149330 (+1356218us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:05.149497 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128822656}
I20260709 12:21:05.175390 28335 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:05.149730 27630 leader_election.cc:400] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 9 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 9. Current term is 10.
I20260709 12:21:05.153342 27737 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:21:05.191741 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:05.192802  2346 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:21:05.193238  2346 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.193638  2346 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:21:05.201805 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 190927us
W20260709 12:21:05.200574 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 176796us
I20260709 12:21:05.203557 28336 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:05.204475 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.206067 28335 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:05.207286 27660 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:05.208973 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.216014 28331 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:21:05.219877 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:05.240372 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:05.249817 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.013s	sys 0.033s
W20260709 12:21:05.250409 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.013s	sys 0.034s
W20260709 12:21:05.250780 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 517013us
I20260709 12:21:05.251922 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:05.258967 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55681.4us
I20260709 12:21:05.254474 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:05.259865 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:05.242383 28295 connection.cc:256] RPC call timeout handler was delayed by -1.0297515899999325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20260709 12:21:05.268218 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.268934 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:05.269636 27584 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:05.271133 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.272287 27581 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:05.273414 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.277544 27581 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:21:05.249773  2549 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.261619  2042 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.264025  2540 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:05.293100 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.264155  2333 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:05.295534 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.296689 27581 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:05.299109 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.298981  2540 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:05.293624  2574 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.320078 28295 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:05.329474 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.298017  2042 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.299367  2549 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.298226  2666 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:05.315826  2667 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.317582  2333 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:21:05.333264  2663 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:21:05.356554  2095 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:05.342358 28335 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:05.357256  2095 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.315486  2557 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:05.343762 27581 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:05.330310  2540 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:05.331588  2581 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:05.339222  2634 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:05.360062 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.343520  2532 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:05.375974 27581 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:05.351305  2666 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:05.382754 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89905.1us
W20260709 12:21:05.403910 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61815.8us
I20260709 12:21:05.405074 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:05.380344 27584 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:05.407873 27704 leader_election.cc:400] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:21:05.408317 27704 leader_election.cc:403] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:05.335538  2574 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:05.411190 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.359349  2557 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:05.411799 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112785us
I20260709 12:21:05.409754 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:05.342697  2659 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.431173 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.375713  2003 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:05.432853 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:05.435940  2664 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:21:05.436416  2671 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
W20260709 12:21:05.437510  2662 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:21:05.438169  2558 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.366356  2581 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.359674  2667 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.440670  2549 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:05.434424  2557 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:05.434743  2531 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:21:05.442044  2042 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.403585  2333 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:05.443516 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.445684  2659 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.447865 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:05.472013 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.472678 27730 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:05.449965 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:05.409049  2666 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:05.462777 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.452764  2676 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:05.376122  2634 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.456503 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.466939 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:05.463701 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.480623 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.480983 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.488061 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.544037 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.545706 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:05.547202 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74404.8us
W20260709 12:21:05.548431 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82588.1us
I20260709 12:21:05.453140  2003 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.480157  2556 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:05.596202 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:05.552511 27730 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:05.553684 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:05.557125  2681 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:05.557801  2546 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:21:05.560127 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 103695us
I20260709 12:21:05.605204 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:05.562858 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.487672  2042 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:05.564980  2680 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:05.566057  2095 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.491904  2532 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:21:05.567185  2569 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.785s	user 0.016s	sys 0.000s
I20260709 12:21:05.624061 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:05.572000  2673 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:21:05.572976 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:05.576081  2346 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.585620  2634 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.587044 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:05.635756 28335 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:05.588852  2667 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.589473  2540 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.591276  2558 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:05.486588  2683 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:05.602936 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.660727 27737 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:05.542783  2682 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:05.624337 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1282) took 2508 ms. Trace:
I20260709 12:21:05.632865 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:05.634485 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.675277 27730 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:05.639063 27584 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:21:05.641871 27729 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:21:05.638044  2095 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.682979 27737 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:21:05.683928 27737 pending_rounds.cc:85] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 13
I20260709 12:21:05.684338 27737 pending_rounds.cc:107] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:21:05.684752 27737 raft_consensus.cc:2889] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:05.688928  2693 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:21:05.690402  2647 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.131s	user 0.005s	sys 0.001s
I20260709 12:21:05.690611 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:05.692318 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:05.693729 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.694461 27584 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:05.706331 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:05.600224  2676 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:21:05.674567 27585 rpcz_store.cc:276] 0709 12:21:03.109683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.109834 (+   151us) service_pool.cc:224] Handling call
0709 12:21:03.130663 (+ 20829us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:04.938770 (+1808107us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b18000efb00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:21:04.938920 (+   150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.938952 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:04.939504 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:21:05.089742 (+150238us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:05.089822 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.089926 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.592267 (+502341us) raft_consensus.cc:1685] finished
0709 12:21:05.592300 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:05.618363 (+ 26063us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12945,"raft.queue_time_us":87850,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":49004,"spinlock_wait_cycles":60665728,"thread_start_us":310033,"threads_started":2,"wal-append.queue_time_us":297632}
I20260709 12:21:05.710544 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.624820  2679 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 12:21:05.689586  2693 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:05.680343  2042 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.736471 27584 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:05.738271 27585 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:05.618126  2681 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.738189 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:05.715418 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 236986us
I20260709 12:21:05.701782  2095 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.743489 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:05.744643  2532 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:05.745189  2095 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.708683  2546 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:05.763577 27664 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:05.764753 27664 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:05.544194  2525 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.142s
I20260709 12:21:05.759063 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.765578 27664 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.742162  2676 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:05.770455 27735 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:21:05.771270  2627 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.579s	user 0.003s	sys 0.000s
I20260709 12:21:05.759413  2532 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:05.772658  2540 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.773082  2380 raft_consensus.cc:993] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:05.709059  2558 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 12:21:05.765707  2674 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:05.765305  2574 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.764071  2556 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:05.764622  2546 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:05.758884  2042 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.777047  2676 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:05.797746  2693 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.799767 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.792287  2427 raft_consensus.cc:1081] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
W20260709 12:21:05.806255 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 201189us
I20260709 12:21:05.790836  2581 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.811633 27735 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:05.785219  2531 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:21:05.819046 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:05.819563 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.820251 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:05.812824  2676 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:05.831688  2532 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.832592 27731 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:05.832535  2666 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.833434  2574 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:05.833688  2042 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.837841  2681 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.856348 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:05.846659  2427 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:05.862242 28335 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:05.865639 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.854598  2003 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.853677  2346 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.884378  2694 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:05.887856 27585 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:05.887090 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.888839 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:05.835938  2546 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.897750 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: [24.13-27.14]
I20260709 12:21:05.899674 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.900337 27737 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:05.901253 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.902403 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:05.902683 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:05.903475 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.904783 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.907853 27729 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:05.904678  2574 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 12:21:05.908874 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.909423 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.910104 27735 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:05.864195  2556 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.905227  2042 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:05.912308  2042 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.915037  2042 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.896944  2592 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20260709 12:21:05.906149 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:05.916460  2666 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.877714  2679 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.029s
I20260709 12:21:05.876323  2427 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.891142  2694 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:05.921407  2546 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:05.921141  2694 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.929086 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:05.930728  2652 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.909s	user 0.001s	sys 0.000s
I20260709 12:21:05.932947 28335 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:05.935107  2679 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:05.935431  2525 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:05.936002 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:05.936259  2525 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:05.936721  2525 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:05.936928  2427 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:05.937247 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.937870 27581 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:05.938170  2577 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:05.938622  2577 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:05.938656 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.939724  2577 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.940897  2577 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:05.941598  2435 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:05.944783 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:05.943379 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.948637 27583 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:05.949517 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:05.950459 27585 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:05.954945 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:05.957171 27587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
W20260709 12:21:05.952282  2661 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.826s	user 0.003s	sys 0.000s
I20260709 12:21:05.958338  2592 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:05.958796  2592 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:05.959424 27587 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:05.959889 27587 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:05.959728  2592 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:05.961172  2592 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:05.962235  2674 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:05.963652 27730 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:05.964818 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:05.968874 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:05.949638 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:05.972258 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2780) took 1416 ms. Trace:
W20260709 12:21:05.972698  2631 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.777s	user 0.000s	sys 0.004s
I20260709 12:21:05.972774 27663 rpcz_store.cc:276] 0709 12:21:04.555840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.623946 (+ 68106us) service_pool.cc:224] Handling call
0709 12:21:04.624274 (+   328us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:04.624702 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.624731 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:04.633770 (+  9039us) log.cc:844] Serialized 52 byte log entry
0709 12:21:04.735628 (+101858us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:04.735682 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.735776 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.930023 (+1194247us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:05.963459 (+ 33436us) raft_consensus.cc:1685] finished
0709 12:21:05.963481 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:05.972167 (+  8686us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":7135,"spinlock_wait_cycles":37258624,"thread_start_us":84399,"threads_started":1,"wal-append.queue_time_us":279251}
I20260709 12:21:05.977717 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.979189 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:05.980803 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.981155 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.982203 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.984019 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.984959 27663 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:05.987210 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:05.982153 27662 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:05.988060 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:21:06.055837 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:05.989580 27658 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:06.010073 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:06.043360 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.991792 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.064131 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.064314 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.064991 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.074576 27657 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [17.9-19.10]
I20260709 12:21:05.901669  2676 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.089936 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.098524 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.098888 27662 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:06.099954 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.101912 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:06.107287 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.109361 27663 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:06.114168 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.114923 27659 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:06.118902 27587 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:05.987795  2042 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:05.927127  2557 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.131942 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.132473 27659 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:06.074088  2375 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.141713 27587 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.142436 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:06.145071 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.145612 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:06.146257 27733 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:06.149758 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.151392 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.152272 28335 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:06.154927 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:06.155548 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:06.164867 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.167838 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:06.169128 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.166452 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:06.167899 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.174963 27738 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:21:06.181913 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.186056 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.188220 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.153342  2657 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:06.191478 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:06.176762 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.157954  2668 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:06.162431  2451 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:06.170770  2574 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:06.199438 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.183744  2412 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:06.202344 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.196979  2435 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:06.186348  2558 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:06.142627  2042 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.224483  2557 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.225243  2668 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:06.225471  2042 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:06.197798  2451 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:06.198660  2556 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:21:06.211347 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.195464  2540 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:06.159551  2375 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.183912  2534 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 12:21:06.216467 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2873) took 2275 ms. Trace:
I20260709 12:21:06.245146 27734 rpcz_store.cc:276] 0709 12:21:03.941478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.941801 (+   323us) service_pool.cc:224] Handling call
0709 12:21:04.765093 (+823292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.772076 (+  6983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.216238 (+1444162us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b54001d24d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:06.216401 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1891978112}
I20260709 12:21:06.246176 27661 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:21:06.220150 27729 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:06.248847 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:06.220268  2689 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:06.250355  2689 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:06.220825  2435 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:06.221616  2525 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.250993  2435 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:06.251442  2689 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.252810  2689 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:06.253087  2525 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.189098  2584 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:06.260540  2659 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:06.261082  2659 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:06.262033  2659 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.204988  2574 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.265565 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2884) took 2031 ms. Trace:
I20260709 12:21:06.265656 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1154) took 1520 ms. Trace:
I20260709 12:21:06.266080 27732 rpcz_store.cc:276] 0709 12:21:04.234630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.053584 (+818954us) service_pool.cc:224] Handling call
0709 12:21:05.054031 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:05.082663 (+ 28632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.265294 (+1182631us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:06.265502 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64242560}
I20260709 12:21:06.267490  2525 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.268093  2525 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:06.270582  2660 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.140s	user 0.000s	sys 0.002s
I20260709 12:21:06.270983 27737 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:06.272251 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:06.272934 27586 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:06.273955  2375 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.266219 28328 rpcz_store.cc:276] 0709 12:21:04.745155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.745969 (+   814us) service_pool.cc:224] Handling call
0709 12:21:04.756030 (+ 10061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.876823 (+120793us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.876877 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:04.876946 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:04.876997 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.265410 (+1388413us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400a1dbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:06.265464 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.265597 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277141248}
I20260709 12:21:06.206717  2202 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:06.200920  2692 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:21:06.206916  2333 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:06.207079  2412 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:06.215008  2693 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:06.311717  2659 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:06.239303 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:06.318367 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:06.243919  2676 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.307123 27586 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:06.231468  2451 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:06.322368 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:06.222587  2572 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.229130 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:06.327749 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:06.328431 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99560us
I20260709 12:21:06.329922 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.288862 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:06.288451 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:06.283308 27662 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:06.283051 27659 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:06.332578 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.280403 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:06.277705 27585 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:06.314291 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:06.312786 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 388127us
W20260709 12:21:06.315850 28292 connection.cc:256] RPC call timeout handler was delayed by -1.152535768000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.188s for a response
W20260709 12:21:06.335103 28292 connection.cc:256] RPC call timeout handler was delayed by -1.1182470070000363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
I20260709 12:21:06.267575  2574 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.332657 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.340878 27659 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:21:06.351673 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.353356 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.354135 27659 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:06.354617 27659 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:06.328543 27656 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:06.357379 27662 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:06.270608  2525 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.316792  2412 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:06.369405 27662 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:06.369931 27729 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:21:06.334329  2693 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:06.360950  2679 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:06.327787  2684 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:06.377796  2684 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:06.378814  2684 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.384905  2412 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.387563  2412 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.391503 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.369081 27659 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.376088 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.392894 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:06.393862 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:06.394512 27657 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:06.368695  2692 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:06.399578 27659 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:06.376667  2693 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:06.408417 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:06.424537 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:06.403036  2679 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:06.447994 27659 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:06.454686 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:06.455027 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.456895 27659 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:06.417109  2692 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:06.468330 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:06.488526 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 125441us
I20260709 12:21:06.439671  2577 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:21:06.491029 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.496820 27659 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:06.494127 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.506374 27656 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: [17.9-19.10]
W20260709 12:21:06.504405  2672 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:21:06.505314  2577 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.516664  2577 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:06.520787 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.512358  2684 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:06.505213  2688 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 12:21:06.513902  2532 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:21:06.549690  2694 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:21:06.550622  2694 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:21:06.554750  2678 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.004s	user 0.002s	sys 0.011s
I20260709 12:21:06.539675  2442 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.556146 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1285) took 1348 ms. Trace:
I20260709 12:21:06.556488 27580 rpcz_store.cc:276] 0709 12:21:05.207572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.226708 (+ 19136us) service_pool.cc:224] Handling call
0709 12:21:05.227163 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:05.227668 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:05.227713 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:05.228379 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:21:05.429564 (+201185us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:05.429596 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.429646 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:06.554108 (+1124462us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:06.555868 (+  1760us) raft_consensus.cc:1685] finished
0709 12:21:06.555889 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.556089 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":952,"raft.queue_time_us":1004,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":16118,"spinlock_wait_cycles":101625600,"thread_start_us":150202,"threads_started":1,"wal-append.queue_time_us":197275}
I20260709 12:21:06.558050 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.558693 28332 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:06.516119  2444 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:21:06.544503 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:06.566078 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2902) took 1081 ms. Trace:
I20260709 12:21:06.566463 27731 rpcz_store.cc:276] 0709 12:21:05.466525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.466682 (+   157us) service_pool.cc:224] Handling call
0709 12:21:06.547548 (+1080866us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54002e9ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:06.547665 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":786788224}
W20260709 12:21:06.527294  2710 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:21:06.551055  2694 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:06.568178  2682 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:06.569257  2444 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:21:06.571589 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.565661 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:06.572024 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.579037 27660 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:06.583752 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.584019 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:06.535523  2692 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:06.586292  2709 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:21:06.584069 27664 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:06.587996 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:06.564087 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.596722 28332 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:06.595518 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51092.6us
I20260709 12:21:06.599085 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.618651 28332 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
W20260709 12:21:06.622049 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 275785us
I20260709 12:21:06.624760 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:06.634910 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.591216  2444 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:06.597596  2288 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:06.601625  2634 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:06.619853  2192 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:06.631047  2685 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:06.654035 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1061) took 1493 ms. Trace:
I20260709 12:21:06.655536 28331 rpcz_store.cc:276] 0709 12:21:05.160433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.160591 (+   158us) service_pool.cc:224] Handling call
0709 12:21:05.215678 (+ 55087us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:05.219413 (+  3735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.653827 (+1434414us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b54005a00d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:06.653976 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35778432}
I20260709 12:21:06.632413  2525 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:21:06.646294  2711 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 12:21:06.639314  2682 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:06.707458 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3484) took 1714 ms. Trace:
I20260709 12:21:06.707893 27582 rpcz_store.cc:276] 0709 12:21:04.992621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.995935 (+  3314us) service_pool.cc:224] Handling call
0709 12:21:04.996250 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:05.176249 (+179999us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:05.176285 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:05.176326 (+    41us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:05.176359 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.707198 (+1530839us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54000889d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:06.707247 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.707387 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413502848}
I20260709 12:21:06.588428  2679 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.645130  2288 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:06.632220  2346 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:21:06.652245  2192 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:06.649559  2634 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:06.706686 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.635381  2003 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.069s
I20260709 12:21:06.727568 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.729666 27730 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:06.655417  2685 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:06.704293  2712 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:06.723726  2288 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:06.651979  2694 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.759676  2634 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.761392  2634 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:06.759724  2694 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:06.733124  2192 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:06.755434  2444 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.736886  2685 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:06.727192  2714 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
W20260709 12:21:06.777710 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.769764  2538 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:21:06.736593  2600 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:06.737715  2718 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:21:06.799156  2718 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:06.799697  2718 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:21:06.804981  2435 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.553s	user 0.007s	sys 0.002s
I20260709 12:21:06.805441  2435 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.807564 27661 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
W20260709 12:21:06.809788 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63527.4us
I20260709 12:21:06.810596 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.811506 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.812258 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.813417  2442 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:06.813933  2442 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:06.814008  2576 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:06.814275  2451 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.814859  2442 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.816298 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2771) took 2801 ms. Trace:
I20260709 12:21:06.816648  2451 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.721879  2682 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.817939  2719 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:06.818370  2719 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:06.818742  2719 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:06.819048  2682 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.814653  2576 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:06.821913  2576 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.822842 27735 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:06.790634  2375 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:06.816818 27655 rpcz_store.cc:276] 0709 12:21:04.014780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.571232 (+556452us) service_pool.cc:224] Handling call
0709 12:21:05.149489 (+578257us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:05.149889 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:05.149921 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:05.150450 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 12:21:05.258821 (+108371us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:05.277168 (+ 18347us) log.cc:844] Serialized 14 byte log entry
0709 12:21:05.418335 (+141167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.418452 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.793850 (+375398us) raft_consensus.cc:1685] finished
0709 12:21:05.793869 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.816030 (+1022161us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b5400343850. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:06.816196 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15651,"prepare.queue_time_us":18358,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":78844,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":181,"spinlock_wait_cycles":1328099712,"thread_start_us":232727,"threads_started":2,"wal-append.queue_time_us":198406,"wal-append.run_cpu_time_us":4153,"wal-append.run_wall_time_us":1441934}
I20260709 12:21:06.830413  2435 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:06.838565 27581 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
W20260709 12:21:06.846872 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.848742 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2799) took 1086 ms. Trace:
I20260709 12:21:06.849334 27664 rpcz_store.cc:276] 0709 12:21:05.762872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.763063 (+   191us) service_pool.cc:224] Handling call
0709 12:21:05.763448 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:06.583693 (+820245us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b58003a1fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:21:06.848397 (+264704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:06.848425 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:06.848467 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:06.848509 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.848581 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.848688 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18021,"raft.queue_time_us":32112,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":6207,"spinlock_wait_cycles":18292480,"thread_start_us":214023,"threads_started":1}
W20260709 12:21:06.852048 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 140301us
I20260709 12:21:06.853264 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:06.856086 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.857407  2346 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:06.857573 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.744058  2666 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.100s
I20260709 12:21:06.861529 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.833704  2526 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:06.768428  2597 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.101s
I20260709 12:21:06.828867  2375 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:06.831489  2720 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:06.834159  2676 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.860008 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.847474  2667 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.846050  2435 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:06.846287  2543 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:06.857964  2346 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.871520  2576 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:06.875093  2526 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:06.917413 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64209.2us
I20260709 12:21:06.876780  2444 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:06.870195  2680 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:06.885948  2380 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:21:06.918427 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58500.3us
I20260709 12:21:06.920132 27553 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:06.908027 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.925895 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.851893  2556 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
W20260709 12:21:06.920193 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58733.3us
I20260709 12:21:06.876219  2578 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:21:06.876355  2600 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.044s
W20260709 12:21:06.923202  2642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.793s	user 0.012s	sys 0.013s
I20260709 12:21:06.889792  2720 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.942704  2720 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:06.862567  2412 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:21:06.922503  2536 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:06.930251 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:06.917111 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 279364us
I20260709 12:21:06.938557 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.936230  2442 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:21:06.942267  2642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.812s	user 0.012s	sys 0.013s
I20260709 12:21:06.927631  2680 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.921701  2721 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:21:06.965869 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45869.3us
I20260709 12:21:06.917397  2435 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.902477  2676 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.960083 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.936729  2546 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:06.913846  2667 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.989926 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:06.962472  2681 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:06.952706  2569 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:06.970770 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.004387 28335 raft_consensus.cc:2376] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEARNER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22. Re-sending same reply.
I20260709 12:21:06.971127 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:06.974560 27588 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:06.954173  2536 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:06.988682 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:06.974138 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:06.973511  2680 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:06.919164  2543 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:06.964949  2526 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:06.994230  2724 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:21:06.996718  2649 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:21:06.997342  2288 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:06.998697  2667 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:06.999718  2412 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.000579 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:07.006356  2536 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:07.027038 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.991429 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:06.973825  2597 raft_consensus.cc:2764] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:21:07.008491  2642 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:06.973285  2721 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:06.989094  2659 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.651s	user 0.013s	sys 0.000s
I20260709 12:21:07.009975 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:06.972584  2442 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:07.057548  2543 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.067771  2720 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.004036  2569 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.075409  2569 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:07.075965  2346 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.076280 27734 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:06.999212  2577 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.079152  2577 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:07.079948 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:07.012667  2693 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.585s	user 0.010s	sys 0.000s
I20260709 12:21:07.080299 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3502) took 1225 ms. Trace:
I20260709 12:21:07.080737 27588 rpcz_store.cc:276] 0709 12:21:05.854770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.955940 (+101170us) service_pool.cc:224] Handling call
0709 12:21:05.956310 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:07.005339 (+1049029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:07.080044 (+ 74705us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:07.080245 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2338180736}
I20260709 12:21:07.081672  2546 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:07.031100 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.083694 27664 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 10.
I20260709 12:21:07.034698 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.013016  2665 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:07.015496  2578 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:21:07.080760  2693 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.090505 27655 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:07.091843  2693 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.093168 27586 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:21:07.095292 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:07.100265  2577 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.102249 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:07.102931 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:07.103729 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:07.105888  2719 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.055743  2380 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:07.055910  2600 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:07.107301  2600 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:07.020810  2526 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:07.107776  2600 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:07.108510  2719 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.030548  2551 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:07.032310  2712 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:07.108518  2679 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:07.112834  2679 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.032081  2412 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.113680  2679 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.012136 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:07.022508  2684 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:07.135982 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:07.071991  2642 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:07.072410  2721 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:07.057582  2442 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.100971  2577 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.105789  2192 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.070019  2387 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:07.115864 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 103804us
I20260709 12:21:07.120858 27664 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:07.115357  2412 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.123818 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:07.184038 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.125185 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.184767 27655 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:21:07.129096 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.052908 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:07.068812  2676 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.138149 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:07.143270 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.115253  2679 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:07.122298 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:07.202266 27731 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:07.172802 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:07.172111 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:07.203416 27664 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:07.204202 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:07.206257 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:07.179683 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:07.181620  2680 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.182183  2451 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:07.126901 27661 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260709 12:21:07.185463 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:07.207777 27661 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 14
I20260709 12:21:07.208190 27661 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:21:07.208442 27661 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:07.208703 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.209643 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:07.189054 28332 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:07.212085 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:07.212718 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74743.5us
I20260709 12:21:07.189834 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.068814  2288 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.192004 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.193692 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.195055 27736 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
W20260709 12:21:07.195441 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 142639us
I20260709 12:21:07.198017 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:07.222795 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 231434us
I20260709 12:21:07.198729 27657 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:21:07.138839  2719 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:07.201625 27656 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:21:07.175103  2718 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.211140 27659 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260709 12:21:07.220527 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:07.229948 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.219740 27664 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:07.223783 27663 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:07.226027  2412 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:07.224291 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.220875  2597 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:07.210803  2451 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.219691 28330 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:07.223580  2693 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:21:07.194557  2387 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.139299  2685 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.258781 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.223855  2727 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:07.228387  2686 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:07.242587  2346 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:21:07.266662  2387 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.267365  2412 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.297175  2412 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.273221  2577 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.302136  2720 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.302933  2720 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:07.303529  2720 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:07.304569  2720 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:07.306584  2694 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.542s	user 0.013s	sys 0.000s
I20260709 12:21:07.273686  2680 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.221043  2634 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:21:07.282907  2727 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:07.309394  2727 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:07.309697  2634 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.310258  2634 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.240310  2665 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:07.243688  2371 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:21:07.277107  2569 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.279057  2451 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.274768  2536 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.287006 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.344740 27586 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:07.289680 27580 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:21:07.339684 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 107633us
I20260709 12:21:07.357419 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:07.361536 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.243268  2659 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 12:21:07.366290 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:07.371076  2346 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:07.371694  2346 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:07.372728  2346 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.368099  2676 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.380118  2676 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.382525  2659 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:07.383085  2659 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.384112  2659 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.315948  2642 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.392602  2536 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.363601  2686 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:07.394778  2686 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.367012  2680 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:21:07.345278  2435 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.364629  2451 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:07.408975 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1211) took 1111 ms. Trace:
I20260709 12:21:07.409454 27735 rpcz_store.cc:276] 0709 12:21:06.297537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:06.297761 (+   224us) service_pool.cc:224] Handling call
0709 12:21:06.333619 (+ 35858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:06.828222 (+494603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:07.408701 (+580479us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:07.408906 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1124620544}
W20260709 12:21:07.410106 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43893.7us
W20260709 12:21:07.419727 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 195528us
I20260709 12:21:07.423158 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:07.426333 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:07.365767  2597 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:07.417636  2387 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:07.427357  2536 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:07.427825  2536 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:07.427702  2387 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.428323  2536 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:07.428002  2597 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.430189  2387 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.414034  2676 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:07.441881 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 234386us
W20260709 12:21:07.453488 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.438802 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:07.467440 27581 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:21:07.468750 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.469480 27581 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:07.473011 27582 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:07.425220  2584 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.435384  2659 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:07.481077 27588 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:07.486816 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.490054 27587 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:07.492280 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.496798 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.497572 27587 raft_consensus.cc:2368] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:21:07.501387 27584 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:07.503449 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.505120 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.481575  2730 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:07.435314  2600 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.436612  2721 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:07.500590  2576 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.001s
I20260709 12:21:07.525115  2536 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.526201  2600 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.529033 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:07.530529 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:07.532274 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:07.537714 27701 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:21:07.506062  2719 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:07.525027  2744 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:07.541585 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:07.509356  2742 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:07.545851  2719 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.483821  2584 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.528297  2634 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.540951  2387 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:07.551975  2536 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.547216  2692 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
W20260709 12:21:07.552394  2738 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:21:07.537124  2718 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.535068  2577 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:21:07.543183  2412 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:07.554939  2387 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.553102  2742 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:07.566195  2387 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:07.559099  2577 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.567668  2577 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:07.553488  2744 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.560788  2412 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.571771  2412 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:07.581195  2721 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.584447  2546 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:07.581794  2569 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.589150  2546 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:07.553805  2741 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 12:21:07.565382  2742 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:07.586180 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1010) took 1002 ms. Trace:
I20260709 12:21:07.585654  2727 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.591387 28330 rpcz_store.cc:276] 0709 12:21:06.583675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:06.583808 (+   133us) service_pool.cc:224] Handling call
0709 12:21:07.577793 (+993985us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b5400a1dbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:07.586105 (+  8312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1284743936}
I20260709 12:21:07.593264  2727 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.583060  2584 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.590029  2546 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.594740  2546 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:07.595057  2577 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.595940  2375 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.596722  2744 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.597350  2577 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.597319  2375 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:07.603425  2192 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.582600  2387 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.605182  2685 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.609678  2387 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.613039  2412 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.615845  2412 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.629161  2742 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:07.664507 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 173507us
I20260709 12:21:07.679524 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.685305 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.686477 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.686795 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.687795 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.688584 27580 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:07.689061 27582 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:21:07.689935 27580 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:07.690750 27582 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:07.691933 27585 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:07.693306 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.695462 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:07.702323 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160661us
I20260709 12:21:07.706218 27660 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:21:07.707558 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:07.691172 27582 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:07.715441 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.721102 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:07.725157  2387 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:07.726468  2192 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.728158 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.727854  2192 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.728444  2387 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:07.729316  2387 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.740372 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.741200  2639 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:07.740378  2540 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.734926  2412 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:07.713357 27582 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.751740  2288 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:21:07.752411  2288 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.752996  2288 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:07.760816 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:07.763852 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.731240  2387 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.767057  2192 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.771293  2714 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:07.776463 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.780905 28337 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:07.780754 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.781149 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1298) took 1879 ms. Trace:
I20260709 12:21:07.767257  2435 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:07.763072  2451 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:07.767664 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:07.787961 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.789741  2742 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.782269  2435 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.795789 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:07.803399 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112322us
I20260709 12:21:07.804140  2742 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:07.807657  2526 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:07.808180  2526 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:07.809423 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.809800  2526 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.810556 28333 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEARNER]: Advancing to term 23
I20260709 12:21:07.813069  2741 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:07.813102  2042 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.814170  2741 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.814417  2042 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.815568  2577 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:07.817128  2577 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:07.817512  2577 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.818197  2042 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.786710  2451 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:07.821136  2451 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.822956  2451 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:07.765269  2540 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.824301 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.824584 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.825713  2540 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.827577 27582 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:07.827808 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.827611 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.833936 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.834898 28328 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:07.831935  2435 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.836354 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.841173 27662 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:21:07.841629  2540 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:07.781698 27583 rpcz_store.cc:276] 0709 12:21:05.886704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.976508 (+ 89804us) service_pool.cc:224] Handling call
0709 12:21:05.977033 (+   525us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:06.595202 (+618169us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:07.098588 (+503386us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b1800241240. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:07.319300 (+220712us) log.cc:844] Serialized 14 byte log entry
0709 12:21:07.606880 (+287580us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b240013aca8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:21:07.680818 (+ 73938us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:07.682353 (+  1535us) log.cc:844] Serialized 26 byte log entry
0709 12:21:07.682599 (+   246us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:07.682657 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:07.682787 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:07.755697 (+ 72910us) raft_consensus.cc:1685] finished
0709 12:21:07.755716 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:07.765706 (+  9990us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":811691,"prepare.queue_time_us":680380,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":867822,"raft.run_cpu_time_us":531,"raft.run_wall_time_us":19258,"spinlock_wait_cycles":68680704,"thread_start_us":373235,"threads_started":2,"wal-append.queue_time_us":231569}
I20260709 12:21:07.843575 27589 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:21:07.843664 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:07.844237 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3514) took 1007 ms. Trace:
I20260709 12:21:07.844620 27589 rpcz_store.cc:276] 0709 12:21:06.837093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:06.839516 (+  2423us) service_pool.cc:224] Handling call
0709 12:21:07.842620 (+1003104us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:21:07.842962 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:07.843981 (+  1019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:07.844186 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4602752}
I20260709 12:21:07.845431  2741 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.846820 27655 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:21:07.848573 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:07.849067 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54998.5us
I20260709 12:21:07.849864  2192 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:07.851585  2192 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.852293  2192 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.853085  2686 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:07.854449  2546 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:07.855455  2685 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:07.855949  2685 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:07.856201  2192 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:07.857306  2685 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.866242  2526 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:07.868758 28332 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:21:07.818729  2577 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.873528 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.859026  2685 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:07.875227 27704 leader_election.cc:400] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:21:07.875710 27704 leader_election.cc:403] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:21:07.876946 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66293us
W20260709 12:21:07.877674 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50592.9us
I20260709 12:21:07.877985 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:07.878911  2683 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:07.879593 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:07.881031  2684 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:07.881438 27655 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:07.881505  2684 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:07.842439  2540 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.885003 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.848939 28333 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEARNER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:07.896347 27582 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:07.907312  2667 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.907552  2412 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:21:07.908172  2667 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.912647  2600 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:07.915810  2667 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:07.883075  2684 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:07.918233  2684 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:07.919951 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:07.923629 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 96550.3us
W20260709 12:21:07.925465 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.931473 27735 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:21:07.932116 27481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 128) took 14335 ms (client timeout 15000 ms). Trace:
I20260709 12:21:07.934145 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:07.935317 27734 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:07.938249 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:07.940333 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.954164 28336 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:21:07.959650 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:07.958353 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:07.900583  2741 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:07.987169 28332 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:07.989373 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:07.990393 28330 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:07.991194  2692 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:07.991945  2692 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:07.994606  2692 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:07.997445 28328 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:21:07.998992 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 121214us
I20260709 12:21:08.007968  2288 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.010677  2288 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:08.012701 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:08.013582 27660 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 14.
I20260709 12:21:08.014644 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:07.932662 27481 rpcz_store.cc:269] 0709 12:20:53.596966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.597163 (+   197us) service_pool.cc:224] Handling call
0709 12:21:07.931756 (+14334593us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:03.015992 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:03.151584 (+135592us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:03.151639 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:21:03.151667 (+    28us) tablet.cc:662] Decoding operations
0709 12:21:03.152750 (+  1083us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:03.152814 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:03.152857 (+    43us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:21:03.257335 (+104478us) tablet.cc:701] Row locks acquired
0709 12:21:03.281370 (+ 24035us) write_op.cc:260] PREPARE: finished
0709 12:21:03.325262 (+ 43892us) write_op.cc:270] Start()
0709 12:21:03.335348 (+ 10086us) write_op.cc:276] Timestamp: P: 1783599663325181 usec, L: 0
0709 12:21:03.340176 (+  4828us) op_driver.cc:348] REPLICATION: starting
0709 12:21:03.398758 (+ 58582us) log.cc:844] Serialized 4074 byte log entry
0709 12:21:03.735113 (+336355us) op_driver.cc:464] REPLICATION: finished
0709 12:21:03.884681 (+149568us) write_op.cc:301] APPLY: starting
0709 12:21:03.884793 (+   112us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:04.162055 (+277262us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:04.162187 (+   132us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:04.493488 (+331301us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:04.535534 (+ 42046us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=20,mrs_lookups=0
0709 12:21:04.543052 (+  7518us) write_op.cc:312] APPLY: finished
0709 12:21:04.543376 (+   324us) log.cc:844] Serialized 97 byte log entry
0709 12:21:04.543547 (+   171us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:04.543683 (+   136us) write_op.cc:454] Released schema lock
0709 12:21:04.543729 (+    46us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:04.544129 (+   400us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:04.544160 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3070434,"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":147030,"apply.run_cpu_time_us":21518,"apply.run_wall_time_us":659442,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45927,"num_ops":10,"prepare.queue_time_us":331560,"prepare.run_cpu_time_us":52735,"prepare.run_wall_time_us":499541,"raft.queue_time_us":171927,"raft.run_cpu_time_us":14975,"raft.run_wall_time_us":181312,"replication_time_us":384871,"spinlock_wait_cycles":2816,"thread_start_us":434479,"threads_started":4,"wal-append.queue_time_us":88027,"wal-append.run_cpu_time_us":74881,"wal-append.run_wall_time_us":2058498}]]}
I20260709 12:21:08.016144 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:08.016932 27656 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:21:08.020125 27581 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.506s	user 0.009s	sys 0.014s
I20260709 12:21:08.020040 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:08.020771 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:08.020983 27664 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:08.023854 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:08.024487 27659 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:08.026983 27658 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:08.027654 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.028199 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:08.028452 27657 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:21:08.030429  2540 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:08.030742 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.033643 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.034335 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:07.976482  2435 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:08.050944 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66414us
I20260709 12:21:07.986750 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:08.053633 27734 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:08.058065 27584 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:08.063344 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:08.067489 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.068264  2435 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:08.068830 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 62297.6us
I20260709 12:21:08.069491  2714 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:21:08.070504 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.070654  2435 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:08.059530 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:08.071305  2683 raft_consensus.cc:2764] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:21:08.084556  2741 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.071564 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.085760 27581 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:08.073668 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.072795 27580 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:08.078279 27737 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:08.072037 27589 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:21:08.083001 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:08.080749 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:08.087242 27733 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:08.076450 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.073390 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:08.087749 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:08.075915 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:08.103825 28332 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:08.073824  2387 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:08.108002 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.081872  2577 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:08.081576  2592 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.082319  2677 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.087221  2741 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:08.081238  2723 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.119973 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:08.123469 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 248742us
I20260709 12:21:08.124733  2540 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:08.124996 27729 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:08.125267  2540 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:08.127954  2540 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:21:08.128239 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 125051us
I20260709 12:21:08.129060 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:08.130591 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.131850 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:21:08.131557  2723 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.134016  2723 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:08.134621 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.067530  2288 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:21:08.135846  2288 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:21:08.135984 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.074275  2692 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:21:08.136368  2288 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:08.137322  2288 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.140976 27660 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:08.073696  2451 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:21:08.145664 27485 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 13 to 16, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 16 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.146704 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:08.147835 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67487.9us
I20260709 12:21:08.148813 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:08.151932  2486 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:08.152442  2486 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:08.153614  2486 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.157891 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:08.160897 28333 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
I20260709 12:21:08.114441  2387 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:08.160656 27585 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:08.165200 28336 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:21:08.126996  2435 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.121758  2685 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:08.169768 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:08.178715 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.131240  2412 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20.
I20260709 12:21:08.181219 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:08.181500 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:08.184489  2747 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:21:08.185168  2747 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:08.185583  2730 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:21:08.187929 27582 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:08.188710 28332 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:08.189014 27589 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:08.189586 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59288.8us
I20260709 12:21:08.187631  2730 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:21:08.192106 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:08.180249 27729 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:08.192149 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.161693  2600 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:08.202437 27660 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 11.
I20260709 12:21:08.168304  2387 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:08.186496  2747 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.205034 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.195986  2692 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:08.219147 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.219825 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:08.209388  2730 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:21:08.135854  2677 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.227943 27738 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:08.209642  2667 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:21:08.210465  2486 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:08.221712 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.232867 27586 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:21:08.167214  2592 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.233019 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.218793  2435 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.241631 27659 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:08.292340 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 101402us
W20260709 12:21:08.293857 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 119237us
I20260709 12:21:08.272105 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:08.289870 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49736.5us
I20260709 12:21:08.214905 27485 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 20. New cstate: current_term: 20 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.219108  2546 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:08.312525 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:08.312764 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.209491  2741 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:21:08.313190 27656 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 16.
I20260709 12:21:08.232275  2667 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:08.290838  2771 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:21:08.242799 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 113857us
W20260709 12:21:08.335516 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63530.6us
I20260709 12:21:08.232928  2692 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:08.342093 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.346187 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.242615 27737 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:21:08.347941 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 222068us
I20260709 12:21:08.393344 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.301072  2677 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:08.298671  2684 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:21:08.264935  2714 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260709 12:21:08.394974 27732 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:08.391212 27485 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.338721  2667 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:21:08.293625  2760 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.188s
I20260709 12:21:08.262444  2444 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.085s
I20260709 12:21:08.338944  2192 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:08.268590  2526 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.124s
I20260709 12:21:08.262727  2683 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.085s
W20260709 12:21:08.402815 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50295us
I20260709 12:21:08.406232 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.415057  2412 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:08.448843 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:08.398974  2546 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:08.454820 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.407670  2192 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:08.480693 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.427603  2686 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:08.493502 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 147455us
I20260709 12:21:08.445850  2584 raft_consensus.cc:2764] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:21:08.473701  2451 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:08.485893  2288 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:21:08.507634 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 245601us
I20260709 12:21:08.489377  2192 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:08.494865  2686 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:08.525657 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.502754  2760 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:21:08.543733 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 153235us
I20260709 12:21:08.555621  2451 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:21:08.559618 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 217605us
I20260709 12:21:08.577260  2192 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.568125  2692 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.587102  2592 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:08.587579  2679 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:08.540835  2760 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:08.596616  2760 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:08.520967  2546 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:08.596808 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 190645us
I20260709 12:21:08.543439  2683 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:08.595973  2435 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:08.536897  2686 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:08.599088 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:08.601509  2782 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:21:08.641662 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:08.652004 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:08.600647  2451 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:08.560263  2741 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:08.638197 27485 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.609194  2577 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:08.678516  2679 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:21:08.656461 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 130957us
I20260709 12:21:08.613605  2683 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:08.599119  2546 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:08.623485  2574 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.625104  2639 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:21:08.691716 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:08.601537  2540 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.700429 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:08.700991 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:08.707947 27585 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:08.699741 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:08.649257  2730 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:08.679131 27485 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.696331 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.699589 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.702821 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.730063 27580 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:08.704581 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:08.710070  2380 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 12:21:08.710382  2574 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.710924 27737 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:21:08.711534 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.711617 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.711966 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.712255 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.712971 27583 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:21:08.713191 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.715878 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.708837 28330 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:08.601657  2597 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.097s
I20260709 12:21:08.729417 27586 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:08.731173 27582 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:08.734988 27658 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 12:21:08.736090 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 290558us
I20260709 12:21:08.738787 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:08.733206 27737 pending_rounds.cc:85] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 10
W20260709 12:21:08.742992 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91095.3us
I20260709 12:21:08.740558 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:08.679986  2600 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
W20260709 12:21:08.754130 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 155133us
I20260709 12:21:08.750239 27729 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:08.741590 27737 pending_rounds.cc:107] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:21:08.758481 27737 raft_consensus.cc:2889] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:08.758538 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.739486  2288 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:08.731989  2192 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:08.757671 27733 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:08.728917  2730 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:08.766856 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:08.755484 27485 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.747165  2741 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:08.714757  2387 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.776790 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.754468  2679 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:08.787263 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.786580 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:08.776048  2730 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:08.788607 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:08.793143 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:08.795264 27485 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 16 to 20, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 20 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.766480  2723 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:08.784948  2435 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:08.787197  2412 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:08.812265 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 99696.8us
I20260709 12:21:08.785487  2679 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:08.813920 27485 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 21, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 21 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:08.790787  2677 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:08.824155  2795 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:08.825652  2784 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:21:08.825822 27614 heartbeater.cc:646] Failed to heartbeat to 127.25.202.62:43715 (5 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.25.202.62:43715 timed out after 15.000s (SENT)
I20260709 12:21:08.827009  2791 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:08.827529  2787 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:21:08.827450 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:08.828313  2790 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:08.828329  2723 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.830708 27730 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:21:08.819509  2435 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:08.835438 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 154644us
I20260709 12:21:08.883096  2793 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:08.834993  2679 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:08.883899 27485 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:08.885089  2795 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.886059  2288 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.820660  2192 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:08.984194 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.788444  2743 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.250s
I20260709 12:21:08.838807  2677 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:08.987041  2435 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:08.997078  2723 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:08.998031  2793 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:08.999672 27485 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 18, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:08.812587  2451 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:08.955005  2790 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:08.999468  2577 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.010088  2677 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:09.038233  2793 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.037690 27485 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 14 to 18, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.039145 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.039723  2783 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:21:09.040169 27586 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:09.040371  2783 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:09.041615  2783 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:09.043020  2785 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.288s	user 0.000s	sys 0.001s
I20260709 12:21:09.045558  2387 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.047578  2574 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.047237  2791 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:21:09.024345  2790 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:09.048602  2540 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.049245  2577 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:09.062557 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 358654us
I20260709 12:21:09.083906 27732 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 12:21:09.084101 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 317344us
I20260709 12:21:08.836439  2683 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.008633  2192 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:09.073987 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 296830us
I20260709 12:21:09.025422  2600 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 12:21:09.110962 28332 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: []
W20260709 12:21:09.114423 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 233173us
I20260709 12:21:09.039389  2794 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.312s
I20260709 12:21:09.098027  2741 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:09.128599 27485 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 14 to 17, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 17 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:21:09.114621  2667 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.001s
I20260709 12:21:09.131012 27485 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 13 to 16, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 16 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.138238 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:09.145262 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.145943 27734 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:09.149869 27657 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:21:09.164306  2577 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:09.165202  2806 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:21:09.170128 27585 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:09.115284  2600 consensus_queue.cc:1243] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 23.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:21:09.173830 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.175249 27485 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 17, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 17 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.180480 27588 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:09.182287 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75253.4us
I20260709 12:21:09.183668 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.188946 27485 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:09.196170 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58013.1us
I20260709 12:21:09.142674  2667 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.139144  2747 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:21:09.202785  2786 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:09.203300  2786 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:09.203722  2786 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:09.208966 27485 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:09.187058  2723 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:09.117219  2412 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.220307  2192 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.186990  2387 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.213177  2795 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.218811  2791 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.219412  2380 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.219704 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:09.226284 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:09.221951 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:09.146021  2679 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.228120 27485 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:09.229803 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.233824  2387 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.233983  2380 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.226651 28331 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:09.239027 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.239781  2642 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:21:09.240264  2642 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:09.240320  2811 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:09.240917  2811 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:09.241276  2811 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:09.240451 27485 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 12 to 13, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 13 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:09.213320  2723 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.211850  2810 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.244864 27485 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 15, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 15 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:21:09.234019  2760 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.637s	user 0.012s	sys 0.000s
I20260709 12:21:09.248924  2667 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.250908 27733 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:09.253456  2679 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.241320  2642 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:09.255014 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79448.2us
I20260709 12:21:09.255156  2380 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.255642  2387 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.257125 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.259178 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:09.244626  2723 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:09.262140 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:09.260572 27485 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 8 to 11, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 11 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.243345  2791 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:09.268585 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:09.273052 28328 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:09.276365 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:09.280369 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:09.276151  2791 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:09.282562  2791 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:09.259428  2192 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.283605  2192 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.286146  2192 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.286718  2780 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:09.287185  2780 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:09.282634  2677 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.291888  2677 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:09.297645  2799 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 12:21:09.301397  2435 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.304311  2435 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.263373  2810 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.305892 27485 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 19, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 19 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.312129  2677 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:09.279284  2380 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:09.335494  2677 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:09.336093  2677 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:09.339795 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.342411 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.343271 27663 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
W20260709 12:21:09.344606 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82604us
I20260709 12:21:09.313406 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.315280  2730 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.262950  2760 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.319286 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.319548 27735 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:09.319933  2822 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:09.322221 27655 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:21:09.322609 27660 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:21:09.322731 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.326016  2679 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:09.326179 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.326150 27661 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:09.280982  2642 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:09.328817  2810 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.329739 28328 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:21:09.332556  2779 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.829s	user 0.017s	sys 0.021s
I20260709 12:21:09.354018 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:09.359651 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47099.8us
I20260709 12:21:09.372675 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.373082 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.376464 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.376843 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.377050 27580 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:21:09.377655 27585 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:09.372782 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.380906 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.383399 27661 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:21:09.384114 27589 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:09.385794 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:09.356432  2714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.014s	sys 0.011s
I20260709 12:21:09.393890 27588 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:09.295114  2811 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.358562  2526 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.275616  2790 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.399926 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:09.397446 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51647us
I20260709 12:21:09.372763  2540 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:09.383538  2584 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.400507 27586 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:09.402244 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.404281 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.406236 27734 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:21:09.381490  2742 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:09.398886  2714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.955s	user 0.014s	sys 0.016s
I20260709 12:21:09.416761 27659 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:09.419437  2574 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:09.420208  2574 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.420830  2574 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.424481 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.392444  2810 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:09.424659 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3394) took 1242 ms. Trace:
I20260709 12:21:09.427503 27661 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:21:09.430830 27663 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 26. Ops: []
I20260709 12:21:09.432057 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:09.435607 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.436743  2790 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.436900  2730 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.438653 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.439342 27588 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:09.343267  2780 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.426939  2742 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.443202  2723 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.444929  2791 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.401711  2800 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:21:09.447486 27582 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:09.447913  2791 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.443444  2810 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.450224  2810 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:09.428812  2686 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.778s	user 0.007s	sys 0.005s
I20260709 12:21:09.451728  2686 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.452169 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.453006 27587 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:09.454089  2686 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.454821 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.455294 27659 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:09.455483 27587 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:09.403754  2526 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.452252  2742 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.460464 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.463822 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.464963 27731 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:09.467656 27580 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:21:09.471769 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.472718 27586 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:09.474092 27588 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:09.447085  2811 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.426229  2574 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.395159  2786 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.412681  2584 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.443202  2780 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:09.474901  2451 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:21:09.479628  2723 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.479740 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.514955 27656 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:09.429577  2822 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:09.462090  2800 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:09.485922 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.427137 27662 rpcz_store.cc:276] 0709 12:21:08.182228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.182368 (+   140us) service_pool.cc:224] Handling call
0709 12:21:08.182691 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:08.394569 (+211878us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:21:08.394609 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:08.411588 (+ 16979us) log.cc:844] Serialized 26 byte log entry
0709 12:21:08.523473 (+111885us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:08.523518 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:09.424223 (+900705us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:09.424276 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:09.424328 (+    52us) raft_consensus.cc:1685] finished
0709 12:21:09.424340 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:09.424574 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44995,"raft.queue_time_us":47940,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":1586,"spinlock_wait_cycles":360309504,"thread_start_us":111507,"threads_started":1,"wal-append.queue_time_us":148139}
I20260709 12:21:09.492862 27729 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:09.493435 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:09.504928 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.507066 27731 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:09.517825  2742 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.518391  2742 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.518720 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.519003  2584 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.519294 27662 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:09.520839  2742 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.522835  2786 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.523514 27659 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:09.524552  2526 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.525466  2540 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:09.526444 27655 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:09.527781 27662 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:09.527467  2822 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.528741 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.529435 27662 raft_consensus.cc:2368] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:21:09.529860  2822 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.530560 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.531143 27662 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:09.530917  2800 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.531654 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:09.532790 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73031.2us
I20260709 12:21:09.533856  2800 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.534113 27730 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:21:09.534476  2800 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:21:09.536190  2781 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.046s	user 0.000s	sys 0.012s
I20260709 12:21:09.536489 27730 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:09.537736 27730 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:21:09.538076 27585 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:09.539789 27738 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:21:09.541190 27738 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:21:09.541085 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.541539 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.542296 27585 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:09.542568 27737 pending_rounds.cc:85] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 11
I20260709 12:21:09.542903 27737 pending_rounds.cc:107] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 23.12
I20260709 12:21:09.543016  2796 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.543705  2796 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.544099 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.510358  2380 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:09.517227  2677 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.549860  2677 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.513737  2412 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:09.512768 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.510481 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.550695 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:09.543218 27737 raft_consensus.cc:2889] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:09.552989 27660 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 17: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:09.540984 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.539733 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.555315 27580 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:09.538090 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:09.557471 27734 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:21:09.558614 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.559719  2600 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:21:09.561203 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.561578  2684 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.561895 27729 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:09.562065  2684 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.562690 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.563759 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:09.564277  2730 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:21:09.564994 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.565434  2794 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.565956  2794 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.566484 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:09.566916 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.567484  2747 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:09.567960 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.568532 27660 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:09.567994  2747 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:09.569327 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.570176  2747 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.550843  2412 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.572822  2412 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.554418  2800 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:09.574609 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.574898  2679 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:09.574715  2800 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.576601 27735 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:09.565039  2730 consensus_queue.cc:1243] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:21:09.570619 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.566366 27588 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
W20260709 12:21:09.570619  2801 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.571s	user 0.000s	sys 0.003s
W20260709 12:21:09.586702 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 200946us
I20260709 12:21:09.583614 27587 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:09.587895 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.587431 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.590059 28332 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:09.590807 27661 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:09.593393 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.593463 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.593997 27656 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:21:09.594153 27580 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:09.594575 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.594750 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.592877 27730 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:21:09.592237  2677 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:09.595382 27656 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:09.596657 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.597636 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.598886 27731 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 14. Current term is 15. Ops: []
I20260709 12:21:09.599902 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.594700 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.600389 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.595769 27730 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 13
I20260709 12:21:09.601038 27730 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:21:09.600734 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.601315 27730 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:09.601699 27588 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:09.601850 27729 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:09.585953  2794 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.598263  2677 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.604509  2684 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.604684  2677 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.606007 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:09.600697 27731 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:09.586877  2679 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:09.608896 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:09.609373 27662 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:09.613273 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.598405  2380 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:09.612208 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.596112  2796 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.615293 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.619519 27656 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:09.616681 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:09.620098 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.620759 27655 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:09.620883 27580 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:09.605535  2783 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:09.614228  2600 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:21:09.625376 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.594041  2526 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.021s
I20260709 12:21:09.612218  2540 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:09.615751  2822 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:09.625761 27662 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:09.628881 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:09.629925  2597 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:09.632882 27655 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:21:09.657658 27586 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:21:09.657413 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.659035 27656 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:21:09.622844  2412 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.610157  2742 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:21:09.625195  2794 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:09.658754 27731 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:09.619158  2741 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:09.618956  2380 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:09.613559  2684 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
W20260709 12:21:09.670274 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54012us
I20260709 12:21:09.677891 27580 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:21:09.631837  2796 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.679949 27662 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:21:09.660991 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.615442  2642 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 12:21:09.662454 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.668401 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.684190 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.684491 27656 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:09.681792 27736 raft_consensus.cc:2376] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25. Re-sending same reply.
I20260709 12:21:09.688912 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.660116  2597 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:09.692035 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57589us
I20260709 12:21:09.670482  2584 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.693503 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.695456  2790 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.695698  2833 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:21:09.696321 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.695528 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.697017  2387 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:09.696900  2807 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.078s
I20260709 12:21:09.697512  2811 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.697667  2821 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:21:09.698056  2809 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.698982 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.703408 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.703526 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.685648  2540 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.691192  2742 consensus_queue.cc:1243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:21:09.681026  2380 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:09.717310 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.630641  2783 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.729326  2795 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:09.729455  2783 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.730223  2795 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.731493  2811 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.733893  2811 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.736116 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3311) took 1026 ms. Trace:
I20260709 12:21:09.736634 27733 rpcz_store.cc:276] 0709 12:21:08.709849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.774302 (+ 64453us) service_pool.cc:224] Handling call
0709 12:21:09.735889 (+961587us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:09.736054 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1048699264}
I20260709 12:21:09.737632  2807 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:21:09.738306  2836 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:21:09.707834 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.738673  2809 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.740535  2793 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:21:09.740991  2793 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:09.741659  2809 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.742085  2793 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.706161  2584 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.748114  2584 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.689639  2796 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.749073 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.750667  2796 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.713863 27656 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:09.752120 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.676661  2794 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.753800  2794 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:09.753278 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:09.743187  2679 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.758260 27734 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:09.759619 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:09.760725 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.761343  2679 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.691865  2822 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:09.711439  2790 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.766134  2822 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.676292  2730 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:21:09.721711  2821 consensus_queue.cc:1243] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:21:09.721312  2540 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.721891 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.802083 27729 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:09.722491  2387 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.724097 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3395) took 1541 ms. Trace:
I20260709 12:21:09.726751  2683 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:09.717599 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.761304 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.771121 27662 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:09.777858 27735 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:09.786285 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.806412 27730 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:09.803951 27658 rpcz_store.cc:276] 0709 12:21:08.182607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.182784 (+   177us) service_pool.cc:224] Handling call
0709 12:21:08.183293 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:08.737006 (+553713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:09.723822 (+986816us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:09.724032 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1267954944}
I20260709 12:21:09.759009  2795 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:09.767710  2790 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:09.809684  2584 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 12:21:09.809620 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.785347 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.690526  2412 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.811276 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:09.690218  2741 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.831007 27586 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:21:09.834355 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.835573 27731 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:09.828606 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:09.836691 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:09.846619 27658 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:09.846550 27580 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:09.815009  2730 consensus_queue.cc:1243] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 26.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:21:09.813989  2793 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:21:09.854077 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:09.848837 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:09.852816 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:09.864050 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.862627 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:09.857304 27584 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:09.867774 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:09.839129  2790 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:09.847944  2684 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:09.846369  2803 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:09.899091 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 221653us
I20260709 12:21:09.868499  2723 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.866683  2791 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:09.908097 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:09.874577  2677 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:09.923578 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.901877  2584 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:21:09.920994 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68319.2us
I20260709 12:21:09.870077  2526 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:21:09.930177  2847 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:09.902647  2790 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:09.859525  2820 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:21:09.939469 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.941607 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:09.931744 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:09.936193 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67480.7us
I20260709 12:21:09.928537  2800 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:09.980587 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48969.5us
I20260709 12:21:09.933405  2791 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:09.936985  2847 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.918675  2803 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:09.924201  2723 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.898532  2597 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.996654 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:09.924015  2380 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:09.933074  2815 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.115s
I20260709 12:21:10.015904  2741 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:10.016878  2741 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:09.947223  2584 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:10.020447 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:09.987164  2679 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:10.030810 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:10.033913  2831 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:10.034580  2831 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.034977  2831 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:10.034986  2412 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:10.034025  2387 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:09.995437  2811 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:10.037147 27660 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.15-28.15]
I20260709 12:21:10.033360 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.039290 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:09.996476  2526 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:09.994887  2847 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:10.018883  2742 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:10.048005 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.046130 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:21:10.041261  2841 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.061s	user 0.003s	sys 0.000s
I20260709 12:21:10.049597 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:10.054535 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 146524us
I20260709 12:21:10.053053 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.055294 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.036085  2810 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:10.058830 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:10.050547  2677 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:10.062963  2844 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:10.063555  2723 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.063310  2677 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.063696  2844 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.066248  2844 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:10.039707  2540 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.068804 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:10.069487 27659 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:10.069808 27730 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:10.075508 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:10.026010  2667 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:21:10.061334  2810 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.053869  2526 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.086552  2760 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.081532  2810 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:10.091663  2854 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:10.093327 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.078197 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:10.094652 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.076459 27659 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
W20260709 12:21:10.102795 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 113093us
I20260709 12:21:10.049678  2857 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:10.097162 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:10.119935 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.121027 28332 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:10.133709 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:10.136592 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:10.082261  2540 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:10.069976  2730 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
W20260709 12:21:10.096773  2842 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:21:10.139948 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.141995 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:10.149612 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:10.140254 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148391us
I20260709 12:21:10.152933 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.103561  2546 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.149243 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:10.076229  2791 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.152165 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3312) took 1428 ms. Trace:
I20260709 12:21:10.156774 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.166759 28330 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:10.158830 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:10.160086 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:10.069345  2790 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.133086  2811 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:10.092341  2854 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.164103 27732 rpcz_store.cc:276] 0709 12:21:08.715362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.777379 (+ 62017us) service_pool.cc:224] Handling call
0709 12:21:09.064059 (+286680us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:09.084648 (+ 20589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.139618 (+1054970us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54002e7cd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:10.143921 (+  4303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":658241920}
I20260709 12:21:10.138759  2597 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:10.174216 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:10.148092  2857 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:10.153401  2684 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:10.193192 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.203444 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.204759 27584 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:10.206028 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:10.206386  2795 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:10.208043 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.209437 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.210692 27580 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:21:10.210763 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69530.3us
I20260709 12:21:10.150756  2486 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:21:10.211279 27580 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:10.213464 27587 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:10.237110 27587 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:10.221066 27662 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:10.223883 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.174463  2811 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:10.246599 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.250483 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:21:10.104671  2584 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.168967  2829 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:10.252206 27664 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:10.194413  2723 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:10.259945 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.261679 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.243212  2714 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:10.225013  2540 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:10.262631 27580 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.275862 27580 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:10.276230 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:10.276419 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:10.171703  2546 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:10.268761 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 121371us
W20260709 12:21:10.288619 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129877us
I20260709 12:21:10.254904  2380 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:10.342612 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184480us
I20260709 12:21:10.273352 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:10.277537  2803 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:10.371891  2841 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:21:10.394487 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:10.395882 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 122593us
I20260709 12:21:10.359359  2851 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:10.252084  2677 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.262573 27587 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.254925  2829 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:10.344486 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.247311  2684 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:10.344983  2843 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:21:10.353219  2856 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:10.356280  2526 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.358237  2546 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.273902  2714 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.265982  2730 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:10.370401 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:10.374265 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.378305 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:10.379635 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.254267  2667 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:21:10.388475  2874 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.372349  2811 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.388769  2486 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:21:10.445210 27587 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:10.430456 28333 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:21:10.471700 28330 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:21:10.470669 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 100368us
I20260709 12:21:10.391750  2815 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:21:10.386687  2857 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.467680  2714 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:10.554023 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95961.1us
I20260709 12:21:10.571046 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:10.487896 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138279us
I20260709 12:21:10.498064 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.573926 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:10.568593 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2928) took 1038 ms. Trace:
I20260709 12:21:10.468107  2865 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:10.560030 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1277) took 1358 ms. Trace:
I20260709 12:21:10.428571 27580 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:21:10.440047  2380 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:10.581588 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:10.439663  2829 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:10.497318 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.445874  2856 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.576355 27735 rpcz_store.cc:276] 0709 12:21:09.530050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.531422 (+  1372us) service_pool.cc:224] Handling call
0709 12:21:10.568407 (+1036985us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:10.568524 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101631744}
I20260709 12:21:10.563910  2796 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:10.473809  2870 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:10.588200 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.588231 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:10.623267 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:10.655793 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1033) took 1086 ms. Trace:
I20260709 12:21:10.445319  2677 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.559012  2854 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:10.620394  2794 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.866s	user 0.013s	sys 0.001s
I20260709 12:21:10.656253 28328 rpcz_store.cc:276] 0709 12:21:09.545545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.558961 (+ 13416us) service_pool.cc:224] Handling call
0709 12:21:09.559233 (+   272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:09.593560 (+ 34327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:09.593589 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:09.609876 (+ 16287us) log.cc:844] Serialized 26 byte log entry
0709 12:21:09.770243 (+160367us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:09.770304 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:09.772801 (+  2497us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:10.631896 (+859095us) raft_consensus.cc:1685] finished
0709 12:21:10.631918 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.632063 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33017,"raft.queue_time_us":29767,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":5813,"thread_start_us":159832,"threads_started":1,"wal-append.queue_time_us":221983}
I20260709 12:21:10.550956  2791 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:10.586519  2380 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.652427 27738 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:10.569383  2838 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:21:10.657466 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69235.4us
I20260709 12:21:10.570647  2730 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.620398  2820 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:21:10.665047 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 77036.7us
I20260709 12:21:10.667157 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:10.663046 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.587158  2793 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:21:10.662985  2777 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:21:10.669059  2838 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:10.669960  2380 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:10.654960  2811 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:10.659106  2822 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:10.689491  2884 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:10.658646  2865 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:10.669133  2873 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 12:21:10.682871 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:10.762391 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140980us
I20260709 12:21:10.682454 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:10.501175  2874 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:10.700129 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.556878  2546 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.105s
I20260709 12:21:10.627065  2856 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:10.700484 27587 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:21:10.631198  2803 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:21:10.703303 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:10.778091 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75850.9us
I20260709 12:21:10.632534  2796 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:10.623378  2868 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.304s
W20260709 12:21:10.711585 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 287853us
I20260709 12:21:10.674136  2791 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.669536  2794 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:10.723613 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.724795  2871 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:21:10.735924  2864 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.350s	user 0.000s	sys 0.001s
W20260709 12:21:10.739601  2877 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:21:10.719810  2838 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.776527  2686 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:10.704587  2870 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:21:10.767431 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:10.774369 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 99022.5us
I20260709 12:21:10.800371 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.776001 27587 pending_rounds.cc:85] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:10.801466 27587 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:21:10.801833 27587 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:10.790542 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.794144 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.724092  2875 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:10.806872  2838 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:10.788739  2822 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:21:10.809345  2831 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.005s
W20260709 12:21:10.816442  2810 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.728s	user 0.012s	sys 0.000s
I20260709 12:21:10.817049  2810 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:10.820004 28336 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:10.797520  2809 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
W20260709 12:21:10.822230  2880 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.006s	sys 0.000s
I20260709 12:21:10.823714  2838 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:21:10.824229 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47827.5us
I20260709 12:21:10.825006 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:10.825402 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.827108 27580 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:10.827559 27580 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:10.827617  2852 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:21:10.827997 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.828756 27586 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:10.828625 27580 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.829748 27580 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:10.829768 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.830862 27586 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:10.831669 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:10.827476  2600 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:10.833074  2600 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:10.832589 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.835099  2600 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:10.835799 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3550) took 1483 ms (client timeout 1914 ms). Trace:
W20260709 12:21:10.836508 27582 rpcz_store.cc:269] 0709 12:21:09.352569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.358442 (+  5873us) service_pool.cc:224] Handling call
0709 12:21:10.835594 (+1477152us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:10.835715 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151690496}
I20260709 12:21:10.836980 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3528) took 1273 ms. Trace:
I20260709 12:21:10.837963 27580 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 12:21:10.838481 27580 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 12
I20260709 12:21:10.838929 27580 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 12:21:10.835207 27585 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:10.839784 27580 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:10.840237 27585 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:10.831135  2882 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:21:10.849646 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:10.841770 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3530) took 1246 ms. Trace:
I20260709 12:21:10.837409 27583 rpcz_store.cc:276] 0709 12:21:09.563161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.567771 (+  4610us) service_pool.cc:224] Handling call
0709 12:21:10.835730 (+1267959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.835919 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.835943 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:10.835984 (+    41us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:10.836013 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.836066 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.836809 (+   743us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001301d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:10.836923 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2915310976}
I20260709 12:21:10.842298 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:10.853896 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:10.851610 27587 rpcz_store.cc:276] 0709 12:21:09.595363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.595506 (+   143us) service_pool.cc:224] Handling call
0709 12:21:09.608823 (+ 13317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.213348 (+604525us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54000d1ea0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:21:10.841541 (+628193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.841717 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232656768}
I20260709 12:21:10.845058 27586 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:10.856926 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.857568 27734 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:21:10.858069 27732 raft_consensus.cc:1217] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Deduplicated request from leader. Original: 16.8->[17.9-17.9]   Dedup: 17.9->[]
I20260709 12:21:10.859071 27582 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:21:10.861274 27738 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:10.861418 27732 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
W20260709 12:21:10.866014  2881 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:21:10.865834 27585 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.867664 27729 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 15. Current term is 16. Ops: []
I20260709 12:21:10.832585  2838 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:10.871944 27659 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:10.877743 27585 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:10.854966  2600 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:10.888268 27738 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:21:10.881481 27583 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:10.883772 27736 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:21:10.822949  2887 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
W20260709 12:21:10.864494  2642 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
W20260709 12:21:10.901952 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 98687.7us
I20260709 12:21:10.870700  2838 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:10.911386 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54597.6us
I20260709 12:21:10.918422 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.904619 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:10.919646 27585 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:21:10.921387 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3525) took 1359 ms. Trace:
I20260709 12:21:10.922484 27585 rpcz_store.cc:276] 0709 12:21:09.562327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.582122 (+ 19795us) service_pool.cc:224] Handling call
0709 12:21:09.585358 (+  3236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.835117 (+1249759us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b540041bca0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:21:10.921126 (+ 86009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.921330 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3588864}
I20260709 12:21:10.891963  2584 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:21:10.903106  2747 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:21:10.899062  2444 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:10.900687  2684 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.942737  2192 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:10.903891  2810 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.904435  2600 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:10.927762  2800 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.945s	user 0.013s	sys 0.001s
I20260709 12:21:10.927139  2584 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.924437  2838 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:10.948670  2747 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.908695  2730 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:10.862360  2796 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:10.959512 27656 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:10.959626 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:10.972193 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67651.5us
I20260709 12:21:10.959878 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:10.960397  2192 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:21:10.960850 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 135621us
I20260709 12:21:10.909190  2677 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:10.940909  2444 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:10.974537 28328 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:21:10.976023 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:10.958053  2584 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:10.962419  2800 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:10.993399  2796 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:11.005996 27737 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:21:11.001187  2883 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 12:21:11.015439  2600 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.015894  2886 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.996519  2879 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:10.971299  2747 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.069873 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67111.7us
I20260709 12:21:11.100382 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:11.018929  2444 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.103031 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 106567us
W20260709 12:21:11.097002  2597 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.908s	user 0.011s	sys 0.011s
I20260709 12:21:11.077603  2192 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:11.123641 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2826) took 1574 ms. Trace:
I20260709 12:21:11.094826  2879 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:11.125577  2910 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:21:11.130316  2747 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.095579  2856 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.132725 27660 rpcz_store.cc:276] 0709 12:21:09.549382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.615385 (+ 66003us) service_pool.cc:224] Handling call
0709 12:21:09.615805 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.044754 (+428949us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.123372 (+1078618us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:11.123582 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":968305920}
I20260709 12:21:11.136426 27588 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:21:10.809804  2831 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.166302  2831 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.166695  2907 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:10.877489  2865 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.141182  2879 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.124223  2895 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:11.167896  2895 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:11.168320  2895 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:11.169423  2879 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.145802 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.130982  2891 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.148s
I20260709 12:21:11.027865  2886 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.154152  2838 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.155942  2893 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 12:21:11.156747  2898 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:10.831483  2875 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.103009  2380 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.191480  2894 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:11.192039  2894 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.193077  2875 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.193220  2380 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.194062  2894 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.174290  2868 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:11.160712 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:11.161208  2884 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:11.161619 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.160732  2815 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:11.163617 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.172616 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:11.213420 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:11.175626  2486 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.156752  2856 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.183758 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:11.136090  2683 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.326s	user 0.014s	sys 0.001s
W20260709 12:21:11.184942 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84680.8us
I20260709 12:21:11.185133 27662 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:11.188033 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:11.188457 27583 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:11.188457 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:11.296377 27583 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:11.296834 28335 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:21:11.297248 27583 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.298344 27583 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:11.317202 28335 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:11.321223 27583 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:11.322674  2895 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.159394  2829 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.329782  2899 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.330403  2899 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.331517 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63585.9us
I20260709 12:21:11.332191  2790 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:11.332757  2790 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.202150 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.334544 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.162520 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:11.335678  2902 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.214985 27655 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:11.148128 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.336160  2902 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.338356  2902 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.136369  2192 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.340348  2192 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.239112 28331 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:11.341121  2526 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.341574  2526 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.342427  2692 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:11.342860  2692 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:11.244531  2867 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.870s	user 0.007s	sys 0.000s
I20260709 12:21:11.344444  2895 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.214583  2815 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:11.352725  2815 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.354301  2815 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:11.211190  2884 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:11.213788  2907 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:11.241533  2486 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.213574  2868 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.252966  2886 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.253902  2838 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.254515  2846 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
W20260709 12:21:11.270133  2847 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.200s	user 0.008s	sys 0.003s
I20260709 12:21:11.377750  2884 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.378929  2847 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.237746  2800 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:21:11.271997  2893 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:11.196316  2584 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.348559  2526 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.348667  2790 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:11.282395  2714 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.001s
I20260709 12:21:11.366546 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:11.408079 28328 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:11.407891  2714 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.410207  2714 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.379347  2884 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:11.217453 27552 leader_election.cc:400] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:21:11.411482 27552 leader_election.cc:403] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:21:11.412317 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 195008us
I20260709 12:21:11.413076 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:11.386348  2847 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.415766 27658 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:11.388849 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:11.388797  2692 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.390273  2486 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.416777  2692 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:11.416828  2584 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.418010  2760 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:21:11.399866 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.418470 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:11.419166 28328 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:10.577725 27657 rpcz_store.cc:276] 0709 12:21:09.201573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.201732 (+   159us) service_pool.cc:224] Handling call
0709 12:21:09.211080 (+  9348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:09.211274 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:21:09.376022 (+164748us) log.cc:844] Serialized 14 byte log entry
0709 12:21:09.380203 (+  4181us) log.cc:844] Serialized 14 byte log entry
0709 12:21:10.555608 (+1175405us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b180030c9a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e99efed70 00007f0e99ef2293 00007f0e99ee7cea 00007f0e99ee823a 00007f0e9a352a2a 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:10.559560 (+  3952us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:10.559631 (+    71us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:10.559696 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.559794 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.559949 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10844,"prepare.queue_time_us":362353,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":12976,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":8627,"spinlock_wait_cycles":3357440,"thread_start_us":164376,"threads_started":1}
I20260709 12:21:11.391314  2907 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.423870 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1195) took 1234 ms. Trace:
I20260709 12:21:11.424311 28336 rpcz_store.cc:276] 0709 12:21:10.189898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.190129 (+   231us) service_pool.cc:224] Handling call
0709 12:21:11.423615 (+1233486us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:11.423774 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1334075136}
I20260709 12:21:11.401600  2829 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.401822 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:11.402257  2838 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.427758  2838 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.429919  2838 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.403867  2899 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.405457 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:11.415756  2853 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:21:11.375216  2891 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.400897  2868 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.435868 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:11.472855 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.466005 28328 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
W20260709 12:21:11.472874 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57968.2us
I20260709 12:21:11.472959 28335 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:11.438205  2800 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:11.478125  2800 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:11.479493 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43735.7us
I20260709 12:21:11.480345  2800 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.480592 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:11.482470 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:11.484002 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:11.443602 27704 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.628s (SENT) [suppressed 11 similar messages]
I20260709 12:21:11.483152 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:11.418545  2760 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:11.488907 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:11.492053 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:11.482115  2444 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:11.493413  2899 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:11.493633 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:11.486882 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:11.494072  2899 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:11.495481 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:11.496687 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:11.496502  2899 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.497589  2833 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:11.498014  2833 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:11.498409  2833 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:11.485855  2887 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:21:11.482118  2790 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:11.506424 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:11.499806  2809 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:21:11.500684  2800 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:11.502084  2892 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:11.504798  2846 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:11.502617  2887 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:11.521740  2578 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:11.533051  2838 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:11.533993  2838 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:11.534365  2838 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:11.522540  2851 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:11.503558  2760 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.523032  2809 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:11.523481  2906 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:11.523826 28335 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:11.524580 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3331) took 1585 ms. Trace:
I20260709 12:21:11.544904  2856 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:11.545924  2856 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:11.546303  2856 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:11.527693  2846 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:11.543000 27737 rpcz_store.cc:276] 0709 12:21:09.938797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.939178 (+   381us) service_pool.cc:224] Handling call
0709 12:21:11.524318 (+1585140us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:11.524482 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2451265280}
I20260709 12:21:11.547693  2846 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:11.536839  2887 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:21:11.547183  2856 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.539227  2851 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:11.520380 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:11.512399  2847 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:11.536834  2838 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.569253  2791 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:11.530912  2743 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 12:21:11.531589 27732 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:11.511628  2790 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:11.538694  2901 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:11.539821 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:11.553206 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:11.616385 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.622237 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69180.8us
I20260709 12:21:11.542274  2892 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.014s
I20260709 12:21:11.550415  2863 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:21:11.547528  2809 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:21:11.576114 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3543) took 1590 ms (client timeout 1628 ms). Trace:
I20260709 12:21:11.753926 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3398) took 2074 ms. Trace:
W20260709 12:21:11.747444 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.624142 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 180593us
W20260709 12:21:11.715595 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.744900 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.591010  2791 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.541793  2853 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.104s
I20260709 12:21:11.484556  2891 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.621675  2807 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:21:11.578531  2901 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:11.809937  2901 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:11.625276  2790 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:11.787469 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.575544  2800 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:11.527019  2907 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.098s
I20260709 12:21:11.754539 27659 rpcz_store.cc:276] 0709 12:21:09.679139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.146137 (+466998us) service_pool.cc:224] Handling call
0709 12:21:10.146493 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.872473 (+725980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.753635 (+881162us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:11.753861 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667720576}
W20260709 12:21:11.753772 27588 rpcz_store.cc:269] 0709 12:21:09.985506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.117838 (+132332us) service_pool.cc:224] Handling call
0709 12:21:11.575782 (+1457944us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:11.576016 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2260197632}
I20260709 12:21:11.797134  2809 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.786028  2444 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.762351 27581 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:21:11.805377  2891 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.806028  2853 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:11.806886  2887 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.761476 27584 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:21:11.744714  2847 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:11.807529 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.760546 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:11.754644  2850 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:11.550390  2856 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:11.763451 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.787792 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 248116us
I20260709 12:21:11.812604  2796 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.826169 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:11.826864 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.830577 27588 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:21:11.815101  2865 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.811172  2833 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.823648  2853 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:11.814785  2791 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.847635 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:11.826908  2692 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:11.852926 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:11.838589 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3573) took 1579 ms. Trace:
I20260709 12:21:11.857359 27589 rpcz_store.cc:276] 0709 12:21:10.258904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.352370 (+ 93466us) service_pool.cc:224] Handling call
0709 12:21:10.352684 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.703229 (+350545us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.703264 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:10.703312 (+    48us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:10.703363 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.838347 (+1134984us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54000bc7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:11.838398 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.838530 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":805625728}
I20260709 12:21:11.859709 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:11.859700 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.863516 27662 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:11.868275 27660 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:11.838454  2794 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.876051 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:11.876995 27655 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:11.831785  2684 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:11.820787  2809 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.822481  2444 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.897629 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.840266  2850 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:11.885187 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:11.885653 27663 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:11.885571 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.890381  2791 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:11.890715 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1088) took 2294 ms (client timeout 1854 ms). Trace:
I20260709 12:21:11.905229 27657 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:11.905967 28331 rpcz_store.cc:269] 0709 12:21:09.596614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.605768 (+  9154us) service_pool.cc:224] Handling call
0709 12:21:11.890482 (+2284714us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400153454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:11.890597 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3754085760}
I20260709 12:21:11.893175  2783 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:11.834549  2815 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:11.884624 27737 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:11.911789  2815 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.911991 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:11.907155  2783 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:11.912879 27729 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:11.914250  2815 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.913952  2783 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.903622 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:11.896536 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2362155630000871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20260709 12:21:11.904655 27731 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:11.884684  2807 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:21:11.915061  2783 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:11.919966 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.833767  2796 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.924027 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:11.845118  2865 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.884817  2892 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:11.929123 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:11.931205  2815 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.003s
I20260709 12:21:11.932523  2578 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:11.933290  2578 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.935120  2846 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.935894 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.932515  2794 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:11.938724 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:11.938599  2794 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.940454 27736 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:11.878593  2692 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.939934  2846 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.949753  2870 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.951880 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.898682  2684 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.956998 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:11.959621 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:11.963625 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:11.942965  2741 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:11.966568 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.969296 27663 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:11.967440 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:11.970578 27734 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:11.946993  2486 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:11.934063  2796 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:11.950533  2850 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.927001  2853 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.984607  2578 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.985283  2578 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:11.985862  2578 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.988734  2692 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:11.989105  2684 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.950534  2870 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:11.992334  2870 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:11.994210  2889 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.045s	user 0.000s	sys 0.010s
I20260709 12:21:11.995131  2444 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.974593  2851 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.996562 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1200) took 1624 ms. Trace:
I20260709 12:21:11.996977 28337 rpcz_store.cc:276] 0709 12:21:10.372385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.372580 (+   195us) service_pool.cc:224] Handling call
0709 12:21:10.380124 (+  7544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.469376 (+ 89252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.469421 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:10.469476 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:10.469520 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.996332 (+1526812us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:11.996382 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.996505 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204589824}
I20260709 12:21:11.997959  2851 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:11.965647  2809 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.000226  2865 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.001828  2809 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.002406  2809 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:12.001995 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:11.965095  2846 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.006119  2540 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.006346  2833 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.009481 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.014880 27662 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:12.017656 27655 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:12.022650 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.026376 27655 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:11.948181  2892 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.030948 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.037266 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:11.976577  2741 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.021261 27589 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:12.002902  2901 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:11.980239  2486 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.044023 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.021162  2865 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:12.060509  2865 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.061589  2793 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:12.062135  2793 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:12.065390 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2942) took 1218 ms. Trace:
I20260709 12:21:12.065801 27735 rpcz_store.cc:276] 0709 12:21:10.846835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.847303 (+   468us) service_pool.cc:224] Handling call
0709 12:21:10.847603 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.854151 (+  6548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.854182 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.854622 (+   440us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.957898 (+103276us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:12.049488 (+1091590us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b18002b2dc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:21:12.049625 (+   137us) log.cc:844] Serialized 14 byte log entry
0709 12:21:12.049901 (+   276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.049974 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:12.050011 (+    37us) raft_consensus.cc:1685] finished
0709 12:21:12.065199 (+ 15188us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:12.065333 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6188,"prepare.queue_time_us":52133,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":2214,"raft.queue_time_us":9078,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":22591,"spinlock_wait_cycles":7764224,"thread_start_us":163114,"threads_started":2,"wal-append.queue_time_us":211169}
I20260709 12:21:12.020624  2846 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.020080  2796 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:21:12.056938  2809 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:12.057914  2794 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:12.046373 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:12.028254  2540 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.051703 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.047274  2887 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.072599 27582 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:12.071539  2642 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.070149  2865 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.079420  2892 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.083755  2796 consensus_queue.cc:1243] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 27.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:21:12.090050 27662 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:12.084569 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.090746 27588 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:12.087813  2847 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:12.095466 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:12.103847 27655 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:12.119908 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.092139  2578 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.094784  2901 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.086764  2642 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.129417 27662 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:12.135179 27655 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:12.138098 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.139276 27655 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:12.140239 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.141186 27655 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:12.145634 27655 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:12.145996 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:12.146945  2578 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:12.142405  2793 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.161031 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:12.167088  2486 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.167632  2642 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.157555  2807 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:12.172250 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.173202 27580 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:21:12.176301  2874 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.168337  2850 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:12.186503  2540 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:12.181960  2642 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:12.183702  2846 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:12.183979  2793 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:12.184473  2876 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:12.164211  2578 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.185225  2760 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:12.186556  2741 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.191651  2869 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:12.194123 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:12.194765 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.194821 27656 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:12.193591 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:12.196664  2876 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.199610 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:12.209906 28337 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:12.232864 27664 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:12.234166 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:12.234838  2800 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:12.235509 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.236176 27664 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:12.236685  2741 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:12.236899 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.237699 27664 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:12.237421  2741 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.238632 27664 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:12.198392 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:12.190968  2807 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:12.206149 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.203649  2869 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:12.248451 27664 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:21:12.248867 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50524.4us
W20260709 12:21:12.249166 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.199611  2760 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.251713 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:12.255213 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:12.256582 27664 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:21:12.251565 27656 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:12.257061 27659 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
W20260709 12:21:12.260329 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.265632  2741 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.265820 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.269868 27660 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:12.273291  2853 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:12.275709 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.276021 27660 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:21:12.266877 27587 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:21:12.241992  2793 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:12.276525 27663 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:21:12.278666 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:12.280016 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.280340 28329 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:12.280968  2891 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:12.280900 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:12.281798 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.281996 28333 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:21:12.282342  2800 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:12.285600 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1287) took 1134 ms. Trace:
I20260709 12:21:12.282872 27657 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:21:12.285993 27658 rpcz_store.cc:276] 0709 12:21:11.142151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.142288 (+   137us) service_pool.cc:224] Handling call
0709 12:21:12.276140 (+1133852us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:12.276877 (+   737us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":469423872}
I20260709 12:21:12.285691  2796 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:21:12.288434 27658 raft_consensus.cc:1217] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Deduplicated request from leader. Original: 22.12->[24.13-25.14]   Dedup: 25.14->[]
I20260709 12:21:12.288926 27658 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: [24.13-25.14]
I20260709 12:21:12.282961  2847 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:21:12.289659  2847 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:12.290001 27658 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:12.290444  2807 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.292246 28334 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:12.294173  2869 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.295945 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.296602 28333 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:12.299517  2793 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:12.300166  2288 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:12.300572  2288 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:12.300468  2760 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.300947  2793 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.302134  2760 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:12.301885  2288 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.303545  2891 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.303823  2288 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:12.304920  2891 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.306733  2891 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:12.309496 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2952) took 1452 ms. Trace:
I20260709 12:21:12.309880 27732 rpcz_store.cc:276] 0709 12:21:10.856689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.862188 (+  5499us) service_pool.cc:224] Handling call
0709 12:21:11.526841 (+664653us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:11.573171 (+ 46330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.309316 (+736145us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b540017f7d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:12.309439 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527789440}
I20260709 12:21:12.310871  2876 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.312332  2876 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:12.316469 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.320144 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:12.322175 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67050.3us
I20260709 12:21:12.323736 28331 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:12.325484 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.323129 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:12.326725 28329 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:12.328210 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.254760  2850 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:12.333905 27655 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:21:12.340588 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.342940 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.346014 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.347579 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.348418 27581 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
W20260709 12:21:12.349916 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.346905 27584 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:12.352494 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.353268 28332 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:12.343959 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.358366 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.360082 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:12.323864  2642 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:12.333179  2868 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:12.377763 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.341416  2809 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:12.334316  2850 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:12.285899  2846 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:21:12.290024  2847 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:12.335492  2741 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:21:12.385361 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62359.4us
W20260709 12:21:12.386325 27702 connection.cc:256] RPC call timeout handler was delayed by -1.076761038000086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1854s for a response
W20260709 12:21:12.386507 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47353.7us
I20260709 12:21:12.388077 27584 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:12.379094 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:12.379721 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.405328 28337 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:12.339871  2667 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:12.353830  2783 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:12.403815 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.384941  2743 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:12.405545 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:12.407989 27585 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:12.368132  2692 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:12.406345  2851 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:12.424273 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.401782  2850 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:12.417122 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.428434 27732 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:12.401238  2869 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:12.421988 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:12.426431 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 180509us
I20260709 12:21:12.388134  2870 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:12.410713  2546 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:21:12.417373 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:12.452105 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:12.428828 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:12.455349 27660 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:12.421710 27661 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:12.453519 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.456233 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.451335 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:12.451118 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.422972 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.396849  2863 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:21:12.419564  2800 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.413633  2868 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:21:12.432700 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64476.9us
I20260709 12:21:12.417675 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.439716 27662 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:12.440892 27659 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:12.421196 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:12.468642 27736 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:12.414952 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.422757 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.458338 27735 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:12.458671 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.427129  2851 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:12.412345  2856 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:12.449345  2743 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:21:12.466212 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:12.472849  2851 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:12.497107  2901 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:12.497601  2901 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.498437  2901 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.464686  2546 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:12.499847 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:12.500797  2546 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.501579  2639 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:12.502367  2863 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:12.502795  2863 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:12.473203  2937 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.271s	user 0.000s	sys 0.001s
I20260709 12:21:12.505082  2856 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:12.475270 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.505023  2800 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.461374  2288 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.506235  2288 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.506436  2856 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.426079  2692 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:12.487452 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.451187  2783 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.503144  2639 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:12.504114  2743 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:12.415189  2820 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:21:12.520206 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.504611  2838 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:12.458798  2870 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.504618  2901 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:12.464764  2847 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.508297  2868 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:12.508723  2288 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:12.412729  2667 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.471658 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:12.459692 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.510548 27585 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:12.477172  2578 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.521072 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:21:12.523840  2800 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:12.503186  2863 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:12.517668  2743 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.525638 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:12.560254 28337 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
I20260709 12:21:12.534873 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.561336 27584 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 23
W20260709 12:21:12.536991 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 208845us
I20260709 12:21:12.561650 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.562498 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:12.550248 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:12.551968 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:12.564437 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.564342 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:12.532975  2868 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:12.567314 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:12.534363  2809 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:12.578712 27589 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:12.522734  2783 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:12.579286 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 202076us
I20260709 12:21:12.583734 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:12.551518  2796 consensus_queue.cc:1175] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 20 status { last_received { term: 20 index: 12 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:12.588044  2870 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.588613  2870 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:12.588449  2743 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.589113  2870 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.589623  2743 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:12.565397  2847 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:12.582232 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:12.595975  2793 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:12.598973 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.599900 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.600682 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.601487 27662 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:21:12.601497 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:12.599278  2793 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:12.606060  2793 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:12.595167 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:12.623767 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.623966 27737 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:12.580370  2642 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:12.628929 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.577680  2800 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:21:12.596903  2741 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:12.645882 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.647509 27589 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:21:12.648340 27586 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:12.623473  2907 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:21:12.624639 27659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:12.651675 27659 raft_consensus.cc:606] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:12.652001 27659 raft_consensus.cc:614] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:12.652334 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.626634  2809 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:12.652992 27581 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:12.656033 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:12.656615 27731 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:12.658037 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.658205 27735 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:12.659171 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.659727 27585 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:12.670468 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.671957 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.672515 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:12.671422 27735 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
W20260709 12:21:12.685330 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 265484us
I20260709 12:21:12.690002 27735 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:12.690364 27585 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:12.692504 27735 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:21:12.692842 27735 pending_rounds.cc:85] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 13
I20260709 12:21:12.693045 27735 pending_rounds.cc:107] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:21:12.693265 27735 raft_consensus.cc:2889] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:12.695215 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.697393 27580 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:12.698057  2667 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.698577  2667 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.701750 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:12.703635 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.705000 27587 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:21:12.706001 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.707816  2741 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.709306 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.710517 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.711315 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:12.711990 27580 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:21:12.713519 27582 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:12.715962 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:12.716243  2931 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:12.716747  2931 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.716917  2540 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:12.707145 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.720175  2540 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:21:12.720539  2938 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.364s	user 0.002s	sys 0.003s
I20260709 12:21:12.721517 27658 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:21:12.723778 28329 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:21:12.725124 28337 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:12.728821  2870 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.729565  2870 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:21:12.730433  2944 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:12.732769 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:12.733609  2540 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:12.733986  2540 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:12.734364  2540 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:12.737133 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.742774 27581 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:12.736277 27729 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 28 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:21:12.697741 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:12.745929 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.746604 27731 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:12.747598 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:12.747897 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.749266 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:12.749964 27730 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:12.745428 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.749286 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.755735 27585 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:12.696436 27735 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:12.761359 27736 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:12.661682  2809 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.773687 27659 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:21:12.698979  2667 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.776412 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:12.806264 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.730875  2870 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:21:12.813669 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.684980  2853 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.108s
I20260709 12:21:12.736364  2949 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.735240  2783 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:12.773751  2878 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:12.815677 27730 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
W20260709 12:21:12.816713 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 252616us
I20260709 12:21:12.817011 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.817973 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.819026 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.691733  2951 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:21:12.822767 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 77460.3us
W20260709 12:21:12.829618 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.822970 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.788502  2829 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:12.695098  2948 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.131s
I20260709 12:21:12.831264 27731 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:12.730922  2850 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.788683  2874 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:21:12.801244  2790 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
W20260709 12:21:12.862661 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.834259  2642 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.833926  2951 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:12.850899  2742 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:21:12.891642 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78047.9us
I20260709 12:21:12.893023 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:12.901376 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.904630 28328 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
W20260709 12:21:12.905478 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 152389us
I20260709 12:21:12.863498  2950 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:12.909060 27588 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:12.914248 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 224947us
I20260709 12:21:12.827663  2949 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.879884  2857 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:12.924237 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.910362 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:12.911237 27586 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:12.919231 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.926342 27589 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:21:12.858367  2871 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:21:12.925609 27587 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:12.917217 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.880326  2951 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:12.900638  2639 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:12.904554  2804 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:12.932729 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:12.941345 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:12.933259 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:12.876160  2820 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
W20260709 12:21:12.942900 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 159174us
I20260709 12:21:12.912896  2950 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:12.907603  2955 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:12.883615  2642 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:12.934610 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 182504us
I20260709 12:21:12.880789  2851 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:12.949150 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:12.950343 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.923451  2760 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:12.944090 27580 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:21:13.014541 27580 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 12
I20260709 12:21:13.014824 27580 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:21:13.015110 27580 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:12.969223 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.928678  2546 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.981586 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:13.016969 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75666.8us
I20260709 12:21:12.934224  2639 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:12.940306  2804 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:13.022733 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:13.024979 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43508.3us
I20260709 12:21:13.020776  2907 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:13.028115 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:12.947110  2742 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:13.037293 27583 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:13.036614  2486 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:13.038545  2486 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:13.038935  2486 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:13.057883 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:13.058801 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:13.061264 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:13.064803 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.072008  2667 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:13.072943  2578 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:13.076810  2642 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.033623  2804 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:13.024353  2546 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.082346  2546 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.082902  2892 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:13.083572  2892 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.085789  2892 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.090219  2901 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:21:13.092175  2954 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
W20260709 12:21:13.093250  2956 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 12:21:13.043640  2743 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:13.094573  2743 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:13.094940  2743 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:13.097051  2852 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:21:13.098795  2958 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:21:13.058475 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:13.104389  2639 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.105646  2639 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:13.115161 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.951077  2952 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.105s
I20260709 12:21:13.162011 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:13.163195 27731 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:21:13.035261  2907 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:13.159112 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:13.170994 27584 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:13.036716  2790 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:13.129945 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71547.4us
I20260709 12:21:13.037487  2742 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:13.174286 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:13.129091  2950 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.109236  2809 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:13.178279 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.127208  2546 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.108788  2486 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.165732  2878 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:13.014967  2955 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.193516 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.187083  2950 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:13.194510 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.173686  2907 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:13.195097 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.190856 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:13.201351 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:13.176959  2742 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:13.209224 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.214344 27735 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:21:13.207890 27731 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:21:13.045761  2863 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.185837  2892 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:13.211730 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 266303us
W20260709 12:21:13.217679 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 441342us
I20260709 12:21:13.218423 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.228507 27734 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:21:13.225512 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:13.230013 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:13.230634 27738 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:13.194336  2878 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:13.226868 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:13.215581  2868 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.624s	user 0.011s	sys 0.001s
I20260709 12:21:13.206401  2960 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.207577  2692 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:13.212299  2851 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.218932  2948 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:13.251637 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.181208  2790 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.246402 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:13.276932  2965 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:13.281695  2863 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:13.284830  2486 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.288178 27734 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:21:13.288931  2793 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:13.289619  2793 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:13.234369  2892 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.293781  2892 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.192601  2546 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.297972  2546 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:13.244124  2878 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:13.304627 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:13.305339 27736 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:13.305508  2962 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.248s	user 0.000s	sys 0.000s
W20260709 12:21:13.307775  2959 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:21:13.309679 27663 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:21:13.316898  2955 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:13.319410  2961 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:21:13.241123  2809 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:13.250160  2868 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.326000  2868 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:13.345956 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.347559 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.347681 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.348472 27656 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:21:13.349210 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.349426 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.287004  2486 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.351805 27659 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:13.353272 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:13.354691 27736 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:13.356299 27736 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:13.388387 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:13.390404 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.392164 27738 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:13.392311 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.393765 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.242357  2949 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:13.263129  2790 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:13.396942 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.397114 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.343573  2874 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:13.399811 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:13.409029 27659 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:13.400662 27663 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:13.392954 27655 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:13.278990  2965 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:21:13.401274 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:13.399507 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.407368 27661 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:13.458513 27734 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:13.394074  2943 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:21:13.348486  2692 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:13.351105  2948 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:13.469211 27657 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.013s	sys 0.017s
I20260709 12:21:13.473078 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:13.290742  2793 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:13.477516 27657 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.013s	sys 0.017s
W20260709 12:21:13.476116 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.396212  2838 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.396569  2578 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.396382  2892 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.398006  2852 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:13.478147 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3431) took 1437 ms. Trace:
I20260709 12:21:13.517342 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.519205 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:13.585980 28337 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:13.533656 27735 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:13.395794  2486 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.527944 28332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:42232
I20260709 12:21:13.407346  2851 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.260527  2960 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.395602  2741 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
W20260709 12:21:13.480858  2540 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.746s	user 0.007s	sys 0.005s
I20260709 12:21:13.402143  2809 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:13.656899 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:13.528728 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55730us
W20260709 12:21:13.604646  2804 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 12:21:13.546052 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:13.557484 27736 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:13.475286  2874 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:13.573654 28329 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:21:13.672816 28332 raft_consensus.cc:606] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:13.673276 28332 raft_consensus.cc:614] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:13.480078  2868 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:13.577945 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 366740us
W20260709 12:21:13.582113 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 344337us
I20260709 12:21:13.517275  2546 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.585572 27657 rpcz_store.cc:276] 0709 12:21:12.041069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.287026 (+245957us) service_pool.cc:224] Handling call
0709 12:21:12.287365 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:12.287538 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:21:12.461516 (+173978us) log.cc:844] Serialized 14 byte log entry
0709 12:21:13.477827 (+1016311us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.477874 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:13.477911 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.477984 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.478099 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15876,"prepare.queue_time_us":136675,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":3172,"raft.queue_time_us":78861,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":2599,"spinlock_wait_cycles":15752832,"thread_start_us":1171445,"threads_started":3,"wal-append.queue_time_us":398312}
I20260709 12:21:13.546551  2870 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:13.528174 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:13.661396 28331 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:13.520794 28333 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:21:13.658382 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:13.659189 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.526438  2892 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.490137  2743 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.682092 27738 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 15: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:13.665165 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.000s	sys 0.009s
I20260709 12:21:13.461426  2857 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.572388 27659 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:13.580680  2838 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.678439 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:13.683995 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.000s	sys 0.011s
I20260709 12:21:13.685491 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:13.702023 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.707765  2948 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.609663  2852 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.716634  2486 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.717204  2852 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.580405  2578 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.720621 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:13.721768 27734 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:13.731489 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:13.738139 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.748524 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.632910  2851 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:13.740342  2907 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.002s
I20260709 12:21:13.656967  2540 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.728008  2870 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:13.679699  2804 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.783522 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:13.697844  2971 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:13.781919 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:13.685690  2868 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.801332 27661 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
W20260709 12:21:13.805388 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84856.2us
I20260709 12:21:13.684478  2546 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:13.833634 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.791726  2870 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:13.834232 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:13.720825  2857 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.837738 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:13.846462 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:13.908309 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.850430 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:13.748957  2978 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 12:21:13.782693  2852 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:21:13.910239 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:13.791419  2907 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:13.917003  2998 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:21:13.918609  2546 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:13.840293  2870 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:13.920120  2985 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:21:13.920399 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.840937  2809 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.983889 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:13.921587  2951 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.946s	user 0.003s	sys 0.009s
I20260709 12:21:13.923152  2540 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:13.923696  2968 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.516s	user 0.001s	sys 0.000s
W20260709 12:21:13.924186  2639 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.818s	user 0.014s	sys 0.000s
I20260709 12:21:13.925203  2851 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.784276  2988 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.101s
I20260709 12:21:13.928776  2804 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:13.849244  2999 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:13.930299  2578 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.004257 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:13.987151  3001 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:13.959683 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 109358us
I20260709 12:21:13.848255  2975 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:14.006642 27734 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:21:13.994334  2871 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:14.002863  2967 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.471s	user 0.005s	sys 0.002s
I20260709 12:21:13.968278  2796 raft_consensus.cc:2764] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:21:13.992762 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:13.852530  3000 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.132s
I20260709 12:21:13.932127  2906 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:21:13.838374  2874 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:13.813812  2973 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.146s
I20260709 12:21:13.957696 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.007911  2793 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:13.990345 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.011s	sys 0.036s
I20260709 12:21:14.040542 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.009639  2829 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:14.009343  2975 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:14.049619  2994 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:14.038401  2988 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.010830 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3604) took 1302 ms. Trace:
I20260709 12:21:14.051654 27587 rpcz_store.cc:276] 0709 12:21:12.708482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.718614 (+ 10132us) service_pool.cc:224] Handling call
0709 12:21:12.925384 (+206770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:12.931278 (+  5894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.010624 (+1079346us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400089dd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:14.010767 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474730880}
I20260709 12:21:14.051465  2988 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.053062 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:14.053206  2857 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.053720 27587 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:14.012334  2892 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:14.055079  2847 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.432s	user 0.012s	sys 0.002s
I20260709 12:21:14.000128  2838 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.016481 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.058110 27657 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:14.016909  2878 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.717s	user 0.009s	sys 0.005s
I20260709 12:21:14.021492  2743 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.022773  2986 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:14.068185  2986 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:14.068048  2878 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.069448  2986 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.070065 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3593) took 1371 ms. Trace:
I20260709 12:21:14.070670  2878 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.070884  2986 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:14.023643  2960 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.071996 27582 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 12:21:14.072356 27582 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 12
I20260709 12:21:14.072575 27582 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:21:14.072820 27582 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:14.073019 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 2967) took 1621 ms. Trace:
I20260709 12:21:14.023844  2987 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:14.073341 27729 rpcz_store.cc:276] 0709 12:21:12.451559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.451693 (+   134us) service_pool.cc:224] Handling call
0709 12:21:14.072827 (+1621134us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540017f2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:14.072967 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462706048}
I20260709 12:21:14.074563 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.026587 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3597) took 1326 ms. Trace:
I20260709 12:21:14.076794  2868 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.078006  2988 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.028256 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3572) took 1164 ms. Trace:
W20260709 12:21:14.028892  2742 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.811s	user 0.010s	sys 0.000s
I20260709 12:21:13.999580  2931 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.029600  2964 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:13.989352  2951 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.015851  3004 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:13.911733  2971 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:13.999753  2984 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.106855  2984 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.037087 27664 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:21:14.008085 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:14.041075 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.011s	sys 0.037s
W20260709 12:21:14.108052 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 293623us
W20260709 12:21:14.109311 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.109385 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.050769  2975 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.012813  3001 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.012528 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:14.056458 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.056209  2829 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:14.060940  2856 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:14.062610  2965 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:21:14.063176  2991 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:21:14.057550  2950 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.849s	user 0.009s	sys 0.003s
I20260709 12:21:14.057616  2793 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.066334 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3606) took 1238 ms. Trace:
I20260709 12:21:14.136657 27585 rpcz_store.cc:276] 0709 12:21:12.829039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.829158 (+   119us) service_pool.cc:224] Handling call
0709 12:21:12.851538 (+ 22380us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:12.855125 (+  3587us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:12.855160 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:12.868042 (+ 12882us) log.cc:844] Serialized 52 byte log entry
0709 12:21:13.129751 (+261709us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:13.129811 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.129907 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:14.066020 (+936113us) raft_consensus.cc:1685] finished
0709 12:21:14.066053 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:14.066258 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24364,"raft.queue_time_us":1990,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":20506,"thread_start_us":214654,"threads_started":1,"wal-append.queue_time_us":511660}
I20260709 12:21:14.138832 27736 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 21 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:21:14.067096 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:14.116010  2931 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:14.068212 27731 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:21:14.070523 27589 rpcz_store.cc:276] 0709 12:21:12.698772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.701471 (+  2699us) service_pool.cc:224] Handling call
0709 12:21:14.069838 (+1368367us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:14.070007 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503094528}
I20260709 12:21:14.170169 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:14.076491 27580 rpcz_store.cc:276] 0709 12:21:12.700041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.712682 (+ 12641us) service_pool.cc:224] Handling call
0709 12:21:12.712939 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.019735 (+306796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.026316 (+1006581us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54000be5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:14.026521 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":530850176}
I20260709 12:21:14.078573 27581 rpcz_store.cc:276] 0709 12:21:12.864330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.866299 (+  1969us) service_pool.cc:224] Handling call
0709 12:21:12.866733 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:12.891430 (+ 24697us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:12.891478 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:12.892091 (+   613us) log.cc:844] Serialized 26 byte log entry
0709 12:21:13.176772 (+284681us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:13.176811 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.176881 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:14.028025 (+851144us) raft_consensus.cc:1685] finished
0709 12:21:14.028048 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:14.028181 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31514,"raft.queue_time_us":33834,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":8818,"thread_start_us":247157,"threads_started":1,"wal-append.queue_time_us":332549}
I20260709 12:21:14.171978 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.172705 27581 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:14.173873 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.174574 27581 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:21:14.105063 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.041177 27584 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:14.111028 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:14.166759 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.115518 27730 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:14.116833 27662 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:21:14.117690 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.119074 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.120042 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.178633 27656 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:14.179664 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:14.121742 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 109708us
I20260709 12:21:14.180907 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:14.116114  2869 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.173s	user 0.022s	sys 0.008s
I20260709 12:21:14.075163 27729 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:14.109200  3004 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.141201 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.141798 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 2969) took 1678 ms. Trace:
I20260709 12:21:14.108073  2964 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.141834 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:14.123752  2965 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:14.142731  2540 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:14.230659  2540 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:14.231043  2965 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.231740  2540 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.232586  2965 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:14.119861  2853 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:14.152135  3014 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:14.177883 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:14.182350 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:14.182736 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.196836  2950 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:14.197836 27584 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:21:14.234901  3014 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.198726 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.199216 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.241132 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3581) took 1196 ms. Trace:
I20260709 12:21:14.241597 27736 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:21:14.135690  2800 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:14.239532 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1342) took 1058 ms. Trace:
I20260709 12:21:14.204808 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.244199 27738 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:14.102226  2742 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.202888 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.128191  2951 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:14.206084  2869 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.023s	sys 0.008s
I20260709 12:21:14.127622  2856 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:14.243283 27584 rpcz_store.cc:276] 0709 12:21:13.181127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.196959 (+1015832us) service_pool.cc:224] Handling call
0709 12:21:14.197411 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.239251 (+ 41840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.239483 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:14.241521 27586 rpcz_store.cc:276] 0709 12:21:13.044308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.213406 (+169098us) service_pool.cc:224] Handling call
0709 12:21:13.213796 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.329025 (+115229us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:13.458932 (+129907us) log.cc:844] Serialized 14 byte log entry
0709 12:21:13.727489 (+268557us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.727558 (+    69us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:13.727607 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.240931 (+513324us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540020cbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:14.240976 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:14.241083 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7834,"prepare.queue_time_us":135905,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":72587,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":14645,"spinlock_wait_cycles":264259968,"thread_start_us":260267,"threads_started":1,"wal-append.queue_time_us":502533}
I20260709 12:21:14.271270 27736 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:14.271389 27734 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:14.272619 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:14.273996 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 93225.9us
I20260709 12:21:14.274775 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:14.276847 27702 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:14.274598 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:14.286319 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.211517 27732 rpcz_store.cc:276] 0709 12:21:12.463411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.463717 (+   306us) service_pool.cc:224] Handling call
0709 12:21:12.512093 (+ 48376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:12.947619 (+435526us) spinlock_profiling.cc:243] Waited 7.56 ms on lock 0x7b180001ef40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:21:12.947753 (+   134us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:12.947777 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:12.979185 (+ 31408us) log.cc:844] Serialized 26 byte log entry
0709 12:21:13.702001 (+722816us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:13.702040 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.702109 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:14.141485 (+439376us) raft_consensus.cc:1685] finished
0709 12:21:14.141505 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:14.141739 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155395,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":18680,"spinlock_wait_cycles":17388800,"thread_start_us":775634,"threads_started":2,"wal-append.queue_time_us":811983}
I20260709 12:21:14.289111 27729 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:14.291692 27586 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:14.288890 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.284657 27587 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:14.297066  2850 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.292057  2288 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:14.299719  3001 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.271788  2540 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:14.296440  2742 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.293118  2743 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.352799  2743 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.278394  2951 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:14.308061  2856 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.355911  2743 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.356688  2856 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:14.342516  2870 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.358481 27733 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:14.309303  2857 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:14.361568  2857 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.364387  2857 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.310354  3004 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.312629  2964 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:14.318282  3014 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.293041  2878 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:21:14.372958  2870 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.334717  2809 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.352159  2288 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.299638  2984 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.382934  2288 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:14.382978  2809 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.383453  2951 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:14.384469 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 107727us
I20260709 12:21:14.391593 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:14.388494 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3346) took 1088 ms. Trace:
I20260709 12:21:14.392465 27735 rpcz_store.cc:276] 0709 12:21:13.300129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.349222 (+ 49093us) service_pool.cc:224] Handling call
0709 12:21:14.388213 (+1038991us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b540044e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:14.388410 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307277056}
I20260709 12:21:14.392871 27589 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:14.390226 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3603) took 1682 ms. Trace:
I20260709 12:21:14.393713 27582 rpcz_store.cc:276] 0709 12:21:12.708226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.714372 (+  6146us) service_pool.cc:224] Handling call
0709 12:21:14.071670 (+1357298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.073215 (+  1545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.389860 (+316645us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001888d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:14.390150 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3120691968}
I20260709 12:21:14.393170 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:14.400691 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.401983 27582 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:14.403162 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.404038  2975 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.404335 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:14.407455 27580 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:14.408434 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.409147 27582 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 11.
I20260709 12:21:14.408874  2288 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.410213 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.411232 27582 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:14.406549  2869 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.406605 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.406042  3004 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:14.411845  2871 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:14.440467  2793 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.440757  2871 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:14.444764  2951 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.445276  2952 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:14.445302  2951 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.446062  2952 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.448060  2984 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.448251  2951 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.448726  2984 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.449649 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:14.450302 27735 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:21:14.412865 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:14.413209 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.453722 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.454298 27658 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:14.413851 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.414461 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.455893 27664 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:14.415174 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.457584 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.458202 27729 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:14.415548  2288 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.415774  2800 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:14.404814  2975 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.417220 27589 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:21:14.459296 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46512.4us
I20260709 12:21:14.455432 27655 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:14.460134 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.408434 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.461258 27580 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:14.462038  3018 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:14.452409  2869 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.462936  3018 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:14.407223  2822 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.430048  3004 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:14.463627  2822 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.463850  3018 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.464653  2869 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.464468  3004 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.465698  2822 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.467375 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:14.467931 27589 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:14.463212 27580 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:14.472425 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.472990 27580 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:14.462024 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:14.475191 27582 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:14.398407  2857 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.437592  2856 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:14.439913  2809 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.397804  2950 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:14.476858 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 204959us
I20260709 12:21:14.478807 27735 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:14.483978 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.459160  2984 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:14.497117 27662 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:14.497892 27589 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:14.498932 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.499200 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.459677  2800 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:14.507982 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.508850 27735 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:14.500799 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.527184 27582 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:14.529261 27658 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:14.530982 27589 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:14.530853 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.531616 27729 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:21:14.528715 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:14.535547 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:14.536173 27735 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:14.537163 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.539127 27589 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
W20260709 12:21:14.540906 27583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3599) took 1840 ms (client timeout 1612 ms). Trace:
W20260709 12:21:14.541281 27583 rpcz_store.cc:269] 0709 12:21:12.700583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.703406 (+  2823us) service_pool.cc:224] Handling call
0709 12:21:14.540704 (+1837298us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:14.540826 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765157760}
I20260709 12:21:14.506606  2800 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:14.545318 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:14.495455  2950 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.538043 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:14.483630  2857 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.484287  2856 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.561887 27582 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:21:14.562039 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:14.541649 27589 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:21:14.566310 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3347) took 1265 ms. Trace:
I20260709 12:21:14.715605 27589 pending_rounds.cc:85] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 10
I20260709 12:21:14.715991 27589 pending_rounds.cc:107] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 20.11
W20260709 12:21:14.625566 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.716495 27589 raft_consensus.cc:2889] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:14.696601  2950 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.559439  2829 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.571091  3008 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:21:14.627990 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 240206us
I20260709 12:21:14.540278  2852 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.715749 27730 rpcz_store.cc:276] 0709 12:21:13.300562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.300748 (+   186us) service_pool.cc:224] Handling call
0709 12:21:13.301095 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.177009 (+875914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.566079 (+389070us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54002e90d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:14.566256 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1872530176}
I20260709 12:21:14.707090 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:14.730278 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.730688 27736 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:14.731222 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 169255us
I20260709 12:21:14.708945 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.712261 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:14.540375  2963 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:21:14.718096 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.703259 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.733433 27735 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 17. (term mismatch)
I20260709 12:21:14.733897 27735 pending_rounds.cc:85] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 16
I20260709 12:21:14.734162 27735 pending_rounds.cc:107] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 21.17
W20260709 12:21:14.734078 27627 connection.cc:256] RPC call timeout handler was delayed by -1.0067515230000481s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1678s for a response
I20260709 12:21:14.734736 27735 raft_consensus.cc:2889] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:14.735248 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:14.735764 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.746387 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.746809 27736 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:21:14.748872 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.750197 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:14.754541 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.755175 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.755652 27738 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:14.751502 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:14.757009 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:14.754786 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.691183  2876 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.768033 27732 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:14.755697 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:14.769217 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.767009 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:14.776765 27734 raft_consensus.cc:2376] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26. Re-sending same reply.
I20260709 12:21:14.775017 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.745576  2870 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:14.787663  2686 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:14.752269  2950 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:14.807627 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3583) took 1581 ms. Trace:
I20260709 12:21:14.807749  2950 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.778198  2963 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:14.808056 27581 rpcz_store.cc:276] 0709 12:21:13.225954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.200721 (+974767us) service_pool.cc:224] Handling call
0709 12:21:14.201373 (+   652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:14.201856 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.201899 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:14.287055 (+ 85156us) log.cc:844] Serialized 26 byte log entry
0709 12:21:14.656799 (+369744us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:14.656844 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.656929 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:14.807397 (+150468us) raft_consensus.cc:1685] finished
0709 12:21:14.807421 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:14.807570 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":575,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":114,"thread_start_us":359729,"threads_started":1,"wal-append.queue_time_us":493855}
I20260709 12:21:14.810107  2950 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.809854  2963 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.754506  2878 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:14.811450  2870 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.776160  2540 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:21:14.796692 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:14.797505 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.813839  2975 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.814112  2870 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.814494 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.785869  2973 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:14.815480  2973 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.815539  2807 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:14.816465  2807 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:14.817157  2829 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.817418  2807 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.818780 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.770401  2793 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:14.783741  3004 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:14.767884  2741 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.820019  3004 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:14.820530  3004 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:14.820325  2741 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.821583  3001 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:14.821993  3001 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.822337  3001 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:14.822723  2741 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:14.756659  3008 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:14.785745  2876 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.824136  3008 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.825177 27585 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:21:14.828006 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.815598  2891 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:14.800832  2857 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:14.831550 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64642.6us
I20260709 12:21:14.834460 27664 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:21:14.834820 27664 pending_rounds.cc:85] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 15
I20260709 12:21:14.835078 27664 pending_rounds.cc:107] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:21:14.835506 27664 raft_consensus.cc:2889] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:14.829947 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.837924 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.838789 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.811914  2878 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:14.849241 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:14.851769 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.807046  2856 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.855679  3018 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:14.856179  2743 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:14.848690  2742 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:14.857582  2857 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.006s
I20260709 12:21:14.844045  2793 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:21:14.848449  2850 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:14.848867  2288 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.011s
I20260709 12:21:14.860620 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:14.861415  2972 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.326s	user 0.007s	sys 0.000s
I20260709 12:21:14.836135  2891 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:14.844237  2984 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:14.855044  3014 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:14.866472  2743 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.862548 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1060) took 2194 ms. Trace:
I20260709 12:21:14.756397  2852 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.867509  2742 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.869843  2793 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.889979  2891 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:14.888783  3014 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.837664  2973 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:14.846032  2906 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:14.853204  2878 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:14.791108  2686 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.870180  2850 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.892352 28328 rpcz_store.cc:276] 0709 12:21:12.667599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.667754 (+   155us) service_pool.cc:224] Handling call
0709 12:21:12.669601 (+  1847us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:12.905082 (+235481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.862217 (+1957135us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400611054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:14.862439 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":540047104}
I20260709 12:21:14.890998 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.922729  2852 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:14.923316  2850 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:14.894820  2743 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:14.893965  2856 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:14.898433  2742 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:14.900002  2793 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:14.920994  2686 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:14.923336 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.923494  2975 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:21:14.890396  2984 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.912024  3014 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:14.864673 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170175us
I20260709 12:21:14.937594 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.944972  3032 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:14.913194  2906 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:14.947592  2906 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:14.956841 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.922961  2871 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:14.971050 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:14.971568 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 188572us
I20260709 12:21:14.966668 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:14.976637 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.978262 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.979923 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:14.980865 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.970362  2870 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.982393 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:14.974507  2856 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:15.002349  3031 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:15.004135 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:15.017338 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:14.955056  3004 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:14.973472  2906 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.006891  2988 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:15.028016  2988 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.028455  2988 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:15.021275 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:15.006736  2870 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.021320  2856 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:15.042060 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3573) took 2177 ms. Trace:
I20260709 12:21:15.044173 27588 rpcz_store.cc:276] 0709 12:21:12.864679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.946340 (+ 81661us) service_pool.cc:224] Handling call
0709 12:21:12.946620 (+   280us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:13.018545 (+ 71925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:13.018577 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:13.019066 (+   489us) log.cc:844] Serialized 26 byte log entry
0709 12:21:13.412504 (+393438us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:13.527083 (+114579us) log.cc:844] Serialized 14 byte log entry
0709 12:21:13.527385 (+   302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.527487 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:15.037949 (+1510462us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b18003817c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:15.041710 (+  3761us) raft_consensus.cc:1685] finished
0709 12:21:15.041847 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.041996 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100432,"prepare.queue_time_us":122287,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":3968,"raft.queue_time_us":153885,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":36207,"spinlock_wait_cycles":55332096,"thread_start_us":379412,"threads_started":1,"wal-append.queue_time_us":578975}
I20260709 12:21:15.047710 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:15.049547  2667 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:15.049984  2667 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:15.050869  2667 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.051868  2667 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:15.049937 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:15.056237 27582 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:15.056402  2871 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.055436  3004 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:15.057060  2871 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:15.057570  2871 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:15.060177  2984 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.058588  2871 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:15.069344 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.087803 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:15.100669  2965 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:15.101220  2965 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:15.101675  2906 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.102200  2965 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.110003 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.113310 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3364) took 1521 ms. Trace:
I20260709 12:21:15.113763 27733 rpcz_store.cc:276] 0709 12:21:13.592070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.198413 (+606343us) service_pool.cc:224] Handling call
0709 12:21:15.113112 (+914699us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002e77d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:15.113246 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271912832}
I20260709 12:21:15.116892  3031 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:15.117712  3031 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:15.118091  2686 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.119622  2907 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:15.121423  3034 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:21:15.121754  2876 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.124809  2874 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:15.125530 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:15.135094 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3612) took 1026 ms. Trace:
I20260709 12:21:15.135573 27586 rpcz_store.cc:276] 0709 12:21:14.108849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.268844 (+159995us) service_pool.cc:224] Handling call
0709 12:21:14.269198 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.316284 (+ 47086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.134809 (+818525us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:15.135017 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39003392}
I20260709 12:21:15.137296  2965 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:15.148741 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.148995 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.149938 27733 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:15.170728 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.214754 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.215745 27736 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:15.217779 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.218650 27736 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:15.082891  3029 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:21:15.118088  3031 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:15.249550  2800 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.688s	user 0.007s	sys 0.004s
I20260709 12:21:15.250181  2800 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.133025  2742 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.250631  3014 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.253521 27589 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:15.265826 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.267453 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.268800 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.273769  2891 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.252183  3031 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:15.251756  3029 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:15.243124  2869 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:15.376526 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.377801 28328 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:15.251924  3001 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.377915 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:15.384155 28337 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:21:15.390451 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.391669 28336 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:15.391806 27729 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:15.394006 28328 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEARNER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:21:15.396515 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:15.397157 28328 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:21:15.376729 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 329107us
I20260709 12:21:15.402961 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:15.404196 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.404285 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:15.404954 27734 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:15.406438 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:15.413630 28337 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:15.416285 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.422204 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3344) took 2153 ms. Trace:
I20260709 12:21:15.417012 27737 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:15.425267 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.426156 27737 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:15.429152 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.378307  3029 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:21:15.422706 27731 rpcz_store.cc:276] 0709 12:21:13.257107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.257585 (+   478us) service_pool.cc:224] Handling call
0709 12:21:13.257965 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.169098 (+911133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.410626 (+1241528us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b54000a46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:15.410795 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1862394624}
I20260709 12:21:15.440871 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.441520 27738 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:15.501225 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78901.5us
I20260709 12:21:15.501435 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.502353 27731 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:15.407608  2874 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:15.407563  3048 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:15.390741  2869 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.414969  2540 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:15.520072 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.553009 28337 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:21:15.557125 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.616225 27731 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:15.616767 27737 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
W20260709 12:21:15.360306  3008 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.002s	sys 0.017s
I20260709 12:21:15.560243 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:15.602422  2850 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.667s	user 0.007s	sys 0.011s
I20260709 12:21:15.563740 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.563990 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.516510  3032 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.098s
I20260709 12:21:15.565337  2989 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:15.566509 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.574129 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:15.582835 28336 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:15.624456 27583 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:15.624650 28335 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:21:15.593449 27735 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:15.594225 27736 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:15.625301 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.627429 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:15.596813 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 207007us
W20260709 12:21:15.516543 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 470886us
W20260709 12:21:15.523202  2743 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.581s	user 0.007s	sys 0.010s
I20260709 12:21:15.631484 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:15.646741 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.647500 27734 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:15.619812  3008 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.002s	sys 0.018s
I20260709 12:21:15.429597  2857 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.142s
I20260709 12:21:15.421012  2907 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:15.612300 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 97598.6us
W20260709 12:21:15.613274 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 104797us
I20260709 12:21:15.781234 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:15.781957 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.782636 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:15.783048 27659 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:15.561874 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.564829 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.619784 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:15.620718 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.004s	sys 0.005s
I20260709 12:21:15.622309 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.784268 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:15.513763  2973 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.002s
I20260709 12:21:15.629918 27732 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:15.633632 27738 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:15.634099 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:15.785236  2973 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.634878 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.628708 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:15.635497 27587 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:21:15.787361 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 158677us
I20260709 12:21:15.636422 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:15.637427 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.788230 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:15.788628  2973 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.638679  3001 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.789346 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:15.643708  2686 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:15.784394 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.004s	sys 0.005s
I20260709 12:21:15.791311  2692 raft_consensus.cc:2764] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260709 12:21:15.791621 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 771112us
I20260709 12:21:15.792282  2692 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:15.792428 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:15.792774  2692 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.603456 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:15.624241  2850 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.614696  2540 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.619642  2906 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.636454  2800 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:15.723985  2878 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.797s	user 0.013s	sys 0.012s
I20260709 12:21:15.620123  3046 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.103s
I20260709 12:21:15.718878 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:15.711066  3038 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.652426  2743 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.646133  2989 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.634562 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.787606 27662 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24.
I20260709 12:21:15.791433 27656 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:21:15.510457 28328 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:15.714764 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.797029 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.799258  2950 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:15.793447  2907 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.839862 28336 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:15.854555  2907 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.819139  3061 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:21:15.808413  2540 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:15.834371  2878 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.813091  3032 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.053s
I20260709 12:21:15.861358  2878 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.798928  2692 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:15.862027  2686 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:15.840827 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.864580 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.865131 27657 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:15.813915  2856 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:15.865933 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.867141  2856 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:15.842344 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1346) took 1873 ms. Trace:
I20260709 12:21:15.866780  2540 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.832064 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.867856 27585 rpcz_store.cc:276] 0709 12:21:13.969136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.212796 (+243660us) service_pool.cc:224] Handling call
0709 12:21:14.213066 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.825604 (+612538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.842076 (+1016472us) spinlock_profiling.cc:243] Waited 612 ms on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:15.842277 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1406886656}
I20260709 12:21:15.869189 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:15.869500 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.870054 28336 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:15.870268 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.870571 28333 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:15.870868 27655 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:15.816969  3058 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.871973  3058 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.872668  2952 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.872922 28329 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:15.873489  2989 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:15.874027  3058 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.818414 27659 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
W20260709 12:21:15.822893 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.875504 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.836689  2742 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.814775  2906 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.815902  2973 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:15.827018  3052 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:21:15.867808  2856 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:15.820607 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3448) took 2473 ms (client timeout 1553 ms). Trace:
W20260709 12:21:15.884613 27660 rpcz_store.cc:269] 0709 12:21:13.346664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.346790 (+   126us) service_pool.cc:224] Handling call
0709 12:21:15.820339 (+2473549us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:15.820490 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2554045696}
I20260709 12:21:15.819067  3046 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 12:21:15.886303  2540 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.886521  3046 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:15.887101  3046 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:15.888995 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2884) took 1474 ms. Trace:
I20260709 12:21:15.889775  2870 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:15.881623  2973 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.891985  2973 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:15.889379 27658 rpcz_store.cc:276] 0709 12:21:14.414641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.454869 (+ 40228us) service_pool.cc:224] Handling call
0709 12:21:14.461445 (+  6576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.529653 (+ 68208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.888772 (+1359119us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:15.888941 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155241856}
I20260709 12:21:15.870291  2743 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.903164  2906 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.904371  2955 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:15.807592  3048 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.109s
I20260709 12:21:15.904893  2955 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.905614  3048 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:15.906037  3048 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:15.842854  3038 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.908010 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:15.909592  3038 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.883750  2950 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:15.910494  3057 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:15.911262  2950 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.910135  2800 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.912348 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3592) took 1454 ms. Trace:
I20260709 12:21:15.913203 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:15.913965  2870 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:21:15.824759  2965 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.686s	user 0.009s	sys 0.000s
I20260709 12:21:15.915045 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:15.906922  2955 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:15.916303 27664 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
W20260709 12:21:15.918315  3031 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.528s	user 0.004s	sys 0.007s
I20260709 12:21:15.818117 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.876796 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.912868 27589 rpcz_store.cc:276] 0709 12:21:14.457706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.753147 (+295441us) service_pool.cc:224] Handling call
0709 12:21:14.753478 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.254000 (+500522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.912114 (+658114us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:15.912292 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1149508992}
I20260709 12:21:15.919688 27659 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:21:15.919836  3031 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.920871 27659 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
W20260709 12:21:15.836669  2793 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.871s	user 0.006s	sys 0.007s
I20260709 12:21:15.922008  3031 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.922293  2793 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:15.832290  2988 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.803s	user 0.013s	sys 0.000s
I20260709 12:21:15.924315  3032 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:21:15.925029  2988 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.823271 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.929190 27661 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:15.928256  2907 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Committing config change with OpId 20.13: config changed from index 11 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:15.931538 27660 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:15.878294 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.878829 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:15.881700 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:15.834302  2850 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.940856 28331 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 16: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:15.942077  2793 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.906862  3048 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.944649  3048 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:15.947427  2874 consensus_queue.cc:1175] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 25 status { last_received { term: 22 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.15-24.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:15.888219  3046 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.957263 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1315) took 1188 ms. Trace:
I20260709 12:21:15.957952  3046 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:15.959496 27663 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:15.962883  3029 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:15.963598 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.963908  2891 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.964098 27655 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:21:15.957799 27662 rpcz_store.cc:276] 0709 12:21:14.768788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.768954 (+   166us) service_pool.cc:224] Handling call
0709 12:21:15.957001 (+1188047us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:15.957188 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2324016000}
I20260709 12:21:15.917837  2856 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.912870  2800 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:15.967592  2869 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:15.968608  2800 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.972350  3014 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:15.972609 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:15.974005  3029 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.976591  2988 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:15.977398  2856 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:15.979547  3014 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:15.980284  3014 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:15.875308 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
W20260709 12:21:15.994647 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 296353us
I20260709 12:21:15.922021 27659 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
W20260709 12:21:15.999819 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 84840.4us
W20260709 12:21:15.807303 27627 connection.cc:256] RPC call timeout handler was delayed by -1.1014028009999492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30010000000000003s for a response
W20260709 12:21:16.001708 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.008210 27659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:16.001096 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.016938 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:16.017577 28329 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEARNER]: Advancing to term 24
I20260709 12:21:16.015940 27481 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 20 to 22, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 22 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.018177 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.021533 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:16.021798 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.021528 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.019093 27481 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.002748  2870 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.044821 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.046860  2989 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:16.041453 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.073571 28329 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:16.056680 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.076355 27481 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 26 to 29. New cstate: current_term: 29 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.030771  2667 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:21:16.083902 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:16.084760 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63309.8us
I20260709 12:21:16.094611 27481 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 10 to 12, term changed from 19 to 21, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to <none>, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 21 committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.096978 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.098707 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.100243 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:16.100769 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 389541us
I20260709 12:21:16.102120 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.103780 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:16.107954 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.108798 27737 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:21:16.021266  2869 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:16.023900  2800 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:16.030236  3014 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:16.115001 27660 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:16.056336  3029 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:16.117532 27729 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:16.118003 27733 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:21:16.097208 27481 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index -1 to 13, term changed from 13 to 17, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 17 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.126055 27481 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 20, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 20 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.119922 27730 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:21:16.120404 27662 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:21:16.131361 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.132448 27663 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:21:16.133700 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.137194 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:16.138118 27662 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:16.141242 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.141615 27738 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:16.141924 27732 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:16.142673 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.128535 27481 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 20, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 20 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.142814 27734 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:16.143869 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:16.143225 27732 raft_consensus.cc:2376] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22. Re-sending same reply.
I20260709 12:21:16.139703 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.145625 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.145982 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.146344 27732 raft_consensus.cc:2376] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23. Re-sending same reply.
I20260709 12:21:16.147107 27733 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:21:16.148104 27736 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:16.031747  3018 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:21:16.157578 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.151722 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:16.143509 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:16.158942 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.159085 27655 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:16.096124  2955 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.156397  3014 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.154194 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.165851 27732 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:16.154470 27481 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 19, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 19 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.151213  2869 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:21:16.096293  2870 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:21:16.153631 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.017s	sys 0.034s
W20260709 12:21:16.168893 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.017s	sys 0.034s
I20260709 12:21:16.167193 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.168694 27481 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 16 to 18, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.084638  2952 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
W20260709 12:21:16.169523 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 541883us
I20260709 12:21:16.169832 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.172603 27734 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:16.153102  2800 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:16.174788 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.179977 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.179849 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:16.171743 27481 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 12 to 15, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.164537 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.180804 28329 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:16.085716  2871 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.078s
I20260709 12:21:16.180608 27729 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:16.159973  2999 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:16.152309  2949 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.190184 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.185271 27481 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 20, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 20 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.170408 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.190248 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.191648 27736 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:16.196220 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.197430 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.198953 27729 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:16.200423 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.201045 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:16.201875 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.202060 27736 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:16.202205 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:16.204404 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:16.205027 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:16.206005 28329 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:21:16.185875 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.204427 27481 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.210310 27481 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 27. New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:16.211552  3018 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.212927 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.213755  3018 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:16.214272  3018 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:16.215197  3018 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.216846  3057 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:16.217247  3057 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:16.218062  3057 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.171366  2870 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:16.219579  2870 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:16.219475 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.220284  2963 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:16.220458 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.221602 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.223028 27481 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.181078  2800 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:16.227517  2963 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:16.229264 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.229975  2963 raft_consensus.cc:2764] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:21:16.231598 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.235201  2963 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:21:16.236696 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53571us
W20260709 12:21:16.237901 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47772.2us
W20260709 12:21:16.239780 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53965.1us
W20260709 12:21:16.241981 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.249782 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:16.243152 27737 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:16.254367 27737 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:21:16.236801 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.278764 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.279397 27737 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:21:16.280972 27737 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:16.219148  3057 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:16.284711 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:16.287992 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67664.2us
I20260709 12:21:16.288316 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:16.288694 27485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 234) took 40194 ms (client timeout 15000 ms). Trace:
I20260709 12:21:16.183835  2871 consensus_queue.cc:1243] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:21:16.170696  2692 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.238408  3014 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:16.302776 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.285558  3038 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 12:21:16.287227  2783 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:21:16.305586  3009 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:16.306108  2783 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:16.306957  3009 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:16.308256  3009 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:21:16.308984  2874 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:16.309574 27553 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:16.309971  2874 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:16.310389  2874 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:16.312623  2288 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:16.313053  2288 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:16.313928  2288 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.314991  2288 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:16.287388  2989 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:16.315840  2989 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:16.311192  2874 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.316725  2989 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.284950  2963 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.317656  2874 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:16.287309  2906 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 12:21:16.321123  2906 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:16.321594  2906 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:16.286228  2876 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:16.322690 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.322948 28331 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: [20.13-23.14]
I20260709 12:21:16.325790 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:16.289202 27485 rpcz_store.cc:269] 0709 12:20:36.044775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:36.045713 (+   938us) service_pool.cc:224] Handling call
0709 12:21:16.238967 (+40193254us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:09.660322 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:09.743921 (+ 83599us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:09.743981 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:21:09.744002 (+    21us) tablet.cc:662] Decoding operations
0709 12:21:09.745939 (+  1937us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:09.746068 (+   129us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:09.746101 (+    33us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:21:10.004974 (+258873us) tablet.cc:701] Row locks acquired
0709 12:21:10.035824 (+ 30850us) write_op.cc:260] PREPARE: finished
0709 12:21:10.052777 (+ 16953us) write_op.cc:270] Start()
0709 12:21:10.076013 (+ 23236us) write_op.cc:276] Timestamp: P: 1783599670044687 usec, L: 0
0709 12:21:10.081583 (+  5570us) op_driver.cc:348] REPLICATION: starting
0709 12:21:10.092283 (+ 10700us) log.cc:844] Serialized 8235 byte log entry
0709 12:21:11.216587 (+1124304us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:11.548014 (+331427us) op_driver.cc:464] REPLICATION: finished
0709 12:21:11.809444 (+261430us) write_op.cc:301] APPLY: starting
0709 12:21:11.809555 (+   111us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:12.208135 (+398580us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:12.208182 (+    47us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:12.724425 (+516243us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b180009f340. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9a0870ec 00007f0e9a0d3c0a 00007f0e9a0d34f7 00007f0e9a0d3490 00007f0e9a08af48 00007f0e9dd9f1bb 00007f0e9dde65d4 00007f0e9dc928da 00007f0e9dc953cb
0709 12:21:13.059888 (+335463us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:13.107792 (+ 47904us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=42,mrs_lookups=0
0709 12:21:13.107856 (+    64us) write_op.cc:312] APPLY: finished
0709 12:21:13.108496 (+   640us) log.cc:844] Serialized 188 byte log entry
0709 12:21:13.468849 (+360353us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:13.627161 (+158312us) write_op.cc:454] Released schema lock
0709 12:21:13.670110 (+ 42949us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:13.703127 (+ 33017us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:13.720626 (+ 17499us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":118828,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":261109,"apply.run_cpu_time_us":56969,"apply.run_wall_time_us":1867482,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91907,"num_ops":21,"prepare.queue_time_us":145331,"prepare.run_cpu_time_us":16795,"prepare.run_wall_time_us":576166,"raft.queue_time_us":203363,"raft.run_cpu_time_us":24230,"raft.run_wall_time_us":219764,"replication_time_us":1466242,"spinlock_wait_cycles":13784704,"thread_start_us":772869,"threads_started":5,"wal-append.queue_time_us":604939,"wal-append.run_cpu_time_us":90296,"wal-append.run_wall_time_us":3407991}]]}
W20260709 12:21:16.329948 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.028s	sys 0.017s
W20260709 12:21:16.330482 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.028s	sys 0.017s
W20260709 12:21:16.331161 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 540276us
I20260709 12:21:16.341631 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:16.326290  2686 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:21:16.352614 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.016s	sys 0.030s
W20260709 12:21:16.355178 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.016s	sys 0.030s
I20260709 12:21:16.286028  2988 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:21:16.285883  2742 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:21:16.303603  3014 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:16.292142  3005 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:21:16.326542  3038 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:16.326545  3052 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:16.350363  3031 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:16.296006  2760 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260709 12:21:16.298916  2743 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.056s
W20260709 12:21:16.383538 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.011s	sys 0.022s
I20260709 12:21:16.325270  2876 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:16.351145  2486 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.017s	sys 0.029s
I20260709 12:21:16.387817 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:16.377302  3004 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.011s	sys 0.012s
W20260709 12:21:16.391472  3053 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.018s	sys 0.015s
I20260709 12:21:16.391571  3063 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:16.392410  3065 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:16.353096  2955 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:21:16.403017  2486 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.017s	sys 0.029s
W20260709 12:21:16.399385 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.011s	sys 0.022s
I20260709 12:21:16.376047  2667 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:16.422132  3004 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.012s	sys 0.012s
I20260709 12:21:16.413334  2743 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.387080  2686 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:16.382192  2973 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:16.426450 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 613189us
I20260709 12:21:16.387787  3014 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:16.404148  2876 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:16.401048  3052 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:16.420637  3031 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:16.422519 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.020s	sys 0.018s
I20260709 12:21:16.405583  3058 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:16.422472  2871 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:16.424005  2857 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.005s	sys 0.026s
W20260709 12:21:16.440799 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.020s	sys 0.019s
I20260709 12:21:16.424265  2955 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:16.422472  3053 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.018s	sys 0.015s
I20260709 12:21:16.420934  3009 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.020s
I20260709 12:21:16.432739  3063 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:16.454694 28292 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.592s (SENT)
I20260709 12:21:16.435802 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:16.434082  3061 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.008s	sys 0.022s
W20260709 12:21:16.455489 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 635244us
I20260709 12:21:16.397943  3064 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:21:16.451526 27730 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:21:16.469340  2955 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.470503  2955 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:16.469367  3031 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.431213  2800 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.480144 27738 raft_consensus.cc:2368] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20.
I20260709 12:21:16.480628  2686 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.467696 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.485397  3009 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:16.456780  2692 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:16.489346  2692 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:21:16.490893  2692 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.491644  3001 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:21:16.491947  3066 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:21:16.492199  3001 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:16.407095  2906 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.493142  3001 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.494490  3001 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:16.508512  2973 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:16.509565  3052 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:16.456027  2857 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.005s	sys 0.027s
W20260709 12:21:16.516660 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:16.527945 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 92191.7us
I20260709 12:21:16.538493 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.539775 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:21:16.546802 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:16.547663 27734 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:16.520102  3061 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.009s	sys 0.023s
I20260709 12:21:16.437052  2743 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:16.553656 27661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.010s	sys 0.017s
W20260709 12:21:16.558828 27661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.010s	sys 0.017s
I20260709 12:21:16.559587 27659 raft_consensus.cc:606] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:16.560002 27659 raft_consensus.cc:614] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:21:16.544195  2950 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.007s	sys 0.017s
I20260709 12:21:16.540628  2988 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:16.459214  2871 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:16.560813 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.565153 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:16.562700  2950 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.008s	sys 0.018s
I20260709 12:21:16.544831  3051 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:16.597985  3051 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:16.563647  2874 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:21:16.598915  3051 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:21:16.579679  3063 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:16.554415  2975 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:16.590747  2743 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:16.594332 27738 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:16.598238  3032 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:16.584102 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:16.575183 27734 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:16.513361  2870 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:16.609520 27658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.003s	sys 0.022s
W20260709 12:21:16.619256 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.112s (SENT) [suppressed 35 similar messages]
I20260709 12:21:16.600198 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:16.639106 27658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.004s	sys 0.022s
I20260709 12:21:16.597996  3058 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:21:16.649711 27735 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.007s	sys 0.023s
W20260709 12:21:16.650213 27735 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.007s	sys 0.023s
I20260709 12:21:16.599296  3051 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:16.667845 27732 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:16.637992  2874 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:21:16.640336  2809 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.016s	sys 0.018s
I20260709 12:21:16.670655 27738 pending_rounds.cc:85] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 10
I20260709 12:21:16.717933 27738 pending_rounds.cc:107] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:21:16.718824 27738 raft_consensus.cc:2889] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:16.764871  2870 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:16.766093 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:16.767659  2800 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:16.770316  3029 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.001s
I20260709 12:21:16.770843  3029 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:16.771998 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.773458  3071 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:16.603103  2988 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:16.799533 27731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.018s	sys 0.026s
W20260709 12:21:16.799997 27731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.018s	sys 0.026s
W20260709 12:21:16.717599  2809 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.016s	sys 0.018s
I20260709 12:21:16.715301  2874 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:16.810348 28336 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
W20260709 12:21:16.810989 27736 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.005s	sys 0.030s
W20260709 12:21:16.835748 27736 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.006s	sys 0.030s
W20260709 12:21:16.877385 27481 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.028s	sys 0.042s
I20260709 12:21:16.871588 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.788589  3072 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:16.807298  3009 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:21:16.877964 27481 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.028s	sys 0.042s
I20260709 12:21:16.798969  2878 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:16.822216  3048 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.873s	user 0.009s	sys 0.004s
I20260709 12:21:16.887576 27659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:16.892691 27659 raft_consensus.cc:606] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:16.893190 27659 raft_consensus.cc:614] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:21:16.882778 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63102.8us
W20260709 12:21:16.818707  3018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.016s	sys 0.031s
I20260709 12:21:16.721491  3058 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:16.951627 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.961196 28331 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:16.881626  2907 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.004s	sys 0.044s
W20260709 12:21:16.948137  3018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.017s	sys 0.031s
W20260709 12:21:16.942508  2952 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.019s	sys 0.061s
I20260709 12:21:16.890610  3057 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:17.018184 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:16.840643  2891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.019s	sys 0.016s
W20260709 12:21:17.024959 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.008s	sys 0.031s
W20260709 12:21:16.968326  2876 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.514s	user 0.008s	sys 0.003s
W20260709 12:21:17.038789 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.008s	sys 0.032s
W20260709 12:21:17.039397 27737 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.015s	sys 0.028s
W20260709 12:21:16.974927  2907 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.005s	sys 0.044s
I20260709 12:21:17.028538 28331 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:16.962023  3048 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:17.048219 27733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.008s	sys 0.019s
W20260709 12:21:17.051728 27733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.009s	sys 0.020s
W20260709 12:21:17.048270 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.013s	sys 0.029s
W20260709 12:21:17.053149 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.014s	sys 0.029s
W20260709 12:21:17.053591 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 699532us
I20260709 12:21:17.054663 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:17.048496 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.012s	sys 0.026s
W20260709 12:21:17.039669 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 784280us
W20260709 12:21:17.057693 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.012s	sys 0.027s
I20260709 12:21:17.041107 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:17.058105 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:16.967916  3072 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:17.063623 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:17.041220 27737 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.015s	sys 0.028s
W20260709 12:21:17.019102  3038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.012s	sys 0.026s
W20260709 12:21:17.039242  2952 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.020s	sys 0.062s
W20260709 12:21:17.039718  3005 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.006s	sys 0.034s
W20260709 12:21:17.087671 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.948856  2878 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:17.041318  2891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.020s	sys 0.016s
W20260709 12:21:17.090266 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.007s	sys 0.040s
I20260709 12:21:17.080453  2949 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:17.039501  2800 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:17.085686  3038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.013s	sys 0.027s
I20260709 12:21:17.105700  3038 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:17.106060  2878 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:17.106817  3038 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.121374  2874 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:17.135974  3084 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:21:17.088014  2950 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:17.137791  2950 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.138288  2950 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:17.138832 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:17.140027  2741 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:17.140928  2949 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:17.140672  2741 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:17.141469  2949 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:17.142187  2741 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:17.145133  3079 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:21:17.040328  3032 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:17.146363  3032 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.147997  3051 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.149479  3051 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:17.146755  2871 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:17.050212  3071 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:17.098426 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.007s	sys 0.040s
W20260709 12:21:17.152658 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 864345us
I20260709 12:21:17.152410  3071 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.154209 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:17.155678  2850 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:21:17.155824 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:17.156468  2988 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:17.157480  2850 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:17.100931 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:17.085373  2692 raft_consensus.cc:2764] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:21:17.158145  2850 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:17.100309  3058 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:17.092535  3005 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.007s	sys 0.035s
I20260709 12:21:17.085786  3057 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:17.159795  2692 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:17.159760  3057 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.160604  3065 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:21:17.161155 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60333.6us
W20260709 12:21:17.105227  3074 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:21:17.049814  2876 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.162539 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:17.088145  3068 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:21:17.160852  3072 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.104897  2800 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.091897  2743 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.165077 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:17.165558 28329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.028s	sys 0.046s
I20260709 12:21:17.164633 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:17.088145  3067 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:17.171808 28329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.029s	sys 0.047s
W20260709 12:21:17.167701 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.021s	sys 0.045s
I20260709 12:21:17.161118  3065 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:17.170138 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.091288  2742 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.016s	sys 0.030s
I20260709 12:21:17.086742  3048 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:17.173781 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.021s	sys 0.046s
W20260709 12:21:17.221299 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.222019 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47319.9us
W20260709 12:21:17.171398  3004 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.017s	sys 0.045s
W20260709 12:21:17.225802 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51059.2us
I20260709 12:21:17.223502 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:17.228685 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:17.171514  2856 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.021s	sys 0.041s
W20260709 12:21:17.229661  2856 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.022s	sys 0.042s
W20260709 12:21:17.232836 28292 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.007s	sys 0.007s
W20260709 12:21:17.233235 28292 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.007s	sys 0.007s
W20260709 12:21:17.240267 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 620974us
W20260709 12:21:17.175557  2742 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.017s	sys 0.030s
W20260709 12:21:17.231638  3014 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.777s	user 0.008s	sys 0.008s
I20260709 12:21:17.192294  2952 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:21:17.228276  3004 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.017s	sys 0.045s
I20260709 12:21:17.248145 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:17.193868  2999 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:21:17.249941 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.882s	user 0.011s	sys 0.005s
W20260709 12:21:17.254068 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.011s	sys 0.005s
W20260709 12:21:17.250258 27735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.009s	sys 0.006s
I20260709 12:21:17.254673 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:17.233978 27729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.001s	sys 0.013s
W20260709 12:21:17.240958  2288 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.910s	user 0.010s	sys 0.003s
W20260709 12:21:17.232664  3031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.002s	sys 0.012s
I20260709 12:21:17.176358  2743 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:17.255096 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.010s	sys 0.006s
W20260709 12:21:17.255482 27729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.001s	sys 0.014s
W20260709 12:21:17.234227  3061 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.002s	sys 0.012s
W20260709 12:21:17.250974  2857 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.000s	sys 0.014s
I20260709 12:21:17.233896  3085 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.271231 28335 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:21:17.273923 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:17.324414 27732 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:17.175251  3067 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:17.248977  2989 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.005s	sys 0.009s
W20260709 12:21:17.295215  2686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.008s	sys 0.024s
I20260709 12:21:17.285305  3090 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:17.325137 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3403) took 1213 ms. Trace:
W20260709 12:21:17.285126  3031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.003s	sys 0.012s
I20260709 12:21:17.325959 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3398) took 1221 ms. Trace:
W20260709 12:21:17.334630 27730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.013s	sys 0.007s
W20260709 12:21:17.318720  3061 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.002s	sys 0.013s
I20260709 12:21:17.258199  3014 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.339560 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:17.342995  2686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.008s	sys 0.024s
W20260709 12:21:17.342275 27738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.017s	sys 0.018s
I20260709 12:21:17.229821  2850 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:17.320045  2857 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.000s	sys 0.015s
I20260709 12:21:17.289672  2876 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:17.347653 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:17.334616  2692 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:17.355790 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:17.334710  2989 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.005s	sys 0.010s
I20260709 12:21:17.356783 27729 rpcz_store.cc:276] 0709 12:21:16.112143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.202605 (+ 90462us) service_pool.cc:224] Handling call
0709 12:21:16.202954 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.282396 (+ 79442us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.282431 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.283028 (+   597us) log.cc:844] Serialized 26 byte log entry
0709 12:21:16.608842 (+325814us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:17.291202 (+682360us) log.cc:844] Serialized 14 byte log entry
0709 12:21:17.324755 (+ 33553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.324843 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:17.324921 (+    78us) raft_consensus.cc:1685] finished
0709 12:21:17.324933 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.325082 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67648,"prepare.queue_time_us":535054,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":128013,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":16156,"spinlock_wait_cycles":31872,"thread_start_us":1007165,"threads_started":2,"wal-append.queue_time_us":683802}
I20260709 12:21:17.375305 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:17.376140 27729 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:17.365619 27659 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:17.366206 27738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.018s	sys 0.018s
I20260709 12:21:17.377758 27655 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:17.359236 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:17.362900 27730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.013s	sys 0.007s
I20260709 12:21:17.368839 27661 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:17.255378  2906 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.010s	sys 0.019s
I20260709 12:21:17.383163 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:17.384941 27729 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:17.388217  3091 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:17.389753  3091 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.390213  3091 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.344462  3097 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:17.252130  3065 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.404114 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:17.405313 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:17.343947  2878 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:21:17.406167 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:17.408800 27657 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:17.357944  3093 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:17.387660 27729 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:21:17.409492 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:17.410771 27729 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260709 12:21:17.395147  2906 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.011s	sys 0.019s
I20260709 12:21:17.412695 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:17.422281 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:17.440332 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:17.444705 27658 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.007s	sys 0.009s
W20260709 12:21:17.445860 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.002s	sys 0.010s
W20260709 12:21:17.446327 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.002s	sys 0.010s
W20260709 12:21:17.446610 27658 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.007s	sys 0.009s
I20260709 12:21:17.448531 28330 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:17.342096  3085 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:17.464990 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3455) took 1666 ms. Trace:
I20260709 12:21:17.465490 27658 rpcz_store.cc:276] 0709 12:21:15.798274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.896640 (+ 98366us) service_pool.cc:224] Handling call
0709 12:21:15.896914 (+   274us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:15.897247 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.897274 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:15.897707 (+   433us) log.cc:844] Serialized 26 byte log entry
0709 12:21:16.639660 (+741953us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:17.449213 (+809553us) log.cc:844] Serialized 14 byte log entry
0709 12:21:17.462555 (+ 13342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.462935 (+   380us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:17.464131 (+  1196us) raft_consensus.cc:1685] finished
0709 12:21:17.464203 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.464847 (+   644us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12578,"prepare.queue_time_us":376182,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":1070,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":1473303,"threads_started":2,"wal-append.queue_time_us":1197350}
I20260709 12:21:17.364770  3008 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
W20260709 12:21:17.461683 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82791.9us
I20260709 12:21:17.397007  2876 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:17.472693 27734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.007s	sys 0.006s
I20260709 12:21:17.473208 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:17.460281 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47663.9us
I20260709 12:21:17.479768 27661 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:17.397976  3018 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:17.440960  2891 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:21:17.486156 27734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.007s	sys 0.006s
W20260709 12:21:17.434192  3005 consensus_queue.cc:1175] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 20 status { last_received { term: 20 index: 13 } last_committed_idx: 13 last_received_current_leader { term: 20 index: 13 } }
W20260709 12:21:17.469015  3074 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20260709 12:21:17.372732  2793 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.002s	sys 0.012s
I20260709 12:21:17.405601  3063 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:17.407439  2857 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:17.456023  2870 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:17.462399  3065 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:17.585816  2793 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.004s	sys 0.019s
I20260709 12:21:17.465734  2783 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.471426  2988 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:21:17.524580 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49489.8us
I20260709 12:21:17.584393  3048 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:17.635746 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:17.555721 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.519794  3018 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.408483  3097 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.522954  2800 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:17.523639 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.469146  3046 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:17.584712  3087 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:17.603252  3060 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.524819  2876 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.778417 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
W20260709 12:21:17.629300  3078 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:21:17.603641  3063 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:17.362112 27735 rpcz_store.cc:276] 0709 12:21:16.104355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.105385 (+  1030us) service_pool.cc:224] Handling call
0709 12:21:16.105721 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.105936 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:21:16.650689 (+544753us) log.cc:844] Serialized 14 byte log entry
0709 12:21:17.325595 (+674906us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.325666 (+    71us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:17.325711 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.325784 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.325905 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33979,"prepare.queue_time_us":982369,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":2174,"raft.queue_time_us":55565,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":6809,"thread_start_us":1146722,"threads_started":2}
W20260709 12:21:17.673759 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 419133us
I20260709 12:21:18.241971 27735 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
W20260709 12:21:18.212976 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:18.243507 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:18.244283 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.814369 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 178651us
W20260709 12:21:18.174578 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.008s	sys 0.003s
W20260709 12:21:18.248005 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.010s	sys 0.004s
W20260709 12:21:18.172161 27737 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.005s	sys 0.005s
W20260709 12:21:18.250337 27737 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.005s	sys 0.008s
I20260709 12:21:17.634907  2857 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.252314  2989 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.252779  3048 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:17.886739  3057 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.023s	sys 0.044s
I20260709 12:21:18.241817 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:18.253711 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45763.6us
W20260709 12:21:17.685662  2949 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.544s	user 0.015s	sys 0.000s
I20260709 12:21:17.639652  2988 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:17.995916  3126 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:21:17.677961  2856 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:17.635516  2907 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.634153  3092 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:17.638518  3070 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:18.356799 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s	user 0.014s	sys 0.045s
W20260709 12:21:18.357111 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.918s	user 0.062s	sys 0.193s
I20260709 12:21:17.865568  3068 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.181524  3116 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.354802 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 113126us
I20260709 12:21:18.181685  3053 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.865458  3061 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.183531  3032 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:18.157217  3115 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
W20260709 12:21:18.411921 27658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.115s	sys 0.206s
W20260709 12:21:18.412400 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.063s	sys 0.193s
W20260709 12:21:18.403002 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.009s	sys 0.017s
W20260709 12:21:18.102850  3123 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.424s	user 0.000s	sys 0.001s
W20260709 12:21:18.407636 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.015s	sys 0.045s
I20260709 12:21:18.427959 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:18.103157  3100 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.512s	user 0.000s	sys 0.001s
I20260709 12:21:18.430688 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:17.856099  2800 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.429929 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:18.051949  3101 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.638s	user 0.000s	sys 0.001s
W20260709 12:21:18.156050  3122 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 12:21:18.471719 27588 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:21:18.470495 27658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.117s	sys 0.206s
I20260709 12:21:18.473198 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:18.165756  3121 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:21:17.927839  3091 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:18.186741  2950 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.048s	user 0.008s	sys 0.005s
W20260709 12:21:18.102319  3112 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.470s	user 0.000s	sys 0.001s
I20260709 12:21:17.678314  2783 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.471597 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:18.087347  3119 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.490s	user 0.001s	sys 0.000s
I20260709 12:21:18.476359 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:17.801681  3018 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:18.478708 27589 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:18.478963 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.010s	sys 0.021s
I20260709 12:21:18.479909 27581 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:18.479960 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:18.595779 27584 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:18.351909  2988 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:17.867062  2686 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.667s
W20260709 12:21:18.087509  3099 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.650s	user 0.000s	sys 0.002s
W20260709 12:21:18.062163  3104 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.592s	user 0.000s	sys 0.000s
I20260709 12:21:18.471261 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1332) took 1066 ms. Trace:
W20260709 12:21:18.483224 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55365.7us
I20260709 12:21:18.483311 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:18.485550 27588 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:21:18.485715 28328 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: [20.13-23.14]
W20260709 12:21:18.045369  3086 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.615s	user 0.000s	sys 0.001s
I20260709 12:21:17.854348  2743 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.481983  3130 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:21:18.489753  3113 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260709 12:21:18.490501  3103 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.394s	user 0.000s	sys 0.000s
I20260709 12:21:17.854851  2952 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.425822  3057 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.263s	user 0.047s	sys 0.065s
W20260709 12:21:18.101642  3106 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.466s	user 0.000s	sys 0.000s
I20260709 12:21:17.865137  3052 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.407227  3092 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:17.998508  3014 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:18.469974  3051 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.320s	user 0.011s	sys 0.002s
I20260709 12:21:18.182518  3085 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:18.163947  3097 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:18.530048 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.995s	user 0.002s	sys 0.017s
W20260709 12:21:18.673657 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.139s	user 0.003s	sys 0.017s
I20260709 12:21:18.674496  2857 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.675184  3120 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.514348 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:18.332698  3060 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:18.532830 27630 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.612s (SENT)
I20260709 12:21:18.677505 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:18.678606 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 145820us
W20260709 12:21:18.572165 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65479.9us
I20260709 12:21:18.352097  2949 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.681818 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3405) took 2569 ms. Trace:
I20260709 12:21:18.682183 27733 rpcz_store.cc:276] 0709 12:21:16.112822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.205491 (+ 92669us) service_pool.cc:224] Handling call
0709 12:21:16.205836 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.285139 (+ 79303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.285171 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.285735 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.052231 (+766496us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:17.052293 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.052408 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.681622 (+1629214us) raft_consensus.cc:1685] finished
0709 12:21:18.681640 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.681759 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78915,"raft.queue_time_us":93146,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":22171,"thread_start_us":766081,"threads_started":1,"wal-append.queue_time_us":1058841}
W20260709 12:21:18.489753  3132 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.393s	user 0.000s	sys 0.000s
I20260709 12:21:18.342986  2989 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.696583  3147 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.696640  3136 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.697479  3147 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.697836  3136 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.698207  2949 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.698222  2989 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:18.699460  3057 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:18.700071  3065 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.068s	user 0.011s	sys 0.000s
I20260709 12:21:18.411703  2856 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.701205  3140 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.701720  3136 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.702745  2856 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.475894  3032 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.496879  2288 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.701759  3140 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.703496  3032 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.703635  2288 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.704229  3142 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.705242  3140 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:17.864768  2878 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:18.427217  3070 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.705747  2878 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.705935  3032 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.435775  3068 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.706354  2878 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:18.707650  3070 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.598384 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3412) took 2156 ms. Trace:
I20260709 12:21:18.627048  2973 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.708174  3068 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:18.529640  2874 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:18.708448  2973 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.710491  2973 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.516949  2906 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.095s
I20260709 12:21:18.594245 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:18.598278 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1236) took 1625 ms. Trace:
I20260709 12:21:18.571730  3138 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:18.492359  2800 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:18.614481  2793 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.016s	sys 0.019s
W20260709 12:21:18.712384  2793 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.017s	sys 0.019s
I20260709 12:21:18.708218 27734 rpcz_store.cc:276] 0709 12:21:16.441652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.790079 (+348427us) service_pool.cc:224] Handling call
0709 12:21:16.790494 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.790957 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:16.790998 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.798028 (+  7030us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.494406 (+696378us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:17.494455 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.494533 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.598124 (+1103591us) raft_consensus.cc:1685] finished
0709 12:21:18.598148 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.598305 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2465,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":12254,"thread_start_us":561978,"threads_started":1,"wal-append.queue_time_us":965913}
I20260709 12:21:18.713228  2793 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.713279  2741 raft_consensus.cc:2764] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:21:18.713706  2800 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.505442  2950 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.682839  2857 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:18.673394  2783 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.468345  3116 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.712067  3138 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:18.678872  3120 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.715790  2950 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:18.475651  3061 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.716408  3138 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.717164  3116 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.602871  3129 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.717832  3061 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.654312  3052 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.717954  3129 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.631778  2743 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:21:18.618979  2850 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:21:18.661489  3118 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:21:18.662402 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1077) took 2349 ms. Trace:
W20260709 12:21:18.666304  3090 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.293s	user 0.010s	sys 0.000s
W20260709 12:21:18.669883  3081 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.294s	user 0.000s	sys 0.024s
I20260709 12:21:18.643790  3130 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:18.482786  2869 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:18.703888  3060 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.704864  3142 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:18.706169  3127 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:21:18.679828  2809 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.468609  3053 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.699512  2989 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.741128  2989 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.742388  3053 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.427649  3048 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.745223  3048 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.745280  3052 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:18.660614  3108 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:18.426155  2907 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.747568  3071 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.705693  2288 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.700414  2692 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.749188  2692 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.749348  2809 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:18.751085  3110 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:21:18.751595  2692 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.751953  3130 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.732524  3090 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:18.732360  2869 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:18.756983  2869 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.758270  2869 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:18.719247  3120 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.713889  2793 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.740079  3147 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:18.700098  3057 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.646888  2952 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.773586  2952 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:18.749104  3152 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.771265  3129 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:18.781085  3121 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.138s	user 0.000s	sys 0.007s
W20260709 12:21:18.786134  3111 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 12:21:18.790268  3082 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.537s	user 0.007s	sys 0.005s
W20260709 12:21:18.542017 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.256s	user 0.013s	sys 0.009s
W20260709 12:21:18.935249 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.649s	user 0.018s	sys 0.015s
I20260709 12:21:18.742496  3092 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.496778 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.943212 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.944244 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:18.945535 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3411) took 2535 ms. Trace:
I20260709 12:21:18.482955 27587 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 12:21:18.856519 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.947181 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1338) took 1526 ms. Trace:
I20260709 12:21:18.947772 27587 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:18.948889 27587 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:21:18.947777 27656 rpcz_store.cc:276] 0709 12:21:17.420866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.471957 (+ 51091us) service_pool.cc:224] Handling call
0709 12:21:17.525440 (+ 53483us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:18.400792 (+875352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.400824 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.400875 (+    51us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:18.400910 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.946445 (+545535us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:18.946570 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.946940 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2012489984}
I20260709 12:21:18.949389 27587 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:18.949757 27584 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:21:18.611560 27655 rpcz_store.cc:276] 0709 12:21:17.404824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.404962 (+   138us) service_pool.cc:224] Handling call
0709 12:21:17.439299 (+ 34337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.439533 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:17.439769 (+   236us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.470837 (+1031068us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.470902 (+    65us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:18.470958 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.471043 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.471195 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":55310,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"thread_start_us":1030722,"threads_started":1}
I20260709 12:21:18.860739 27659 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:21:18.952245 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1275) took 2636 ms. Trace:
W20260709 12:21:18.954134 27659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1335) took 1548 ms (client timeout 1619 ms). Trace:
I20260709 12:21:18.952699 27737 rpcz_store.cc:276] 0709 12:21:16.316245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.330534 (+ 14289us) service_pool.cc:224] Handling call
0709 12:21:16.330911 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.331301 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:17.071161 (+739860us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.399232 (+1328071us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.399542 (+   310us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.427731 (+ 28189us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.427907 (+   176us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:18.427944 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.428038 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.780733 (+352695us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b24002c5b38. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:21:18.951836 (+171103us) raft_consensus.cc:1685] finished
0709 12:21:18.951879 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.952091 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23745,"prepare.queue_time_us":1083379,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":105,"raft.queue_time_us":430853,"raft.run_cpu_time_us":664,"raft.run_wall_time_us":109515,"spinlock_wait_cycles":6653824,"thread_start_us":1973827,"threads_started":3,"wal-append.queue_time_us":918540}
I20260709 12:21:18.863165 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1337) took 1442 ms. Trace:
I20260709 12:21:18.867626 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3484) took 1532 ms. Trace:
W20260709 12:21:18.719682 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 125503us
I20260709 12:21:18.955438 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.956398 27737 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:18.955538 27658 rpcz_store.cc:276] 0709 12:21:17.420485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.467792 (+ 47307us) service_pool.cc:224] Handling call
0709 12:21:17.468063 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.468241 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:21:17.488034 (+ 19793us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.861780 (+1373746us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.862283 (+   503us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.862364 (+    81us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:18.862492 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.862608 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.862916 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4489,"prepare.queue_time_us":218852,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":96,"raft.queue_time_us":276612,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":338,"thread_start_us":1372305,"threads_started":2,"wal-append.queue_time_us":1174357}
I20260709 12:21:18.958294 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.661871 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3404) took 2549 ms. Trace:
I20260709 12:21:18.959048 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.959275 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.686759 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1276) took 2370 ms. Trace:
I20260709 12:21:18.959838 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.872568 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3402) took 2762 ms. Trace:
I20260709 12:21:18.711485 28336 rpcz_store.cc:276] 0709 12:21:16.972811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.974416 (+  1605us) service_pool.cc:224] Handling call
0709 12:21:16.974818 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:17.025283 (+ 50465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.025330 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:17.026024 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.488333 (+462309us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:17.488928 (+   595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.489196 (+   268us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.598000 (+1108804us) raft_consensus.cc:1685] finished
0709 12:21:18.598027 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.598208 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":50114,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":15806,"thread_start_us":396656,"threads_started":1,"wal-append.queue_time_us":567549}
I20260709 12:21:18.960784 27737 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:18.960914 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.726735  2850 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:18.954702 27659 rpcz_store.cc:269] 0709 12:21:17.407020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.409104 (+  2084us) service_pool.cc:224] Handling call
0709 12:21:18.953776 (+1544672us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:18.954016 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3332959488}
I20260709 12:21:18.727787 28329 rpcz_store.cc:276] 0709 12:21:16.312585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.312790 (+   205us) service_pool.cc:224] Handling call
0709 12:21:16.313241 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.313739 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.313785 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.399152 (+ 85367us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b2c000471f8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:21:16.402558 (+  3406us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.172436 (+769878us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:17.172536 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.172618 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.662105 (+1489487us) raft_consensus.cc:1685] finished
0709 12:21:18.662125 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.662334 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":490,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"spinlock_wait_cycles":119284992,"thread_start_us":744964,"threads_started":1,"wal-append.queue_time_us":1006987}
I20260709 12:21:18.962000 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.880252  2760 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:18.963203 27737 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:18.963586  2760 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:18.964380 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.955626 27663 rpcz_store.cc:276] 0709 12:21:17.335310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.336198 (+   888us) service_pool.cc:224] Handling call
0709 12:21:17.336626 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:17.343827 (+  7201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.343869 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:17.353793 (+  9924us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.572285 (+218492us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:17.572358 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.572499 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.867008 (+1294509us) raft_consensus.cc:1685] finished
0709 12:21:18.867054 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.867379 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7104,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":14718,"thread_start_us":203896,"threads_started":1,"wal-append.queue_time_us":648047}
I20260709 12:21:18.965195 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:21:18.966166 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.965452  2760 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.960289 27730 rpcz_store.cc:276] 0709 12:21:16.316565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.560512 (+243947us) service_pool.cc:224] Handling call
0709 12:21:16.560872 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.561249 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.561288 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.561842 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.380033 (+818191us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:17.380094 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.380192 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.686535 (+1306343us) raft_consensus.cc:1685] finished
0709 12:21:18.686567 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.686701 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":84,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8300,"spinlock_wait_cycles":1829760,"thread_start_us":816996,"threads_started":1,"wal-append.queue_time_us":1484588}
I20260709 12:21:18.967121 27737 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:18.884266  2793 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.887799  2667 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.892031  2963 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.003s	sys 0.011s
I20260709 12:21:18.896106  2907 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.900753  2988 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:18.760242  2989 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.902058 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.008210 27733 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:19.007591  2667 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.009668 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.010546 27733 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:18.904418 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.011720 27734 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:19.012023 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:18.909184  2743 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.012957 28331 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:18.911221 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:18.914686 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3417) took 1564 ms. Trace:
I20260709 12:21:19.013561  2743 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.014245 27729 rpcz_store.cc:276] 0709 12:21:17.350398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.411625 (+ 61227us) service_pool.cc:224] Handling call
0709 12:21:17.411873 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:18.913771 (+1501898us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b180005b900. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:21:18.914079 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.914105 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.914163 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:18.914188 (+    25us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.914286 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.914488 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87233,"raft.queue_time_us":172800,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":18757,"spinlock_wait_cycles":43127552,"thread_start_us":172082,"threads_started":1}
I20260709 12:21:18.791757  3004 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:18.794523  3009 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:21:19.022606  3159 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:21:18.909178 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1238) took 1935 ms (client timeout 1745 ms). Trace:
W20260709 12:21:18.916334 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.922036 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.924677 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:18.929097 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.800482  3057 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.933485  3063 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.028173 27588 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:21:18.798257  3067 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:18.793124  3083 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.539s	user 0.015s	sys 0.012s
I20260709 12:21:19.028996  3004 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:19.029659 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:18.933883 28332 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260709 12:21:19.024629 28335 rpcz_store.cc:269] 0709 12:21:16.973544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.040684 (+ 67140us) service_pool.cc:224] Handling call
0709 12:21:18.908542 (+1867858us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:18.908847 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486157056}
I20260709 12:21:18.933750  2876 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:18.801177  3072 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.009s	sys 0.037s
W20260709 12:21:19.032917  3072 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.036s	user 0.009s	sys 0.038s
I20260709 12:21:19.032992 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1240) took 2058 ms. Trace:
I20260709 12:21:19.033530 28332 rpcz_store.cc:276] 0709 12:21:16.974239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.018866 (+ 44627us) service_pool.cc:224] Handling call
0709 12:21:17.026674 (+  7808us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.031605 (+2004931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.032528 (+   923us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:19.032866 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4383868288}
I20260709 12:21:19.024760  3009 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:18.529814 27580 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:19.035926  3009 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.961382  3120 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:19.037531 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.037856  3120 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.026533  2907 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:19.039543 27660 raft_consensus.cc:606] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:19.024091  3154 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.040951  3009 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:21:19.041404  3154 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.042033 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.962311  2850 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:18.959733 27736 rpcz_store.cc:276] 0709 12:21:16.112498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.202647 (+ 90149us) service_pool.cc:224] Handling call
0709 12:21:16.202954 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.872295 (+669341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.872346 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.872985 (+   639us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.446755 (+573770us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:17.446810 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.446905 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.661630 (+1214725us) raft_consensus.cc:1685] finished
0709 12:21:18.661662 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.661817 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80354,"raft.queue_time_us":92511,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":5931,"spinlock_wait_cycles":62720,"thread_start_us":1126091,"threads_started":2,"wal-append.queue_time_us":1786506}
I20260709 12:21:19.048333 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.049290 27736 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:18.940574  3092 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:19.049873  2743 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.050436 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:18.968461  2760 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:19.052273  2988 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:18.962189 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:18.972694 28333 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:21:19.022680 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:18.975193  3058 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:18.975894 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:18.974572 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.057125  3058 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.877094 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3414) took 1527 ms. Trace:
I20260709 12:21:19.058369 28328 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:18.977393 27731 rpcz_store.cc:276] 0709 12:21:16.110259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.110431 (+   172us) service_pool.cc:224] Handling call
0709 12:21:16.110855 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.111337 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.111391 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:16.120346 (+  8955us) log.cc:844] Serialized 26 byte log entry
0709 12:21:16.800435 (+680089us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:16.800506 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.800626 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.669265 (+1868639us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:18.871607 (+202342us) raft_consensus.cc:1685] finished
0709 12:21:18.871661 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.872245 (+   584us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10105,"raft.queue_time_us":5454,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":22395,"spinlock_wait_cycles":36804352,"thread_start_us":618017,"threads_started":1,"wal-append.queue_time_us":1099767}
I20260709 12:21:19.058787  3120 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.059587 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.060360  3008 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.981178 28334 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:18.983137 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:18.802605  3152 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.946298 27738 rpcz_store.cc:276] 0709 12:21:16.410503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.651201 (+240698us) service_pool.cc:224] Handling call
0709 12:21:16.670388 (+ 19187us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:16.720011 (+ 49623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.720044 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:16.720794 (+   750us) log.cc:844] Serialized 52 byte log entry
0709 12:21:17.377987 (+657193us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:17.419629 (+ 41642us) log.cc:844] Serialized 14 byte log entry
0709 12:21:17.455644 (+ 36015us) log.cc:844] Serialized 14 byte log entry
0709 12:21:17.455849 (+   205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.455931 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.713011 (+1257080us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:21:18.785555 (+ 72544us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:18.945081 (+159526us) raft_consensus.cc:1685] finished
0709 12:21:18.945120 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.945333 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49353,"prepare.queue_time_us":54706,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":97,"raft.queue_time_us":422675,"raft.run_cpu_time_us":8023,"raft.run_wall_time_us":1030910,"spinlock_wait_cycles":116188288,"thread_start_us":656807,"threads_started":1,"wal-append.queue_time_us":1985904}
I20260709 12:21:19.063187 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.064424 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.051311 27736 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:19.031044 27588 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:18.988451  2999 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.616s	user 0.012s	sys 0.036s
I20260709 12:21:18.997135  3018 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.999506 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.012897 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.026633 27583 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:21:19.036646  3142 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.039713  2907 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:18.882505  2809 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:18.976842 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.061224  3008 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.088400 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.088773 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.089457 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.089684 28333 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:19.090615 28336 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:21:19.088423 28334 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:21:19.091071 27658 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 14
I20260709 12:21:19.094472 27658 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:21:19.061107 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:18.990574 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 311118us
I20260709 12:21:19.073241 27738 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:21:18.961930 27585 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:19.058296 27732 rpcz_store.cc:276] 0709 12:21:17.349324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.351435 (+  2111us) service_pool.cc:224] Handling call
0709 12:21:17.351873 (+   438us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:17.352346 (+   473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.352402 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:17.361202 (+  8800us) log.cc:844] Serialized 52 byte log entry
0709 12:21:18.248607 (+887405us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:18.248669 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.248771 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.778710 (+529939us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b24002c5b38. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:21:18.876228 (+ 97518us) raft_consensus.cc:1685] finished
0709 12:21:18.876265 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.876815 (+   550us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29726,"raft.queue_time_us":21537,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":15793,"spinlock_wait_cycles":4801408,"thread_start_us":856856,"threads_started":1,"wal-append.queue_time_us":1282621}
I20260709 12:21:19.081732  3154 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.081871  3058 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.082638 27730 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:19.083638 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.082872 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.084303  2667 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:19.085542 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.088846 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.092151 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.094861 27658 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:19.041250 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.030949  3004 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.103437 27734 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:19.103518 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.104847 28328 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:19.106001 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:19.106559 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.107074 28335 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:19.115036 28332 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
W20260709 12:21:19.107223 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.108290  3152 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:19.108961 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:19.120656 28333 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:21:19.120857  3130 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:19.121956  3130 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.109380  3004 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:19.126941 27737 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
W20260709 12:21:19.093115  2999 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.728s	user 0.013s	sys 0.036s
I20260709 12:21:19.119612 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.127878  2999 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:19.129748  2999 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.131048 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.123135  3018 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.133080  3005 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:19.133920 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3485) took 1796 ms. Trace:
I20260709 12:21:19.134027 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:19.135658 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.136775 28336 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:19.134538 27662 rpcz_store.cc:276] 0709 12:21:17.337378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.337538 (+   160us) service_pool.cc:224] Handling call
0709 12:21:17.337912 (+   374us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:17.346273 (+  8361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.346312 (+    39us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:18.671851 (+1325539us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:21:18.672559 (+   708us) log.cc:844] Serialized 78 byte log entry
0709 12:21:19.022522 (+349963us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:19.023203 (+   681us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.024325 (+  1122us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.024589 (+   264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.074722 (+ 50133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.133400 (+ 58678us) raft_consensus.cc:1685] finished
0709 12:21:19.133426 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:19.133662 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12366,"prepare.queue_time_us":1150,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":10787,"raft.queue_time_us":9113,"raft.run_cpu_time_us":458,"raft.run_wall_time_us":16635,"spinlock_wait_cycles":5959552,"thread_start_us":349184,"threads_started":1,"wal-append.queue_time_us":412237}
I20260709 12:21:19.109994 27730 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:21:19.111199 27655 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:21:19.113812 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.119612 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.139604 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.141251  3057 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:19.143823  3057 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:21:19.145605  2667 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:19.146191  2667 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:19.147651 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68384us
I20260709 12:21:19.147924  2667 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.150033 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.150954 27580 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:19.170419 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.174283 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.175235 27656 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:19.177718 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.178604 27662 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:19.189541 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.172340 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.195837 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.194475 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.194598  2874 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:19.214668 27662 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:19.169739  2988 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.217422 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.221112  3138 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:19.221789  3138 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.223539 27585 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:19.224715 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.225674  3130 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.225937 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:19.230470  2878 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.014s
I20260709 12:21:19.225688 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.231925 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.232566 27659 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:19.232815 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:19.236320  2988 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.238832 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:19.240900 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.248324 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.251742 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.255549 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.251456 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.259002 28331 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:19.215178  2809 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.262493 28330 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:21:19.239183  2988 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.276806 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.287961 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.298219 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.313222 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.315605 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:19.319681 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 158012us
I20260709 12:21:19.322302 27655 raft_consensus.cc:2410] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:21:19.231065  2878 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.324867 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.314066 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:19.331537  3142 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:19.332289  3142 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.335729 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:19.339437  2809 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.340076  3005 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:21:19.340612  3005 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:19.341019  3005 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:21:19.342280  2809 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:19.343858 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46099.4us
I20260709 12:21:19.344698 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:19.348435 27588 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:19.298154  3154 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:19.352831 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.340129 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.316257 27664 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
W20260709 12:21:19.358115 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 151228us
I20260709 12:21:19.350310 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:19.343218 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.370424  3150 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.370995 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.371968 27583 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:19.371284  3150 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.373083 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.370518  3052 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:19.374033 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.374363  3052 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:19.374823  3052 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:19.363158  2809 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 12:21:19.376514  2809 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:19.376886  2809 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:19.377586 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.378101  2809 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.366050  3154 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:19.379295 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.379863  3154 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:19.389505 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.393857 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.395521 27657 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Committing config change with OpId 20.13: config changed from index 11 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:19.413376 27729 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:19.266644  2874 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:19.418962 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.419513 28333 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:19.415264 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:19.421787 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 99134.9us
I20260709 12:21:19.402374 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.400887 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:19.442363 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.422058 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:19.447561 27584 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:19.316943  3105 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:19.428722  2760 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:19.453496 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67792.4us
I20260709 12:21:19.411922  3008 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:19.457451  3170 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
W20260709 12:21:19.457792 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91109.5us
I20260709 12:21:19.458226 27585 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:19.459411 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.460314  3072 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:19.461990 27659 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:19.400171  3142 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:19.464221 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.421298  2878 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.467132 27738 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:19.459131 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:19.270583  2850 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.459518  3120 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:21:19.473585 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:19.494266 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.495625 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.497702 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.499650 27581 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:21:19.495990 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:19.504449 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.443399  2809 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:19.511641 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:19.513394 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.514330 27662 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:19.515266 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:19.526794 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 74157us
I20260709 12:21:19.481900  3120 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:19.533301 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.466207  2760 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.495990  3071 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:19.531180 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.537761 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.466255  3072 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.556463 27729 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:19.533700  3120 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:19.572220 27656 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:19.542171 27663 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:21:19.581773  3172 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.362s	user 0.002s	sys 0.000s
W20260709 12:21:19.582696 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 181980us
I20260709 12:21:19.584013 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.538451  3071 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:19.586521 27589 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:19.588240  2850 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.589176 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:19.590790 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44183.7us
I20260709 12:21:19.592104 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.590503  2988 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:19.587370  3071 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:19.609584 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.611666 27731 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:19.535215  3105 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.599905  2869 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:19.622720  3105 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.623637 27655 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:19.624857 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:19.627343 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:19.632308  3018 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:19.632923  3018 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:19.633455  3018 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:19.637135 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.610985  2988 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:19.640385 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.593835  3052 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.638535 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:19.646121 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.635812  3004 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:21:19.664116 27630 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 2.605s (SENT)
I20260709 12:21:19.667871  2667 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:19.681164 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.681738 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
W20260709 12:21:19.682678 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53531us
I20260709 12:21:19.688911 27588 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:19.690373 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:19.698807 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 91183.8us
I20260709 12:21:19.645569  2988 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:19.712656 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.716493 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.683151  2874 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.682636  3004 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:19.732664 27655 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:21:19.734587 27656 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:21:19.738498 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.739862 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.744606 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.726317  3004 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:19.749909 27731 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:21:19.752351 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.756515 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:19.757324 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.764009 27663 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:19.765481 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:19.767021 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 102980us
I20260709 12:21:19.779906 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.782094 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:19.783788 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.730935  3142 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:21:19.808552 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.814507 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.819303 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.779176  2850 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.808997  2760 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.825407 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.829595 27656 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:19.809715  3063 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.853195  3063 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.854218 27588 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:19.857646  2809 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:19.860548 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:19.866607 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:19.894364 27658 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24.
I20260709 12:21:19.835817 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.895597  3150 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.895985 27656 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:19.836071  3168 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.839231  3005 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.897656  3168 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:19.898221 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3609) took 1425 ms (client timeout 1882 ms). Trace:
W20260709 12:21:19.898690 27586 rpcz_store.cc:269] 0709 12:21:18.472758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.472897 (+   139us) service_pool.cc:224] Handling call
0709 12:21:19.897882 (+1424985us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000bc7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:19.898049 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2807971072}
I20260709 12:21:19.899062  3005 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.899804 27583 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 11. Current term is 12. Ops: []
I20260709 12:21:19.900015 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.901252 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.901572 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.902403 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.902702 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.899824  3168 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.846793 27737 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:19.903169 27586 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:19.904562 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1331) took 2500 ms (client timeout 1830 ms). Trace:
I20260709 12:21:19.904971 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.905387 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.905812 27737 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
W20260709 12:21:19.904987 27664 rpcz_store.cc:269] 0709 12:21:17.403716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.403849 (+   133us) service_pool.cc:224] Handling call
0709 12:21:19.904351 (+2500502us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:19.904481 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3741156096}
I20260709 12:21:19.906148 27586 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:19.906653 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.907224 27664 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:21:19.847740 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.909343 27664 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:19.862515 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:19.848496 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.849249 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:19.849342  3052 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:19.851478  3092 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.801s	user 0.007s	sys 0.007s
I20260709 12:21:19.852519 27731 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:21:19.852274  2850 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:19.921883 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:19.922866 27586 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:19.923344  2850 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.832453 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:19.923669  2907 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.924160 27655 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:21:19.924880 27588 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:21:19.925316  3072 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.926126 27583 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:19.863158  3105 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:19.869060 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:19.928001 27582 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:19.929111 27582 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:19.872562  3063 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:19.876453 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54366 (request call id 35) took 2099 ms. Trace:
W20260709 12:21:19.884305  3179 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:21:19.932219 27660 rpcz_store.cc:276] 0709 12:21:17.777152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.777958 (+   806us) service_pool.cc:224] Handling call
0709 12:21:19.876131 (+2098173us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b5400211e54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:19.876307 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1826480512}
I20260709 12:21:19.888125 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:19.933724 27584 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 12: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:19.936822 27584 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:21:19.938254 27584 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:21:19.845606 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:19.939479 27734 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:19.924825 27580 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:21:19.895488 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:19.948758 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86380.7us
I20260709 12:21:19.959182 27659 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:21:19.914399 27738 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
W20260709 12:21:19.916760  3180 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.076s	user 0.002s	sys 0.000s
I20260709 12:21:19.919823  3138 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:19.968106 27663 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
W20260709 12:21:19.979732 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 223775us
I20260709 12:21:19.994011  2878 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:19.994769  2878 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:19.999274  2878 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:20.025000 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.027442 27580 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:21:20.029234 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:20.030514 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.030673 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.038816 27584 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:20.042912 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.044631 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:20.014293  3071 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:20.045606  3154 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.579s	user 0.012s	sys 0.000s
I20260709 12:21:20.061690 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:20.056753  3138 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:20.068032 27656 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
W20260709 12:21:20.065784 27661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.021s	sys 0.016s
W20260709 12:21:20.077258 27661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.021s	sys 0.017s
I20260709 12:21:20.079008 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:20.058548  2667 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:20.077373 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:20.078755 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 451506us
I20260709 12:21:20.070887  2809 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:20.089952 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:20.073338  3138 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:20.093811 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.063062  3154 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:20.107926 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73302us
I20260709 12:21:20.073208  3057 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.022s
I20260709 12:21:20.090214  2809 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:20.140004 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.099776  3138 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:20.148442 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:20.163763 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.164750 27584 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:20.166472 27662 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:20.161428 27487 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 24, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.168299  3071 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:20.170677 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85099us
I20260709 12:21:20.129690  3105 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:20.130926  3063 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:20.172930 27586 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:20.099558  2667 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.177496 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:20.149501  2809 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:20.176467  2878 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:20.180763 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.183663 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.130342  3004 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.130614  3018 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.185130 27580 raft_consensus.cc:2376] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24. Re-sending same reply.
W20260709 12:21:20.188154 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47061.9us
I20260709 12:21:20.171622  3057 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:20.188686 27481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 129) took 8104 ms. Trace:
I20260709 12:21:20.188508 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:20.191802 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:20.205210 27586 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:20.206192 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.208303 27487 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 21, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:20.195012 27481 rpcz_store.cc:276] 0709 12:21:12.085756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.094738 (+  8982us) service_pool.cc:224] Handling call
0709 12:21:20.162494 (+8067756us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:17.362480 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:17.612328 (+249848us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:17.629917 (+ 17589us) write_op.cc:435] Acquired schema lock
0709 12:21:17.631904 (+  1987us) tablet.cc:662] Decoding operations
0709 12:21:18.167205 (+535301us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:18.167295 (+    90us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:18.167360 (+    65us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:21:18.168850 (+  1490us) tablet.cc:701] Row locks acquired
0709 12:21:18.184387 (+ 15537us) write_op.cc:260] PREPARE: finished
0709 12:21:18.184579 (+   192us) write_op.cc:270] Start()
0709 12:21:18.184709 (+   130us) write_op.cc:276] Timestamp: P: 1783599678184518 usec, L: 0
0709 12:21:18.184734 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:21:18.185496 (+   762us) log.cc:844] Serialized 5622 byte log entry
0709 12:21:18.812929 (+627433us) op_driver.cc:464] REPLICATION: finished
0709 12:21:18.816902 (+  3973us) write_op.cc:301] APPLY: starting
0709 12:21:18.816975 (+    73us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:18.822185 (+  5210us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:18.822272 (+    87us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:19.182874 (+360602us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:19.257028 (+ 74154us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 12:21:19.273788 (+ 16760us) write_op.cc:312] APPLY: finished
0709 12:21:19.423429 (+149641us) log.cc:844] Serialized 129 byte log entry
0709 12:21:19.423763 (+   334us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:19.424119 (+   356us) write_op.cc:454] Released schema lock
0709 12:21:19.424212 (+    93us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:19.424902 (+   690us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:19.424941 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":84234,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":3732,"apply.run_cpu_time_us":24858,"apply.run_wall_time_us":607870,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62647,"num_ops":14,"prepare.queue_time_us":1008046,"prepare.run_cpu_time_us":28854,"prepare.run_wall_time_us":1188267,"raft.queue_time_us":155550,"raft.run_cpu_time_us":3226,"raft.run_wall_time_us":186535,"replication_time_us":627983,"thread_start_us":1233246,"threads_started":4,"wal-append.queue_time_us":468767}]]}
I20260709 12:21:20.210428 27487 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 23. New cstate: current_term: 23 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.225140 27487 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 21 to 24, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 24 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:20.227743 27487 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 23, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 23 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.231192 27487 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 18, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.192783 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:20.239233 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.243907 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:20.244127 28336 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:20.190871  3063 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:20.249797 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58090.6us
W20260709 12:21:20.256469  3120 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.000s
I20260709 12:21:20.257114  3120 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.258651 27655 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:20.252974  2907 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:20.260237  2907 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:20.260600  2667 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:20.261337  2667 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:20.261642  2907 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.271060 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:20.272769 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:20.264909 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:20.273562 27586 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:20.278708  3004 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:20.280027  3183 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:21:20.274952  3071 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:20.284056  3154 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:20.285516  3063 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.237563  3057 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:20.298084 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 105372us
I20260709 12:21:20.261698  2667 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:20.240988  2988 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.295974 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:20.328855 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76298.2us
W20260709 12:21:20.341030 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.342133 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.348635 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:20.335781 27487 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 22, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 22 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:20.353178 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53846.7us
W20260709 12:21:20.348769 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.008s	sys 0.029s
W20260709 12:21:20.355019 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.008s	sys 0.029s
W20260709 12:21:20.355531 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 662198us
I20260709 12:21:20.355818 27664 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:20.335860  3004 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:21:20.372967 27587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.017s	sys 0.028s
W20260709 12:21:20.375131 27587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.017s	sys 0.029s
I20260709 12:21:20.361591 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:20.380550 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:20.382257 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:20.338806  3105 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:20.376137 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:20.383541  3046 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.009s	sys 0.023s
W20260709 12:21:20.383980  3046 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.009s	sys 0.023s
W20260709 12:21:20.384912 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.026s	sys 0.036s
W20260709 12:21:20.385342 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.026s	sys 0.036s
W20260709 12:21:20.385761 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.385569 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:20.386579  3092 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:20.372082  3004 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.387075 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:20.375351 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.014s	sys 0.026s
W20260709 12:21:20.392704 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.015s	sys 0.026s
I20260709 12:21:20.383186  3105 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.401677 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1365) took 1461 ms. Trace:
I20260709 12:21:20.387172  3092 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:20.390998  3004 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:20.389427  3072 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:20.397758  3005 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:20.389083  3154 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:20.408106 27487 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 18 to 19, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 19 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:20.443120 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.469313 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82320.5us
I20260709 12:21:20.469588  3120 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:20.470325 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:20.470463 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88285.3us
I20260709 12:21:20.471928 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:20.472954 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2906) took 1079 ms. Trace:
I20260709 12:21:20.473219 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:20.473575 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:20.474545 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 849702us
I20260709 12:21:20.475657 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:20.401366  3071 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:20.404861  3105 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:20.494277 27586 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:20.473461 27661 rpcz_store.cc:276] 0709 12:21:19.393387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.432675 (+ 39288us) service_pool.cc:224] Handling call
0709 12:21:19.433082 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.433669 (+   587us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:21:19.434013 (+   344us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.078030 (+644017us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:20.088407 (+ 10377us) log.cc:844] Serialized 26 byte log entry
0709 12:21:20.088596 (+   189us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:20.088653 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.088760 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.472645 (+383885us) raft_consensus.cc:1685] finished
0709 12:21:20.472676 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.472874 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":773,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":5940,"raft.queue_time_us":261,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":6421,"thread_start_us":643412,"threads_started":1,"wal-append.queue_time_us":1035849}
I20260709 12:21:20.418279  3092 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:20.410022  3130 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:20.407734 27587 rpcz_store.cc:276] 0709 12:21:18.940506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.428299 (+487793us) service_pool.cc:224] Handling call
0709 12:21:19.428717 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.429480 (+   763us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.429522 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.430580 (+  1058us) log.cc:844] Serialized 26 byte log entry
0709 12:21:19.701693 (+271113us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:20.389080 (+687387us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.392078 (+  2998us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.393176 (+  1098us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.399139 (+  5963us) raft_consensus.cc:1685] finished
0709 12:21:20.399167 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.401605 (+  2438us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83,"raft.queue_time_us":8933,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":3309,"thread_start_us":939211,"threads_started":2,"wal-append.queue_time_us":440670}
I20260709 12:21:20.473245 27487 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.502455 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:20.466095  3168 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:20.488867  3046 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:20.498062  3120 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:21:20.522018  3185 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:21:20.522501 27487 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:21:20.548108 28330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.814s	user 0.011s	sys 0.031s
W20260709 12:21:20.549129 28330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.012s	sys 0.032s
I20260709 12:21:20.536273  3142 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:20.537461 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1369) took 1584 ms. Trace:
I20260709 12:21:20.539537  3168 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.530406  3018 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:20.496364  3154 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:21:20.544727  3071 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:20.545629  3120 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.551574 27589 rpcz_store.cc:276] 0709 12:21:18.952734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.594699 (+641965us) service_pool.cc:224] Handling call
0709 12:21:19.603750 (+  9051us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.650390 (+ 46640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.650421 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.650905 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 12:21:20.078725 (+427820us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:20.078772 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.078846 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.537231 (+458385us) raft_consensus.cc:1685] finished
0709 12:21:20.537254 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.537399 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18276,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5161,"thread_start_us":427444,"threads_started":1,"wal-append.queue_time_us":884365}
I20260709 12:21:20.489341  2667 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.553196  3130 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:21:20.569949 27733 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.018s	sys 0.028s
W20260709 12:21:20.590211 27733 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.019s	sys 0.028s
W20260709 12:21:20.588511 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.000s	sys 0.046s
W20260709 12:21:20.591878 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.000s	sys 0.046s
W20260709 12:21:20.569113 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.025s	sys 0.023s
W20260709 12:21:20.593302 27628 thread.cc:641] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.010s	sys 0.037s
W20260709 12:21:20.596451 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.025s	sys 0.023s
W20260709 12:21:20.596585 27628 thread.cc:608] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.011s	sys 0.037s
W20260709 12:21:20.596819 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 769136us
W20260709 12:21:20.592247 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 898171us
I20260709 12:21:20.594421 27487 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 16 to 18, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 18 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:21:20.578851 27730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.017s	sys 0.030s
W20260709 12:21:20.600456 27730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.018s	sys 0.030s
I20260709 12:21:20.595263 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:20.578943 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s	user 0.015s	sys 0.023s
W20260709 12:21:20.601431 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.016s	sys 0.023s
I20260709 12:21:20.566555  3071 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:20.595276  3150 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.007s	sys 0.050s
I20260709 12:21:20.572728  3120 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:20.612331 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 21, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 21 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:20.611184 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1091) took 1264 ms. Trace:
I20260709 12:21:20.615852 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:20.611984  3150 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.007s	sys 0.050s
I20260709 12:21:20.567337  3142 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.617401 28333 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:20.588723  2988 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:20.599829 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3022) took 1186 ms. Trace:
W20260709 12:21:20.625204 27588 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.067s	sys 0.127s
W20260709 12:21:20.626165 27588 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.067s	sys 0.127s
I20260709 12:21:20.635460 28332 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:20.566735  3046 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:21:20.613940  3071 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:20.614042 28330 rpcz_store.cc:276] 0709 12:21:19.347033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.347743 (+   710us) service_pool.cc:224] Handling call
0709 12:21:19.375501 (+ 27758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.375852 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:19.701404 (+325552us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.606478 (+905074us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:20.606562 (+    84us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:20.610778 (+  4216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.610891 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.611065 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":618669,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"thread_start_us":1186523,"threads_started":2}
I20260709 12:21:20.610035  3057 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:20.640059 27582 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.000s	sys 0.026s
W20260709 12:21:20.647065 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51925.5us
W20260709 12:21:20.647578 27582 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.000s	sys 0.026s
I20260709 12:21:20.639432 27487 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 18, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.621830 27733 rpcz_store.cc:276] 0709 12:21:19.412997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.413175 (+   178us) service_pool.cc:224] Handling call
0709 12:21:19.413592 (+   417us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:19.414075 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.414104 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:19.414886 (+   782us) log.cc:844] Serialized 52 byte log entry
0709 12:21:19.726014 (+311128us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:20.599074 (+873060us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.599401 (+   327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.599477 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.599533 (+    56us) raft_consensus.cc:1685] finished
0709 12:21:20.599553 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.599750 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":270270,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":117,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":157,"thread_start_us":1157873,"threads_started":2,"wal-append.queue_time_us":673276}
I20260709 12:21:20.637013  2667 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:20.664223 27583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.000s	sys 0.028s
I20260709 12:21:20.617946  3130 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.038s
W20260709 12:21:20.667268 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44186.7us
W20260709 12:21:20.667048 27585 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.002s	sys 0.026s
I20260709 12:21:20.642390  2874 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:21:20.670099 27585 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.002s	sys 0.026s
W20260709 12:21:20.666602 27583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.000s	sys 0.028s
I20260709 12:21:20.664721 27487 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 12 to 10, term changed from 21 to 19, leader changed from <none> to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from VOTER to NON_VOTER. New cstate: current_term: 19 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:21:20.685911 27487 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 21 to 23, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 23 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:20.688179  3005 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:20.689025  3005 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.690886 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:20.691500  3154 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:20.691807 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1368) took 1739 ms. Trace:
I20260709 12:21:20.693097  3154 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:20.693935 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:20.694911 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:20.696316  3046 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:20.674058  3198 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:20.675909  3057 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:20.692312 27581 rpcz_store.cc:276] 0709 12:21:18.952245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.594157 (+641912us) service_pool.cc:224] Handling call
0709 12:21:19.608087 (+ 13930us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.648851 (+ 40764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.648919 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.649605 (+   686us) log.cc:844] Serialized 26 byte log entry
0709 12:21:20.079093 (+429488us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:20.079151 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.079248 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.691537 (+612289us) raft_consensus.cc:1685] finished
0709 12:21:20.691559 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.691719 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3444,"raft.queue_time_us":8419,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"thread_start_us":429023,"threads_started":1,"wal-append.queue_time_us":956059}
I20260709 12:21:20.688349 27487 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 24, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.695016  3154 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:21:20.727919 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:20.704810  2874 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:20.710152  3198 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:20.748360 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.005s	sys 0.014s
W20260709 12:21:20.748852 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.005s	sys 0.014s
I20260709 12:21:20.728830  2988 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:20.739403  3153 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.904s	user 0.014s	sys 0.008s
I20260709 12:21:20.763078 27487 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 17, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 17 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:21:20.750219 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.806344 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.005s	sys 0.009s
W20260709 12:21:20.806982 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.772101 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.009s	sys 0.009s
W20260709 12:21:20.808044 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.009s	sys 0.009s
W20260709 12:21:20.771643 28335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.002s	sys 0.015s
I20260709 12:21:20.736573  2874 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:20.737062  3009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.003s	sys 0.024s
I20260709 12:21:20.808959 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3026) took 1394 ms. Trace:
W20260709 12:21:20.806804 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 542033us
W20260709 12:21:20.808881 28335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.002s	sys 0.015s
I20260709 12:21:20.817456  3168 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:20.821092  2874 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:20.823753  3196 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:20.831600  3196 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.834375  3196 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:20.834609  3092 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.728641  3057 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.835601  3142 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:20.852887 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 158058us
I20260709 12:21:20.747692  3198 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:20.760082  3046 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:20.830530 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:20.824911  2869 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:20.855877  3196 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:20.805639  2760 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:21:20.824311  3193 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:20.861198 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3632) took 1638 ms. Trace:
I20260709 12:21:20.861842  3142 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:21:20.859144  3194 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:20.826735 27487 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 15 to 22, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 22 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:20.863626  2988 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:20.864351 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:20.803992  3005 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:20.804224  3154 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.864410  3092 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:20.866737  3057 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:20.828344  3178 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.868610 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:20.882047  2878 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:20.870947 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:20.910156 27656 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:20.872207 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:20.873443 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:20.874421 27472 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.003s	sys 0.024s
I20260709 12:21:20.875121 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:20.876436 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:20.876725  3201 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.880077 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:20.822331  3009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.004s	sys 0.025s
W20260709 12:21:20.805267  3153 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.099s	user 0.015s	sys 0.009s
I20260709 12:21:20.896293 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:20.893823 27583 rpcz_store.cc:276] 0709 12:21:19.222323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.926841 (+704518us) service_pool.cc:224] Handling call
0709 12:21:19.927089 (+   248us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.929254 (+  2165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.929282 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.929809 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:21:20.670783 (+740974us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:20.670824 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.738036 (+ 67212us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:20.738103 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.860898 (+122795us) raft_consensus.cc:1685] finished
0709 12:21:20.860918 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.861138 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2253,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":4198656,"thread_start_us":740572,"threads_started":1,"wal-append.queue_time_us":910484}
I20260709 12:21:20.919155 27658 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:20.826423 27736 rpcz_store.cc:276] 0709 12:21:19.414883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.421850 (+  6967us) service_pool.cc:224] Handling call
0709 12:21:19.422207 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.422583 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:19.735978 (+313395us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.808559 (+1072581us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:20.808624 (+    65us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:20.808678 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.808763 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.808897 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":573266,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":47215,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":45,"thread_start_us":1164715,"threads_started":2}
I20260709 12:21:20.895911  3105 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.903800 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:20.905308 27487 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 14, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 14 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:20.869915 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:20.906870 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:20.922928 27662 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:20.923619 27738 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
W20260709 12:21:20.910353  2809 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.722s	user 0.014s	sys 0.000s
W20260709 12:21:20.907871 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.015s	sys 0.020s
I20260709 12:21:20.891022  3046 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:20.912887 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
W20260709 12:21:20.918382  3138 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.736s	user 0.011s	sys 0.003s
I20260709 12:21:20.914824 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:20.915822 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:20.938684 27733 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:20.911880 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:21:20.913547 27472 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.004s	sys 0.024s
I20260709 12:21:20.921787 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:20.936729 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:20.898065  3194 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:20.941043 27734 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:20.897457  2874 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:20.899748  3196 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.940377 27655 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:20.970464 27733 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
W20260709 12:21:20.941699 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.016s	sys 0.020s
I20260709 12:21:20.956347 27658 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:21:20.955338 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75360.9us
I20260709 12:21:20.996743 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:20.943521 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 24, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:20.936498 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3028) took 1471 ms. Trace:
I20260709 12:21:21.000937 27730 rpcz_store.cc:276] 0709 12:21:19.467534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.471586 (+  4052us) service_pool.cc:224] Handling call
0709 12:21:19.494494 (+ 22908us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.494871 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:19.784955 (+290084us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.936150 (+1151195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:20.936206 (+    56us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:20.936258 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.936324 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.936434 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":501272,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":41,"raft.queue_time_us":254469,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":55,"thread_start_us":1387709,"threads_started":3,"wal-append.queue_time_us":1037200}
W20260709 12:21:21.003878 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 378166us
I20260709 12:21:21.016443 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:21.009001 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72785.3us
I20260709 12:21:21.017401 27736 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:21.021632 27738 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:20.921869  3195 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:21:21.058230 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 449504us
W20260709 12:21:21.049299 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.014s	sys 0.022s
I20260709 12:21:20.917555  2878 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:21.061861 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.015s	sys 0.022s
W20260709 12:21:21.071991 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 571538us
W20260709 12:21:21.043087  2907 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.017s	sys 0.025s
I20260709 12:21:20.904754  3193 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.922139  2988 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.077203 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:20.935442  3154 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.015725  2667 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:20.916827  3178 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:20.935717  2809 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.091357 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:21.096133 28292 thread.cc:641] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.034s	sys 0.037s
I20260709 12:21:20.954322  3138 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.085696 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:21.049736  2869 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:21.100459 27655 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:21:21.080005  2907 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.018s	sys 0.025s
I20260709 12:21:21.104339 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:21.115196 28292 thread.cc:608] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.034s	sys 0.037s
I20260709 12:21:20.922297  3201 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:21.129171 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43867.2us
I20260709 12:21:21.080348  3168 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:21.065147  3205 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:21.045696  2760 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:21:21.126271 27736 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:21.135093 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.080376  3142 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.138463 27664 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
W20260709 12:21:21.149581 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 676026us
I20260709 12:21:20.953610  3046 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.103694  3195 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:21.074743  3005 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
W20260709 12:21:21.128664  3207 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:21:21.167893 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77244.4us
I20260709 12:21:21.143201 27655 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:21.125437  3004 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.647s	user 0.013s	sys 0.001s
I20260709 12:21:21.126964  2869 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:21.123157 28331 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.501s	user 0.012s	sys 0.008s
I20260709 12:21:21.160758 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.175563 28331 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:21:21.171545 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2908) took 1777 ms. Trace:
I20260709 12:21:21.175853 27734 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:21.139143  3168 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:21.176378 27657 rpcz_store.cc:276] 0709 12:21:19.394311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.394495 (+   184us) service_pool.cc:224] Handling call
0709 12:21:19.394910 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.395248 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:19.692841 (+297593us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.043564 (+1350723us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.170386 (+126822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:21.170589 (+   203us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:21.170795 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.171083 (+   288us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.171439 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7598,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":10627,"thread_start_us":1392844,"threads_started":3,"wal-append.queue_time_us":1200573}
I20260709 12:21:21.180250 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.149260  3009 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:21.191385 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.198056 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.199065 27657 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:21.200726 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.201572 27736 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:21.209411 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.209718 27732 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:21.142632  2667 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.210987 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.216436 27737 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:21.212627 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.217664 27729 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:21.180078  3046 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:21.222740 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.229672  3193 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:21.233052  2988 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:21.214308  3228 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:21.233148 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3641) took 1763 ms. Trace:
I20260709 12:21:21.199621  3009 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.234335  3228 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.192361  3195 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:21.245200 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:21.149597  3205 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:21.288131 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107971us
I20260709 12:21:21.167196  3142 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.129091  3194 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.187790  3004 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.314749 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.251875  3009 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:21.236395 27582 rpcz_store.cc:276] 0709 12:21:19.470071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.930996 (+460925us) service_pool.cc:224] Handling call
0709 12:21:19.931290 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.948061 (+ 16771us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:20.648126 (+700065us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.216729 (+568603us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.228289 (+ 11560us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:21.232850 (+  4561us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:21.232907 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.232977 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.233092 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18305,"prepare.queue_time_us":904580,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":25580,"raft.queue_time_us":101797,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":10646,"thread_start_us":1189018,"threads_started":3}
I20260709 12:21:21.292986 27657 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
W20260709 12:21:21.318689 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 218477us
I20260709 12:21:21.321274 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:21.322013 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.322117 27735 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:21:21.325544 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138306us
I20260709 12:21:21.326472 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:21.329890 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 333244us
I20260709 12:21:21.295091  2907 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:21.292541  3105 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:21.341956 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 150347us
I20260709 12:21:21.346055 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3642) took 1863 ms. Trace:
I20260709 12:21:21.354807 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.360426 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.346751 27585 rpcz_store.cc:276] 0709 12:21:19.482465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.932682 (+450217us) service_pool.cc:224] Handling call
0709 12:21:19.932957 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.933155 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:21:20.674522 (+741367us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.143279 (+468757us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.234837 (+ 91558us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b1800067c60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:21:21.345558 (+110721us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:21.345640 (+    82us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:21.345709 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.345810 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.345976 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39736,"prepare.queue_time_us":871218,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":2896,"raft.queue_time_us":145028,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":12488,"spinlock_wait_cycles":79116544,"thread_start_us":1259182,"threads_started":3,"wal-append.queue_time_us":584709}
I20260709 12:21:21.337319  3193 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:21:21.376129 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45432us
W20260709 12:21:21.379017 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.378857 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.379768 27732 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
W20260709 12:21:21.380805  3217 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 12:21:21.381740  3232 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:21:21.382150  3218 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:21.382828  3218 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.383616 27628 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.384034  3228 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:21.384163  3168 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.385344  3218 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:21.386292 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:21.387562 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1127) took 1318 ms. Trace:
I20260709 12:21:21.376955 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:21.392036 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.392962 27657 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:21.399312 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:21.387996 28335 rpcz_store.cc:276] 0709 12:21:20.069068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.072910 (+  3842us) service_pool.cc:224] Handling call
0709 12:21:20.083258 (+ 10348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:20.104384 (+ 21126us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:20.104417 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:20.142444 (+ 38027us) log.cc:844] Serialized 26 byte log entry
0709 12:21:20.877621 (+735177us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:20.877682 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.877776 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:21.387300 (+509524us) raft_consensus.cc:1685] finished
0709 12:21:21.387351 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.387497 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13903,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":4495,"thread_start_us":645037,"threads_started":1,"wal-append.queue_time_us":917053}
I20260709 12:21:21.403228  3250 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.005s
I20260709 12:21:21.404282  3250 consensus_queue.cc:1243] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 26.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:21:21.398972  3243 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:21.367871  2878 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:21.370257  2874 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:21.380697  3142 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:21.433658 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.414336 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.420153 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:21.390040  3154 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:21.431540  3214 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.292s	user 0.001s	sys 0.001s
I20260709 12:21:21.455518 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.459972 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.460548 27732 raft_consensus.cc:2410] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:21:21.409983  3138 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:21.414659  3150 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:21.411917  3057 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:21.475513 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:21.434057 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:21.465154 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.466674 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.409055  3178 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:21.485968  2869 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.487150 27731 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:21.488618 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:21.487950 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:21.431748  3201 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:21.412467  3241 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:21.478940 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
W20260709 12:21:21.519795 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.519647 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.520550 27732 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:21.463254  3154 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:21.443552  3250 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:21.497663 27584 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:21:21.451043  3218 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:21:21.500476 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86221us
I20260709 12:21:21.629686 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:21.501873 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81813.5us
I20260709 12:21:21.502796 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.631227 27662 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:21:21.631609 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:21.513355 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:21.501920 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.518569 27737 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:21:21.523528 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 151771us
I20260709 12:21:21.559355 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:21.559834 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:21.561789 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.559675 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:21.634363 27585 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:21.635528 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:21.469174  3057 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.626855 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:21.641600 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 128343us
I20260709 12:21:21.627286 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.627611 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:21.643311 28331 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:21.643273 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 235763us
I20260709 12:21:21.503461  2667 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:21.633606 27580 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:21.645345 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.638566 28330 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:21.646586 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.564220  3138 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:21.654927 27732 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:21.655084 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:21.656046 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.656893 27732 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:21.657385 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:21.649449  2869 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:21.649389 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.650252  3150 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:21.650753  3052 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:21.651758 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:21.669180  3052 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.669656  3052 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:21.666962 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:21.628227  3241 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:21.627986 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.408519  3195 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.675657 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:21.661556 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:21.644222 27658 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:21.667966 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:21.668614  2874 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:21.682744 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:21.670918 27585 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:21.657766 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:21.640277 28292 connection.cc:256] RPC call timeout handler was delayed by -1.0755314680000083s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
W20260709 12:21:21.684466 28292 connection.cc:256] RPC call timeout handler was delayed by -1.0737314680000054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
I20260709 12:21:21.675527  3178 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:21.684584 27732 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:21.642633  3057 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:21.677044 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.691181 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:21.686450 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:21.684897 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.646s (SENT) [suppressed 20 similar messages]
I20260709 12:21:21.629062  3250 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.667533  3150 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:21.736183 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.735296 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:21.737428 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.673061  2667 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:21.738364 27732 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
W20260709 12:21:21.738722 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77234us
W20260709 12:21:21.738835 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70951.1us
I20260709 12:21:21.745684 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:21.746423 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.740207 27735 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:21:21.682165  3154 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:21.746608 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:21.734299  3120 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.072s	user 0.009s	sys 0.014s
W20260709 12:21:21.760214 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:21.763865 27734 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.570s	user 0.007s	sys 0.004s
I20260709 12:21:21.764366 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:21.765687 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:21.766700 27734 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:21.757159  3198 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.850s	user 0.013s	sys 0.000s
W20260709 12:21:21.768014 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85419.9us
I20260709 12:21:21.763278 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:21.769389 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:21.772804 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:21.683198  3138 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:21:21.737782  3150 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:21.701067  3205 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:21.739041  2869 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:21.745138  3153 raft_consensus.cc:2764] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:21:21.743647  2667 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.775509 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:21.775873 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.787302 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.776705 28333 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23.
I20260709 12:21:21.776969 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:21.780723 28329 raft_consensus.cc:1217] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Deduplicated request from leader. Original: 22.17->[24.18-27.20]   Dedup: 24.18->[25.19-27.20]
I20260709 12:21:21.789768 27658 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:21.786424 27729 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.546s	user 0.007s	sys 0.002s
I20260709 12:21:21.749748  2988 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:21.802560 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:21.773928  3178 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:21.789762 28329 pending_rounds.cc:85] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 18
I20260709 12:21:21.772118  3046 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:21.811568 28329 pending_rounds.cc:107] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:21:21.803682 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:21.808801 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 123898us
I20260709 12:21:21.809795 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:21.811925 28329 raft_consensus.cc:2889] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:21.812462 27661 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17.
I20260709 12:21:21.806031 27729 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
W20260709 12:21:21.812647 28292 leader_election.cc:336] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.112s (SENT)
I20260709 12:21:21.813239 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:21.785295  2869 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.790970  2667 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:21.771039  3120 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:21.833446 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45435.3us
I20260709 12:21:21.834335 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:21.810793  2988 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:21.876073 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.773612  3198 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:21.831143  2869 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:21.811442  3178 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:21.911164 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:21.914287 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101102us
I20260709 12:21:21.837168  3046 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:21.809060  3154 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:21.814864  3241 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.910682 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.905483 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.921520 27582 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:21.917361 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:21.928393 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116014us
W20260709 12:21:21.904704  3251 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
W20260709 12:21:21.904276  3071 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.241s	user 0.022s	sys 0.005s
W20260709 12:21:21.896304  3267 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:21:21.884570  3266 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:21:21.938251 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:21.941864 27656 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 12: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:21.887785  3242 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:21.949839  3242 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:21.950129 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 652725us
I20260709 12:21:21.950304  3242 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:21.950783 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.952576 28332 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:21.899426  3222 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.940415 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.924288  3178 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:21.883733  3153 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:21.947885  3196 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:21.948356 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 703217us
I20260709 12:21:21.949257  3204 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:21:21.950392  3256 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:21:21.940481 27655 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:21.915789  2988 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:21.956722 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.979755 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.951802 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:21.942945 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:21.961316 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.964399 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:21.968498 27658 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:21.970059 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3629) took 2748 ms. Trace:
I20260709 12:21:21.981520 27661 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
W20260709 12:21:21.971179  3272 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:21.973048 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:21.941123  3046 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:21.999461  3046 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.000905  3120 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:21.974545  3153 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.001775  3153 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:21.971179  3269 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:21:21.996930 28331 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:21.998407  3009 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.661s	user 0.007s	sys 0.004s
I20260709 12:21:21.935945  3234 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.246s
W20260709 12:21:22.003119 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60208.3us
I20260709 12:21:22.003463 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:21.969080  3138 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:22.017119  3250 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.020493 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:22.021752 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:22.022259 27738 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:22.022455  3243 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.578s	user 0.005s	sys 0.003s
I20260709 12:21:22.022643  3194 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:22.022895  3243 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:22.024499  3198 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.026602 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:22.003497 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86219.8us
I20260709 12:21:22.027516  3018 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:22.027925  3018 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.028337  3018 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:22.031705  3234 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:22.032119  3234 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:22.032685  3009 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.005101 27733 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:22.006480 27656 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:21.953097  3071 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:21.981273 27588 rpcz_store.cc:276] 0709 12:21:19.221276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.925696 (+704420us) service_pool.cc:224] Handling call
0709 12:21:19.926000 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.926269 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:20.626615 (+700346us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.336778 (+710163us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.969601 (+632823us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:21.969675 (+    74us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:21.969735 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.969825 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.969978 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":632535,"prepare.queue_time_us":804799,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":111,"raft.queue_time_us":731322,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":23894,"thread_start_us":1370295,"threads_started":3,"wal-append.queue_time_us":523135}
I20260709 12:21:21.971819  2850 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:22.010739 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.009039  3008 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:22.013778 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.044201 27586 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:21:22.014672  3221 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.754s	user 0.002s	sys 0.002s
I20260709 12:21:22.014539 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:22.047488 27661 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:22.027830 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:22.005741 27658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51410
W20260709 12:21:22.040994 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.049932 27663 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:21.967438  3222 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.060115 27738 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:22.065335 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:22.049280  3008 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:22.085636 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:22.053736  3204 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:22.089706  3204 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:22.094162 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 142458us
I20260709 12:21:22.094936 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:22.062132  3244 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:22.051488  3057 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.108367 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:22.080904  3138 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:21:22.112768 28292 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 16 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 16. Current term is 17.
I20260709 12:21:22.113536 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.123342  3265 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 12:21:22.124583  3265 consensus_queue.cc:1243] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:21:22.125274  3265 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:21:22.125732  3265 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:22.126562  3229 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:22.132917 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:22.141746 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.143074 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:22.151079  3278 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:21:22.153033  3234 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.154604  3234 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:22.144868  3257 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:22.208971  3231 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:22.212404  3222 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:22.219488  3138 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:22.220088  3138 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.220527  3138 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:22.233687  3279 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:21:22.235591  3071 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.135543  3196 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:22.236655  3196 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.237097  3196 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:22.240144  2869 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.115221  3244 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.244928  3150 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:22.251516  3052 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.581s	user 0.011s	sys 0.001s
I20260709 12:21:22.254459  3052 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.257280  3168 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:22.257664 27660 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:22.258034 27658 raft_consensus.cc:606] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Received request to transfer leadership
I20260709 12:21:22.258543 27658 raft_consensus.cc:614] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:22.115422  3008 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.256750  3057 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:22.268443  3225 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:22.269554  3244 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.291550 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.271034 28328 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:22.272754  3009 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:22.278209 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:22.334760 27734 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:22.280650 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:22.281764  3072 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:22.281942  3005 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:21:22.282007 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 172162us
I20260709 12:21:22.284977 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.285674 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.287110  3247 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.287925  2809 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.288412  3246 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:22.303099  3264 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.546s	user 0.000s	sys 0.001s
I20260709 12:21:22.291888  3225 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:22.322300 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:22.324481 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.344264 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:22.346490 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:22.347376 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 204365us
I20260709 12:21:22.362073 28332 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:22.247737  3150 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.354501 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.353098 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.371270 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3450) took 1131 ms. Trace:
I20260709 12:21:22.371471 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:22.372403 27663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3500) took 1498 ms (client timeout 1685 ms). Trace:
I20260709 12:21:22.372624 27628 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23.
W20260709 12:21:22.372812 27663 rpcz_store.cc:269] 0709 12:21:20.874253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.874817 (+   564us) service_pool.cc:224] Handling call
0709 12:21:22.372225 (+1497408us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:22.372348 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2615293824}
I20260709 12:21:22.374176 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.375358 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.375932 27588 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:22.377035 27588 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:21:22.341475  3046 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:22.379813 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45911.4us
I20260709 12:21:22.366219 27585 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:22.385666 27581 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:22.371918 27736 rpcz_store.cc:276] 0709 12:21:21.239974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.242368 (+  2394us) service_pool.cc:224] Handling call
0709 12:21:21.291908 (+ 49540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:21.386124 (+ 94216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:21.386156 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:21.386723 (+   567us) log.cc:844] Serialized 26 byte log entry
0709 12:21:21.494623 (+107900us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:21.494674 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.302215 (+807541us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:22.359834 (+ 57619us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:22.359877 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:22.370971 (+ 11094us) raft_consensus.cc:1685] finished
0709 12:21:22.370999 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:22.371193 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218690688,"thread_start_us":107512,"threads_started":1,"wal-append.queue_time_us":369840}
I20260709 12:21:22.380476 27589 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:22.381160 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.344951  3071 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.354772  3004 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:22.263870  2667 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.390058  3265 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:22.413095  3280 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:21:22.413383 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.414626 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.382426  3246 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.421382  3246 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.421481  3052 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.356742  3247 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.408321 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:22.392890 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:22.424057 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.424803 27587 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:22.425961 27582 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:22.404248 27586 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:22.341519  3072 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:22.413519  2667 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:22.363487  3243 raft_consensus.cc:2764] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election decision vote started in defunct term 22: won
I20260709 12:21:22.427014 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.378051  3231 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:22.206874  3242 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.415232 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:22.429410 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.432032 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:22.435225 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50692.5us
W20260709 12:21:22.436038 27535 auto_rebalancer.cc:893] Removing replace marker failed: Replica 9c09719959a54e8eb66d2041dd9c3113 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 19 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:21:22.442381 27535 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 12:21:22.437094 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.406314  2869 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.423591  3057 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:21:22.472522 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101159us
I20260709 12:21:22.476850 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:22.442773 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:21:22.478641 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:22.490482 28332 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:22.388549  3259 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:21:22.495085 27662 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24.
I20260709 12:21:22.431879  3250 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:22.493260 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:22.515534 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 373869us
I20260709 12:21:22.405439  3046 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:22.526736 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.423640  3071 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:22.535552 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46619.2us
W20260709 12:21:22.535874 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 108945us
I20260709 12:21:22.536301 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:22.540961 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.541440 27582 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:22.494338  3228 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:22.545248 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.556885 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.561163 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3455) took 1039 ms. Trace:
I20260709 12:21:22.511379  3245 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:21:22.536520  2988 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.536s	user 0.011s	sys 0.000s
I20260709 12:21:22.561841 27738 rpcz_store.cc:276] 0709 12:21:21.521852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.061077 (+539225us) service_pool.cc:224] Handling call
0709 12:21:22.061491 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.061801 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:22.100494 (+ 38693us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.344514 (+244020us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.560716 (+216202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:22.560789 (+    73us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:22.560847 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.560944 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:22.561093 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21344,"prepare.queue_time_us":53014,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":6047,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":4179,"spinlock_wait_cycles":7470336,"thread_start_us":465812,"threads_started":3,"wal-append.queue_time_us":400838}
I20260709 12:21:22.511883  3250 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:22.571458 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:22.544528  3018 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.004s
W20260709 12:21:22.575389  3153 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.000s
I20260709 12:21:22.574088  3057 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:21:22.597791 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61311.9us
I20260709 12:21:22.606097 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.607072 27581 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:22.607553 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.612704  3204 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:21:22.616230  3284 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:21:22.623349  3057 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.623947  3057 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:21:22.635998  3289 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:22.584215  3250 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:22.648253 28332 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:21:22.652032 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:22.653875  3204 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:22.655253  3242 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.585278  2667 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 12:21:22.660908 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.662294 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:22.664201 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.665575 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.665748 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:22.666229 27589 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:21:22.669306  3247 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.675223  3046 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.678663 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.682011  2869 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:22.683985  2809 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:22.684439  2809 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:22.684954  2809 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:22.689447  3252 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.939s	user 0.000s	sys 0.003s
I20260709 12:21:22.694288 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:22.695573 27587 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:22.698436 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:22.698894 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91466.4us
I20260709 12:21:22.700766 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.706060 27581 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:22.599788  3219 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:22.713593 28332 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:22.634501  3194 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.611s	user 0.003s	sys 0.006s
I20260709 12:21:22.583423  2988 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.705153 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:22.583160 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:22.724005 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:22.706517  3245 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:22.729549  2988 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.715939  3229 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:22.733271 28335 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:22.732997  3057 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:22.736783  3178 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.767s	user 0.006s	sys 0.006s
I20260709 12:21:22.737051  3071 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.737233  3178 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.566506  3072 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.740631  3072 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:22.743626  3243 consensus_queue.cc:1175] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" responder_term: 22 status { last_received { term: 22 index: 17 } last_committed_idx: 17 last_received_current_leader { term: 22 index: 17 } }
I20260709 12:21:22.712915  2850 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:21:22.592901  3018 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.750612  3057 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:22.750931 28330 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
W20260709 12:21:22.751936  3138 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.531s	user 0.008s	sys 0.004s
I20260709 12:21:22.610427  3153 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.752317  3219 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:22.753283  3219 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.753886  3153 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:22.754596  3138 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.756142 27731 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:22.757809  3138 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:22.751987  3018 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:22.764554 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59494us
W20260709 12:21:22.716250  3241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.032s	sys 0.075s
I20260709 12:21:22.770335  2850 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:22.773257  3241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.033s	sys 0.075s
W20260709 12:21:22.803804 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 220720us
I20260709 12:21:22.819355 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:22.812598 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1301) took 1151 ms. Trace:
W20260709 12:21:22.916620 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 97358.4us
I20260709 12:21:22.917616 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:22.918677 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 100557us
I20260709 12:21:22.769192  3178 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:22.923360 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.915189 27730 rpcz_store.cc:276] 0709 12:21:21.660703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.072106 (+411403us) service_pool.cc:224] Handling call
0709 12:21:22.072418 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.072634 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:22.384271 (+311637us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.812115 (+427844us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:22.812191 (+    76us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:22.812254 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.812347 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:22.812520 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19310,"prepare.queue_time_us":49927,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":103986,"thread_start_us":686157,"threads_started":2}
I20260709 12:21:22.925961 27589 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:22.921905 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:22.929524 28332 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:22.939172 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.807799  3253 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:22.962672 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:22.962824 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 156769us
I20260709 12:21:22.965071 28332 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:21:22.964108 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:22.915838  2988 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:22.915648  3194 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.966383 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:22.915632  3153 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:22.926261  3231 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:22.775126  3204 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.182s
I20260709 12:21:22.932300  3138 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.932580  3057 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.925710  3259 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
W20260709 12:21:22.980423 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.976404  2869 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:22.932107  3130 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:21:22.969483  3244 raft_consensus.cc:2764] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:21:22.971220  3299 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:22.977026  3231 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:22.995456 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 329742us
I20260709 12:21:22.967489  3247 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:21:23.033089 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:23.035393  2667 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 12:21:23.036791  3222 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:23.036975  2667 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:23.037245  3222 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:23.037768  3222 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:23.042685  3231 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.044090 28332 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
W20260709 12:21:22.990783 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.044231 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:22.972368  2988 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.048112  3071 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.049068  2988 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.048218 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:23.049409  3071 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.050570  2869 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:21:23.050789  3293 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:23.051054  2869 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:23.052176  3293 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:23.051960  2869 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.052577  3293 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:23.052964  3293 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:23.053056  2869 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:23.053788  3071 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.055090 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:23.055933 28332 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 12.
I20260709 12:21:22.993916 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1350) took 1185 ms. Trace:
I20260709 12:21:23.057511 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:23.057672 27662 rpcz_store.cc:276] 0709 12:21:21.808457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.808601 (+   144us) service_pool.cc:224] Handling call
0709 12:21:22.993625 (+1185024us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:22.993830 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1514753280}
I20260709 12:21:22.998847  3245 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:22.972491  3194 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:22.974478  3153 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:22.977329  3138 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:23.066238 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63123.4us
W20260709 12:21:23.061302  3196 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.797s	user 0.015s	sys 0.000s
I20260709 12:21:22.979221  3057 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.082168  3071 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:21:23.050514  3222 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.117530  3222 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.120484  3130 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:23.128396 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1355) took 1118 ms. Trace:
W20260709 12:21:23.128841  3301 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
W20260709 12:21:23.130332  3304 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:21:23.145238  3138 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.157366  3153 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.164306 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1258) took 1404 ms. Trace:
I20260709 12:21:23.172083 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:23.128412  3298 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:21:23.085287  3196 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:23.164707  3195 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.048s	sys 0.041s
I20260709 12:21:23.128892 27659 rpcz_store.cc:276] 0709 12:21:22.010306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.013409 (+  3103us) service_pool.cc:224] Handling call
0709 12:21:22.013763 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:22.013986 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:22.014015 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:22.368190 (+354175us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:21:22.368602 (+   412us) log.cc:844] Serialized 26 byte log entry
0709 12:21:22.571306 (+202704us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:23.094926 (+523620us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.095698 (+   772us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.095799 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:23.095857 (+    58us) raft_consensus.cc:1685] finished
0709 12:21:23.128162 (+ 32305us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.128328 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":96308,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":320,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":13990,"spinlock_wait_cycles":47981696,"thread_start_us":690937,"threads_started":2,"wal-append.queue_time_us":361675}
I20260709 12:21:23.173188 28329 rpcz_store.cc:276] 0709 12:21:21.760372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.760512 (+   140us) service_pool.cc:224] Handling call
0709 12:21:21.780484 (+ 19972us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:21.812791 (+ 32307us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:22.328407 (+515616us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b180041e4e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:22.535834 (+207427us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.003737 (+467903us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:23.006103 (+  2366us) log.cc:844] Serialized 52 byte log entry
0709 12:21:23.046188 (+ 40085us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:23.046634 (+   446us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.060614 (+ 13980us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.060852 (+   238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.061008 (+   156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:23.164003 (+102995us) raft_consensus.cc:1685] finished
0709 12:21:23.164034 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.164223 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32255,"prepare.queue_time_us":212705,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":3893,"raft.queue_time_us":16253,"raft.run_cpu_time_us":501,"raft.run_wall_time_us":45058,"spinlock_wait_cycles":32028800,"thread_start_us":610552,"threads_started":2,"wal-append.queue_time_us":543900}
I20260709 12:21:23.164398  3071 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.207782  3071 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:21:23.204079  3195 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.048s	sys 0.042s
W20260709 12:21:23.228899 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 311360us
W20260709 12:21:23.223340  2809 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.000s
W20260709 12:21:23.224758 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.014s	sys 0.010s
W20260709 12:21:23.234336 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.014s	sys 0.010s
W20260709 12:21:23.236220 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 584661us
I20260709 12:21:23.242641 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.244459 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:23.246779  2809 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.249540 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:23.271806 27738 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:23.251958 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.273339 27663 raft_consensus.cc:2376] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23. Re-sending same reply.
I20260709 12:21:23.252704 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.255781 27662 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:21:23.261441  3072 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:23.287902 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45373.3us
W20260709 12:21:23.265457 28292 connection.cc:256] RPC call timeout handler was delayed by -1.108649816000252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
W20260709 12:21:23.289409 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 205620us
I20260709 12:21:23.289525 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:23.266574 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.264024 27733 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:23.291337 27730 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 21: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:23.294821 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.302795 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.322310 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:23.287266  3194 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.295727  2869 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:23.310629  3299 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:23.351050 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.353058 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:23.356324 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:23.358616 27732 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:23.359189 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.359993 27661 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:23.360976 27736 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:23.361095 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:23.361518 27656 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:23.333690  3138 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:23.339370 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.365195 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.319028  3263 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:23.330641  3236 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:23.310043  3259 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:21:23.364286 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.364588 27732 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:23.295043  3072 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.366112 27661 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:23.365654 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:23.369624 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:23.366885 27552 connection.cc:256] RPC call timeout handler was delayed by -2.7841752820002057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
W20260709 12:21:23.372498 27552 connection.cc:256] RPC call timeout handler was delayed by -2.7154700950000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1734s for a response
I20260709 12:21:23.367347 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:23.373689 27735 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:23.374495 27733 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:23.374608 27660 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:23.366914 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.375828 27656 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:23.376049 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:23.378602  3246 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:23.339010  2869 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:23.385594  2869 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:23.379173  3246 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:23.337132 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 288944us
I20260709 12:21:23.397265 27730 raft_consensus.cc:2368] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:23.346798  3299 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:23.401162 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.336424  3245 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:23.363111  3250 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.646s	user 0.006s	sys 0.005s
I20260709 12:21:23.364665  3057 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.413594 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.414214 27658 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:23.397591 27552 connection.cc:256] RPC call timeout handler was delayed by -2.5080693090001205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18830000000000002s for a response
I20260709 12:21:23.416780 27737 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:23.367525  3236 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:23.417788  3250 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.417788  3306 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:21:23.425459 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 358136us
W20260709 12:21:23.422595 27627 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.045s	user 0.029s	sys 0.074s
I20260709 12:21:23.392968  3246 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.395915  3318 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:23.438700 27627 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.061s	user 0.036s	sys 0.076s
I20260709 12:21:23.439196 27738 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 12:21:23.449306 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.1174e+06us
W20260709 12:21:23.427165 27552 connection.cc:256] RPC call timeout handler was delayed by -2.35556396700008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
W20260709 12:21:23.450275 27552 connection.cc:256] RPC call timeout handler was delayed by -2.335036213000194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20260709 12:21:23.450711 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3338700950000657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2158s for a response
W20260709 12:21:23.451138 27552 connection.cc:256] RPC call timeout handler was delayed by -1.9833760730002723s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
W20260709 12:21:23.451560 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5427859280000575s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
W20260709 12:21:23.451960 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3249859280001601s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20260709 12:21:23.428368 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:23.442224 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.411778  3299 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:23.437988  3138 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:23.465167 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.469921 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:23.471520  3057 raft_consensus.cc:2764] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260709 12:21:23.478348  3308 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.274s	user 0.000s	sys 0.002s
I20260709 12:21:23.481649  2850 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:21:23.485199 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.427604  3236 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:23.508802 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.501350 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.504246 27738 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:21:23.512671  3194 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.513296  3194 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.516047  3194 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.399642  3225 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.460628  3005 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:23.543507 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73745us
I20260709 12:21:23.539383 27738 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:21:23.522001 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:23.550963 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 187458us
W20260709 12:21:23.552116 27701 connection.cc:256] RPC call timeout handler was delayed by -1.5333657789999506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
I20260709 12:21:23.554512 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.519233  3228 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:23.518347  3154 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.442931  3318 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.507141  2850 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:23.536526  3222 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:23.538964  3246 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:23.537073  3229 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:23.467108  3293 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:23.568325 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 255242us
I20260709 12:21:23.559267  3057 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.560665  2809 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.581532 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.562690  2988 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.582933 27581 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:23.583609 27588 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:23.585080 27588 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:23.585803 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.568239  3138 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:23.571041  3322 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.567888  3178 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:23.568827  3225 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:23.570243  3150 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:23.550894  3005 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.583916  3195 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:23.625243 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47366.5us
I20260709 12:21:23.602818 27588 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:23.636890 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51755.9us
W20260709 12:21:23.641454 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.568619  3154 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.648411 27588 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:23.649430  3072 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.601166  3229 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:21:23.684666  3228 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:23.685856 28292 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24.
I20260709 12:21:23.686318  2809 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.686550  3231 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:23.686932  2809 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.690435  2809 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.684803  3306 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.585088  3057 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.632473  3138 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:23.685469  3228 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.644410  3328 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:23.589483  2988 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.711580 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.712560 27589 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
W20260709 12:21:23.697129  3314 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260709 12:21:23.695250  3247 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:23.632477  3178 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.636241  3150 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.582942  3245 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.701229  3250 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.732434  3250 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.732973  3250 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.600865  3322 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:23.728060  3323 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
W20260709 12:21:23.728843  3324 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:21:23.730077 27734 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:23.720072  3247 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:23.646062  3195 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.725176 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:23.774554 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49474.2us
I20260709 12:21:23.632578  2850 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.710788  3194 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:23.749467 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.777631 27589 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:23.701134  3306 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.735311  3250 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.796552  3057 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.797837  3228 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.801623 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
W20260709 12:21:23.801954  3071 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.594s	user 0.005s	sys 0.005s
I20260709 12:21:23.731184  3150 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:23.768862 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 192315us
I20260709 12:21:23.770371 27588 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:21:23.839562 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.840230 27588 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:23.720850  3072 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:23.772536 27734 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:23.803232 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:23.841563 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.804838 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:23.806509  3195 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.851023 27589 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:23.807554  3247 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.748286  3336 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.804515 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.856752 27581 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:23.750231 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61037.6us
I20260709 12:21:23.811249  3250 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:23.804638  3322 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.821062  3306 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.823418  2988 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.824380  3154 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.829098  3178 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.772810  3138 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.830576  3005 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.809691  3194 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.807686  3293 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.853346 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.876531 27586 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:23.862243 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.892475 27588 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:23.892709 27581 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:23.818878  2869 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.863927 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3695) took 1573 ms. Trace:
I20260709 12:21:23.848168  3150 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:23.858541  3247 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:23.887833 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46344.3us
I20260709 12:21:23.891606 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.888640 27589 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:23.894131 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.856034  3072 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.840952  3071 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.911957 27587 rpcz_store.cc:276] 0709 12:21:22.291055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.291228 (+   173us) service_pool.cc:224] Handling call
0709 12:21:23.863775 (+1572547us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400087ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:23.863875 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292883840}
I20260709 12:21:23.843156  3259 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:21:23.917976 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:23.918673 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.918834 27587 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:23.858870  3250 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.920223 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1395) took 1635 ms. Trace:
I20260709 12:21:23.862475 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.921317 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.920837 27582 rpcz_store.cc:276] 0709 12:21:22.284935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.285430 (+   495us) service_pool.cc:224] Handling call
0709 12:21:23.919991 (+1634561us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:23.920148 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183869824}
I20260709 12:21:23.923630 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.924703 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.925683 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.926200 27582 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:23.927500 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.895262  3322 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:21:23.922067 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.864470  3244 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:23.916090  3326 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:23.937835 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1383) took 2285 ms. Trace:
I20260709 12:21:23.938354  3326 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:23.928369 27664 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:21:23.928602 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.929104 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.929947 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.933426 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:23.933717 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.935105 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.942382 27663 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:23.943150 27655 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 12. Current term is 13. Ops: []
I20260709 12:21:23.943461 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.941898 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.944235 27658 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:23.922340  3150 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:23.936059 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.945007 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.943392 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.872327  3225 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:21:23.860795  3336 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.945513 27662 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:23.922945  3250 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:23.951242 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.953819 27663 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:23.957752 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.874950  3339 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.084s
I20260709 12:21:23.958146 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:23.957818 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.961048 27659 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:23.960457 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.961217 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.938249 27584 rpcz_store.cc:276] 0709 12:21:21.652375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.671008 (+ 18633us) service_pool.cc:224] Handling call
0709 12:21:21.671398 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:22.035579 (+364181us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:22.051287 (+ 15708us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.280277 (+228990us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b1800305d40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9e21c58f 00007f0e9725b571 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd
0709 12:21:22.364214 (+ 83937us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.364419 (+   205us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:22.410813 (+ 46394us) log.cc:844] Serialized 26 byte log entry
0709 12:21:22.428610 (+ 17797us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:22.428830 (+   220us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.434691 (+  5861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.441609 (+  6918us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:22.679927 (+238318us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b1800305d40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:22.984689 (+304762us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:21:23.923198 (+938509us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b1800305d40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:23.937607 (+ 14409us) raft_consensus.cc:1685] finished
0709 12:21:23.937626 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.937756 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":431026,"prepare.queue_time_us":267881,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":17032,"raft.queue_time_us":433229,"raft.run_cpu_time_us":8616,"raft.run_wall_time_us":581792,"spinlock_wait_cycles":51627904,"thread_start_us":259624,"threads_started":1,"wal-append.queue_time_us":360896}
I20260709 12:21:23.961805 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:23.940670  3322 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:23.932488  3318 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.984025  3322 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.987418 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:23.988091 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:23.988224  3299 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:23.989336 27663 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:23.965278 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:23.965160 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:23.990710  3243 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:23.990949  3299 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.991255  3243 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:23.992264  3243 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:23.993381  3328 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:23.960541  3259 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:23.997762  3265 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:23.998876 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.968133 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:23.999913 27663 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:23.968931 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:23.970356  3194 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.951267  3153 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:21:23.971787  2869 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.971199  3241 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:21:23.977588  3341 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:21:23.980068  3336 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:23.980517  3071 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.982246  3072 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:23.984544  3196 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:23.965925 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:23.966570 27664 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:24.002245 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.012320  3318 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:24.032305 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64212.1us
I20260709 12:21:23.953178  3244 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:24.071269 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72476us
I20260709 12:21:24.076526  3194 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:24.074366 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.090430 27662 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:24.002127 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85398.2us
I20260709 12:21:24.070883  3241 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:24.106817 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3646) took 1681 ms (client timeout 1540 ms). Trace:
W20260709 12:21:24.107398 27586 rpcz_store.cc:269] 0709 12:21:22.425916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.429180 (+  3264us) service_pool.cc:224] Handling call
0709 12:21:24.106659 (+1677479us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:24.106747 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3208213888}
I20260709 12:21:24.108675 27586 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:24.109177 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.110285 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.033810 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.073935  3153 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:24.035912  3259 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:24.089885 27582 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
W20260709 12:21:24.116082 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.116571 27736 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:21:24.116945 27736 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 15
I20260709 12:21:24.117174 27736 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:21:24.117563 27736 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:24.093755  3241 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:24.118080 27664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:21:24.118693 27664 raft_consensus.cc:606] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:24.110744 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1262) took 1412 ms. Trace:
I20260709 12:21:24.119251 28330 rpcz_store.cc:276] 0709 12:21:22.698339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.698470 (+   131us) service_pool.cc:224] Handling call
0709 12:21:22.718827 (+ 20357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.751422 (+ 32595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:24.110532 (+1359110us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b54006bc6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:24.110691 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73397632}
I20260709 12:21:24.086701 27658 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
W20260709 12:21:24.112658 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 114709us
I20260709 12:21:24.115633 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.119581 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.121654 27729 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.122565 27582 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:21:24.112285 27586 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:21:24.122841 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.090821  3150 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.150369  3228 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:24.152797 27737 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:21:24.151081 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.173522 27732 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:21:24.173925 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.153180 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.174778 27737 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:24.169872  3347 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.175715 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:24.175552  3347 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.176265 27588 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:21:24.175874  3228 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.157817  3253 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:24.178552 28332 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:24.178887 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.179483 27732 raft_consensus.cc:2376] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26. Re-sending same reply.
I20260709 12:21:24.133518 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.180401 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.125075 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.090543  3318 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.181015 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:24.156297 28329 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.184773 27662 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
W20260709 12:21:24.187963 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81779.7us
I20260709 12:21:24.189966  3228 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.192911 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.191299 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.197799 27732 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:24.187588 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.198922 27736 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:24.125114 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.201628  3247 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:24.202167 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.203011  3247 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:24.203572 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.202597 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.204360  3247 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:24.204800 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.205617  3345 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:24.205819 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.206339  3345 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:24.207062 27732 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:21:24.207496  3245 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:24.207965 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.208578 27732 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:24.204377  3318 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.209352 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.210949 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.111780  3130 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:21:24.121812  3299 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:24.211822  3130 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:24.211956 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.212203  3130 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:24.212978  3130 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.214202  3130 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:24.217278  3250 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.218246  3244 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.212981 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:24.160431  3246 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.585s	user 0.005s	sys 0.005s
I20260709 12:21:24.225459 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.117928  3243 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.227195  3347 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.227025  3243 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:24.191045  3236 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.001s
I20260709 12:21:24.228638  3236 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.231011  3236 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.193451  3253 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:24.224038 27662 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:21:24.122594  2869 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.117656  3071 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.231598 27730 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:24.172022  3153 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:24.231793 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3467) took 1954 ms (client timeout 1595 ms). Trace:
I20260709 12:21:24.232965 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.247825 27585 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:21:24.247524 27731 rpcz_store.cc:269] 0709 12:21:22.277616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.280304 (+  2688us) service_pool.cc:224] Handling call
0709 12:21:24.231514 (+1951210us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:24.231670 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":967312128}
I20260709 12:21:24.233662 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.111379  3225 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.113s
I20260709 12:21:24.234542 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.252830 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.235476  2850 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:24.238546 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.256986 27656 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:24.257068 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.258126 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.239869 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.120330  3315 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:24.241705 27729 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:24.223501 27734 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:24.262346 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.263221 27659 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:24.265251 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.267740 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.243499 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:24.125213  3293 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:24.269407 27657 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:24.249127 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:24.269613  3293 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.270777 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.233875 27733 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:21:24.250844 27664 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:24.241122 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.273208 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:24.274245 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.274967 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:24.275970 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:24.259378 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.246126  3247 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:24.274843 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:24.272476 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:24.278432 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.284957 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.283814 27732 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:24.286518 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.279673 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.279522 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.288350 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.290195 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.291126 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.294041 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:24.295680 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.305558 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.261535  3306 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:24.307757 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.303220 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.250670  3244 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:21:24.277510  3236 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:24.261104  3315 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:24.281674  3072 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:24.252447  2869 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.252573  3071 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.331830  3072 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.318410 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.282251  3228 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:24.253207  3231 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:21:24.279996  3336 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:24.316166  3225 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:21:24.334110 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.281908  3347 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:21:24.317602  3333 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:24.349920 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.338122  3072 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:24.299536  3339 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:24.282439  3250 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.279906  3344 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:24.304350  3318 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:24.304144  2988 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:24.303860  3150 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:24.370110  3150 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.370574  3150 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.363498  3344 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:24.371783  3344 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.336621  3236 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:24.374428  3236 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.374883  3236 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.375653  3243 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.316461  3057 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:24.376361  3057 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.354190  3253 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.376796  3057 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.316790  3342 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:24.378314  3342 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:24.379356  3342 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:24.380017  3354 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:21:24.284509  3138 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:24.382413  3138 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.382895  3138 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:24.351428 27732 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:21:24.391960 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.393045 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.393950 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.394069  3228 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:24.394578  3228 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.394939  3228 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.400415  3333 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:24.401055  3333 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.402158  3333 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.408149 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.409997 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:24.416559 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.331043  3322 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:21:24.330022  3245 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:21:24.340458  3247 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:24.365450 27552 leader_election.cc:400] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:24.359179  3293 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:24.340497  3225 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:24.369593  2988 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.417534 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:24.418270 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.333990 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:21:24.420327 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.351540  3336 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:24.448019  3336 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.426165 27734 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:21:24.358299  3229 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:24.453186 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.328974  3306 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.359117  3339 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.454025  3339 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:24.454296 27729 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:24.427774 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:21:24.430940 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.427995 27552 leader_election.cc:403] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:21:24.455646 27734 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:24.456620 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.457489 27734 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:24.460305 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:24.461501 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:24.468961  3339 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:24.478214 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 175060us
W20260709 12:21:24.475353 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57935us
W20260709 12:21:24.428874  3356 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:21:24.490226 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 127166us
W20260709 12:21:24.491549 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.492637 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.495456 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:24.496263 27586 raft_consensus.cc:2376] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21. Re-sending same reply.
W20260709 12:21:24.427937  3355 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:21:24.499948 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.425151  3318 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:24.502213 27586 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:24.507347 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.445849  3225 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:24.475757  2988 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:24.514372  3204 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:24.476845  3009 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.518095  3241 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.522425 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:24.526274 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:24.503245  3354 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:21:24.530992 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.540635 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:24.501029  3247 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:24.535733 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:24.536701 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56743.3us
I20260709 12:21:24.542901 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:24.505288  3245 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.535476  3229 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:24.557004  3320 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:24.558048  3229 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:24.558406  3229 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:24.559268  3229 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.560738  3229 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.563969 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.530478  3318 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.531849  3196 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:24.532298  3293 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:24.410797  3315 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.595546  3374 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:24.596140  3374 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.583865 27736 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:24.587481  3371 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:24.591989  3372 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:24.577610  2869 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:24.563149  3228 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.598038  3320 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.543305  3009 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.597079  3374 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.546445  3204 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.576546  3245 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.589959  3345 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:21:24.614784  3071 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:24.630630 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73890.9us
I20260709 12:21:24.625898  3320 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.641947 27738 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:24.617849  3371 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:24.641287 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 98462.7us
I20260709 12:21:24.620175  3293 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:24.589661  3365 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:21:24.678122 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.677047  3057 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.685479  3057 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:24.685619  3320 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.675436  3321 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:21:24.686496  3321 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:24.687754  3321 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.689167  3321 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:21:24.679252  3293 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.706799  3204 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.707794  3245 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.707988  3371 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:24.714541  3386 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:21:24.718945  3150 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.719795  3228 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:24.720458  3228 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:24.720837  3228 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.721169  3228 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:24.721664  3150 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.722579  3365 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:24.723003  3365 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.725292  3178 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.725194  3072 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.725912  3178 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.726409  3365 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.727598  3339 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.727909  3178 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.728055  3365 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:24.690598  3318 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.731675  3150 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:21:24.733119 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 314919us
I20260709 12:21:24.733342  3384 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:24.733793 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.734541  3138 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.734776  3333 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:24.735492  3333 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.735596  3243 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:24.736091  3243 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:24.736477  3243 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:24.737561 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62141.6us
I20260709 12:21:24.740249 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.738409 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.759441  2988 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.761709  2988 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.718950  3225 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.736572  3333 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.763270  3057 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:24.764055  3333 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:24.764225  3225 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.764997 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.768024  3293 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.769332  3293 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.771584  3332 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:24.771938  3342 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:24.753182  3367 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:21:24.772408  3342 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.774019 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:24.774472 27732 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:21:24.774611 27587 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.776741  3138 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:24.778491 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:24.788677 27588 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:21:24.791473 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:24.805706 27589 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.810313 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.811029 28332 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
W20260709 12:21:24.813299 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:24.814236 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3700) took 1260 ms (client timeout 1556 ms). Trace:
W20260709 12:21:24.814642 27585 rpcz_store.cc:269] 0709 12:21:23.554001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.554181 (+   180us) service_pool.cc:224] Handling call
0709 12:21:24.814080 (+1259899us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b540011bf54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:24.814184 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1591828864}
I20260709 12:21:24.816529 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.817426 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.780520  3201 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.818361 28331 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:24.784065  3072 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.821755 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.822131 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:24.825628 28333 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:24.826457 28331 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:24.819243 27587 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:24.834823 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.753008  3250 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:21:24.837844 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.845793 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.846629 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.846980 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.847611  3376 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:21:24.848026 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.824206 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.847643 28333 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:24.850267 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.856674 27589 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:24.838631  3365 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:24.859375 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:24.861207 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.861903 28334 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:24.859560 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.859143 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.864199 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.864734  3138 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:24.865700 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.866358 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:24.840732  2667 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:24.870000 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:24.870517  3057 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:24.871186  3380 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:24.873015 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.873729 27657 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:24.875206 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.875531 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.877183 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.875914 28333 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:24.881319  3245 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:21:24.838196  3201 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.882176  3245 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 23.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:21:24.881904  3243 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.882679  3245 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:24.883754  3245 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:24.884114  3245 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:24.884464  3332 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:24.885182  3332 consensus_queue.cc:1243] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 23.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:21:24.884943  3245 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.885641  3332 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:24.885972  3332 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:24.886471  2850 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:24.868144  3371 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:24.886870  2850 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:24.887705 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.887832  2850 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:24.867354  3004 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.022s
I20260709 12:21:24.889334  2850 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.892676 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:24.892670 28332 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:24.894933 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:24.895818 28337 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:24.902006  2667 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.903100  2667 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.907979  3342 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.858593  3318 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:24.876168 27657 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:24.913229 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.913595 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:24.912325  3318 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.919883  3318 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:24.862385  3365 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.920622 27738 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:21:24.920907  3365 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:24.919902  3306 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:24.925376  3344 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.553s	user 0.000s	sys 0.010s
I20260709 12:21:24.925784  3344 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.926772 27584 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
W20260709 12:21:24.881805  3236 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.000s
I20260709 12:21:24.932982 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:24.933142 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.933817 27659 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.941177 27655 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260709 12:21:24.941545 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64425.7us
I20260709 12:21:24.942273 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.943051 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.945230 27659 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
W20260709 12:21:24.895850  3383 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:21:24.946435 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:24.947790 27628 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:24.957037 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.962958 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.966632 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.966749 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:24.967507 27664 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:24.971410 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:24.973129 27656 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:21:24.886179  3245 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:24.911701  3138 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:24.977205 27660 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:24.980803 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:24.949373  3072 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:24.981588  3345 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:24.961748  3376 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.968017  3153 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:24.962133  3225 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:24.976007  3071 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:24.974583  3320 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:24.966552  3317 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:24.977660  3194 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:24.948426  3236 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:24.946420  3229 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:21:25.003041 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.006690 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.006552  3228 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.007565 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.008695 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.009027  3228 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:25.009357 27656 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:25.011533 27664 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:25.011911 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.012730 27664 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:25.014717 27664 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:25.017988 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.011013 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.019171  3243 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.993068  3071 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.020074  3071 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:25.019490 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.019930 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.023624  3138 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.024381 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:25.021492 27585 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:21:25.025043 27664 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:21:25.027408  3344 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:25.028103 27664 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:21:24.993659  3306 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:25.028807  3318 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.024700 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.029533  3306 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.031198 27656 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:21:25.031700  3374 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:25.032188 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.032850  3306 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:24.998919  3194 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.012779 27588 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:25.033787  3194 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:25.033365 27656 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:25.034022 27664 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:25.039027 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:25.039427  3388 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:21:25.040652 27664 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:21:25.041723 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.044275 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:25.045223  3326 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:21:25.045701 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:25.048640 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:25.048750  3246 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:25.048825  3333 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:25.049263  3246 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.050288  3342 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:25.050550 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3706) took 1493 ms (client timeout 1927 ms). Trace:
I20260709 12:21:25.050902  3342 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.051312  3342 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:25.051126 27580 rpcz_store.cc:269] 0709 12:21:23.556919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.862232 (+305313us) service_pool.cc:224] Handling call
0709 12:21:25.050337 (+1188105us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54002bacd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:25.050493 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779851008}
I20260709 12:21:25.049641  3333 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:25.055446  3333 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.059520  3263 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:25.061503  3263 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:25.061833 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.063598  3344 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:25.064348  3344 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:25.064424 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.066198  3243 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.068063 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:25.068385  3342 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.069530  3234 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:25.069968  3234 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:25.070384  3234 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:25.077605  3342 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:25.063282 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.079526  3247 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:25.080530  3345 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:25.081415  3247 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:25.081777 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.081813  3247 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.082736  3345 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:25.083091  3247 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.084575  3247 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:25.084733  3320 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:25.002113  3317 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.087072  3320 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.088297 27655 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:25.089268  3320 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:24.991899  3376 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.089975 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 15.
I20260709 12:21:25.087090 27661 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:25.085772 27659 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:21:25.092738 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.080355  3130 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:25.095877 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.091383 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.117398 27734 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:25.096872 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:25.097028 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:25.120959 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.050287  3246 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.082844 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.127381 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.128232 27729 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:25.112592 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.133795 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.136823 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.135928 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.139855 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.106648 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:24.994342  3225 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.143170 27738 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:25.110219 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.144436 27729 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:21:25.144666 27735 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:25.144809 27729 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 11
I20260709 12:21:25.145123 27729 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:21:25.145432 27729 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:25.116039 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.146211 27732 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:25.124092 27736 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:21:25.140941 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.150964 27737 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:25.117563  3317 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.138348  3320 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:25.138096  3306 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.140611  3384 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.137930  3120 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:25.165974 27730 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:25.173458 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:25.103367 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.122354  3204 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:21:25.177992 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.150780  3228 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:25.196411 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.138589  3321 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:21:25.198908 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.208621 27736 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:25.159590  3243 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:25.200165 27738 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:25.161901  3246 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:25.163271  3057 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.201954 27580 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:25.143009  3130 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:21:25.213752 27737 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:25.199545  2988 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:25.203672 27732 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:25.164544  3318 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.164902  3320 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.199739  3229 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.195961  3228 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:25.179973 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52070us
I20260709 12:21:25.210817 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.216327 27664 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:21:25.216439 27738 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:25.226338  3201 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:25.228087  3320 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.229067  3228 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:25.229631  3320 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:25.234594  3345 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:25.235394  3229 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.214207  3057 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.237774  3229 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:25.239789  3399 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:21:25.241626  2988 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:25.243654  3376 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:25.245116  3057 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.247987 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.208835  3120 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:21:25.254432 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.251235 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.251863 27583 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:25.252422 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.222111 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.270098 28332 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:21:25.274241 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.277231 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.278128 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.278563 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.278798 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.279379 27730 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:21:25.279536 27664 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:21:25.280728 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.281560 27732 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 15.
I20260709 12:21:25.283095 27736 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:21:25.288750 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.292676  3243 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:25.293989  3243 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:25.294340  3243 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:25.295614 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.296444 27729 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:25.297169  3150 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.014s
I20260709 12:21:25.298496  3150 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:25.298892  3150 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:25.269515  3130 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:21:25.299719  3150 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.270692  3178 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:25.301146  3150 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:25.311041 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:25.254665  3321 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:25.318089 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.318970 28337 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:21:25.312326 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.319285 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.316042 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:25.320206 28332 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:25.314462 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:25.321018 27586 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:21:25.323163 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.326114 27736 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:21:25.330581 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.331779 27586 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:25.333947 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.334631 27586 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:25.336413  3225 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.338132 27589 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:25.339838  3194 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.342221  3194 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.342840  3194 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:25.343760 27662 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:21:25.343706 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.344717 27586 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:21:25.345119  3204 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:21:25.345602  3398 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:21:25.312166  2988 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:21:25.329600  3236 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:25.345870 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.317752  3321 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.348112 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.327793  3344 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:25.349850 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3065) took 1980 ms. Trace:
I20260709 12:21:25.353497 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.389779 27733 rpcz_store.cc:276] 0709 12:21:23.369177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.807281 (+438104us) service_pool.cc:224] Handling call
0709 12:21:24.233614 (+426333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:24.271857 (+ 38243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.349642 (+1077785us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b540029f750. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:25.349792 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":979528448}
I20260709 12:21:25.343223  3194 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.357424 28332 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:25.360065  3342 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:21:25.370970 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:25.327158  3376 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.395658 28331 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:25.295173  3243 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.333778  3130 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:21:25.393622 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.329974  3345 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:25.392719  2988 consensus_queue.cc:1243] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:21:25.331645  3402 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:21:25.410307  3315 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:25.400396  3057 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:21:25.416989 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.418170 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.419657 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.420773 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.396910  3194 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:25.421250 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3068) took 1018 ms. Trace:
I20260709 12:21:25.421692 27734 rpcz_store.cc:276] 0709 12:21:24.402555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.458333 (+ 55778us) service_pool.cc:224] Handling call
0709 12:21:24.458725 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.123839 (+665114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.421037 (+297198us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54000a5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:25.421197 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1514269952}
I20260709 12:21:25.425060 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.426193 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.426744 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:21:25.437872 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.369087  3204 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.439414  3153 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.440105  3204 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.435923 27583 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:21:25.394398 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:25.442265 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47954.2us
I20260709 12:21:25.385314  3326 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:21:25.443535 27660 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:25.437788  3228 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.443450  3008 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:25.446688 28337 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 11
I20260709 12:21:25.447247 28337 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:21:25.447201  3008 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:25.454202 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:25.400007 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78124.3us
I20260709 12:21:25.448797 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:25.464951 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:25.329298  3120 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:21:25.419185 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.451355 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:25.468914 27661 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:25.452983 27734 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
W20260709 12:21:25.470213 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 151012us
I20260709 12:21:25.458142 28331 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:25.458662 27582 raft_consensus.cc:2376] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22. Re-sending same reply.
I20260709 12:21:25.471639 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.447917 27580 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:25.448971 28337 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:25.391160  3241 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:25.472899 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.473522 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:25.474336 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.475064 27585 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:25.471835  2988 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:21:25.474105 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:25.480141  3153 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:25.480643  3153 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:25.481427  3153 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.477512 27582 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 13 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:21:25.492401 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:25.506134 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:25.507294 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:25.507576 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 221020us
I20260709 12:21:25.403857  3345 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.512197 27734 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:25.447772  3228 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.515875 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.517367 27730 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:21:25.396217  3376 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.543722 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.508805  2988 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:25.552148 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:25.553429 27585 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:25.541822  2850 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.643s	user 0.013s	sys 0.000s
I20260709 12:21:25.476364  3336 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:21:25.563289 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.504362  3315 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:21:25.530448  3241 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:25.532455  3130 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:21:25.570871  3408 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.524200  3071 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.616652 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:25.610572 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.620708 27660 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:21:25.620923 27585 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:21:25.622185 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.580873  3376 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.584451 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3669) took 1091 ms. Trace:
I20260709 12:21:25.507201  3321 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.591856 27589 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:21:25.595827  3247 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.596238  3008 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.596671 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.597692  3229 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:25.601718  3401 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.255s	user 0.003s	sys 0.000s
W20260709 12:21:25.602972  3404 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 12:21:25.603868  3317 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:25.596282 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:25.608083 27588 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:21:25.608250 27586 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:21:25.623459  3241 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:25.626196 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.628176 27663 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:25.628031 27584 rpcz_store.cc:276] 0709 12:21:24.493101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.504446 (+ 11345us) service_pool.cc:224] Handling call
0709 12:21:24.504794 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.927143 (+422349us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.584202 (+657059us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b540006f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:25.584387 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":970027904}
W20260709 12:21:25.678042 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81827.4us
I20260709 12:21:25.564090  3201 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:25.634001 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:25.553867  3234 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.689167  3234 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.648373 27583 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:21:25.648730 27731 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:21:25.609623  2988 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:25.649663 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53076us
I20260709 12:21:25.651087 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.691740  3057 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:25.692230 28335 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:25.692371 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.629045  3138 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.693328  3246 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:25.567552  3194 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.627683  3384 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:21:25.694427 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68349.6us
I20260709 12:21:25.565471 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.686221  3333 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:25.695111  3333 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.695458  3194 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:25.686914  3380 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.697067  3380 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.697968  3071 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.650647  3244 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.693621 27552 leader_election.cc:400] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:21:25.699740 27552 leader_election.cc:403] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:25.693147  3138 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.674774 27586 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:21:25.701478 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.695533  3333 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:25.702502 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:25.708300 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.672384  3317 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:25.713577  3244 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:21:25.681577  3153 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:21:25.719894  3138 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.723096 27589 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:21:25.691645  2988 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.727237  3380 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.714018  3402 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:25.729113 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.729859  3138 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:25.730222  3138 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:25.730509  3204 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:25.730589  3138 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:25.731283 27701 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.735414 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:25.731853  2667 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:25.634550  3228 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:21:25.737272 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:25.740754 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 210784us
I20260709 12:21:25.741015 28328 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:25.742393 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:25.748132 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:25.748832 28329 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEARNER]: Advancing to term 25
I20260709 12:21:25.748545 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.624063  3408 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.753110 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.753780 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:25.754838 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.654064  3120 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.084s
I20260709 12:21:25.684103  3201 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:25.648234  3247 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.736011  3380 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:25.757808  3365 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.836s	user 0.013s	sys 0.000s
I20260709 12:21:25.749786  3194 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.769316 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.770682 27585 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:21:25.655805  3320 consensus_queue.cc:1175] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 23 status { last_received { term: 21 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:25.773731  3201 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:25.783041 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 315376us
I20260709 12:21:25.784327 27655 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:21:25.784763 27733 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:21:25.789100 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.784144 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:25.782021  3201 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:25.767966  3153 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:25.776258  3380 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:25.782428  3150 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:25.792111  2667 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:25.800860 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:25.809278  3408 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.810505 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.811046 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.809128  3333 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.813163  3332 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.814177  3332 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:25.814512  3333 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.803117  3299 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:25.777304  3365 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.815183  3299 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.774111  3236 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:21:25.799263  3072 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:25.817731 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.806596 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.821137 28329 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEARNER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:25.805765  3018 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:25.804777  2667 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.805760  3380 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:25.803356  3384 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:25.824913 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.797606  3229 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:21:25.833976  3247 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:25.831451  3416 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.835629 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:25.836259 27737 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:25.836989 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:25.818746  3299 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:25.839545 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:25.837474 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:25.843334 28329 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:25.793712  3241 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.847826 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:25.825690  3322 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:25.822959  3414 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.822824  3408 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:25.817354  3326 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.025s
I20260709 12:21:25.826112  3018 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:25.837800  2667 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:25.856508 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.842484  3247 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:25.859707 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 122503us
I20260709 12:21:25.845824  3416 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:25.848480  3229 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:25.848547  3072 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:25.855430  3120 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:25.874464 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.876389 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:25.877803 27737 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:25.846575  3231 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
W20260709 12:21:25.880391 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 198071us
W20260709 12:21:25.882579 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 142326us
I20260709 12:21:25.883653 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.888033 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:25.871799  3229 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.906992  3419 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:25.909930 27734 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:21:25.909847  3419 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:21:25.913196  3423 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:25.913882  3423 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.914335  3423 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:25.890725 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.863842  3408 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.928081 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:25.872207  3247 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.886744  3333 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:25.910857  3229 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:25.959291  3415 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
W20260709 12:21:25.962165  3418 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:21:25.964416  3409 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:21:25.966717  3424 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:25.974504  3365 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:25.938623  3411 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:21:25.912305  3322 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:25.979640  3417 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:21:25.924284  3263 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:21:25.866293  3219 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Committing config change with OpId 15.11: config changed from index 5 to 11, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:25.989895  3138 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.993044 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1148) took 1171 ms. Trace:
I20260709 12:21:25.881884  3380 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.921648  3018 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.039237  3229 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:25.931154  3228 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:26.052547  3228 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:26.053436  3228 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.054699  3228 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:26.016736  3246 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:26.055634  3246 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.057760  3246 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.059100  3018 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:25.975483  3365 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:26.063521 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:26.028702  3402 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:26.064184 28337 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:26.064540  3402 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.033054  3247 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.007016  3322 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:25.924906  3130 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:21:26.068470  3322 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.070008  3322 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.070864 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:26.063733 27580 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:21:26.070628  3322 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.075455  3322 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.078392 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.078634  3322 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:26.082844  3250 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.083606  3250 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.089614 28337 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
W20260709 12:21:26.090962  3410 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260709 12:21:26.091178 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:26.092181  3229 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:26.204717 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 113670us
I20260709 12:21:26.204890  3229 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:26.205680  3229 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:26.105554  3345 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.208246  3345 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:26.113807  3009 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.042793  3244 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20.
I20260709 12:21:26.124956  3243 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:26.010876 28328 rpcz_store.cc:276] 0709 12:21:24.821706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.821922 (+   216us) service_pool.cc:224] Handling call
0709 12:21:25.992779 (+1170857us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:25.992940 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1533690752}
I20260709 12:21:26.126811  3234 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:26.220199  3234 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:26.221244  3234 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.222468  3234 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:26.222604  3245 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.127390 27733 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:26.223416  3245 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.226625  3245 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:26.231812 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:26.232758 28328 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:26.206806  3229 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:26.237341 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.239743 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:26.127851  3138 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:26.130359 28329 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:26.040313  3408 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.136366 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3497) took 1031 ms. Trace:
I20260709 12:21:26.139086  3317 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.139767 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:26.244917 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:26.284683 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 396691us
I20260709 12:21:26.151268  3247 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.152129  3412 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:21:26.153537  3263 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:21:26.157150 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.158046 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.301904 27734 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:26.158862 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.303218 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:26.304509 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.305253 27732 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:26.161940  3198 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:21:26.163800  2850 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.164938  3414 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:26.165161 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.166301 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.166798 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.167582 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.168087 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.169380 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.170291 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:26.173086 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.173607 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.317022 27663 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:26.174777 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:26.074196  3321 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:26.184013  3402 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.272449  3244 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:26.184948  3250 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.190209  3299 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.240684 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:26.246145 27733 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:26.246243 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:26.352139 27589 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:26.247728 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:26.353170 27586 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:26.248240 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.250276 27659 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:26.273762 27731 rpcz_store.cc:276] 0709 12:21:25.104905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.105072 (+   167us) service_pool.cc:224] Handling call
0709 12:21:25.136586 (+ 31514us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:25.690783 (+554197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.136028 (+445245us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54001d1fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:26.136287 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1167413504}
W20260709 12:21:26.141912  3413 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.341s	user 0.001s	sys 0.001s
I20260709 12:21:26.142033  3315 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:26.213315  2667 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.288902 27580 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260709 12:21:26.314520  3408 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:26.291183 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 151521us
I20260709 12:21:26.291925 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:26.292919 27584 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:21:26.369127 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:26.264858 27656 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 5 to 11, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:26.294898 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:26.287629 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:26.309327 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:26.332623 27655 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:26.333003 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.332603  3245 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.335915 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.364377 27580 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 14
I20260709 12:21:26.381717 27580 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:21:26.382063 27580 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:26.289646 27582 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:21:26.336571  3204 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.392997  3204 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.396507 27734 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:26.396888  2667 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:26.184551  3320 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:26.401499 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 92305.6us
W20260709 12:21:26.375566 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 447531us
I20260709 12:21:26.377184 27586 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:26.332578  3414 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:26.338367  3244 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.408037 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.409987 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:26.425426 27663 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:26.382206 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 94659.7us
I20260709 12:21:26.392307  3138 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:26.402292 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:26.423502 27658 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:21:26.439458 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:26.424485 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:26.300412  3247 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:26.458812 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.357506  3402 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:26.468721 27659 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:21:26.337958  3150 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.091s
I20260709 12:21:26.337743  2850 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:26.473446 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 235) took 25458 ms (client timeout 15000 ms). Trace:
I20260709 12:21:26.382903  3072 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:26.462706 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:26.486873 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.485750 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:26.497323 27658 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:26.464047  3130 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:26.498654 28331 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:21:26.499233  3407 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:21:26.492899  3246 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:26.500217  3246 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.503480  3225 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.504120  3225 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.506827  3225 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.492998  2667 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:21:26.498678  3130 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:26.510123  3130 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:26.503095  3246 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.512252 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:26.529464 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.530628 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:26.532967 27730 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:21:26.438298  3244 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:26.537367 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.544881 27731 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:26.449208  3344 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:26.545179 27734 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.19-23.19]
I20260709 12:21:26.466173  3204 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:26.477348  3405 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:21:26.466080  3322 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:26.540723 27589 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:26.473124  3231 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:26.519097  3317 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:26.380831  3408 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.571147 27660 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:26.525223  3138 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:26.542866  3228 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.422766  3245 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:26.480428 27487 rpcz_store.cc:269] 0709 12:21:01.001937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.004554 (+  2617us) service_pool.cc:224] Handling call
0709 12:21:26.459705 (+25455151us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:21.883786 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:21.999258 (+115472us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b5c00183d8c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea053a331 00007f0ea053a221 00007f0ea0534f6c 00007f0e9dd92d69 00007f0e9dd7e0a6 00007f0e9dd7dd58 00007f0e9dd829f1 00007f0e9dd829a9 00007f0e9dd82939 00007f0e9dd82901 00007f0e9dd81b2d
0709 12:21:22.038310 (+ 39052us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:22.047976 (+  9666us) write_op.cc:435] Acquired schema lock
0709 12:21:22.047990 (+    14us) tablet.cc:662] Decoding operations
0709 12:21:22.228659 (+180669us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:22.228788 (+   129us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:22.228831 (+    43us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:21:22.230915 (+  2084us) tablet.cc:701] Row locks acquired
0709 12:21:22.230977 (+    62us) write_op.cc:260] PREPARE: finished
0709 12:21:22.231209 (+   232us) write_op.cc:270] Start()
0709 12:21:22.359317 (+128108us) write_op.cc:276] Timestamp: P: 1783599682231118 usec, L: 0
0709 12:21:22.366016 (+  6699us) op_driver.cc:348] REPLICATION: starting
0709 12:21:22.422846 (+ 56830us) log.cc:844] Serialized 7887 byte log entry
0709 12:21:23.149435 (+726589us) op_driver.cc:464] REPLICATION: finished
0709 12:21:23.592770 (+443335us) write_op.cc:301] APPLY: starting
0709 12:21:23.639930 (+ 47160us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:23.984042 (+344112us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:23.984100 (+    58us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:24.521019 (+536919us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:24.521322 (+   303us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=40,mrs_lookups=0
0709 12:21:24.521369 (+    47us) write_op.cc:312] APPLY: finished
0709 12:21:24.521771 (+   402us) log.cc:844] Serialized 180 byte log entry
0709 12:21:24.546951 (+ 25180us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:24.590145 (+ 43194us) write_op.cc:454] Released schema lock
0709 12:21:24.600311 (+ 10166us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:24.626266 (+ 25955us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:24.629784 (+  3518us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":923903,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":425381,"apply.run_cpu_time_us":36833,"apply.run_wall_time_us":1032055,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87727,"num_ops":20,"prepare.queue_time_us":536480,"prepare.run_cpu_time_us":21822,"prepare.run_wall_time_us":725428,"raft.queue_time_us":475924,"raft.run_cpu_time_us":53261,"raft.run_wall_time_us":266789,"replication_time_us":783208,"spinlock_wait_cycles":6988672,"thread_start_us":896840,"threads_started":5,"wal-append.queue_time_us":276066,"wal-append.run_cpu_time_us":46427,"wal-append.run_wall_time_us":2461779}]]}
W20260709 12:21:26.558791  3403 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:21:26.552886  3225 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.550330  3344 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:26.460141  3320 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.479450  3402 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.568882  3322 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:26.586311  3241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.012s	sys 0.024s
I20260709 12:21:26.573841  3231 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.586457  3344 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:26.599627  3322 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:26.582026  3228 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:26.629807  3241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.012s	sys 0.024s
I20260709 12:21:26.658178  3322 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.587149  3225 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:26.678963 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.030s	sys 0.037s
W20260709 12:21:26.706452 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.031s	sys 0.037s
I20260709 12:21:26.567629  3414 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.641450  3407 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.710512 27661 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20260709 12:21:26.707577 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 758056us
W20260709 12:21:26.709499 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 397361us
I20260709 12:21:26.718611 27589 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:26.732758 27589 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
W20260709 12:21:26.718789  3423 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.804s	user 0.002s	sys 0.012s
I20260709 12:21:26.741925 27659 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:26.743151 27589 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:21:26.745036 27658 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:21:26.748091  3247 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:26.749351  3231 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.751508 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:26.752190 27588 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:21:26.752490  3345 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.544s	user 0.007s	sys 0.004s
W20260709 12:21:26.753736  3219 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.710s	user 0.008s	sys 0.003s
I20260709 12:21:26.756071  3228 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:26.757160  3408 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.757819  3320 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:26.758838  3299 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.693943  3138 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.759810  3402 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:26.760924  3429 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:26.761039 27588 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:26.762485  3423 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:26.729235  3407 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:26.766314 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.768199 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:26.769001 28331 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 12.
I20260709 12:21:26.769152 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:26.770326 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:26.771092 28329 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:26.771899 27584 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:21:26.776459 27734 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:26.781270 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:26.801631 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:26.802388 27584 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:26.807930 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:26.809397 27738 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:26.733515  3414 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:26.815908 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:26.823490 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.854403 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:26.813036  3431 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.847384  3408 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.870160 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:26.850137  3384 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.811853  3204 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:26.887950 27584 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:26.799188  2850 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.813577  3402 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:21:26.849319  3365 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:26.763235  3130 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:26.906984 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:26.908711 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.913314 27704 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:26.913628 27704 leader_election.cc:403] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:21:26.916083 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:26.917128 28328 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEARNER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:26.919286 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:26.920153 28332 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:26.880228  3342 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:26.885299  3225 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:26.885903  3430 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:21:26.947780 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54166.5us
I20260709 12:21:26.972354 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:27.010480 27584 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:27.012646 28332 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
W20260709 12:21:26.976823 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.004s	sys 0.025s
W20260709 12:21:27.014633 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.004s	sys 0.029s
W20260709 12:21:27.014931 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 583217us
W20260709 12:21:26.918512  3229 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.026s	sys 0.055s
I20260709 12:21:26.856904  3018 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:21:26.910759  3402 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.905624  3427 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.125s
I20260709 12:21:27.031816 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:26.905216  3320 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.036571 28328 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:21:27.054081 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:26.935976 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 512365us
I20260709 12:21:26.919513  2850 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.045989 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 560326us
I20260709 12:21:26.948542  2667 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:26.996199  3419 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:26.948151  3332 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:26.892015  3236 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.137s
I20260709 12:21:27.070614 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:27.027837  3229 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.030s	sys 0.055s
I20260709 12:21:26.890588  3408 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.022907  3225 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.911742  3057 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.134s
I20260709 12:21:27.076210 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:27.047173 27589 raft_consensus.cc:2955] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 5 to 11, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:21:27.048115  3321 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.709s	user 0.011s	sys 0.000s
I20260709 12:21:26.919917  3376 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.132s
I20260709 12:21:26.909101  3384 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.021132  3228 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.919165  3365 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.057916  3402 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:21:26.891865  3431 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:27.090920 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49241us
I20260709 12:21:27.013381  3342 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:27.094424 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75194.6us
W20260709 12:21:27.087230  3317 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.002s
I20260709 12:21:27.093649  3138 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:27.090329  3322 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.075958  3320 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.092593  3407 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:27.126945 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50666.7us
I20260709 12:21:27.095836  3219 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52006 (request call id 0) took 2990 ms. Trace:
I20260709 12:21:27.136991 27587 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
W20260709 12:21:27.141578 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.087159  3443 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:21:27.143224 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:27.144721 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:27.109098  3423 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.085513  2850 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.161780 27585 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:21:27.103982  3241 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:21:27.163301 27584 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:21:27.165009 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51756.8us
I20260709 12:21:27.109798  3332 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.151225  3448 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.172966  3435 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:21:27.109310  3419 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.177595 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:27.170125 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:27.114228  3445 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:21:27.118916  3225 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.161223  2667 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:27.185333 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:27.163151  3345 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.179152  3438 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:21:27.149678  3228 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.142828  3444 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 12:21:27.191303 27584 raft_consensus.cc:2410] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:21:27.189987  3236 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:27.200238  3236 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:27.201115  3236 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.201618 27588 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:21:27.204793 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3095) took 1049 ms. Trace:
I20260709 12:21:27.183424  3130 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.205396 27738 rpcz_store.cc:276] 0709 12:21:26.165593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.170000 (+  4407us) service_pool.cc:224] Handling call
0709 12:21:27.204478 (+1034478us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:27.204689 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1115895552}
I20260709 12:21:27.210248 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:27.208729 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:27.190748  3317 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:27.211232 28328 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:27.213814 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:27.190053  3322 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.206889  3243 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:27.190433  3430 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:21:27.229691 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.243779 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.187016  3448 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.253983 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.233453 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.255470  3342 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.193778  3345 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:27.242527  3344 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.574s	user 0.010s	sys 0.000s
I20260709 12:21:27.231168 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.210537  3453 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.258569 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3519) took 1371 ms. Trace:
I20260709 12:21:27.219504  3427 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:27.274669  3427 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.275103  3427 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:27.219301  3442 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
W20260709 12:21:27.278827  3451 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.072s	user 0.001s	sys 0.001s
I20260709 12:21:27.279191  3448 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.262038 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.161015  3219 rpcz_store.cc:276] 0709 12:21:24.102244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.105728 (+  3484us) service_pool.cc:224] Handling call
0709 12:21:24.240247 (+134519us) log.cc:844] Serialized 494 byte log entry
0709 12:21:27.090184 (+2849937us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3439,"raft.run_cpu_time_us":1154,"raft.run_wall_time_us":1156,"thread_start_us":62383,"threads_started":1,"wal-append.queue_time_us":126744,"wal-append.run_cpu_time_us":1997,"wal-append.run_wall_time_us":1005331}
I20260709 12:21:27.279945 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:27.262912 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:27.224102  3057 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:27.263881 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.224010  3444 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:27.222992  3376 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:27.296278  3419 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.293038  3130 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:27.296674  3376 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.297441  3130 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.256156  3332 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:27.271683  3447 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:21:27.272563  3458 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
W20260709 12:21:27.273068  3244 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.000s
I20260709 12:21:27.300112  3244 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.302012 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:27.302796 27581 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:27.258997  3455 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:21:27.247251  3408 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.268383 27737 rpcz_store.cc:276] 0709 12:21:25.886849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.886992 (+   143us) service_pool.cc:224] Handling call
0709 12:21:25.887365 (+   373us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:26.007790 (+120425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.007840 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:26.009233 (+  1393us) log.cc:844] Serialized 52 byte log entry
0709 12:21:26.588992 (+579759us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:26.589060 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.113985 (+524925us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b540029fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:27.114073 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.256736 (+142663us) raft_consensus.cc:1685] finished
0709 12:21:27.256755 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.258481 (+  1726us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2868,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":44027,"spinlock_wait_cycles":275580160,"thread_start_us":440727,"threads_started":1,"wal-append.queue_time_us":664651}
I20260709 12:21:27.233453 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:27.309494 27729 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:27.223677  3138 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:21:27.293152  3317 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.302222  3244 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.273391  3453 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.245003  2850 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.293162  3444 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:27.311388 27731 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:27.299476  3243 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.330410 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:27.223516  3322 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.320745 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:27.314361 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:27.271188  3320 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.352460 27589 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:27.286417  3344 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:27.367427 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 199246us
W20260709 12:21:27.368815 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 211512us
I20260709 12:21:27.368997 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:27.326624  3448 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:27.325948  3408 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.352550  3130 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.365839  3244 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:27.325647  3317 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.378858 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:27.380084 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:27.370190  3228 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.329465  3402 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.347870  3243 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.388513  3243 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.388633  3402 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.389127  3243 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:27.379951  3384 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.400992  3380 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.385955  3472 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:27.445406  3470 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:27.446370  3322 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:27.517539  3234 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.295s	user 0.012s	sys 0.014s
I20260709 12:21:27.400005  3243 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:27.383569  3431 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.458783 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:27.378002  3408 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.510172  3317 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.488118  3344 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.446419  3453 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.376699  3448 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.514815  3244 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.373736  3241 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:21:27.525709  3228 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:27.551515 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.532649  3419 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.536741  3384 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:27.567869 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 417802us
I20260709 12:21:27.563119 27580 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:27.542163  3322 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.564801 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184778us
I20260709 12:21:27.557775  3243 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:27.535593  3380 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.589016  3219 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.591894  3487 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.592217  3219 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.592499  3487 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.594727  3219 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.594839  3487 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.572695  3234 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.565227  3244 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.595899  3234 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.597421  3244 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.597556  3243 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.601383  2988 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:27.603773  3488 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.603917 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:27.604686 28337 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:27.604343  3488 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.605515 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:27.606038 28331 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:21:27.606544  3488 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:27.608280 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.608448  3342 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:27.608886  3342 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:27.609639  3342 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.610728  3342 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:27.566687  3241 consensus_queue.cc:1243] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:21:27.611589 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:27.611933 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:27.614579 27588 raft_consensus.cc:2368] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14.
I20260709 12:21:27.620951  3408 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.621346 27701 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:27.613773  3018 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.622260  3018 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.622928  3380 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.569430  3228 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.623811  2667 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:21:27.625144  3018 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.625612  3228 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.628654 28331 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:21:27.572541  3384 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.633097  3384 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.584514  3299 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.639724  3488 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.608259  3234 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.571228  3419 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:27.647181 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57219.7us
I20260709 12:21:27.648177 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:27.637588  3120 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.591087  3322 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.653043 27664 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:21:27.637439  3332 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:27.627168  3241 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:21:27.670975 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.656575 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.658352 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.672320 27663 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:27.671819 27657 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:27.673390 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.674011 27664 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:27.674867 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 2966) took 1504 ms. Trace:
I20260709 12:21:27.674875 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:27.677206 27587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3740) took 2363 ms (client timeout 1994 ms). Trace:
W20260709 12:21:27.677651 27587 rpcz_store.cc:269] 0709 12:21:25.313607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.420528 (+106921us) service_pool.cc:224] Handling call
0709 12:21:27.676969 (+2256441us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:27.677101 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3944575872}
I20260709 12:21:27.679949 27587 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
W20260709 12:21:27.680045 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 2965) took 1511 ms (client timeout 1894 ms). Trace:
W20260709 12:21:27.680473 27660 rpcz_store.cc:269] 0709 12:21:26.168891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.169051 (+   160us) service_pool.cc:224] Handling call
0709 12:21:27.679873 (+1510822us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:27.679988 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580131200}
I20260709 12:21:27.647025  3470 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.681293 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.672407 27659 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:27.680385 27587 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:27.681954 27655 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:27.682443 27587 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:21:27.682796 27587 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:27.675271 27661 rpcz_store.cc:276] 0709 12:21:26.170267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.170407 (+   140us) service_pool.cc:224] Handling call
0709 12:21:26.170765 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:26.711268 (+540503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.674616 (+963348us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54001e5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:27.674805 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1240354176}
I20260709 12:21:27.656113 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.685894 27658 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:27.686764 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:27.691242 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:27.691430 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:27.654239  3219 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:27.647343  3299 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.670270  3349 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:21:27.694412  3431 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:27.698797 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:27.657691 27472 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7274fd8a2115405c9b78c61214bff736 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:27.654392  3004 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.648216  3488 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.667373  3234 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.654472  3198 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:21:27.684715  3246 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:27.725582 27472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67897.1us
I20260709 12:21:27.654650  3120 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.732362 28337 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:27.689455  3228 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
W20260709 12:21:27.715888  3478 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:21:27.684377  3332 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.697526  3138 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:27.689855  3470 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.757620  3488 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.819492 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:27.810731  3138 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.820426 28332 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:27.820499  3138 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:27.731077  3234 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.752604  3299 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.725507  3452 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:27.717375  3349 consensus_queue.cc:1243] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:21:27.741245  3448 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.723906  3424 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:27.734546  3246 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.724673  3004 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:27.749230  3480 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:21:27.692304  3427 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.723570  3431 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:27.826193 27630 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.994s (SENT) [suppressed 13 similar messages]
I20260709 12:21:27.826503 28332 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 12:21:27.832659 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 143304us
I20260709 12:21:27.856058 28295 leader_election.cc:400] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14.
I20260709 12:21:27.827524  3426 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.823652  3402 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.856371 28295 leader_election.cc:403] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:27.827862  3407 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:27.859478 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:27.863945 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:27.827415  3380 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:27.868749 28331 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:21:27.839874  3424 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.740571  3472 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:27.868893 28328 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:27.852689  3376 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.863497 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:27.887862 27734 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:27.869684  3426 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.892068  3472 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.892978  3322 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.873034  3402 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.893940  3426 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.894548  3426 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.895434  3402 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.889813  3228 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:27.896571  3228 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.898648  3427 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.899344  3427 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.899860  3427 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.902176  3427 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.855031  3243 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:27.903049  3243 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.858784  3431 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:27.898890  3228 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.871970  2850 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.867568  3492 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:27.891079  3488 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.914149 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3704) took 1110 ms. Trace:
I20260709 12:21:27.871712  3299 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 13. Current term is 14.
I20260709 12:21:27.888979  3424 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:27.850001  3452 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.924569 27584 rpcz_store.cc:276] 0709 12:21:26.804242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.197319 (+393077us) service_pool.cc:224] Handling call
0709 12:21:27.222737 (+ 25418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.223079 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.223116 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:27.224644 (+  1528us) log.cc:844] Serialized 26 byte log entry
0709 12:21:27.536525 (+311881us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:27.536573 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.536644 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.913894 (+377250us) raft_consensus.cc:1685] finished
0709 12:21:27.913926 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.914084 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":576,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":25324,"thread_start_us":311522,"threads_started":1,"wal-append.queue_time_us":418842}
I20260709 12:21:27.928225  3489 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.928660  3497 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:27.928941  3489 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.929407  3497 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.931960  3489 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.932914  3497 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.934633  3488 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.937479  3488 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.941516  3365 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:27.914218  3004 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.885830  3407 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.895047  3426 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:27.915884  3402 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:27.887987  3380 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.922626  3150 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.927435  3120 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.944492  3234 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:27.944461  3246 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:27.925261  2850 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.981465  3402 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:27.927554  3492 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:27.988003  3150 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.010337 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.010893 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.010635 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:28.013146 27729 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:27.999841  3402 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:28.014849 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.015763 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:28.014545 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.016934 27737 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:28.017158 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 191008us
I20260709 12:21:28.017186 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.020450 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.020982 27733 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:28.025839  3380 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.026782 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:28.028999 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:28.029803 27664 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:28.031452 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.032248 27663 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 13: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:28.033025  2850 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:28.033836  3496 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:21:28.036244 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.036556 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.037024 27658 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:28.037211 27737 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:28.037737 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.037931 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.038611 27737 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:21:28.039036 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.040259 27733 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:21:28.040377 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.041038 27661 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:28.041152 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.042208 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.042701 27658 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:28.043507 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.010588  3332 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.044289 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.009845  3245 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.046183 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.047694 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.048503 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.048198 27658 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:28.027717  3470 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.044371  3246 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.049397  3503 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.056041  3503 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.058385  3503 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.050348 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.060799 27661 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260709 12:21:28.051716 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.062496 27657 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:28.027984  3243 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.076828 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.077759 27658 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 13.
I20260709 12:21:28.080349 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.080796 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:28.081540 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.082393  3219 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:28.082526 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.082892  3219 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:28.083887  3236 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:28.084923  3236 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:21:28.085773  3225 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.087396  3426 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.087703 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.090287 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.092048 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.092391 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.092772 28329 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:28.093644 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.095269 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.095863 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:28.096196 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.095865 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:28.097012 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:28.097613 27731 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:21:28.097690 27663 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:28.098315  3245 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.098860  3245 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.100055 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.100555 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.100674 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:28.101136  3245 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.101658  3245 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:21:28.102267 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.102531 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:28.103425 27663 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:28.103559 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:28.104285 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.105192  3225 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:28.105633 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.105758 27734 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:21:28.102664  3245 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:28.108067 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.107007 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:28.108947  3441 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.986s	user 0.005s	sys 0.011s
I20260709 12:21:28.109298 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.109570 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.109965 27658 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:28.110143 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3758) took 1862 ms. Trace:
I20260709 12:21:28.110155 27659 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:28.111099  3245 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.083191 27737 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:28.114676 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.114890 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.115496 27584 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:28.115576  3443 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:28.115645  3452 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.116055  3443 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.116464  3443 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:28.116990 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.117861 27584 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:21:28.105600  3225 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.119275  3225 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:28.119684  3150 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.117704 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.123651 27588 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:28.110546 27583 rpcz_store.cc:276] 0709 12:21:26.247721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.282627 (+ 34906us) service_pool.cc:224] Handling call
0709 12:21:26.306853 (+ 24226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:26.316707 (+  9854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.316752 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.340478 (+ 23726us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.810237 (+469759us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:26.810448 (+   211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.810544 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:28.108413 (+1297869us) spinlock_profiling.cc:243] Waited 9.78 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:28.109874 (+  1461us) raft_consensus.cc:1685] finished
0709 12:21:28.109895 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:28.110087 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22503168,"thread_start_us":424837,"threads_started":1,"wal-append.queue_time_us":711146}
I20260709 12:21:28.112238 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.128000 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:28.132957 28331 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:21:28.129558 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.129928 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.130244 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:28.049985  3234 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:28.142432 27664 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:28.135207 27663 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:21:28.110852 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:28.136231 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.145695 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:28.129571 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.141451 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.148237 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:28.142297 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.131522 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.151532 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:28.153404 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.160912 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:28.157073 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.161535 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.166038 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.167902 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.170288 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.170967 27589 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:28.135723  3426 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:28.142274  3234 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.142056  3492 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.188185 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:28.188630 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.189010 27589 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:28.189286 27588 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:28.172857 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.177731 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.052173  3246 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.178841 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.130645  3431 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.190997 27659 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:28.142417  3236 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.194540 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.200521 27659 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:28.163897  3150 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:28.150477  3505 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:28.168002  3437 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:28.207581 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.207800 27663 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 14.
I20260709 12:21:28.209067 27589 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:28.188153  3234 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:28.220399 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.187315  3332 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:28.211545 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.156638  3424 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.222249 27733 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:28.228186 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.230613 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.232887 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.233599 27663 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:28.202149  3492 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:28.235610 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:28.248409 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.249682 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.250485 28331 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:28.221391  3150 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:28.187611  3219 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.243814 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.244714 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.246289 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:28.246678 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86799.2us
I20260709 12:21:28.248574 27731 raft_consensus.cc:606] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:28.239995 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.207170  3407 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.260317  3344 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.264840 27729 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:28.187512  3426 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.267482 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.248402  3492 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.294947 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.251627  3322 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.267807  3150 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.272760 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.242996  3246 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.225734  3332 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:28.210408  3489 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:21:28.226477  3437 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:28.247094  3503 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:28.218478  3501 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:21:28.289189  3448 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:21:28.299748 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62326.4us
I20260709 12:21:28.302510 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.300482  3492 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:28.208469  3236 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.217837  3505 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.324864 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:28.253151  3508 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:28.209487  3444 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.298763  3138 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.330497 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:28.303593 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.343531 28331 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:21:28.327132 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77041.7us
I20260709 12:21:28.317016  3503 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.372709 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.376323 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.337443 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.308625  3244 raft_consensus.cc:2764] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:21:28.266371  3402 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.383507  3402 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.314302  3332 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:28.384675  3426 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.338542  3365 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:28.385497  3365 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:28.386006  3365 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:28.335394 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.342558  3508 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.387748  3508 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:28.338579  3246 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:28.389322  3246 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.389722  3246 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:28.349331  3138 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.350808 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.351611  3448 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:28.391907 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.392154 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.392730  3448 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:28.393124  3448 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:28.392197 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.393567  3448 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:28.394223 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.395411 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.324086  2988 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:28.357064  3150 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.359582  3484 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:28.403800  3484 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:28.403644  3446 consensus_queue.cc:1175] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 21 status { last_received { term: 21 index: 12 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.12-20.12]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:28.404825  3484 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.405946  3484 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:28.364342  3424 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.366899  3236 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.295495  3344 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.346118 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:28.369998 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96205.9us
I20260709 12:21:28.417662 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.417932 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:28.400213 28329 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:28.410435  3153 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:28.413856  3225 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.308774  3519 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:21:28.314051  3322 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:28.455929 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109898us
I20260709 12:21:28.461074 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.456684  3009 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:28.373996  3503 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:28.466859 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 194349us
W20260709 12:21:28.463217 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.454803 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:28.469323 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:28.416392  3150 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.416389  3244 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:28.459019  3402 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:28.478835 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61266.9us
I20260709 12:21:28.460626  3250 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:28.479790 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:28.489578 27630 leader_election.cc:336] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 2.632s (SENT)
I20260709 12:21:28.466493  3009 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:28.416265  2988 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.491956 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:28.493774 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:28.475031  3244 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.461994  3443 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.495489  3522 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:28.462425  3153 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.488282  3472 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:28.483026  3402 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.483920  3431 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.490576  3243 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:28.497219  3523 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:28.524807  3523 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.525902 27731 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
W20260709 12:21:28.530650 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 135330us
I20260709 12:21:28.518260 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:28.533108 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:28.537847 28330 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:28.533520 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:28.528596 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.554247 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:28.608533 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:28.609699 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.529384 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:28.642714 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88544.7us
I20260709 12:21:28.643560 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:28.648666 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119364us
I20260709 12:21:28.513640  3443 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.490729  3521 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:28.496838  3009 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:28.507021  3244 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:28.515849  3402 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:28.514166  3522 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.514582  3414 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:28.477268  3150 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.502979  3505 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.655273 27731 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:21:28.527288  3523 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.519115  3431 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:28.664661 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.521785  3384 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.515628  3472 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.518829  3526 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:28.715366 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50781.1us
I20260709 12:21:28.505643  3071 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.159s
I20260709 12:21:28.654145  3138 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:28.723591 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 254311us
I20260709 12:21:28.723891 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:28.724892 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:28.647506  3380 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:28.523949  3365 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.663460  3521 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:28.724637 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:28.735729 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.663720  3448 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.716924  3472 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:28.712106  3431 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.752524 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.753494 27589 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:28.753593 27586 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:28.658360  3245 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:28.491695  3437 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.754290 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:28.720621  3526 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:28.674973  3522 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:28.723312  3138 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:28.768154 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:28.769183 27580 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:28.773456 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.774379 27587 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:28.743958  3521 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:28.757786  3431 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:28.809777  3542 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:28.717489  3384 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:28.812013  3534 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:21:28.812577  3384 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.755718  3228 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:28.831890 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.763772  3245 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.764709  3344 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.772691  3138 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:28.854802  3234 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.622s	user 0.009s	sys 0.001s
I20260709 12:21:28.855461  3448 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.855249  3234 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.855649  3322 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.856411  3322 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:28.856988  3322 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:28.860646  3531 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:21:28.807471  3225 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:28.770035  3380 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.863402  3380 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.864744  3365 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.837276 27587 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:28.864574  3508 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.865422  3153 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.764881  3526 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:28.838091 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:28.840322  3518 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:28.840533  3530 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:28.841382  3532 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:28.843931  3426 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:28.844726  3520 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.328s	user 0.002s	sys 0.000s
I20260709 12:21:28.845114  3516 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:28.848340  3228 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:28.882725  3539 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:21:28.882267  3516 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.836566  3384 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.872278  3245 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:28.902699  3516 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:28.903898  3228 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:28.909777  3246 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.520s	user 0.008s	sys 0.002s
I20260709 12:21:28.876399  3532 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.911412  3384 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:28.912883  3150 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.884974  3426 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:28.917810 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1287) took 1061 ms. Trace:
I20260709 12:21:28.919420  3542 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:28.928213 27731 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:28.963065 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.929021 28330 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:21:28.933007 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47316.6us
W20260709 12:21:28.933461 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51753us
I20260709 12:21:28.936676 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.967103 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.968596 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:28.936869 27580 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
W20260709 12:21:28.934595  3332 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20260709 12:21:28.959280 28332 rpcz_store.cc:276] 0709 12:21:27.856626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.856829 (+   203us) service_pool.cc:224] Handling call
0709 12:21:27.857124 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.885777 (+ 28653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.885809 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:27.885850 (+    41us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:27.885884 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.917576 (+1031692us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:28.917622 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:28.917753 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65423104}
I20260709 12:21:29.005138  3554 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.033118  3321 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:29.048411  3321 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.050763  3321 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.032009  3322 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.053985  3426 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.057569  3419 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:29.059285 27581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3698) took 2290 ms (client timeout 2636 ms). Trace:
W20260709 12:21:29.059736 27581 rpcz_store.cc:269] 0709 12:21:26.768694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.768863 (+   169us) service_pool.cc:224] Handling call
0709 12:21:29.059058 (+2290195us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54001fe5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:29.059222 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1225127680}
I20260709 12:21:29.060976 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.061900 27581 raft_consensus.cc:2410] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:21:29.062849 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:28.973755 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.941466  3532 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:28.986816 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.065714 27581 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:28.961990  3508 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:28.990653  3384 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:29.071297 27588 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:21:29.072023 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:29.007736 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 265389us
I20260709 12:21:28.987633  3380 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:29.011853  3244 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.017233  3345 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:29.041539  3554 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.064319 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.941545  3505 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:29.070277 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:29.072682 27581 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:29.080199 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.099138 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.100737 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:29.101249  3494 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.100s	user 0.003s	sys 0.000s
I20260709 12:21:29.101997 27581 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:21:29.102840 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.103856 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.104487 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.104734 27581 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:29.105698 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:29.106720 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:28.943573  3246 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.038210  3225 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:29.107859  3501 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:29.108409  3501 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:29.109210 27582 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:29.109283  3501 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.110471 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.085608  3244 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:29.088007 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.111227 27731 raft_consensus.cc:2368] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:21:29.111828 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3532) took 1078 ms. Trace:
I20260709 12:21:29.112067 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:29.112177 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.087293 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.112982 27586 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
W20260709 12:21:29.091532  3493 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.092s	user 0.003s	sys 0.001s
I20260709 12:21:29.112217 27729 rpcz_store.cc:276] 0709 12:21:28.033001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.037562 (+  4561us) service_pool.cc:224] Handling call
0709 12:21:29.111680 (+1074118us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:29.111780 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":520684288}
I20260709 12:21:29.088116  3009 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.039535  3332 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.107511  3225 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.084610 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3579) took 1434 ms. Trace:
I20260709 12:21:29.030210  3526 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.120373 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.126636 28331 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:21:29.127352 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:29.128086 27729 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:29.129050 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.104616  3554 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.124106 27662 rpcz_store.cc:276] 0709 12:21:27.649918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.651280 (+  1362us) service_pool.cc:224] Handling call
0709 12:21:27.651687 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.842249 (+190562us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:21:27.842298 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:27.842889 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 12:21:27.952553 (+109664us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:27.952596 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.084242 (+1131646us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54001e4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:29.084292 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:29.084338 (+    46us) raft_consensus.cc:1685] finished
0709 12:21:29.084351 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.084549 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":946,"raft.queue_time_us":10095,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":28835,"spinlock_wait_cycles":434767744,"thread_start_us":109319,"threads_started":1,"wal-append.queue_time_us":166753,"wal-append.run_cpu_time_us":1610,"wal-append.run_wall_time_us":1022749}
W20260709 12:21:29.136319 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.139554  3241 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.082252  3138 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.145998  3241 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.146282 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.147765  3138 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:29.148051  3241 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.146032 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:29.150025  3512 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.815s	user 0.000s	sys 0.002s
I20260709 12:21:29.151050  3234 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:29.152886  3517 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.749s	user 0.002s	sys 0.000s
I20260709 12:21:29.153048  3225 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.155138 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.155896 27582 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:29.156790 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.157439 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:29.160534 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.161368 27664 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:29.164160 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.165688 28337 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:29.166100  3521 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.166767 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3569) took 2763 ms. Trace:
I20260709 12:21:29.156155 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3719) took 1037 ms. Trace:
I20260709 12:21:29.168561 27584 rpcz_store.cc:276] 0709 12:21:28.118740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.120868 (+  2128us) service_pool.cc:224] Handling call
0709 12:21:28.121201 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:28.132108 (+ 10907us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:28.132133 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:28.132544 (+   411us) log.cc:844] Serialized 26 byte log entry
0709 12:21:28.231555 (+ 99011us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:28.231598 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.231669 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:29.152120 (+920451us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:29.155794 (+  3674us) raft_consensus.cc:1685] finished
0709 12:21:29.155824 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.156084 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11217,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7867,"spinlock_wait_cycles":13239040,"thread_start_us":55004,"threads_started":1,"wal-append.queue_time_us":245035}
I20260709 12:21:29.172528 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.173652 27584 raft_consensus.cc:2368] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:21:29.174701 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.167127 27655 rpcz_store.cc:276] 0709 12:21:26.403693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.435994 (+ 32301us) service_pool.cc:224] Handling call
0709 12:21:27.681633 (+1245639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.682589 (+   956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.166559 (+1483970us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000c2950. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:29.166688 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2863631104}
I20260709 12:21:29.176652 27737 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:29.179607 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:29.181257 27584 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:21:29.183727  3532 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.167490  3009 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:29.185158 27660 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:29.193100 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.198182 27729 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:29.199525 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:29.200637 27730 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:29.203907 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.191280  3444 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:29.185493  3505 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:29.212427  3505 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:29.203312  3402 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.002s
I20260709 12:21:29.194409 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:29.183878  3521 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:29.186470  3526 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:29.223855 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.225673 27731 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:29.226953 27664 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:29.186403  3532 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:29.228761 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3751) took 3064 ms (client timeout 2632 ms). Trace:
W20260709 12:21:29.229172 27585 rpcz_store.cc:269] 0709 12:21:26.164648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.239702 (+ 75054us) service_pool.cc:224] Handling call
0709 12:21:29.228577 (+2988875us) spinlock_profiling.cc:243] Waited 791 ms on lock 0x7b540020cbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:29.228701 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1819842304}
I20260709 12:21:29.229970 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:29.230909 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.231760 27738 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:29.232309 27584 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:21:29.232759 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.239238 27738 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:29.240448 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.242158 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.243403 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.246702 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:29.247543 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.019s	sys 0.020s
W20260709 12:21:29.247888 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.019s	sys 0.020s
I20260709 12:21:29.248004 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.249504 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:29.250336 28331 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:29.248783 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:29.251704 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.255189 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.257550 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:29.228103  3414 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.530s	user 0.002s	sys 0.011s
I20260709 12:21:29.258450 27661 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:29.263502 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:29.264942 27731 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:29.267092 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.267050 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.193714  3431 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.268265 27664 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:29.259253  3503 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.772s	user 0.009s	sys 0.003s
I20260709 12:21:29.199211  3472 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:29.271559 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.275885 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:29.290815 27589 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.019s	sys 0.008s
W20260709 12:21:29.292191 27589 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.019s	sys 0.009s
I20260709 12:21:29.189628 27480 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 18 to 19, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 19 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:29.286254 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.302126  3246 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:29.252686 27581 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:29.296187 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.308139 27655 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:29.270172  3492 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.926s	user 0.010s	sys 0.004s
W20260709 12:21:29.311796 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.312388 27735 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:29.312747 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.313606 27661 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:29.316108 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.316890 27661 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:29.233656  3402 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.320387 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:29.252799  3522 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.324553  3522 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:29.325186  3431 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.327111 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.293974  3503 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.327754 27657 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:29.329033 27658 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:29.330279  3332 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:29.332034  3525 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:21:29.270282  3414 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:29.253108  3554 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:21:29.334462 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:29.339732 27655 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:29.342402 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:29.358392  3547 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:21:29.360483 27656 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260709 12:21:29.357035 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:29.361040 27656 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 11
I20260709 12:21:29.361370 27655 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:29.361377 27656 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:21:29.361974 27656 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:29.362854  3402 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:29.363516  3150 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.364117  3150 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.368810  3503 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.376780 28331 raft_consensus.cc:2368] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:21:29.387123 27702 leader_election.cc:400] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:21:29.425366 27702 leader_election.cc:403] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:21:29.391875 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.427990 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.392364 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.404532 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.433723 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.434835 28336 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:21:29.435895 27664 raft_consensus.cc:1217] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Deduplicated request from leader. Original: 20.15->[22.16-25.18]   Dedup: 22.16->[23.17-25.18]
I20260709 12:21:29.436698 27664 pending_rounds.cc:85] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 16
I20260709 12:21:29.437692 27664 pending_rounds.cc:107] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:21:29.438854 27664 raft_consensus.cc:2889] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:29.383797 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:21:29.447847 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60943.3us
I20260709 12:21:29.399268 27657 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:29.458137 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:29.396914 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:29.478461 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:29.459379 28334 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:29.491518 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3537) took 1457 ms. Trace:
I20260709 12:21:29.492162 27733 rpcz_store.cc:276] 0709 12:21:28.034496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.041002 (+  6506us) service_pool.cc:224] Handling call
0709 12:21:29.491235 (+1450233us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:29.491435 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414481664}
W20260709 12:21:29.493476 27587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.015s	sys 0.014s
W20260709 12:21:29.493996 27587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.015s	sys 0.014s
I20260709 12:21:29.495493 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.496373 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:29.498791 28335 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:29.502503 27655 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:29.494006 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3582) took 1842 ms. Trace:
I20260709 12:21:29.506445 27656 rpcz_store.cc:276] 0709 12:21:27.651424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.651542 (+   118us) service_pool.cc:224] Handling call
0709 12:21:27.651826 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.362428 (+1710602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.493153 (+130725us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000e4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:29.493927 (+   774us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3928586624}
I20260709 12:21:29.506881 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:29.510229 27730 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:29.428377  3345 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:29.318228  3492 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.622658 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:29.623906 28331 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:29.419166  3244 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:29.498598  3443 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.494786  3521 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:21:29.403234  3402 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:29.644891 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 248048us
I20260709 12:21:29.656698 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:29.348937  3246 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:29.600059  3430 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.710s	user 0.028s	sys 0.079s
I20260709 12:21:29.418294  3241 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:29.653146  3244 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.659752 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:29.668597 28328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.870s	user 0.039s	sys 0.111s
W20260709 12:21:29.669932 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.866s	user 0.013s	sys 0.050s
W20260709 12:21:29.677064 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.014s	sys 0.051s
W20260709 12:21:29.669380 28329 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.007s	user 0.020s	sys 0.022s
W20260709 12:21:29.677423 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.007s	sys 0.038s
W20260709 12:21:29.678696 28329 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.022s	sys 0.022s
W20260709 12:21:29.676600 28328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.039s	sys 0.112s
W20260709 12:21:29.669131 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.999s	user 0.009s	sys 0.062s
W20260709 12:21:29.680115 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.010s	user 0.010s	sys 0.063s
W20260709 12:21:29.678828 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.880s	user 0.007s	sys 0.039s
I20260709 12:21:29.744535 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1299) took 1353 ms. Trace:
I20260709 12:21:29.745265 28328 rpcz_store.cc:276] 0709 12:21:28.391170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.396803 (+  5633us) service_pool.cc:224] Handling call
0709 12:21:28.397245 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.397546 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:28.700531 (+302985us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.744030 (+1043499us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:29.744119 (+    89us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:29.744187 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.744292 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.744456 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18311,"prepare.queue_time_us":462242,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":39,"raft.queue_time_us":65345,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"thread_start_us":1202761,"threads_started":3,"wal-append.queue_time_us":161477}
I20260709 12:21:29.793016 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:29.637717  3549 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:21:29.664970  3430 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.774s	user 0.028s	sys 0.079s
I20260709 12:21:29.667784  3527 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:29.636409  3345 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.664773  3427 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:29.840521  3549 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:29.864003  3228 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.025s	sys 0.027s
I20260709 12:21:29.864351  3427 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:29.866480  3228 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.025s	sys 0.027s
I20260709 12:21:29.866562  3549 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.868306  3427 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:29.869149  3427 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:29.870162  3427 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:29.871668  3427 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:29.862550 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:29.660980  3443 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:29.669025  3402 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.666776  3246 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:29.677235  3241 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:29.883931  3437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s	user 0.033s	sys 0.038s
W20260709 12:21:29.918885  3505 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.706s	user 0.006s	sys 0.006s
W20260709 12:21:29.914815  3470 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.013s	sys 0.028s
W20260709 12:21:30.080261  3470 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.014s	sys 0.029s
W20260709 12:21:29.989584  3437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.034s	sys 0.038s
I20260709 12:21:29.917950  3538 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:30.081677  3538 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:30.082186  3538 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:30.024051  3492 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.009840  3505 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.089459  3505 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:30.051987  3443 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:30.073124  3519 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.000s	sys 0.024s
W20260709 12:21:30.090369  3519 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.000s	sys 0.024s
W20260709 12:21:30.077656  3501 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.015s	sys 0.009s
I20260709 12:21:30.050802  3402 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:30.091532  3501 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.016s	sys 0.009s
I20260709 12:21:30.096971  3538 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:30.146696 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.151708 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.619s	user 0.015s	sys 0.019s
W20260709 12:21:30.153815 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 493985us
W20260709 12:21:30.311765 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.779s	user 0.015s	sys 0.019s
W20260709 12:21:30.311990 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.312319 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.77983e+06us
I20260709 12:21:30.313750 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:30.154155 27583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.024s	sys 0.012s
W20260709 12:21:30.314637 27583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.368s	user 0.025s	sys 0.012s
I20260709 12:21:29.643887 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.315588  3558 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:30.315990 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3734) took 1564 ms. Trace:
I20260709 12:21:30.316329 27584 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:30.316483 27583 rpcz_store.cc:276] 0709 12:21:28.751700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.754723 (+  3023us) service_pool.cc:224] Handling call
0709 12:21:28.755094 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.755353 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:28.945920 (+190567us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.315296 (+1369376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.315429 (+   133us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:30.315521 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.315628 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.315844 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":562629,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":1457,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":308,"thread_start_us":1551606,"threads_started":2,"wal-append.queue_time_us":1087532}
I20260709 12:21:30.318328 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:30.156438 27586 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.041s	user 0.000s	sys 0.021s
W20260709 12:21:30.319589 27586 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.204s	user 0.000s	sys 0.021s
I20260709 12:21:30.319739 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:30.158803 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:30.320966 27583 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:30.321619 27660 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
W20260709 12:21:30.159811 27662 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.004s	user 0.001s	sys 0.001s
I20260709 12:21:30.322978 27583 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
W20260709 12:21:30.323632 27662 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.168s	user 0.001s	sys 0.001s
W20260709 12:21:30.160315 27663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s	user 0.001s	sys 0.001s
I20260709 12:21:29.504724 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:30.325378 27663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.148s	user 0.001s	sys 0.001s
W20260709 12:21:30.326197 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 821302us
I20260709 12:21:30.326996 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3595) took 1181 ms. Trace:
I20260709 12:21:30.164332 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.166529 27585 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:21:30.168540 27588 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:30.176342 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:29.793035 27661 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:21:30.180717 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.651s	user 0.017s	sys 0.092s
I20260709 12:21:30.182096 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1347) took 1078 ms. Trace:
I20260709 12:21:30.183439 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1345) took 2029 ms. Trace:
I20260709 12:21:30.184326 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.185473 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.153839 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:30.209949 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:30.194742 27480 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:30.216964 28332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.100s	user 0.008s	sys 0.026s
I20260709 12:21:30.273252  3345 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.275259  3246 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.277122  3244 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.280253  3444 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:30.259618  3505 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:30.306649  3236 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:21:30.163681 27701 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.008s	sys 0.008s
I20260709 12:21:30.332105  3546 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:30.332593  3225 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:30.335453 27584 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:30.327549 27663 rpcz_store.cc:276] 0709 12:21:29.158065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.158204 (+   139us) service_pool.cc:224] Handling call
0709 12:21:29.171772 (+ 13568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.176670 (+  4898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.176701 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.177200 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.326265 (+1149065us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:30.326394 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.326562 (+   168us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.326646 (+    84us) raft_consensus.cc:1685] finished
0709 12:21:30.326662 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.326902 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17260,"raft.queue_time_us":5391,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":10653,"thread_start_us":1148381,"threads_started":1,"wal-append.queue_time_us":886251}
I20260709 12:21:30.323124 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:30.334522 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:30.335995  2850 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:30.341627  3402 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:30.349139  2988 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.353437  3472 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.356997 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:30.004678  3299 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.021s	sys 0.031s
I20260709 12:21:30.310915  3538 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.328599  3241 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.377842 27656 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:30.316257  3558 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:30.401261 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.007s	sys 0.018s
W20260709 12:21:30.403858 27701 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.093s	user 0.009s	sys 0.008s
W20260709 12:21:30.393059 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.863s	user 0.017s	sys 0.092s
W20260709 12:21:30.402019 28332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.285s	user 0.008s	sys 0.027s
I20260709 12:21:30.382473 27735 rpcz_store.cc:276] 0709 12:21:29.103033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.111808 (+  8775us) service_pool.cc:224] Handling call
0709 12:21:30.181426 (+1069618us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:30.181782 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":459249920}
I20260709 12:21:30.408004 27583 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:30.409585 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:30.410856 27655 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:30.412925 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.008s	sys 0.018s
W20260709 12:21:30.427619 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.004s	sys 0.022s
I20260709 12:21:30.427980 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.428579 27657 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:30.429198 27655 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:30.430016 27655 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:21:30.431682 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:30.428221 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.004s	sys 0.022s
I20260709 12:21:30.394860  3236 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:30.414426 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:30.433185 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98923.3us
I20260709 12:21:30.432400 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:30.397878  3546 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:30.435339 27580 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:30.407546  3558 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:30.427649 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1414) took 1101 ms. Trace:
W20260709 12:21:30.449812 28329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.006s	sys 0.018s
I20260709 12:21:30.447136 27658 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:30.448000 27661 rpcz_store.cc:276] 0709 12:21:29.313053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.314177 (+  1124us) service_pool.cc:224] Handling call
0709 12:21:30.407535 (+1093358us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.443969  3546 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:30.462392 28329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.007s	sys 0.018s
W20260709 12:21:30.471477 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.16051e+06us
I20260709 12:21:30.457214  3345 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:30.461416 27589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.168s	user 0.013s	sys 0.036s
W20260709 12:21:30.471057 27587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s	user 0.019s	sys 0.027s
I20260709 12:21:30.461826  3150 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.383029 27732 rpcz_store.cc:276] 0709 12:21:28.153781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.153909 (+   128us) service_pool.cc:224] Handling call
0709 12:21:28.154158 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.367646 (+213488us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:21:28.669697 (+302051us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.680514 (+1010817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:29.680590 (+    76us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:29.680643 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.182504 (+501861us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54001d10d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:30.182617 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.183055 (+   438us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4442,"prepare.queue_time_us":85840,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":188368,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":490405248,"thread_start_us":1312059,"threads_started":2,"wal-append.queue_time_us":1354825}
W20260709 12:21:30.474756 27582 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.179s	user 0.025s	sys 0.023s
I20260709 12:21:30.475286  3345 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.484849  3345 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:30.477537 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.016s	sys 0.011s
W20260709 12:21:30.484208 27589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.191s	user 0.014s	sys 0.036s
I20260709 12:21:30.489933 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.490958 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3554) took 1962 ms. Trace:
I20260709 12:21:30.491365 27736 rpcz_store.cc:276] 0709 12:21:28.528044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.528152 (+   108us) service_pool.cc:224] Handling call
0709 12:21:28.528416 (+   264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:28.528782 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:28.528821 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:28.529404 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.487241 (+1957837us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:30.490448 (+  3207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.490546 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.490608 (+    62us) raft_consensus.cc:1685] finished
0709 12:21:30.490620 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.490871 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":776,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":1876716,"threads_started":1,"wal-append.queue_time_us":1507399}
I20260709 12:21:30.495422  3414 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:30.496274  3414 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.496927  3439 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.497133 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:30.497913  3439 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:30.502542 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 188187us
W20260709 12:21:30.503435  3299 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.330s	user 0.022s	sys 0.032s
I20260709 12:21:30.504825  3250 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.505852  3250 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.507881 28332 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Committing config change with OpId 20.13: config changed from index 11 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:30.511714  3444 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:21:30.512368 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 302826us
I20260709 12:21:30.514257  3444 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:21:30.516709 27582 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.301s	user 0.025s	sys 0.023s
W20260709 12:21:30.487313 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.020s	sys 0.028s
I20260709 12:21:30.519824 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3609) took 1092 ms. Trace:
W20260709 12:21:30.520823 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 727852us
I20260709 12:21:30.520398 27657 rpcz_store.cc:276] 0709 12:21:29.427668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.457685 (+ 30017us) service_pool.cc:224] Handling call
0709 12:21:30.401256 (+943571us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.429190 (+ 27934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.519531 (+ 90341us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54001e41d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:30.519744 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2138057472}
I20260709 12:21:30.522766  3553 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:30.525063  3553 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:30.525471  3553 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:30.527655  3228 raft_consensus.cc:993] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:30.527556 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:30.528573  3228 raft_consensus.cc:1081] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:30.475090  3538 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:30.529655  3538 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.533468 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3592) took 1390 ms. Trace:
W20260709 12:21:30.489528 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.017s	sys 0.011s
I20260709 12:21:30.517514 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:30.485430 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1434) took 1450 ms. Trace:
W20260709 12:21:30.545876 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46205.8us
I20260709 12:21:30.522684  3219 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:30.568715 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:30.570510 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3738) took 1817 ms. Trace:
I20260709 12:21:30.545076 27662 rpcz_store.cc:276] 0709 12:21:29.143512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.143958 (+   446us) service_pool.cc:224] Handling call
0709 12:21:29.144292 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.144537 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:30.324399 (+1179862us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.532758 (+208359us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.532881 (+   123us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:30.532986 (+   105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.533132 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.533395 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13419,"prepare.queue_time_us":918381,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":13590,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"thread_start_us":1330018,"threads_started":2}
I20260709 12:21:30.578712 27589 rpcz_store.cc:276] 0709 12:21:28.752942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.754428 (+  1486us) service_pool.cc:224] Handling call
0709 12:21:28.754780 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.755034 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:29.292847 (+537813us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.570012 (+1277165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.570112 (+   100us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:30.570192 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.570286 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.570456 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5276,"prepare.queue_time_us":512097,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":3857,"raft.queue_time_us":6321,"thread_start_us":1732920,"threads_started":2,"wal-append.queue_time_us":1279690}
W20260709 12:21:30.582593 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 130424us
I20260709 12:21:30.583674 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:30.586148 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:30.587641 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:30.479557  2850 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.589929 28334 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:30.590884  2850 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.591081 28331 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:30.595803 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.597450 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3776) took 1470 ms. Trace:
I20260709 12:21:30.597870 27582 rpcz_store.cc:276] 0709 12:21:29.127467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.178395 (+ 50928us) service_pool.cc:224] Handling call
0709 12:21:29.214335 (+ 35940us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.214902 (+   567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.214949 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.215650 (+   701us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.584880 (+1369230us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:30.584950 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.585053 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.597218 (+ 12165us) raft_consensus.cc:1685] finished
0709 12:21:30.597238 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.597393 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":3668,"thread_start_us":1323375,"threads_started":1,"wal-append.queue_time_us":1378842}
I20260709 12:21:30.599612  3535 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:30.600682 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.609598 27582 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:30.603796  3236 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.619598  3587 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:30.620540  3236 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:30.620451  3587 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.608228  3454 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:30.623842 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.608536 27630 leader_election.cc:400] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
W20260709 12:21:30.559688 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 659706us
I20260709 12:21:30.608680 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1177) took 2077 ms. Trace:
I20260709 12:21:30.602818  3250 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.610311  3439 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:30.612588 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:30.544008 27586 rpcz_store.cc:276] 0709 12:21:29.034620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.114500 (+ 79880us) service_pool.cc:224] Handling call
0709 12:21:29.114742 (+   242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.114925 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:21:30.320334 (+1205409us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.484788 (+164454us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.484866 (+    78us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:30.484990 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.485095 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.485342 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":250427,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":39,"raft.queue_time_us":7927,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":12039,"thread_start_us":1359530,"threads_started":2}
I20260709 12:21:30.578872 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:30.558339 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:30.630332 27582 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:21:30.477792  3505 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.635308 27584 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:21:30.640376 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.526929  3553 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.643867 27586 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:30.626313 28329 rpcz_store.cc:276] 0709 12:21:28.531160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.531299 (+   139us) service_pool.cc:224] Handling call
0709 12:21:28.654199 (+122900us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.679220 (+1025021us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.679259 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.683377 (+  4118us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.593310 (+909933us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:30.593396 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.593516 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.608417 (+ 14901us) raft_consensus.cc:1685] finished
0709 12:21:30.608441 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.608587 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142761,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":34382,"thread_start_us":1638283,"threads_started":2,"wal-append.queue_time_us":808424}
I20260709 12:21:30.623277  3587 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.489185  3402 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:30.661881 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 138743us
I20260709 12:21:30.668748 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.673914 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:30.681660 27581 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:30.691509  3581 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:30.692487  3581 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.694569 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:30.700112 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184909us
I20260709 12:21:30.703133 28337 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:30.704219  3374 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:21:30.705570 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93200.7us
I20260709 12:21:30.705678  3588 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:30.706348  3588 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:30.707401  3491 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.707674  3588 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.708025  3491 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.710032  3505 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:30.710248  3491 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.711444  3587 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:30.711879  3587 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:30.712144 27704 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:30.712312  3587 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:30.712493 27704 leader_election.cc:403] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:21:30.713877  3376 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:30.715449  3380 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:30.716101  3380 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.718364  3380 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.724888 27480 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 19 to 24. New cstate: current_term: 24 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:30.708274  3549 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.704702  3492 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:30.678613  3414 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.707156  3439 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:21:30.736166  3427 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.818s	user 0.005s	sys 0.019s
I20260709 12:21:30.740473  3250 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:30.761768 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:30.749205  3538 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:30.770699  3402 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.780947 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.754118  3439 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:30.739118  3549 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.784430 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.786293  3402 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.783809  3581 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.784196 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.769027  3553 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:30.783082  3558 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:30.797241  3245 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.857s	user 0.008s	sys 0.012s
I20260709 12:21:30.840977 27588 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:30.840696  3245 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.835990 27662 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 14.
I20260709 12:21:30.798064 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:30.803767  3250 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:30.804378 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:30.804845 27656 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
W20260709 12:21:30.807680 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 166120us
I20260709 12:21:30.844326  3245 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:30.849195 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:30.849529 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51622.9us
I20260709 12:21:30.814458 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.847437 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:30.837167 28337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1304) took 1600 ms (client timeout 1973 ms). Trace:
I20260709 12:21:30.787236  3402 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.825155 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:30.845439 28329 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:30.809067 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.847761  3439 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.763278  3427 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.853776  3402 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.815666  3380 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:30.853077 27657 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:30.846822 27588 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:21:30.855676  3380 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.855535 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:30.856889 28333 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:30.855131  3071 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:30.851281 27584 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:30.858567  3380 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:30.855242  3427 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:30.848169 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.810400 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:30.846545  3250 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:30.860942  3250 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:30.845808  3492 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:30.862993  3250 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.863876 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:30.864638  3250 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:30.850487 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:30.850483 27661 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
W20260709 12:21:30.851490 28337 rpcz_store.cc:269] 0709 12:21:29.237189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.243039 (+  5850us) service_pool.cc:224] Handling call
0709 12:21:30.836820 (+1593781us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:30.837045 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3354515712}
I20260709 12:21:30.816078 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:30.866271 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:30.867621 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:30.825464 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:30.815543 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:30.864758 27732 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:30.859001 27581 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:21:30.837132  3549 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:30.813891  3376 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:30.844026  3558 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:30.907253 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.916862 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66551.3us
W20260709 12:21:30.917461 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.898597 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:30.918105 28333 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:30.869670  3427 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:30.920615 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95345.7us
W20260709 12:21:30.925197 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 251413us
I20260709 12:21:30.809039  3345 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.929279  3596 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:30.939939  3596 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:30.940421  3596 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:30.941357  3345 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.942323 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3740) took 2183 ms. Trace:
I20260709 12:21:30.943570 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:30.942744 27587 rpcz_store.cc:276] 0709 12:21:28.759005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.865674 (+106669us) service_pool.cc:224] Handling call
0709 12:21:28.866033 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.866220 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:21:29.494546 (+628326us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.903494 (+1408948us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.921737 (+ 18243us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.926576 (+  4839us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:30.929942 (+  3366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.942055 (+ 12113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.942242 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1534,"prepare.queue_time_us":615954,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":6597,"raft.queue_time_us":30413,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":5285,"thread_start_us":1830721,"threads_started":3,"wal-append.queue_time_us":1108824}
I20260709 12:21:30.945434 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:30.945840  3505 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:30.946349  3505 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.946699 27585 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:30.947273  3246 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:30.948000  3246 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.926262  3581 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:21:30.950908  3581 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:30.951406  3581 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:30.951872  3581 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:30.926481  3299 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:30.939158 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 355686us
I20260709 12:21:30.933477  3427 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:30.935290 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.959620 27580 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.501s	user 0.003s	sys 0.009s
W20260709 12:21:30.964781 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.968760 27580 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:30.957341  3150 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.970422  3529 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:30.971058  3529 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:30.970826  3150 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:30.971583  3529 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:30.963742  3299 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.974587  3299 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:30.975507 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:30.975562 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:30.906160  3219 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:21:30.978449 27736 raft_consensus.cc:606] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:30.946753  3505 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:30.964143  3380 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:30.930379  3241 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:30.981690  3241 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:30.982995  3241 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:30.987372  3508 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:30.989497 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1415) took 1676 ms. Trace:
I20260709 12:21:30.973944  3246 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:30.994443 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:30.996461 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:30.817408 27480 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 21 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:31.005203 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 227029us
I20260709 12:21:31.007673 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:30.989988 27664 rpcz_store.cc:276] 0709 12:21:29.313343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.435088 (+121745us) service_pool.cc:224] Handling call
0709 12:21:29.435623 (+   535us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:29.521766 (+ 86143us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:30.293510 (+771744us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b180044a880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:30.518829 (+225319us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.905040 (+386211us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:30.985820 (+ 80780us) log.cc:844] Serialized 52 byte log entry
0709 12:21:30.985987 (+   167us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:30.986021 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.986138 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.988933 (+  2795us) raft_consensus.cc:1685] finished
0709 12:21:30.989076 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.989275 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116110,"prepare.queue_time_us":543550,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":232517,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":14856,"spinlock_wait_cycles":8933120,"thread_start_us":465719,"threads_started":2,"wal-append.queue_time_us":401359}
I20260709 12:21:31.035086 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1303) took 1966 ms. Trace:
I20260709 12:21:31.004108  3150 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.035387 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.037963 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.039575 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.035665 28332 rpcz_store.cc:276] 0709 12:21:29.068221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.068385 (+   164us) service_pool.cc:224] Handling call
0709 12:21:29.069133 (+   748us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:29.081510 (+ 12377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.081558 (+    48us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:29.116504 (+ 34946us) log.cc:844] Serialized 608 byte log entry
0709 12:21:30.507512 (+1391008us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:30.731789 (+224277us) log.cc:844] Serialized 14 byte log entry
0709 12:21:31.004651 (+272862us) log.cc:844] Serialized 14 byte log entry
0709 12:21:31.004972 (+   321us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.026404 (+ 21432us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:31.026452 (+    48us) raft_consensus.cc:1685] finished
0709 12:21:31.026466 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.035022 (+  8556us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6014,"prepare.queue_time_us":88451,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":46,"raft.queue_time_us":26747,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":18598,"thread_start_us":1544339,"threads_started":2,"wal-append.queue_time_us":941331}
I20260709 12:21:31.037400 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:31.039358 27701 connection.cc:256] RPC call timeout handler was delayed by -1.1614704729997811s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
I20260709 12:21:31.042716 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.043871 27657 raft_consensus.cc:2368] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:21:31.046834 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.047608 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.048094  3376 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.048516 27661 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:31.049045  3376 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.041951 27656 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:31.072252  3004 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:31.073211  3004 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.074584 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:31.075384 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:31.079764 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:31.081089 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:31.082602 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.083429 27736 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:31.086182  3228 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:31.087869  2667 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:31.088351  2667 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:31.089437  2667 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.094000 27730 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:21:31.095651 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.096297 27663 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:31.097062 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.099822 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.101334 27662 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:21:31.101760 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 250528us
I20260709 12:21:31.102381  2667 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:31.103137 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.103341 27480 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 11 to 22, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 22 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.104727 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:31.106527 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:31.106768  3508 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:31.107465 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.109899  3529 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.110739 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.111451 28330 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:31.111606 27580 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:21:31.112268 27580 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:31.112651 27580 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:21:31.117013 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85218.1us
I20260709 12:21:31.117702 27732 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:31.040740 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.122151 27734 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:31.122236 27661 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:31.106873 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:31.135035 27659 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:31.142709 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:31.128661  3546 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.602s	user 0.007s	sys 0.001s
I20260709 12:21:31.148187 27733 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:31.146693 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:31.121835  3071 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:31.148980 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.156939 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.136785  3491 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.152367 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:31.159127 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:31.155293  3236 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.534s	user 0.004s	sys 0.008s
I20260709 12:21:31.167744 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:31.170601 27585 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:31.169312 27588 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:31.172695 27732 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:31.173940 27736 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:31.175789  3201 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:21:31.176469  3201 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:31.177139  3201 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:21:31.178268  3243 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:21:31.178727  3243 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:31.179438  3588 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:21:31.179557 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72824.4us
I20260709 12:21:31.188090  3376 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.191488  3349 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:21:31.168634 27580 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.192340 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.155709  3546 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.192586 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.213635 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:31.179119  3243 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:21:31.303562 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 197156us
I20260709 12:21:31.323686 27730 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:31.243883 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.274117  3201 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.336016  3546 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.336803  3201 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.238382  3529 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.344575 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.346071 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.346865 27730 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:31.348680 27583 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:21:31.315632 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.316423 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.381083 27732 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:31.312229 27733 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:31.229501 27589 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:31.192764  3538 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:31.164978  3587 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.168747  3414 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:31.261060  3581 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.331523  3376 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.401152 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.401763 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:31.409457 27580 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:31.417551  3581 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.220337  3491 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.336839  3427 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.395309 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.419883 27735 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:31.282267  3553 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260709 12:21:31.420725 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.424481 27732 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
W20260709 12:21:31.424602 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 271653us
I20260709 12:21:31.378214  3374 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:31.430500 27552 connection.cc:256] RPC call timeout handler was delayed by -2.4015575740002078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:21:31.432695 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3061575740000535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 12:21:31.433089 27552 connection.cc:256] RPC call timeout handler was delayed by -2.270688396000196s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
W20260709 12:21:31.433384 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1540575740000349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20260709 12:21:31.433681 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1126575740001954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260709 12:21:31.434023 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1050883960001556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20260709 12:21:31.434374 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0843883960001222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20260709 12:21:31.434609 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0046575740000208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20260709 12:21:31.434844 27552 connection.cc:256] RPC call timeout handler was delayed by -1.9934355180000694s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17090000000000002s for a response
W20260709 12:21:31.435091 27552 connection.cc:256] RPC call timeout handler was delayed by -1.9875575740002205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
W20260709 12:21:31.435374 27552 connection.cc:256] RPC call timeout handler was delayed by -1.96913551800003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
W20260709 12:21:31.435899 27552 connection.cc:256] RPC call timeout handler was delayed by -1.9430883960001211s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
W20260709 12:21:31.436218 27552 connection.cc:256] RPC call timeout handler was delayed by -1.8917883960000381s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20260709 12:21:31.436626 27552 connection.cc:256] RPC call timeout handler was delayed by -1.7580575740000768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2215s for a response
W20260709 12:21:31.437005 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6295069539999076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
W20260709 12:21:31.437429 27552 connection.cc:256] RPC call timeout handler was delayed by -1.594871742000123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
I20260709 12:21:31.437741 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:31.437968 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5780126870001823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2126s for a response
I20260709 12:21:31.438782 27586 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:31.429217 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 248851us
I20260709 12:21:31.377707  3546 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:31.391796  3538 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:31.395972  3414 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.440184 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:31.438800 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4340575740000077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2575s for a response
W20260709 12:21:31.447116 27552 connection.cc:256] RPC call timeout handler was delayed by -1.28212768100002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
W20260709 12:21:31.447638 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2675575740001932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27599999999999997s for a response
I20260709 12:21:31.446583 27738 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
W20260709 12:21:31.456000 27552 connection.cc:256] RPC call timeout handler was delayed by -1.126323143000036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24780000000000002s for a response
I20260709 12:21:31.448127  3538 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:31.447778  3414 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:31.467665 27704 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:31.472671 27704 leader_election.cc:403] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:21:31.482647 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.483734 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.472745 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.457566  3588 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:31.443997  3376 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.471992  3529 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:31.520362 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.520784 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.521275 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.516770 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:31.518096 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:31.452886  3374 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.462707  3546 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.525072 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:31.529518 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 133657us
I20260709 12:21:31.523857 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.534430 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:31.517939  3505 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.536s	user 0.014s	sys 0.000s
I20260709 12:21:31.535811 27662 raft_consensus.cc:2368] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:21:31.543915 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.544204 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.544675 27660 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:31.545099 27656 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:31.548035 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.548709 27661 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:31.551618  3376 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.553920 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:31.555074  3614 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:31.555757  3614 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.556255  3614 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:31.556468 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:31.557797  3427 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:31.559919 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.520839  3246 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:31.531081  3228 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:31.565099 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.566267 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.585886 27664 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:31.598688 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.588891 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.603087 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:31.588478  3596 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.647s	user 0.009s	sys 0.007s
I20260709 12:21:31.601118  3553 raft_consensus.cc:993] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:31.546854  3505 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:31.610347 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 285522us
I20260709 12:21:31.611454 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.613628 27659 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:31.611836 28332 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:21:31.613811 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:31.614147 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:31.617576 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.619810 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.622669 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:31.603245  3587 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.631590 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:31.638612 27734 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:31.602104  3004 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:31.606264  3617 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:31.623427  3553 raft_consensus.cc:1081] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
W20260709 12:21:31.649143 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 317950us
I20260709 12:21:31.632802  3250 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:31.622335  3246 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 12:21:31.668834 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.665756 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.670187 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:31.667923 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81200.7us
I20260709 12:21:31.672370 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.656177  3491 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:31.676406 27662 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:31.612782  3596 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.655748  3427 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:31.675783 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:31.677757 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 120348us
I20260709 12:21:31.682056 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.683981 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.684847 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.686156 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.687597 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.688469 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:31.690030 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.700938 27657 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:31.700956 27663 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:31.707736 27661 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:31.709131  3596 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:31.709355  3250 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:31.709967  3250 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.711490 27658 raft_consensus.cc:2376] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22. Re-sending same reply.
I20260709 12:21:31.712028  3250 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:31.712570 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.713310 27656 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:31.713402  2667 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.610s	user 0.009s	sys 0.005s
W20260709 12:21:31.714512  3615 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
W20260709 12:21:31.715569  3613 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260709 12:21:31.716725  3616 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260709 12:21:31.718175  3618 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:21:31.723361 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.723634 27661 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:31.724319 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:31.728195 27585 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.553s	user 0.008s	sys 0.004s
I20260709 12:21:31.728564 27585 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:31.731048 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.738253 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.739706 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:31.737404 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:31.635748  3588 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.741341 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.763538  2667 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:21:31.764652 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 291980us
I20260709 12:21:31.729768  3508 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:31.764820 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:31.732973  3552 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:31.726078  3529 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:31.771852 28333 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:31.772651 27736 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:31.546239 27480 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 10 to 12, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 19 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:31.775532 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 221734us
W20260709 12:21:31.777477 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 160032us
I20260709 12:21:31.777740 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:31.768574  3299 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.793s	user 0.012s	sys 0.003s
W20260709 12:21:31.779119 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52940.1us
I20260709 12:21:31.732142  3241 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:21:31.754066  3620 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:31.785280 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:31.787423 27664 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:31.789601 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.790216 27662 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:31.796518 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.791891 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:31.799968  3376 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:31.800621  3376 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:31.801839  3376 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.807869 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:31.809535  3617 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:31.809588 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:31.813454  3553 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:31.813953  3553 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:31.814399  3553 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:31.780385  3546 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:31.820302 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:31.822039  3609 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:31.822109  3241 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:31.828903  3241 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:31.829818  3241 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.789291  3508 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:31.825699  3630 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:31.831444 27733 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:31.831768  3630 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:31.832370  3630 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:31.832500 27587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:21:31.833128 27587 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:31.833496 27587 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:31.833619 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:31.834226 27586 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:31.822721 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.828167 27480 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 18 to 19, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 19 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:31.836216 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:31.836777 27584 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:31.823825 27582 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:31.789898  3299 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.841741  3299 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.844426 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:31.844419 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:31.845944 27663 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:31.852504 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.853169 27729 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:31.854676 27663 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:31.827530  3623 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:21:31.828150 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44666.5us
I20260709 12:21:31.858282 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:31.831807  3627 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:21:31.872380 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74832.1us
I20260709 12:21:31.822506 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.831071  3241 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:31.847179  3228 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:21:31.789636  3374 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.824055 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:31.834139 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.875281 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:31.876067 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:31.824139  3505 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.793401  3622 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:31.878921 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:31.889912 27586 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:31.880765 27587 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.899506  2667 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:31.901551 27580 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:31.878659  3546 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:31.875949  3508 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:31.907505 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 83530.3us
I20260709 12:21:31.908486 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.909077 27731 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:31.910578 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:31.911708 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:31.873271  3529 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:21:31.916471  3508 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:31.919476  3243 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.002s
I20260709 12:21:31.919615  3635 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
W20260709 12:21:31.920441 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45306.8us
I20260709 12:21:31.954167 27730 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:31.921049 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3748) took 1582 ms. Trace:
I20260709 12:21:31.954736 27583 rpcz_store.cc:276] 0709 12:21:30.338520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.623537 (+285017us) service_pool.cc:224] Handling call
0709 12:21:31.920811 (+1297274us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54000f00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:31.920938 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1664461568}
I20260709 12:21:31.921269 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:31.955729 27585 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:31.907521  3631 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:21:31.965200 27701 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:31.922117 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.966041 27732 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:31.925844  3630 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.926728  3596 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:21:31.927248  3611 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.440s	user 0.000s	sys 0.005s
I20260709 12:21:31.929951 28335 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:21:31.937635  3345 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:21:31.931814 27584 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:31.913990  3620 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:31.932497  3414 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:31.922932  3553 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.937634 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:31.982416 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44864.1us
I20260709 12:21:31.938017 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:31.984293 28333 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27.
I20260709 12:21:31.939821 27731 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:31.985780 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:31.940339 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 216161us
I20260709 12:21:31.941020 27587 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:31.941846 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:31.947978 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:31.948163 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:31.951499  3629 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:31.990774 27733 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
W20260709 12:21:31.976341 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 101635us
I20260709 12:21:31.976912 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.991652 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:31.906546  3546 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:31.930881  3614 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:31.987437 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:31.990394 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:31.992517 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44615.8us
I20260709 12:21:31.992857 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.968669  3596 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.995820 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:31.997077 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:31.997318 28334 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:31.998324 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.004156 27738 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:21:32.000128  3546 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:32.008312  3427 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:31.980384  3345 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:31.982688  3620 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:32.027591 27736 raft_consensus.cc:606] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:32.028291 27736 raft_consensus.cc:614] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:32.028999 27737 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:32.014007 27731 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:21:32.017546 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 173255us
I20260709 12:21:32.018652 27729 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:21:32.019286 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71203.5us
I20260709 12:21:32.010937  3414 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.022897 27732 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:32.023238 27553 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27.
I20260709 12:21:32.031404 27553 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:21:32.025619  3614 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:32.027195  3427 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:32.031667 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:32.052028 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49424.9us
I20260709 12:21:32.033329  3627 raft_consensus.cc:2764] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election decision vote started in defunct term 27: won
W20260709 12:21:32.037855 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50763.5us
I20260709 12:21:32.038452  3646 raft_consensus.cc:2764] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260709 12:21:31.999966  3626 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:32.043278 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:32.045029  3553 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:32.048058  3630 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:31.997298  3581 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:32.020042  3558 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:21:31.964731  3243 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.064714  3635 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:32.068245  3635 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.068605  3635 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:32.069258  3243 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:32.035606  3424 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:32.069856  3299 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:21:32.070964  3299 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:32.071288  3299 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:32.071681 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76209.1us
I20260709 12:21:32.072017  3581 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.072190  3299 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.073012 27701 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:21:32.073519  3299 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:32.058990  3617 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 12:21:32.075163 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.075480 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.065403  3626 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:32.076076 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.077450  3342 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:32.077028  3626 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.19, Last appended by leader: 19, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.077847  3342 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:32.077937  3631 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:32.078261  3342 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:32.078555  3631 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.079052  3631 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:32.079401  3617 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:32.052353 27585 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:32.055671 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:32.055950 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.020148  3596 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:32.087764 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:32.061730 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:32.056901 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:32.090813 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:32.091231 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.091846 27662 raft_consensus.cc:2376] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25. Re-sending same reply.
I20260709 12:21:32.089856 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.093381 28335 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:32.095296 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:32.099720  3424 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:32.100276  3424 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:32.090739  3630 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:32.101195  3424 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:32.103768 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50130.9us
I20260709 12:21:32.104452  3491 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:32.104959  3491 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.105880  3491 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.107506  3529 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:32.108414  3529 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:32.110363  3553 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:32.111263  3553 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:32.111630  3553 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:32.112634 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.112499  3553 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.113205 27660 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:32.113714  3553 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:32.116669 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.117271 27663 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:21:32.117314 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 127031us
I20260709 12:21:32.119928 27655 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:32.107690 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:32.100289 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.119874 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.121169 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.123450 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:32.124833 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:32.135680 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 106165us
I20260709 12:21:32.136422 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:32.087939  3246 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27.
W20260709 12:21:32.094327  3538 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.609s	user 0.005s	sys 0.006s
I20260709 12:21:32.089888  3201 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:21:32.150797 27663 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:21:32.151209 27661 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
W20260709 12:21:32.166779 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91386.1us
I20260709 12:21:32.168361 27701 leader_election.cc:400] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:21:32.169447 27655 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:21:32.170532 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:32.171535  3641 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.109s	user 0.002s	sys 0.000s
I20260709 12:21:32.174576 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:32.175661  3582 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:32.176370  3582 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.177500  3582 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.180148  3004 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:32.180902 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:32.181671  3004 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:32.182176  3004 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.184660 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.174568  3345 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.185608  3508 raft_consensus.cc:2764] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:21:32.186358  3345 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:21:32.154659  3546 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:32.189083 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69357.4us
I20260709 12:21:32.127586  3236 raft_consensus.cc:2764] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:21:32.146579  3558 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:32.191756  3558 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:32.192564  3546 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.192674  3558 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.155488 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:32.151436  3538 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.197017 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.197403 28329 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:32.197829 27658 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:32.198433  3538 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.201339  3588 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:32.202875  3588 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:21:32.073074  3581 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.203703  3588 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:32.204259  3588 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:32.205045  3588 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.206193  3588 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:32.145769  3228 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:32.190996 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.225111 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:32.227451 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:32.231997 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81842.1us
I20260709 12:21:32.234867  3228 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:32.244191 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97234.2us
I20260709 12:21:32.244400  3228 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.149534  3342 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.249472 27585 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
W20260709 12:21:32.252782 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 146110us
I20260709 12:21:32.250653 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.250034 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.253619 27584 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:32.244426 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:32.254933 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95547.6us
W20260709 12:21:32.257007 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.234681  3546 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:32.231515  3630 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:32.268343 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.270427 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.271307 27587 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:32.279018 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88165.4us
I20260709 12:21:32.280949 27584 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
W20260709 12:21:32.284555 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 110070us
I20260709 12:21:32.278959  3629 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:32.298240  3376 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.280478  3538 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:21:32.183192  3004 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.313269  3004 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:32.314322  3620 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.315375 28335 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:32.263284  3250 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:21:32.317613 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.310576 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.311565  3538 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.311697  3644 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.329037  3655 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:32.328743  3644 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.330277 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:32.312120 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67772.2us
I20260709 12:21:32.331568 27580 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:32.331969 27589 pending_rounds.cc:85] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 11
I20260709 12:21:32.317077  3636 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 12:21:32.332293 27589 pending_rounds.cc:107] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 12:21:32.332615 27589 raft_consensus.cc:2889] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:32.333103  3636 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:32.321066  3376 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.333600  3636 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:32.323606 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:32.307624  3552 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:21:32.334424 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.323819  3549 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 12:21:32.335242 27584 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:21:32.336011  3653 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:32.308347  3629 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.319008 27660 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:32.336961  3629 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:32.337363 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.307853  3241 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:21:32.338254  3241 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:32.338590  3241 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:32.339195  3345 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:32.339390  3241 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.293939  3427 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.340829  3241 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:32.340946 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.341768  3596 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.342536  3427 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:32.328687  3538 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.335095  3552 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:21:32.343755 27585 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:21:32.336520 27583 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 12:21:32.333029 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.298532  3635 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.345194 27657 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:32.324864  3651 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:32.331823 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.343652  3552 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.345963  3651 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:32.346864  3552 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.347577  3427 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:32.347306  3651 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.335261  3376 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:32.347263 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:32.348908  3552 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:32.350848  3342 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.350919 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.351943 27582 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:32.345885 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.361838 27588 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:32.373951 27658 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:32.375603 27662 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:32.345168  3596 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.377974 27660 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:32.334972  3631 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.396667 27587 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:32.396441 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.399106 28333 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
W20260709 12:21:32.400799 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59965.6us
I20260709 12:21:32.374115  3376 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:32.402283  3620 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.409992  3631 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.414194  3619 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.414700  3236 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:32.415462 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.447430 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.444278  3529 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.450181  3529 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.446745  3546 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:32.452131  3644 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.457670  3635 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:32.461227 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.462515 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:32.464210 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:32.467546  3655 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:32.468261 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:32.468434  3655 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:32.468781  3655 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:32.468918 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:32.416846 27587 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:32.418954 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:32.427894 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:32.430120 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:32.471146 27661 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:32.432232 27730 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:32.432823 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:32.432483  3636 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.432971 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.433480 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:32.471810 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:32.472309 27731 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:32.434212 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:32.435041 27733 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 26 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:21:32.434947 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:32.439394 27736 raft_consensus.cc:606] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.474977 27736 raft_consensus.cc:614] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:32.415498  3380 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:21:32.474486 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.475268 27582 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:32.475277 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.482748 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:32.421485  3376 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.483227 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.469715 27732 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Committing config change with OpId 20.13: config changed from index 11 to 13, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:32.484108 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:32.485970 27630 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:21:32.486442  3654 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:32.487149 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:32.485973  3635 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 12:21:32.489915  3635 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:32.490759  3635 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:32.491266 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.491751 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.499289 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.500113 27662 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:32.501938 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.499711 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:32.493232 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:32.503835 27657 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:32.504953 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:32.491145  3635 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:32.505558 27656 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:32.509295 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:32.508184  3626 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:32.517813 27583 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [22.13-24.14]
I20260709 12:21:32.520679 27657 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:32.519106 27583 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:21:32.493121  3614 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:32.493685  3376 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:32.549002 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79022.2us
I20260709 12:21:32.521040 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:32.555699 27587 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:32.559942 27581 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:32.560278 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:32.445910  3619 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:32.564200 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.576428 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:32.524782  3636 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:32.581681 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.551117  3651 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:21:32.561750  3654 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.017s
W20260709 12:21:32.583235 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 137783us
W20260709 12:21:32.582366 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 166964us
I20260709 12:21:32.589550 27738 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:32.593885 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:32.594385 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:32.538091  2667 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:32.585393 27659 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:21:32.587507 28331 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:32.597019  2667 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:32.598019  2667 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.600016 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:32.495045  3629 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.605746 27664 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:21:32.580958  3620 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:32.565451  3614 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:32.609174 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.613431 27587 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:21:32.613874 27581 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:32.616078 27552 leader_election.cc:400] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:21:32.617872 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:32.619155 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.593691  3631 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:32.622797  2667 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:32.625186 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:32.625855 27656 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:21:32.626572 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:32.630455 27734 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:32.615404  3505 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:32.635846  3529 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:32.644210 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.645267 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:32.622988  3620 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.623345  3654 consensus_queue.cc:1243] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:21:32.631711  3243 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:32.631963  3582 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:32.536690  3655 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.656266 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.635860  3643 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:32.659422  3624 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:21:32.660310  3656 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:21:32.659570  3614 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.659448 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
W20260709 12:21:32.660989  3661 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:21:32.637182  3505 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:32.668375 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:32.642650  3631 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.669718 27736 raft_consensus.cc:606] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:32.670594 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:32.623946  3635 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.677598 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:32.655055  3670 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.675390 27736 raft_consensus.cc:614] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:32.653247  3588 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:21:32.655767  3243 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:32.680477 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97643.7us
I20260709 12:21:32.682585 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:32.662840  3643 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:32.675218  3582 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:32.682451  3424 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:32.673202  3228 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:32.675806  3505 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:32.672375  3654 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:32.681458  3655 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:32.695732  3617 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:32.686247  3243 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:32.725548  2667 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:32.738034  3617 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.809831  3617 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:32.812191  3674 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
W20260709 12:21:32.813964  3669 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.121s	user 0.001s	sys 0.001s
I20260709 12:21:32.815692  3643 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.818863  3631 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.819446 28331 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 20. (index mismatch)
I20260709 12:21:32.822345  3617 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.823200  3620 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:32.823788 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97222.7us
I20260709 12:21:32.825488  3635 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:32.826748  3629 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:32.687142  3670 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.837755  3553 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:32.841284  3619 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:32.872977  3553 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:32.875428  3553 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:32.810801  3582 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:32.883456 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:32.894191 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211660us
I20260709 12:21:32.895467 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:32.908772  3376 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:32.918720 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 197791us
I20260709 12:21:32.954414 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:32.955241 28331 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:33.016543  3670 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.022058  3243 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:33.027618 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 132347us
I20260709 12:21:33.028513 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.026690 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.031246 28331 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:32.920435  3651 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:21:33.068261 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 342935us
W20260709 12:21:33.072594 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44194us
I20260709 12:21:33.073706 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:33.067890 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.024053  3376 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.082290 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:33.037431  3241 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:33.093075 27582 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:21:33.096312 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:33.031414  3246 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:33.098819 27661 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:33.066632  3655 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:33.101841 27587 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:33.101941 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:33.109362 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.083643  3671 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:33.114524 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:33.115305 27655 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:33.095523  3651 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:21:33.119175 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.109057  3246 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.131938  3246 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:33.135675 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 255880us
I20260709 12:21:33.136637 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:33.137619  3687 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:21:33.139859 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:33.140471 27658 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:33.141026 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.145171 28331 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:33.145624 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.104302  3617 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.079389  3424 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
W20260709 12:21:33.111199  3636 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.523s	user 0.010s	sys 0.000s
I20260709 12:21:33.113147  3671 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.093153  3546 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.053s
I20260709 12:21:33.112104  3655 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:33.159463 27658 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:33.152953 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.147678  3626 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:33.165472  3682 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:33.166312  3682 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:21:33.179010  3644 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:33.179545  3644 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:33.162968  3241 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.180634  3644 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.182206  3644 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:33.155144  3671 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:33.179611  3685 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:21:33.180656  3241 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.191466  3241 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.176736  3246 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:21:33.210645 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 115504us
I20260709 12:21:33.201762  3241 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.188571  3505 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.219969 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.221211  3694 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:33.222900 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.222783  3651 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:33.259378 28333 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:33.236510  3241 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.260864 28335 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:33.261646  3241 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.263990  3505 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.264770  3243 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.229750  3655 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.226750  3529 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:33.269847 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:33.231623  3508 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:33.235637 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.237485  3636 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:33.240944 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:33.242817  3588 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:33.243427  3670 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:33.243746  3228 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:33.245710 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:33.288638 27588 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:33.246445 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.289739 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:33.247493 27580 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:33.248104 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.292961 27584 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:33.250471 27585 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:21:33.250939  3546 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:33.294051 27585 pending_rounds.cc:85] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:33.296620 27585 pending_rounds.cc:107] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:21:33.296909 27585 raft_consensus.cc:2889] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:33.251579 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:33.252276 27589 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:33.254935  3626 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:33.298908 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47429.4us
I20260709 12:21:33.299137  3626 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.255565  3694 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:33.310237  3546 raft_consensus.cc:2764] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:21:33.317601 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.259397  3692 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:33.331107  3692 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:33.289023  3619 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:21:33.332254  3692 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.333833  3692 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:33.338430  3250 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:33.338935  3250 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:33.347853  3689 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:21:33.348361  3694 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:21:33.348234 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69848.9us
I20260709 12:21:33.349340  3694 consensus_queue.cc:1243] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:21:33.351926  3250 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.353446  3250 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:33.353929  2667 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:33.352051 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.355937  3380 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:33.356339  3380 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:33.356750  3380 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:33.357334 27584 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:21:33.359901 27733 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:33.361234 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:33.361860 27737 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:33.319201 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.278836  3241 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:33.287917  3588 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.321907 27655 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:33.322522  3670 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.279354  3636 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.381465 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:33.289901  3695 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
W20260709 12:21:33.370043 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.376792 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:33.378662 27582 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:21:33.385408  2667 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:33.396275  3004 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:21:33.396770 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:33.397385 27657 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:21:33.397524 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:33.397845 27657 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 13
I20260709 12:21:33.398126 27657 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:21:33.398368 27657 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:33.410772 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:33.413596 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 174352us
I20260709 12:21:33.416373 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.415102 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:33.417979 27581 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:21:33.418552 27582 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:33.375921  3243 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:33.384858  3491 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.421676 27582 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:33.428587 27582 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:33.423257 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:33.426131 27655 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:33.385572  3546 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:33.430611 27655 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:33.429498 27582 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.434481 27582 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:33.431299  3243 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.410223  3201 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 12:21:33.448764 27655 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.450307 27655 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:33.463835 27655 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:21:33.464373 27588 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:21:33.476982 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.469871 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.488355 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:33.491312 27582 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 16. (term mismatch)
I20260709 12:21:33.491750 27582 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:33.491989 27582 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:21:33.492229 27582 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:33.493921 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 176393us
I20260709 12:21:33.417596  3631 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:21:33.469372  3619 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:33.509997 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.510495 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.513450 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.434432  3491 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.508025  3228 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:33.540534 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.533767  3617 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.545114  3617 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.500999  3505 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:21:33.515005  3619 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:33.579313 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.578970  3376 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:33.589087  3376 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.592507  3670 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.557677  3508 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:33.593802  3508 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:33.594326  3508 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:33.528475  3546 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:33.602780  3617 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.603495  3653 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:33.606343  3491 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:33.607606  3380 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.538849  3696 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:33.621408 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.622310  3376 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.622637 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.623600  3619 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.628823 27589 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:33.631163 27737 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:21:33.639954 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 129823us
I20260709 12:21:33.658660 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:33.660092 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.632651  3653 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:33.638671  3696 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:33.638847  3380 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.657673  3546 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:33.703116 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 162707us
I20260709 12:21:33.677310  3491 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.648677  3617 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:21:33.709836 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.658035  3670 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:21:33.682132  3709 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.648627  3708 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:21:33.680190  3653 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.717260  3246 raft_consensus.cc:2764] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:21:33.713114  3491 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.721757  3491 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.723954  3653 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.724833 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.725512 28335 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:33.726045 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.726509 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.726841 28329 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:21:33.728339  3701 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:21:33.728554 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:33.729967  3706 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:21:33.730888  3246 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:33.685148  3694 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:33.748785 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:33.718086 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.752197 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.754686 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:33.705670  3546 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.747506  3246 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:33.748078  3614 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.712896  3697 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.112s
I20260709 12:21:33.789333 27656 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260709 12:21:33.757589  3671 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.559s	user 0.009s	sys 0.002s
I20260709 12:21:33.790632  3671 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.792431  3626 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.767699  3711 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:33.793009  3626 raft_consensus.cc:2764] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:21:33.793399  3711 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:33.793617 27664 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:33.794330  3711 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.796465 27663 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:21:33.753763  3694 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.797573 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.798255 27658 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:21:33.799016  3694 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.780530  3553 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:33.801398 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.802126 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:33.804494 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:33.806048  3529 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:33.806813  3529 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:33.810698  3692 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:33.811100 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.811213  3692 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.813563  3692 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.815212 27655 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:21:33.815510  3671 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.815791  3243 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.816210  3671 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:33.771775  3246 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:33.818830  3588 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.819465  3672 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.820075  3672 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.822592  3672 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.782243 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:33.785655  3635 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
W20260709 12:21:33.786307  3710 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:21:33.786760  3643 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:33.754926  3655 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:21:33.763573  3700 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.262s	user 0.001s	sys 0.001s
I20260709 12:21:33.771564  3696 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.782171  3546 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.831478 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:33.876631 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65391.7us
I20260709 12:21:33.807174  3529 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:33.850086 28331 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:33.884441 27657 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:21:33.731992  3709 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.826689 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 15.
I20260709 12:21:33.888513 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:33.889185 27585 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:21:33.879552 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.891809 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:33.893401 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:33.880087  3694 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:33.896337 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.899690 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:33.895370  3694 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.878291  3655 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:33.905258  3655 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:33.905939 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.904757  3694 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:33.912750  3636 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:33.840150  3614 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.923245  3508 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.924643 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.934590  3376 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:33.935084 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.935141  3376 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:33.935873 27582 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:33.936432  3552 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:21:33.936844  3552 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.935707 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:33.926863  3646 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:33.937407  3552 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:33.930426  3672 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.936765 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.938848  3672 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.939275 27582 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:33.940310 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.940944  3672 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.937470  3376 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:33.941458  3671 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:33.942116  3671 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.942472  3671 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:33.948362 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.949128 27582 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:33.949723 27658 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:21:33.950305  3694 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:33.950747 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.951807 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.953229 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.954569 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:33.926381  3614 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:33.956418 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.958700 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.960119  3376 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:33.960386 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.964928 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:33.976433 27658 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:33.976683 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:33.977893 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44121.4us
I20260709 12:21:33.980669 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:33.972919 27582 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:33.979758 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:33.965823 27585 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:33.973768 27657 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:33.989516 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.989959 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:33.995496 27585 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:21:33.999720 27661 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:34.000856  2667 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:34.001472  2667 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:34.002410  2667 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.022619 27657 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:21:34.033448 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:33.976032  3380 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:34.039683  3505 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:33.998188  3553 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:33.998953  3708 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.021263  3672 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.035354  3299 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:34.061950 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:34.038177  3380 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.998234  3004 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 12:21:33.998039  3201 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:21:33.998340  3631 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:21:34.037061  3643 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.036791  3646 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.005157  3623 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:21:33.985625  3376 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.071733  3380 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:34.058477  3709 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.058298  3553 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:34.028003  3619 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:34.061599  3632 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:21:34.079592  3614 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:34.131948  3614 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.132360  3614 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:34.132460  3596 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:34.133030  3596 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.074942  3643 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.063237  3708 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.064527  3228 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:21:33.938480 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:34.118916  3529 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.124359  3508 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.125624  3671 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.072026  3672 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.097512  3491 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.103617  3655 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.130391  3694 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.131623  3552 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.105087  3246 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.139506 27585 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:21:34.139536 27584 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:21:34.164938 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:34.135149  3596 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.102738  3553 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:34.127059  3682 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:34.135278  3299 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:34.167495  3299 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:34.168705  3682 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.173722 27582 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:34.174626 27580 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:34.180356  3349 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.181134  3349 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.186306  3349 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.186789 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.188016 27580 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:34.079630  3646 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.222142 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.222784 27735 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:34.209172 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 270761us
I20260709 12:21:34.223732 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.225206 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:34.212127 27732 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:34.213618 27737 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:21:34.218185 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.219547 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:34.227421 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.227573 27738 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:34.220049 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.228681 27731 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:34.231807 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.232663 27584 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:34.234694 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:34.236240 27588 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:34.240303 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:34.252593 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:34.253306 28329 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:34.291770 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.232504  3709 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:34.232326  3670 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:34.163986  3491 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.236071  3694 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:34.236073  3636 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:34.253703  3626 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.252663  3617 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:34.191643  3619 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.326169  3709 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:34.363382 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.315482  3529 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:34.350870 27736 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:34.353120 27735 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:34.314863  3635 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:34.358120  3619 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.395057  3635 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.396317  3646 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.396948  3646 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:34.397485  3619 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.397723 27734 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:21:34.398595  3635 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.399171 27735 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:34.401126 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.401859  3553 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.403245 27735 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:21:34.404048  3655 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.404474 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.405287 27736 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:21:34.345634  3631 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.407878  3631 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.408684  3619 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:34.409747  3619 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.410112  3619 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:34.347600  3376 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.345037  3694 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.413620  3552 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.414093  3694 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.351871  3626 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.336841  3670 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.359879  3709 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:34.359577  3617 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.425410  3617 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.393292 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.428798 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:34.429829 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.402236 27738 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:34.430389  3626 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:34.328284  3711 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.009s	sys 0.015s
I20260709 12:21:34.419020 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:34.444909 27588 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:21:34.422039  3670 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.427175  3299 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.402107 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:34.430447 27584 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:34.440189 27582 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:34.446391 27729 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
I20260709 12:21:34.455616 27729 pending_rounds.cc:85] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 16
I20260709 12:21:34.455960 27729 pending_rounds.cc:107] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:21:34.456254 27729 raft_consensus.cc:2889] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:34.458081 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.018s	sys 0.024s
W20260709 12:21:34.458482 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.018s	sys 0.024s
W20260709 12:21:34.444500  3711 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.010s	sys 0.015s
W20260709 12:21:34.483618 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.355170  3636 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.527763 27588 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:34.530530 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:34.538766 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 136734us
I20260709 12:21:34.540601 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:34.543623  3588 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:34.546156 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:34.518915  3299 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:34.569268 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:34.528644  3626 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.567742 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:34.593407 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:34.594928 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:34.590915 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:34.598222 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.539666  3643 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.542398  3670 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:21:34.532406  3696 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.013s	sys 0.023s
I20260709 12:21:34.655889 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.662892 27738 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:34.663415 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.649115 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:34.588258  3682 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.569886  3671 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.647710 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.629280  3529 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:34.673772 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:34.666939 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.020s	sys 0.022s
W20260709 12:21:34.662631  3696 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.014s	sys 0.024s
I20260709 12:21:34.672298 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:34.674634 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:34.675052 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.021s	sys 0.023s
I20260709 12:21:34.670400 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.587067  3654 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:21:34.660373  3672 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.662307  3643 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:34.661659  3246 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.709645 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.569490  3694 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.095s
I20260709 12:21:34.692082 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:34.649336  3553 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.693665 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.716997 27736 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:21:34.698603 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 472494us
I20260709 12:21:34.672727  3631 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.689395  3219 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:34.692349  3201 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:21:34.711869 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.011s	sys 0.015s
W20260709 12:21:34.699142  3719 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.384s	user 0.001s	sys 0.001s
I20260709 12:21:34.703486  3635 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:34.708897  3651 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:34.714140 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.013s	sys 0.033s
I20260709 12:21:34.695936  3711 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:34.733105  3711 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:34.734265  3711 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.600459  3626 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:34.721376 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.011s	sys 0.015s
I20260709 12:21:34.713497 28329 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
W20260709 12:21:34.737821 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.013s	sys 0.034s
W20260709 12:21:34.742200 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 709293us
W20260709 12:21:34.740820 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 396486us
I20260709 12:21:34.723507  3219 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.720593  3643 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:34.742091 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:34.772274 28333 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:21:34.748733  3731 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 12:21:34.751806  3729 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:21:34.774175 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:34.757496  3380 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.680s	user 0.007s	sys 0.003s
I20260709 12:21:34.758172  3491 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.759917  3626 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.746500 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:34.744668  3635 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:34.741609  3651 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.724887  3682 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.745416  3201 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:34.779044 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.018s	sys 0.033s
W20260709 12:21:34.811965 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.022s	sys 0.040s
I20260709 12:21:34.783383 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:34.727262  3630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.024s	sys 0.025s
I20260709 12:21:34.773313  3219 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:34.809665 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.810652 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:34.817269 27731 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:34.818368 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.815598  2667 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:21:34.822678 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76226.7us
I20260709 12:21:34.807101  3635 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.825435  3635 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:34.808454  3380 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.826861  3004 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:21:34.827792  3004 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:21:34.828840 27733 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:34.776806  3643 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.831693  3201 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.833241  3643 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.808743  3636 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.781544 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.812960 27588 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:34.779036  3553 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:21:34.776510  3718 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.332s	user 0.001s	sys 0.000s
I20260709 12:21:34.778796  3724 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:34.813386  3682 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.842404  3635 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.879979  3635 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.880616  3635 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:34.880731  3299 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:34.881290  3299 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:34.819180  3727 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:21:34.882225  3299 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.845839  3619 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.848258  2667 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:34.883617 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.883973  2667 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:34.884151  3682 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.815244  3671 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:34.810709 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 463813us
I20260709 12:21:34.884902 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.884821  2667 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.885509  3619 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:34.858767 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.097s	user 0.009s	sys 0.027s
I20260709 12:21:34.886137  3552 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:21:34.886592 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.011s	sys 0.028s
W20260709 12:21:34.886945 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.13483e+06us
I20260709 12:21:34.884914  3671 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.888088 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.888700  3629 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:34.889474  3671 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.826547  3745 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.870792 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.890154  3299 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:34.890151  3745 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.890151  3626 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:34.891955  3626 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:34.885872  2667 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:34.892817  3626 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.848021  3424 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.895779  3424 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.897948  3424 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.898447  3745 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.828903  3380 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.889645 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.900477  3246 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:34.900815 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.902511 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:34.903098  3653 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:34.903580  3726 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:21:34.814215  3630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.026s	sys 0.025s
I20260709 12:21:34.904275  3726 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:34.904452  3651 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:34.877827  3670 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.905435  3670 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.905287  3726 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.907630  3670 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:34.883476 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1496) took 1429 ms. Trace:
I20260709 12:21:34.908712 27589 rpcz_store.cc:276] 0709 12:21:33.453721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.646221 (+192500us) service_pool.cc:224] Handling call
0709 12:21:33.646634 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:33.658698 (+ 12064us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:33.658734 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:33.678737 (+ 20003us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.106285 (+427548us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:34.106323 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.106388 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.883209 (+776821us) raft_consensus.cc:1685] finished
0709 12:21:34.883233 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.883411 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9474,"raft.queue_time_us":9583,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8042,"thread_start_us":424266,"threads_started":1,"wal-append.queue_time_us":646397}
I20260709 12:21:34.909488  3635 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:34.910574 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.911055  3636 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:34.911444  3636 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:34.911788  3553 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:34.912235  3636 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.913420  3636 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:34.913029  3553 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.885567 27584 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:34.916832 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:34.918226 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.918948 27584 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:34.889257  3629 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:34.920339 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3688) took 1022 ms. Trace:
I20260709 12:21:34.920853 27655 rpcz_store.cc:276] 0709 12:21:33.897915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.898085 (+   170us) service_pool.cc:224] Handling call
0709 12:21:33.927969 (+ 29884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:33.928392 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:33.928444 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:33.929008 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.458866 (+529858us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:34.458919 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.458992 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.920099 (+461107us) raft_consensus.cc:1685] finished
0709 12:21:34.920127 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.920274 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5808,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":2289,"thread_start_us":529417,"threads_started":1,"wal-append.queue_time_us":846251}
W20260709 12:21:34.926374  3736 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 12:21:34.927207 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.928332 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:34.928678  3721 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:21:34.900997  3246 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:34.930332  3246 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:34.938431 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:34.939254 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:34.940337 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.941288 27657 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: [23.17-25.19]
I20260709 12:21:34.941540 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.942446 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:34.943035 27658 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:34.943089 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.944065 27737 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:34.944618 27584 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 12. (term mismatch)
W20260709 12:21:34.948050 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.873441 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1497) took 1419 ms. Trace:
I20260709 12:21:34.922077  3629 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:34.869493 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1499) took 1265 ms. Trace:
I20260709 12:21:34.954741 27587 rpcz_store.cc:276] 0709 12:21:33.603808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.802289 (+198481us) service_pool.cc:224] Handling call
0709 12:21:33.802599 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:33.802896 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:33.802923 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:33.832118 (+ 29195us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.325468 (+493350us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:34.325514 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.325578 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.869294 (+543716us) raft_consensus.cc:1685] finished
0709 12:21:34.869322 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.869440 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29643,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":25088,"thread_start_us":493002,"threads_started":1,"wal-append.queue_time_us":832263}
I20260709 12:21:34.865623 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:34.957127 27582 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:34.903663  3653 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:34.875306 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:34.950420 27581 rpcz_store.cc:276] 0709 12:21:33.454156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.678880 (+224724us) service_pool.cc:224] Handling call
0709 12:21:33.706302 (+ 27422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:33.706714 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:33.971344 (+264630us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.702355 (+731011us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:34.702921 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.703086 (+   165us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:34.703131 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.703213 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.873174 (+169961us) raft_consensus.cc:1685] finished
0709 12:21:34.873203 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.873368 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1862,"prepare.queue_time_us":64112,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":3126,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":14939,"spinlock_wait_cycles":1180928,"thread_start_us":980880,"threads_started":2,"wal-append.queue_time_us":849361}
I20260709 12:21:34.898505  3424 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:21:34.852823  3730 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 12:21:34.960731 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.973513 27731 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:34.975041 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:34.975426 27552 connection.cc:256] RPC call timeout handler was delayed by -2.5881086280001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260709 12:21:34.975842 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3910086280000087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260709 12:21:34.945032 27584 pending_rounds.cc:85] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 11
I20260709 12:21:34.976027 27734 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:34.964711 27655 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:34.978118 27737 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:21:34.979647 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:34.967734 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:34.980710 27735 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:34.980841 27657 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:34.981976 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:34.952214  3614 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.819s	user 0.012s	sys 0.000s
I20260709 12:21:34.976398 27584 pending_rounds.cc:107] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:21:34.984647 27584 raft_consensus.cc:2889] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:34.964109 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.991657 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:34.991765 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:34.992616 27658 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:34.992803 27655 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
W20260709 12:21:34.976186 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3910086280000087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260709 12:21:35.002766 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3100086280001051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
W20260709 12:21:35.003116 27552 connection.cc:256] RPC call timeout handler was delayed by -2.2983086280000862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
I20260709 12:21:34.961702  3724 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:35.005343 27733 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 28. Ops: []
I20260709 12:21:34.963559  3424 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.010944 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.017855 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:35.011876 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.012830 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.020576 27660 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:35.012938 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:35.015786 27552 connection.cc:256] RPC call timeout handler was delayed by -2.2434086279999974s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20260709 12:21:35.021358 27552 connection.cc:256] RPC call timeout handler was delayed by -2.2128086279999479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260709 12:21:35.021850 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1759086280001156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1962s for a response
W20260709 12:21:35.022261 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1741086280001127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
W20260709 12:21:35.022641 27552 connection.cc:256] RPC call timeout handler was delayed by -2.1507086280000749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
W20260709 12:21:35.023030 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0922086279999803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20550000000000002s for a response
W20260709 12:21:35.023439 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0895086279999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
I20260709 12:21:35.023672 27738 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:35.026998 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.028221 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:35.027149 27552 connection.cc:256] RPC call timeout handler was delayed by -2.0643086279999352s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
W20260709 12:21:35.030558 27552 connection.cc:256] RPC call timeout handler was delayed by -2.035508628000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
W20260709 12:21:35.030953 27552 connection.cc:256] RPC call timeout handler was delayed by -1.9212086279999312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22450000000000003s for a response
W20260709 12:21:35.031378 27552 connection.cc:256] RPC call timeout handler was delayed by -1.8375086280000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2338s for a response
W20260709 12:21:35.031767 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4658086280001044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2751s for a response
I20260709 12:21:35.028895 27587 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
W20260709 12:21:35.032181 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0770086279999305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3183s for a response
I20260709 12:21:35.034463 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:34.995622  3614 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:35.037695 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.038091 27552 leader_election.cc:400] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
I20260709 12:21:35.045624 27552 leader_election.cc:403] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:21:35.046656 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.024681  3424 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:35.061165 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3835) took 1188 ms. Trace:
I20260709 12:21:35.061621 27583 rpcz_store.cc:276] 0709 12:21:33.873140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.204910 (+331770us) service_pool.cc:224] Handling call
0709 12:21:34.205250 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:34.205634 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.205662 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:34.206089 (+   427us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.406450 (+200361us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b180047b7e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:21:34.819347 (+412897us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:34.819415 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.819477 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.060876 (+241399us) raft_consensus.cc:1685] finished
0709 12:21:35.060901 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.061103 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1674,"raft.queue_time_us":85,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":13741,"spinlock_wait_cycles":40437632,"thread_start_us":412705,"threads_started":1,"wal-append.queue_time_us":826579}
I20260709 12:21:35.036265  3653 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:35.071990 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:35.072840 27580 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:35.074431  3219 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.049043  3725 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22.
W20260709 12:21:35.078912 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.052850 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3812) took 1228 ms. Trace:
I20260709 12:21:35.079916 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:35.081090 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.081911 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.082576 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:35.083393 27583 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:35.084533 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:35.085987 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:35.079610 27585 rpcz_store.cc:276] 0709 12:21:33.824016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.172610 (+348594us) service_pool.cc:224] Handling call
0709 12:21:34.174071 (+  1461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:34.177360 (+  3289us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:34.238800 (+ 61440us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b180033e8c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:34.314416 (+ 75616us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.412400 (+ 97984us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:21:34.820134 (+407734us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:34.877333 (+ 57199us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.877507 (+   174us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:34.877556 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.877637 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.052638 (+175001us) raft_consensus.cc:1685] finished
0709 12:21:35.052658 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.052792 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33986,"prepare.queue_time_us":142128,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":941,"raft.run_cpu_time_us":3676,"raft.run_wall_time_us":235685,"spinlock_wait_cycles":77775872,"thread_start_us":457348,"threads_started":1,"wal-append.queue_time_us":666247}
I20260709 12:21:35.088064 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:35.093648 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55837.6us
I20260709 12:21:35.099514  3743 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:35.099555 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.100445 28335 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:35.100811 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.101094 27663 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:35.102136 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.104656 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.105712 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.106338 28334 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:21:35.106703 28334 pending_rounds.cc:85] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 18
I20260709 12:21:35.106895 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.107617 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.106910 28334 pending_rounds.cc:107] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:21:35.108767 28334 raft_consensus.cc:2889] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:35.109865 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.111390 28335 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:35.111652 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:35.112412  3219 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:35.113441  3737 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.305s	user 0.001s	sys 0.000s
I20260709 12:21:35.113210 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.114218 28328 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:35.118057 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.119415 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:35.119609 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:35.120914 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:35.045722 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:35.126469  3750 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:35.053186  3724 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.136197 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:35.136763 28330 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:35.129590 27729 raft_consensus.cc:2368] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
W20260709 12:21:35.054515  3709 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.632s	user 0.008s	sys 0.004s
I20260709 12:21:35.050375  3529 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:21:35.142362 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 323563us
I20260709 12:21:35.133261 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:35.133525 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.134053 27586 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:35.126483  3670 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:35.149116 27587 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:21:35.144037 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:35.130576 27658 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:35.132313  3723 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:35.018810  3629 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:21:35.143213 27731 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 20. (index mismatch)
I20260709 12:21:35.131467 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.130728 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:35.146548 27660 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 28 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:21:35.128939  3228 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:35.165551  2667 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:35.166486  2667 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:35.147543 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:35.148108 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 435676us
I20260709 12:21:35.168109  2667 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:35.169728 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:35.170665  2667 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:35.151883 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:35.157877 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:35.174793 27738 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:21:35.188158 27660 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:35.188665 27733 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:35.189843 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:35.199365 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.202895 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.196781 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.206357 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:35.212175 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:35.165169  3636 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:35.151489  3755 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:35.144934  3709 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:35.219201 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:35.165124  3228 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.226425 27660 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:35.262392 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.264119 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.149035  3670 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.202404  3723 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:35.200838  3682 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:35.195456  3725 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.196558  3491 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:21:35.251217 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:35.196779  3735 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.073s
I20260709 12:21:35.252864 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.293395 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:35.200644  3726 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:35.273754  3670 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:35.266832 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:35.271369 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54270.4us
I20260709 12:21:35.271399 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.315824 27663 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:35.316692 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.317261 27663 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:35.250176 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.299960 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.287374 27734 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:35.322064 27657 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:35.251472 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.270707  3228 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:35.266330  3630 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:35.264927 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:35.218436  3636 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.336488 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.323426 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.328769 27734 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:35.330991 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.338680 27660 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:35.334190 27733 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:35.339215 27657 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:21:35.346873 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 189113us
I20260709 12:21:35.348527 27656 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:21:35.350200 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:35.350831 27656 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:35.294333  3626 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.364872 27660 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:35.362756 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:35.363108 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:35.363485 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 163049us
I20260709 12:21:35.260769  3755 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:35.447609 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79732.4us
I20260709 12:21:35.312860  3753 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.334937  3201 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:35.338333  3651 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:35.363757  3749 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
I20260709 12:21:35.292238  3725 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.345808  3636 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:35.505928 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.524144 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.562101 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.458321 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:35.359002  3508 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:21:35.626195 27659 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.578s	user 0.010s	sys 0.000s
W20260709 12:21:35.524187 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.664551  3782 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:35.665752  3757 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:35.666534  3757 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:35.669024  3757 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:35.663864 27659 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:35.458050 27663 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:35.635084 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:35.680498 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 332516us
I20260709 12:21:35.708986 27734 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:35.709383 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:35.560379 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:35.720322 27584 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 28
W20260709 12:21:35.573827  3748 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 12:21:35.729913  3508 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.730535  3508 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:35.731527 27655 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 14. Current term is 15. Ops: []
I20260709 12:21:35.733774  3755 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:35.593657  3651 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.598737  3753 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.736977  3753 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:35.738809 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1162) took 1195 ms. Trace:
W20260709 12:21:35.699960  3760 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:35.701475  3778 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.026s
W20260709 12:21:35.702317  3756 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.461s	user 0.000s	sys 0.001s
I20260709 12:21:35.596529  3614 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:35.625545 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:35.771993 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62733.8us
W20260709 12:21:35.637631 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 187774us
I20260709 12:21:35.455111  3626 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.727317  3725 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:35.600332  3711 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:35.726927  3228 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:35.743950 28330 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.603s	user 0.009s	sys 0.001s
I20260709 12:21:35.741698 28331 rpcz_store.cc:276] 0709 12:21:34.543896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.544079 (+   183us) service_pool.cc:224] Handling call
0709 12:21:34.544400 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:34.544602 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:21:34.832269 (+287667us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.736779 (+904510us) spinlock_profiling.cc:243] Waited 6.71 ms on lock 0x7b18003d1aa0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:35.738181 (+  1402us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.738399 (+   218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:35.738456 (+    57us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:35.738511 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.738608 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.738750 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":691385,"prepare.queue_time_us":1199967,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":2692,"raft.queue_time_us":544,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":15439104,"thread_start_us":457993,"threads_started":2,"wal-append.queue_time_us":214253}
I20260709 12:21:35.784420 28330 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:35.676450  3201 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.753067 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:35.608299  3708 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:35.791555 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:35.773948 27733 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:21:35.774897 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:35.790638 28329 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.002s	sys 0.011s
I20260709 12:21:35.779181 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.781183  3626 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:35.787617 27735 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:35.806921 28335 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
W20260709 12:21:35.789994 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 331942us
W20260709 12:21:35.591760  3758 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:21:35.777189 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:35.768832  3709 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:35.776321  3755 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:35.787030  3581 raft_consensus.cc:2764] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:21:35.792574  3201 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:35.786834  3711 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:35.839931 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62825.6us
I20260709 12:21:35.845841 27661 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:21:35.865173 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:35.820880 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46343us
I20260709 12:21:35.827459 27736 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:35.824741 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:35.819000 28329 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.005s	sys 0.013s
W20260709 12:21:35.824553 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.011s	sys 0.030s
W20260709 12:21:35.870389 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.011s	sys 0.031s
I20260709 12:21:35.872996 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:35.873765 27659 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:35.886750 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.890440 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:35.860965  3694 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.139s	user 0.024s	sys 0.053s
I20260709 12:21:35.840106  3626 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:35.898370 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.839625  3791 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:35.856396  3651 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:35.839365  3788 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.859473  3775 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:35.911975  3726 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.930095  3726 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.932533  3726 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:35.887920  3725 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:35.934525  3786 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:35.935009  3786 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.935361  3786 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:35.897292  3780 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:35.844750  3636 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:35.900038 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:35.921610 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:35.899248  3694 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.178s	user 0.025s	sys 0.053s
I20260709 12:21:35.958968 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:35.904997  3626 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.891705  3708 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
W20260709 12:21:35.969219 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60940.4us
I20260709 12:21:35.971362 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:35.925722  3651 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.975566 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:35.960481  3788 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:35.978183  3636 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:35.965950  3228 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:35.958884  3709 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:35.988737  3711 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:35.989270  3785 raft_consensus.cc:2764] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
W20260709 12:21:35.989806  3424 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.934s	user 0.010s	sys 0.002s
I20260709 12:21:35.940969  3791 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.007207  3782 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:35.964637  3643 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:21:35.966413  3626 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:35.964926  3725 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:21:36.022186 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46731.9us
I20260709 12:21:36.023669  3726 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:36.043049 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:36.049505  3787 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.115s	user 0.002s	sys 0.000s
I20260709 12:21:36.034889  3424 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.019238  3651 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.030545  3709 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.058243 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.064270 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.061594  3726 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:36.073414 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.086503  3791 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.064883  3725 raft_consensus.cc:2764] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:21:36.076316  3782 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:36.095000  3788 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.090584  3424 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:36.097110  3723 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.820s	user 0.005s	sys 0.009s
I20260709 12:21:36.093134  3780 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:36.107625  3653 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.097965  3782 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.101577  3651 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:36.110133  3508 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.110669 28329 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:36.089399 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.097406  3726 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.135277  3653 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:36.139671 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60226.4us
W20260709 12:21:36.140394 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.135921  3709 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:36.143083 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.144084 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:36.137424  3723 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:36.145361 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.145745  3788 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.146282  3788 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.146631  3653 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.148301 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.146070  3725 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:21:36.141599  3775 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:36.152341 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3152) took 1213 ms. Trace:
I20260709 12:21:36.152738 27729 rpcz_store.cc:276] 0709 12:21:34.938841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.941161 (+  2320us) service_pool.cc:224] Handling call
0709 12:21:36.152155 (+1210994us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54001d24d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:36.152284 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337129728}
I20260709 12:21:36.156754  3508 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:36.163049 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.163846 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:36.163997  2667 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.164582  2667 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.165294 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:36.166301  3671 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.166802  3671 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.168848  3671 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:36.175726  3782 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.167088  2667 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.182066 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3842) took 1220 ms. Trace:
I20260709 12:21:36.182503 27582 rpcz_store.cc:276] 0709 12:21:34.961383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.974763 (+ 13380us) service_pool.cc:224] Handling call
0709 12:21:36.181851 (+1207088us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:36.181996 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1822571904}
I20260709 12:21:36.185410 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3160) took 1067 ms. Trace:
I20260709 12:21:36.185819 27736 rpcz_store.cc:276] 0709 12:21:35.117572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.117782 (+   210us) service_pool.cc:224] Handling call
0709 12:21:36.185210 (+1067428us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:36.185348 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612097280}
I20260709 12:21:36.140152  3780 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.148358  3788 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:36.188025  3780 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.157119  3626 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.191154  3629 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.191439  3626 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.191882  3629 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.144182  3786 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.194159  3629 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:36.194679  3629 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.195237  3786 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:36.197744  3630 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.198287  3630 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.198968 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.200634  3630 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.219058 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:36.219455  3796 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.162s	user 0.000s	sys 0.002s
I20260709 12:21:36.215597 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:36.220507 27656 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:36.220796 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3852) took 1047 ms. Trace:
I20260709 12:21:36.221220 27586 rpcz_store.cc:276] 0709 12:21:35.173170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.289809 (+116639us) service_pool.cc:224] Handling call
0709 12:21:35.332679 (+ 42870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.741496 (+408817us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b18004aaa00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:21:35.741636 (+   140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.741660 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.742172 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.897518 (+155346us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:35.897583 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.897679 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:36.208412 (+310733us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b18004aaa00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b557 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:36.218867 (+ 10455us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:36.220577 (+  1710us) raft_consensus.cc:1685] finished
0709 12:21:36.220601 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:36.220735 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34066,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":83311,"spinlock_wait_cycles":86539136,"thread_start_us":277367,"threads_started":2,"wal-append.queue_time_us":291984}
I20260709 12:21:36.223483  3775 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.224746  3505 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:36.224659  3775 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.225526  3505 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:36.229697 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:36.230479 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.203439  3380 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.203896  3697 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.204792  3723 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.159137 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:36.241183 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82135.6us
I20260709 12:21:36.208647  3246 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:36.209933  3696 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:36.149537  3670 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.811s	user 0.013s	sys 0.000s
I20260709 12:21:36.214196 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:36.245697 27659 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:36.214488  3797 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.277s	user 0.003s	sys 0.000s
I20260709 12:21:36.215508  3646 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:36.247751 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.248129 27735 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:21:36.250344 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:36.216542 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.229697 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.230693 27660 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:36.226686  3505 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.203665 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.233215 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.233891 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.235143 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.264963 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:36.266712 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:36.267277 27660 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:36.270248 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:36.270972 27658 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.271555 27658 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:21:36.273319 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.240432  3653 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.267849  3508 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:21:36.202817 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57531.8us
I20260709 12:21:36.242659 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:36.241748 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:36.281044 27734 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:36.281517 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.243594 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.211678  3219 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:36.258118 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:36.249322 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:36.258782 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:36.259486 27586 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:36.247221  3623 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:36.260309 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.228222  3783 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.284844 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:36.336335 27658 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:36.286212 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:36.280057  3508 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.292984 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:36.231343 27589 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:36.349941 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.350056 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:36.350828 27589 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:21:36.195127  3629 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.238874 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:36.354882 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116115us
I20260709 12:21:36.356945 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:36.299872 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.357929  2667 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:36.301719 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.358681 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:36.302963 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.361828 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:36.305413 27734 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:36.363521 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.364938 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.245177  3697 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.247398  3724 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.066s
W20260709 12:21:36.371363 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85238.6us
I20260709 12:21:36.372328 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.315088  3201 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.318785 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.258745  3670 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.321794 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.259352  3696 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.322515 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.245363  3723 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.323211 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.408946 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:36.409989 27731 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:36.410624 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:36.411677 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54818.7us
I20260709 12:21:36.412199 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:36.413013 27737 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:36.324802 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1228) took 1134 ms. Trace:
I20260709 12:21:36.413367 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:36.328205 27735 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:36.328991  3424 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:36.329566 27482 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 15 to 17, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 17 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:36.247341  3646 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:36.345793 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:36.436797 27659 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:36.295678 27586 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.352140 28337 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:36.369365 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:36.245545  3380 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.373471 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.259892  3246 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.374152 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:36.377501 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:36.294027  3219 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:36.399791 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.450558 27588 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:36.077111 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:36.451478 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:36.451867 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:36.408629 27583 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:36.353267  3508 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:36.435245 27734 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:36.413996 28328 rpcz_store.cc:276] 0709 12:21:35.190448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.190946 (+   498us) service_pool.cc:224] Handling call
0709 12:21:35.211152 (+ 20206us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.693922 (+482770us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:35.783600 (+ 89678us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.901120 (+117520us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.901728 (+   608us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.901896 (+   168us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:35.901939 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.902032 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:36.213965 (+311933us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:36.324525 (+110560us) raft_consensus.cc:1685] finished
0709 12:21:36.324621 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:36.324747 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103659,"prepare.queue_time_us":146553,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":156100,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":25628,"spinlock_wait_cycles":58060032,"thread_start_us":522099,"threads_started":2,"wal-append.queue_time_us":121860}
I20260709 12:21:36.441035 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:36.441365 27661 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:36.442529 27662 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:21:36.353585  3623 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:36.447115 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73713.8us
I20260709 12:21:36.385003  3753 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:36.446152 27659 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:21:36.299594  3725 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.108s
I20260709 12:21:36.459143 27732 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:36.459885 27738 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:21:36.461072 27656 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:36.466297 28331 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:36.336668  3653 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.384752  3651 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:36.467157 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:36.453533 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.447803 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:36.468652 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91249.7us
I20260709 12:21:36.489912 27729 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:36.469187 27733 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:36.482527 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.495025 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:36.444052  3742 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:36.434655  3672 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.360075  3783 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.461503  3788 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:36.489845 27731 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:21:36.460796  3424 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:36.497619 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49904.5us
I20260709 12:21:36.505156 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.505797 27588 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.506428 27588 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:21:36.498013 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:36.507990 27735 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:36.469789  3671 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:36.486378  3529 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:36.509378 27588 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:36.496098 27581 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:36.533034 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:21:36.374231  3726 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.496881  3588 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:36.560103 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:36.502784  3746 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:36.502183  3630 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:21:36.574496 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.575400 28328 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
W20260709 12:21:36.581472 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 114384us
I20260709 12:21:36.504382  3782 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.501637  3582 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:21:36.514146  3671 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.542732  3791 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:36.596964  3782 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.597759  3755 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:36.598423  3755 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:36.570437  3588 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.499550  3742 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.617304 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3619) took 1367 ms. Trace:
I20260709 12:21:36.617866 27738 rpcz_store.cc:276] 0709 12:21:35.250075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.250693 (+   618us) service_pool.cc:224] Handling call
0709 12:21:35.251041 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:36.488075 (+1237034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.616969 (+128894us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:36.617228 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2779472256}
I20260709 12:21:36.625607 27735 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:36.515079  3672 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.649101 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:36.627513 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44534.6us
W20260709 12:21:36.628332 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 215055us
I20260709 12:21:36.648079  3742 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:36.586221  3746 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:36.599198  3782 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:36.658296 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:36.611959  3724 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:36.602541  3791 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:36.600122  3651 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.611047  3671 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:36.538921  3529 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.576514  3631 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:21:36.612392  3588 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:36.662916  3582 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:36.665480  3724 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:36.671574 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.719763 27733 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:36.687151 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:36.669559  3651 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:36.740171 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:36.660873  3299 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:21:36.599586  3755 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.711445  3529 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:36.589325  3623 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:36.746191 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74691.2us
I20260709 12:21:36.605329  3246 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.758424 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.763376 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:36.765220 27589 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:21:36.771230 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 319446us
I20260709 12:21:36.774618 27704 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:21:36.775076 27704 leader_election.cc:403] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:36.775619 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.701618  3682 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 12:21:36.742327  3670 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:36.773312 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:36.735584  3630 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:36.785530 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.748978  3697 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:36.753363  3788 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:36.794080 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:36.668234  3424 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.796836 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:36.742969  3725 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 28.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:21:36.753194  3723 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:36.770503  3626 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.801049  2667 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:21:36.801614 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 143395us
W20260709 12:21:36.760331 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:36.802516 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.781462  3635 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:36.770933  3696 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:36.811074 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.785591  3785 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:36.818094 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:36.795498  3643 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:21:36.805184  3808 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
I20260709 12:21:36.835250 27736 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:36.799767  3798 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:36.758162  3793 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:36.826278  2667 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:36.820516 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:36.784319  3380 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.862725  2667 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.816152  3725 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:36.821087  3788 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:36.861716  3653 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.891870  3725 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.894167  3725 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:36.801221  3724 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.896603  3724 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:36.823961  3709 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:21:36.874085  3814 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
W20260709 12:21:36.874809  3815 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.212s	user 0.000s	sys 0.000s
W20260709 12:21:36.875766  3812 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:21:36.877192  3819 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:36.878625  3817 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:21:36.879426  3810 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:21:36.753618  3746 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.883744 27656 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:21:36.883558  3630 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.825484  3821 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:36.858165  3793 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:36.887133  3822 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:21:36.814730  3697 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:21:36.758752  3791 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.840945  3723 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:21:36.904065 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.841236  3696 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:36.918208 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.922616  3582 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:36.871901  3811 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.262s	user 0.001s	sys 0.001s
I20260709 12:21:36.820436  3626 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.001443 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:36.925946  3635 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:21:36.915134  3798 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:36.955554  3808 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:21:36.914866  3709 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:36.891871  3653 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.081s
I20260709 12:21:36.998322 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:36.958990 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122538us
I20260709 12:21:36.998883  3682 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.026893  3682 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.029048  3682 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:37.012600 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.977617  3793 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:36.977993  3821 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:36.970316  3819 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:36.981393  3782 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:36.991848  3672 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:36.997741  3696 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.018478  3349 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:21:37.024686  3788 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:36.994352  3838 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:37.261204 27581 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:21:36.996910  3785 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:37.019428  3709 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:37.284060  3785 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.034334 27582 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:21:37.008924  3791 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:37.114984 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57352.7us
I20260709 12:21:37.098194 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:37.064997 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:37.006026  3636 consensus_queue.cc:1175] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 27 status { last_received { term: 27 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 27 index: 16 } }
I20260709 12:21:37.060528  3819 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:37.222710 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:37.231634 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 371135us
I20260709 12:21:37.300194 27587 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: [28.15-28.15]
I20260709 12:21:37.010128  3798 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.303638  3798 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:37.310318  3709 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.303112 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:37.335176 27736 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:37.339078 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:37.311147  3588 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.574s	user 0.005s	sys 0.001s
I20260709 12:21:37.243598  3696 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:37.368815 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73039.6us
I20260709 12:21:37.380282 27664 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:37.266074  3672 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:37.381865 27732 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:21:37.242297  3821 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.241729  3782 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:37.439215 27583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.021s	sys 0.051s
W20260709 12:21:37.440685 27583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.021s	sys 0.051s
I20260709 12:21:37.442512 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3862) took 1122 ms. Trace:
I20260709 12:21:37.475798 27583 rpcz_store.cc:276] 0709 12:21:36.319589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.506358 (+186769us) service_pool.cc:224] Handling call
0709 12:21:36.506634 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.506816 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:36.720516 (+213700us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.441380 (+720864us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.441447 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:37.441518 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.441615 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.442423 (+   808us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":304533,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":107,"thread_start_us":897631,"threads_started":2}
I20260709 12:21:37.275414  3788 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.365398  3588 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:37.510368  3849 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 12:21:37.511866  3588 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:37.313975  3838 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.514576  3838 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.521322  3788 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.521559  3782 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.521963  3788 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.522471  3788 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.523900  3709 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.420212  3846 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.117s
I20260709 12:21:37.524595  3709 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.509073  3791 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.525893  3791 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.528069  3791 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:37.524049  3651 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.773s	user 0.005s	sys 0.005s
I20260709 12:21:37.528827  3651 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.529083  3798 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.508343  3853 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:37.525111  3709 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.273093  3582 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:37.537873 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.552743 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3869) took 1168 ms. Trace:
I20260709 12:21:37.562824 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:37.566613 27580 rpcz_store.cc:276] 0709 12:21:36.384237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.014696 (+630459us) service_pool.cc:224] Handling call
0709 12:21:37.014966 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.033843 (+ 18877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.033874 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.033912 (+    38us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:37.033941 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.552414 (+518473us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:37.552478 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.552662 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42979584}
I20260709 12:21:37.580626 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1487) took 1148 ms. Trace:
W20260709 12:21:37.534884  3508 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.047s	user 0.008s	sys 0.000s
W20260709 12:21:37.534394  3742 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.868s	user 0.008s	sys 0.000s
I20260709 12:21:37.581219 27660 rpcz_store.cc:276] 0709 12:21:36.432678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.440370 (+  7692us) service_pool.cc:224] Handling call
0709 12:21:36.440655 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.492509 (+ 51854us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:36.527184 (+ 34675us) log.cc:844] Serialized 14 byte log entry
0709 12:21:36.593086 (+ 65902us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:36.593153 (+    67us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:36.593206 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.580422 (+987216us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b5400291b54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:37.580461 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.580573 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106430,"prepare.queue_time_us":51014,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3719,"raft.queue_time_us":83775,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":49185,"spinlock_wait_cycles":118693760}
W20260709 12:21:37.596797 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.422009  3672 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.589226 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:37.591005 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.009s	sys 0.005s
W20260709 12:21:37.614156 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.009s	sys 0.005s
W20260709 12:21:37.546507  3724 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.649s	user 0.012s	sys 0.000s
W20260709 12:21:37.536351  3529 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.783s	user 0.008s	sys 0.003s
I20260709 12:21:37.545120  3788 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.531287  3798 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.555675  3588 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.539733  3709 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.618502 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:37.622444 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 145939us
W20260709 12:21:37.623283 27589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.051s	sys 0.095s
W20260709 12:21:37.640519 27589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.051s	sys 0.095s
I20260709 12:21:37.535595  3853 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:37.589003 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 285946us
I20260709 12:21:37.607280  3508 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.620846  3529 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.641973 27701 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:21:37.646626 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1526) took 1315 ms. Trace:
I20260709 12:21:37.629186  3827 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.613844  3742 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.649808 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:37.661703 27589 rpcz_store.cc:276] 0709 12:21:36.330873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.766071 (+435198us) service_pool.cc:224] Handling call
0709 12:21:36.766405 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.766633 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:36.954178 (+187545us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.646295 (+692117us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.646348 (+    53us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:37.646390 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.646468 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.646572 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5277,"prepare.queue_time_us":45056,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":6159,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":10195,"thread_start_us":873674,"threads_started":2}
W20260709 12:21:37.651502  3630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.054s	sys 0.102s
I20260709 12:21:37.665788 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:37.651568  3246 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:37.670320 27737 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:21:37.669788 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:37.661397  3631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.008s	sys 0.008s
W20260709 12:21:37.665827  3630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.054s	sys 0.103s
I20260709 12:21:37.689178 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:37.702963 27580 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:37.676658 27552 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:21:37.700416 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:37.701377 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:37.701845 27657 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:37.664294  3724 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.666730  3867 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:37.744460 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74738.2us
I20260709 12:21:37.726735 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:37.729094  3424 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:37.747699  3424 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:37.748093  3788 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.727349 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:37.700273  3791 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.728564 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:37.666574  3826 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.670747  3821 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:37.744124 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44648.3us
W20260709 12:21:37.727865 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62159us
I20260709 12:21:37.700470  3782 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.057s
I20260709 12:21:37.752044 27585 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:37.663030  3827 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.645848  3588 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.750680 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:37.695955  3798 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:21:37.763304 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:37.759792  3793 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.729s	user 0.012s	sys 0.000s
I20260709 12:21:37.727078  3653 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:21:37.713304  3838 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:21:37.790072 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:37.793458 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:37.793965 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.800313 27585 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:21:37.800612 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.795909 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.796718 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.798265 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:37.802356 27661 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:37.804490 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.804733 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.807519 27662 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:37.807896 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57342.2us
W20260709 12:21:37.819161 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.808648 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:37.759977  3791 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.829296 27552 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:21:37.791365  3818 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:37.809113  3671 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.105s	user 0.009s	sys 0.004s
I20260709 12:21:37.757272  3788 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.849689  3788 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.850258  3788 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.850730  3788 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.852066 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:37.833395 27552 leader_election.cc:400] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:37.853368 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:37.762466  3826 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.746908  3724 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.837349 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:37.806201  3798 consensus_queue.cc:1243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 29.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:21:37.765858  3821 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.767081  3853 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.812626  3424 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.857100  3866 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:37.857240  3838 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:37.859278  3827 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.860656  3788 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.864631  3629 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.864925  3791 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.867836 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:37.873291 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:37.889864 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:37.875027 27585 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:37.752612  3867 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.878453  3742 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.878499 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:37.880261  3529 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.799854  3793 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.882642 27581 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:37.880996  3588 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:37.853645  3671 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.881512  3838 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:37.922567  3838 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.890350  3653 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:37.893816 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:37.895129 27660 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:37.889042  3508 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:37.907492  3864 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
W20260709 12:21:37.907687  3858 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:21:37.907832  3848 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 12:21:37.908085  3875 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:37.909119  3821 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.894801  3845 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:21:37.903793  3870 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:21:37.903951  3855 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.236s	user 0.000s	sys 0.001s
W20260709 12:21:37.902649  3860 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:21:37.902891  3874 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.964965  3874 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:37.903087  3863 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:21:37.967587  3784 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.890612  3862 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:37.901515  3629 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:37.903770  3646 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.913046  3671 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:37.903595  3865 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.971652  3865 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.973623  3626 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.974229  3865 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:37.913650  3861 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:21:37.902549  3782 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:21:37.914665  3878 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:37.915196  3793 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:37.975625  3878 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:37.903399  3869 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
W20260709 12:21:37.919052  3696 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.520s	user 0.008s	sys 0.000s
I20260709 12:21:37.976665  3696 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.892082  3819 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:37.979537  3819 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:37.979969  3819 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:37.923223  3723 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.926s	user 0.011s	sys 0.000s
I20260709 12:21:37.980880 27656 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:37.981446  3651 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:37.982718 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.983180  3823 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:37.961814  3582 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:37.984560  3582 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:37.986948  3582 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.895524  3827 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
W20260709 12:21:37.942996 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49283.9us
I20260709 12:21:37.931627 27586 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.870453  3818 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:37.901860  3780 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:21:37.931449  3838 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:37.967607  3874 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:37.929081  3653 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:37.976137  3878 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:38.124073 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:38.124558 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:37.979270  3696 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:38.140916  3874 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:38.141778  3874 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:38.152701 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:38.153144 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:38.156004 28332 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:21:38.159296  3793 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:38.035992  3845 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:37.990800  3826 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:38.018357  3651 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.980826  3723 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 16. Current term is 17.
I20260709 12:21:38.029136 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:37.968317  3784 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:38.375367  3818 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:38.426578  3784 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:38.427515  3866 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:21:38.032959 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 138610us
I20260709 12:21:37.957417  3875 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:38.430254  3875 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:38.431782  3875 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:38.037135 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47797.8us
I20260709 12:21:37.989174 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:38.441656 27661 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:38.378293  3876 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:38.448657  3866 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:38.448418  3876 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:38.449229  3866 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:38.374888  3529 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:38.160256 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:38.367193 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:38.450856  3876 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:38.469292 27661 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:21:38.473795  3755 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.456542  3724 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:38.474382  3755 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:38.475309  3845 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:38.476928  3755 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:38.370990 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:38.464298 28332 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:38.465927  3876 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:38.380452 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.357908  3880 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.399s	user 0.001s	sys 0.000s
I20260709 12:21:38.131708  3833 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:38.134245  3862 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:38.200853  3508 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.424232 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:38.375207  3750 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:38.492486 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58766.1us
I20260709 12:21:38.074044  3653 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:38.160105  3793 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:38.134693  3834 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.604s
I20260709 12:21:38.496836 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.951711 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.953197 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.954624 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.644164 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.654278 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.653841 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.463967  3651 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:38.569706 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.020s	sys 0.029s
W20260709 12:21:38.965713 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.020s	sys 0.042s
I20260709 12:21:38.965489  3724 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.967846  3724 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:38.968317  3793 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:38.969003  3793 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:38.969548  3793 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:38.970554  3724 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:38.971417  3879 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.972074  3879 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:38.464939  3867 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:38.557128  3755 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:38.452785  3818 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.431s
I20260709 12:21:38.463833  3866 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:38.500798 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.502794 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:39.005002 27656 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.005s	sys 0.005s
W20260709 12:21:39.050379 27656 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.005s	sys 0.005s
I20260709 12:21:39.031806 27738 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:39.005980 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:39.063695 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.001s	sys 0.022s
I20260709 12:21:39.006819 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:39.068499 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61785.4us
I20260709 12:21:38.478489  3875 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:38.135967  3823 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:39.013269  3876 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:38.557695 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3635) took 2257 ms. Trace:
I20260709 12:21:39.015100  3862 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:39.015746 27658 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.028s	user 0.003s	sys 0.007s
I20260709 12:21:38.525332 27736 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
W20260709 12:21:39.019712 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.03051e+06us
I20260709 12:21:39.026844  3750 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:39.027669 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.077s	user 0.014s	sys 0.040s
W20260709 12:21:39.190982 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.250s	user 0.014s	sys 0.041s
W20260709 12:21:39.067821 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.001s	sys 0.022s
W20260709 12:21:39.191982 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.011878  3833 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.182513 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1340) took 1067 ms. Trace:
I20260709 12:21:39.192869 28328 rpcz_store.cc:276] 0709 12:21:38.121235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.121441 (+   206us) service_pool.cc:224] Handling call
0709 12:21:39.182421 (+1060980us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:39.191672 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:38.558745  3614 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:39.003612  3724 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:39.120342  3424 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:39.233641  3626 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:39.262768 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:39.119757  3879 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.079133  3742 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:39.237226 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1397) took 2862 ms. Trace:
W20260709 12:21:39.237347  3883 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.262s	user 0.000s	sys 0.000s
W20260709 12:21:39.239946  3893 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.289s	user 0.006s	sys 0.003s
I20260709 12:21:39.078895  3696 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:39.243440  3882 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:39.245078  3724 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.190145  3875 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:39.195225  3866 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.457340  3529 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.208407  3894 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.272678 27737 rpcz_store.cc:276] 0709 12:21:36.375285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.505603 (+130318us) service_pool.cc:224] Handling call
0709 12:21:36.560453 (+ 54850us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.697259 (+1136806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.236875 (+1539616us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:39.237152 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2551754496}
I20260709 12:21:39.226845  3793 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:39.270651  3845 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.025s	sys 0.045s
I20260709 12:21:39.292415  3882 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:39.282625  3742 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.333266  3219 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.356736  3201 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.298969  3696 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.440270  3882 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.441506  3882 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:39.432829  3845 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.027s	sys 0.045s
I20260709 12:21:39.260087  3874 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.371760  3552 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:21:39.379673 27658 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.392s	user 0.006s	sys 0.009s
I20260709 12:21:39.121280 27738 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:39.381620 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:39.479492 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 97958.2us
I20260709 12:21:39.263032  3626 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:39.199671  3823 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:39.417924  3809 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.748s	user 0.020s	sys 0.057s
I20260709 12:21:39.418708  3897 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:21:39.082330  3878 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.927s	user 0.005s	sys 0.006s
I20260709 12:21:39.382210 27730 rpcz_store.cc:276] 0709 12:21:36.300885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.301012 (+   127us) service_pool.cc:224] Handling call
0709 12:21:36.301303 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:36.462854 (+161551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:36.462899 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:36.463459 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:21:36.661880 (+198421us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:36.661948 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.492651 (+1830703us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54001d38d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:38.493215 (+   564us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:38.493457 (+   242us) raft_consensus.cc:1685] finished
0709 12:21:38.493470 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:38.557563 (+ 64093us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370784512,"thread_start_us":134160,"threads_started":1,"wal-append.queue_time_us":307325,"wal-append.run_cpu_time_us":2061,"wal-append.run_wall_time_us":1085380}
I20260709 12:21:39.419761  3894 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.437737  3219 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:39.331508  3875 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.505314 27731 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:39.063838  3867 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:39.477150  3833 net_util.cc:396] Time spent resolve address for 127.25.202.1: real 0.235s	user 0.024s	sys 0.020s
I20260709 12:21:39.432562  3834 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:39.511909 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.063695  3755 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.435189  3424 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:39.425554  3866 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:39.505157 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 242509us
I20260709 12:21:39.496062 27738 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:21:39.505564 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.041s	sys 0.163s
W20260709 12:21:39.507910 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 483117us
I20260709 12:21:39.487730  3614 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:39.515678 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.041s	sys 0.163s
I20260709 12:21:39.065649  3508 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.518226 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:39.519917 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:39.521654 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:39.522112 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:39.562951 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.002s	sys 0.024s
W20260709 12:21:39.563657 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.002s	sys 0.024s
W20260709 12:21:39.567058 27704 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.565s (SENT) [suppressed 16 similar messages]
W20260709 12:21:39.573621 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 89916.4us
W20260709 12:21:39.577744 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54226.1us
I20260709 12:21:39.579468 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:39.579766 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:39.580595 28333 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:39.582172 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:39.377161  3862 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.589758 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:39.599040  3742 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.522109  3626 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.599738  3742 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:39.600170  3626 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:39.603988  3838 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.569s	user 0.001s	sys 0.018s
I20260709 12:21:39.515982  3897 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:39.577801  3651 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.605093  3897 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.605530  3424 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.604624  3838 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:39.504856  3878 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:39.609227  3879 consensus_queue.cc:1175] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 25 status { last_received { term: 21 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:39.612484  3876 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.613014  3878 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.613751  3878 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:21:39.614041  3819 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.634s	user 0.008s	sys 0.012s
I20260709 12:21:39.614400  3878 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:39.614463  3819 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.512742  3867 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.619057 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:39.516595  3834 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.620234  3724 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.624233  3894 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.625026  3866 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.628446  3696 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.608968  3755 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.631935  3838 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.658277 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:39.491001  3750 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.660589 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:39.659045 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:39.661559 27736 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:39.660670 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:39.666450 27732 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:39.667235  3899 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:21:39.668464  3906 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.669009  3906 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.658157  3529 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.671617  3908 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.672173  3908 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.673892  3725 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:39.674314  3725 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:39.675206  3725 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.676795 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:39.677884 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:39.679656 28333 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:39.680614 28332 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: [26.18-26.18]
W20260709 12:21:39.659785  3653 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.652s	user 0.015s	sys 0.003s
I20260709 12:21:39.658828  3778 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:39.675038  3908 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.659036  3900 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:39.671188  3906 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:39.731489 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.733757 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.734409 27659 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:39.686000  3653 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.664247  3838 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.743778 27662 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:39.742429 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.684176  3778 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.746706 27659 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:39.747725 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.748442 27659 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:39.748966  3778 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:39.733868  3891 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.156s	user 0.000s	sys 0.002s
I20260709 12:21:39.690397  3742 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:39.749356 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.688038  3910 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:39.754637 27659 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
W20260709 12:21:39.755196 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 221149us
W20260709 12:21:39.759111 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 101457us
I20260709 12:21:39.762210 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:39.763293 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.764065 27659 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:39.766644 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:39.767650 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 200618us
I20260709 12:21:39.769187 27738 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:39.769197 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.776247 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3741) took 1921 ms. Trace:
I20260709 12:21:39.776634 27662 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:39.776654 27658 rpcz_store.cc:276] 0709 12:21:37.854815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.987035 (+132220us) service_pool.cc:224] Handling call
0709 12:21:37.987303 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.987542 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:39.475742 (+1488200us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.775913 (+300171us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.775971 (+    58us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:39.776019 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.776089 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.776199 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":122215,"prepare.queue_time_us":913900,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"thread_start_us":1665311,"threads_started":2,"wal-append.queue_time_us":263037}
I20260709 12:21:39.781257 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:39.781618 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50117.6us
I20260709 12:21:39.782089 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:39.783315 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.784310 27659 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:39.751106  3898 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:39.750032  3876 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:39.788288 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 205117us
I20260709 12:21:39.757289  3742 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.788908 27658 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:21:39.792119 27730 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:39.791193 27738 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:39.796599 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:39.797367 27658 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:39.788812 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.797847 27659 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
W20260709 12:21:39.774420  3635 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.755s	user 0.041s	sys 0.049s
I20260709 12:21:39.799513 27662 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:39.813167 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3868) took 3429 ms. Trace:
I20260709 12:21:39.811049 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1500) took 1824 ms. Trace:
I20260709 12:21:39.822844 27732 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:39.832966 27660 rpcz_store.cc:276] 0709 12:21:37.986489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.018245 (+1031756us) service_pool.cc:224] Handling call
0709 12:21:39.018695 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:39.477919 (+459224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.477977 (+    58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.478059 (+    82us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:39.478103 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.810797 (+332694us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:39.810847 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.810988 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055418240}
I20260709 12:21:39.811733 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1495) took 2986 ms. Trace:
I20260709 12:21:39.834653 27664 rpcz_store.cc:276] 0709 12:21:36.825509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.826975 (+  1466us) service_pool.cc:224] Handling call
0709 12:21:36.863101 (+ 36126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.380826 (+517725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.811496 (+2430670us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b5400344754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:39.811652 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1188920576}
I20260709 12:21:39.835588 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.831341 27588 rpcz_store.cc:276] 0709 12:21:36.383964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.015318 (+631354us) service_pool.cc:224] Handling call
0709 12:21:37.015611 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.537800 (+522189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.537846 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.537896 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:37.537939 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.812906 (+2274967us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54000893d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:39.812955 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.813081 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1200408320}
I20260709 12:21:39.778851  3651 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:39.836417 27736 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:39.689385  3900 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.791774  3724 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:39.788578  3912 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:39.853040 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.803061  3901 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:39.837448 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
W20260709 12:21:39.806351  3635 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.790s	user 0.042s	sys 0.049s
W20260709 12:21:39.847275 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 89287us
I20260709 12:21:39.851398 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3881) took 2096 ms. Trace:
I20260709 12:21:39.858549 27588 rpcz_store.cc:276] 0709 12:21:37.754712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.850540 (+2095828us) service_pool.cc:224] Handling call
0709 12:21:39.850845 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:39.851026 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.851055 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.851103 (+    48us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:39.851136 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.851190 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.851306 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:39.812155  3653 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.795426  3742 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:39.795730  3902 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:39.796093  3823 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.852182 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.814523  3897 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.816928  3821 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:39.859747 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.853883 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:39.809365  3910 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:39.865448 27662 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:39.863250 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3882) took 2108 ms. Trace:
W20260709 12:21:40.089152 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 403197us
I20260709 12:21:40.089092 27588 rpcz_store.cc:276] 0709 12:21:37.754968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.862459 (+2107491us) service_pool.cc:224] Handling call
0709 12:21:39.862760 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:39.862938 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.862962 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.863001 (+    39us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:39.863035 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.863090 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.863197 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:40.090158 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:39.838235 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:39.868573 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.928961  3907 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.858763  3912 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.801597  3876 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.851162  3651 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.923555  3834 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.998185 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3863) took 3190 ms. Trace:
I20260709 12:21:40.096410 27587 rpcz_store.cc:276] 0709 12:21:36.815823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.028675 (+212852us) service_pool.cc:224] Handling call
0709 12:21:37.299928 (+271253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.300695 (+   767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.997933 (+2697238us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54001fead0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:39.998110 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623079680}
I20260709 12:21:39.869710  3838 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.047545 27660 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
W20260709 12:21:40.091039  3915 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:21:40.100206 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:39.869333  3902 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.107313 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:39.993837 27580 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:21:40.111634 27580 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:40.112198 27580 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:21:40.112672 27580 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:40.113642 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3884) took 2355 ms. Trace:
I20260709 12:21:40.114143 27580 rpcz_store.cc:276] 0709 12:21:37.757746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.992914 (+2235168us) service_pool.cc:224] Handling call
0709 12:21:39.993354 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.113317 (+119963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.113572 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:39.803385  3898 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:39.850809  3833 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:39.912006  3867 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:39.996876 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3878) took 2243 ms. Trace:
I20260709 12:21:39.861733  3901 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.085731  3821 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.086701  3819 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.119971 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.118919 27583 rpcz_store.cc:276] 0709 12:21:37.753834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.753984 (+   150us) service_pool.cc:224] Handling call
0709 12:21:37.754284 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.882053 (+127769us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.882092 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.882135 (+    43us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:37.882178 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.996543 (+2114365us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:39.996606 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.996795 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293371648}
I20260709 12:21:40.125888 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.091528 27658 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:40.099264 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:39.988875  3783 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.101130 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:40.128122 27729 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:40.106086  3823 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:40.102510 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.128748 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.103739 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:40.104727 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.105348 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.106240 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.129285 27662 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:40.130357 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:40.130786 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.131675 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:39.933748 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.132266 27738 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:39.853631  3724 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:39.947638  3643 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:21:40.004302 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.136644 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.136950 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.138450 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.139037 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:40.139354 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.139796 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.140420 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:40.140650 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:40.141199 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.141376 27588 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:40.143682 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:40.144407 27732 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:39.952963  3824 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 3.104s	user 0.013s	sys 0.046s
I20260709 12:21:40.131385 27662 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:40.119431 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.134938 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.095080  3910 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.145363 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:40.147482 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.125262  3903 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:40.136327  3643 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:40.153143 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:40.137641  3823 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:40.096626  3907 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:40.161942 27731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3176) took 1669 ms (client timeout 2190 ms). Trace:
W20260709 12:21:40.219717 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.273581 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:40.167105 27581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3871) took 2467 ms (client timeout 1547 ms). Trace:
I20260709 12:21:40.273990 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:40.274454 27581 rpcz_store.cc:269] 0709 12:21:37.699418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.701062 (+  1644us) service_pool.cc:224] Handling call
0709 12:21:40.166921 (+2465859us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54000be5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:40.167017 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":408534400}
I20260709 12:21:40.253189 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.158310 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1488) took 3718 ms. Trace:
I20260709 12:21:40.238282 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:40.272677 27731 rpcz_store.cc:269] 0709 12:21:38.492173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.494420 (+  2247us) service_pool.cc:224] Handling call
0709 12:21:40.161771 (+1667351us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:40.161883 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1255601536}
I20260709 12:21:40.248567 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.244285 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.275964 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:40.247457 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:40.272665 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.276049 27657 rpcz_store.cc:276] 0709 12:21:36.440271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.465989 (+ 25718us) service_pool.cc:224] Handling call
0709 12:21:36.468583 (+  2594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.705149 (+1236566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.158093 (+2452944us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:40.158256 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2835769600}
I20260709 12:21:40.279126 27659 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:40.280419 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.168468 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.280566 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:40.156656  3921 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 12:21:40.155827  3614 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.640s	user 0.008s	sys 0.003s
I20260709 12:21:40.283272 27588 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:21:40.277060 27664 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:40.245765  3219 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.737s	user 0.012s	sys 0.000s
I20260709 12:21:40.284804 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.286270 27657 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:21:40.296232 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.297489 27731 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:40.297964 27657 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:21:40.298715 27582 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 21. (term mismatch)
I20260709 12:21:40.299170 27582 pending_rounds.cc:85] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 20
I20260709 12:21:40.299419 27582 pending_rounds.cc:107] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 30.21
I20260709 12:21:40.299700 27582 raft_consensus.cc:2889] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:40.300416 27657 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:21:40.301839  3614 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:40.288419  3913 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:21:40.310693 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.311026 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.311470 28332 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:40.315614 27657 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:40.315740 27658 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:21:40.243086  3885 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.390s	user 0.003s	sys 0.012s
W20260709 12:21:40.157137  3922 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260709 12:21:40.157852  3884 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.663s	user 0.001s	sys 0.001s
I20260709 12:21:40.152047 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.163843 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3183) took 1618 ms. Trace:
W20260709 12:21:40.156701  3882 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.715s	user 0.013s	sys 0.007s
I20260709 12:21:40.293736 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:40.292196 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.278123 28329 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 17.
I20260709 12:21:40.277326 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.329577 27588 raft_consensus.cc:2376] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32. Re-sending same reply.
I20260709 12:21:40.331676 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:40.332048 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:40.333096  3424 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:40.333559  3424 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:40.334021  3424 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:40.334590 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3737) took 2525 ms. Trace:
I20260709 12:21:40.337878 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.296211  3879 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.164690  3004 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:40.317459 27658 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:40.317621 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.319048 27657 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:40.323108 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:40.352922 27732 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:40.354331 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:40.355681 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.321192 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.322225 28331 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:21:40.325692 27662 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:40.357172 28331 pending_rounds.cc:85] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 18
I20260709 12:21:40.357553 28331 pending_rounds.cc:107] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:21:40.324898 27730 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:40.357818 28331 raft_consensus.cc:2889] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:40.324900 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:40.359477 27659 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:40.335091 27656 rpcz_store.cc:276] 0709 12:21:37.808861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.982908 (+174047us) service_pool.cc:224] Handling call
0709 12:21:37.989428 (+  6520us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:38.018186 (+ 28758us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:38.018230 (+    44us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:38.019086 (+   856us) log.cc:844] Serialized 78 byte log entry
0709 12:21:38.134995 (+115909us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:39.051048 (+916053us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.051619 (+   571us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.144154 (+ 92535us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.672989 (+528835us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b1800436cc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:21:39.781262 (+108273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.781437 (+   175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.242311 (+460874us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:40.334336 (+ 92025us) raft_consensus.cc:1685] finished
0709 12:21:40.334364 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.334522 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":300816,"prepare.queue_time_us":928592,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":147,"raft.queue_time_us":1821254,"raft.run_cpu_time_us":444,"raft.run_wall_time_us":45059,"spinlock_wait_cycles":12920320,"thread_start_us":900654,"threads_started":3,"wal-append.queue_time_us":647889}
I20260709 12:21:40.149860  3783 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:40.325210  3912 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.296722  3819 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:40.344416 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.326393 27734 rpcz_store.cc:276] 0709 12:21:38.545158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.182788 (+637630us) service_pool.cc:224] Handling call
0709 12:21:39.183191 (+   403us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:39.576919 (+393728us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.576952 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:39.609152 (+ 32200us) log.cc:844] Serialized 52 byte log entry
0709 12:21:39.801932 (+192780us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:40.109603 (+307671us) log.cc:844] Serialized 14 byte log entry
0709 12:21:40.123690 (+ 14087us) log.cc:844] Serialized 14 byte log entry
0709 12:21:40.123928 (+   238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.124006 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.163633 (+ 39627us) raft_consensus.cc:1685] finished
0709 12:21:40.163652 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.163790 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":235348,"prepare.queue_time_us":291281,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":249386,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":2282,"spinlock_wait_cycles":6044928,"thread_start_us":575373,"threads_started":3,"wal-append.queue_time_us":220767}
I20260709 12:21:40.374446 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.379137 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:40.379940 27734 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21. Re-sending same reply.
I20260709 12:21:40.381098 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.362130 27656 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:40.319625  3928 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:40.319746  3653 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:40.340240 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.368249 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:40.369220 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.389973 27732 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:40.383888 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.356969  3724 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.360371  3913 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:40.351261  3004 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:40.396763  3004 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.397742  3004 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:40.400768  3907 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.401988  3898 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.402618  3821 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.364219  3881 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.403650 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.404925  3875 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.405975  3929 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:40.367130  3783 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.364143  3927 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.407626  3927 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:40.408244  3890 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.508s	user 0.004s	sys 0.000s
I20260709 12:21:40.410073  3927 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.410763  3902 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.366448  3906 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.364790  3924 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:40.411576  3906 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.413311  3742 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.403645  3881 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.351534  3635 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:21:40.415241  3635 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:40.415602  3635 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:40.416587 27738 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:21:40.416378  3635 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.417514  3635 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:40.393715  3913 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:40.415618 27735 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:40.420424  3876 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.360447  3879 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.393988  3653 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.394913 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:40.424466 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.411651  3924 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.413725  3906 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:40.395370  3928 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.428947 28333 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:40.429857 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.434860 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:40.400556 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.437175 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.430945 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.431460 27737 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:40.385769 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.423800  3653 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:40.419167 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.440646 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:40.437304 27736 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:40.432475  3879 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.443588 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.446261  3754 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:40.419165  3913 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:40.453671 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:40.369458  3823 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.452389 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:40.454957  3734 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.003s
I20260709 12:21:40.434708  3723 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:40.388250  3819 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.463131  3924 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:40.463704  3928 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.465526  3552 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:40.465994  3552 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:40.460318  3219 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:40.468161  3219 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:40.468775  3219 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:40.450793  3841 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:40.432297  3424 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.474041  3901 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.475006  3900 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.478201  3508 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.460860  3723 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.485384  3723 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:40.512889  3643 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.471282  3742 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.324826 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:40.340205 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:40.330865 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.368626 27658 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:40.432868 27731 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:21:40.658536  3856 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:40.479365  3841 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:40.674734  3723 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.495232  3646 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:21:40.641676 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 171034us
I20260709 12:21:40.685913  3643 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:40.688544 27586 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:40.690775 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:40.692166 27659 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:40.692333 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 367389us
I20260709 12:21:40.692982 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:40.579111 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.694308 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.673929 27735 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
W20260709 12:21:40.695541 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.694383 27662 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:21:40.697428 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1341) took 2574 ms. Trace:
I20260709 12:21:40.526813  3913 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.596031 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3739) took 2748 ms. Trace:
I20260709 12:21:40.529836  3881 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:40.700441 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.699697 27655 rpcz_store.cc:276] 0709 12:21:37.846974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.983828 (+136854us) service_pool.cc:224] Handling call
0709 12:21:37.984115 (+   287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.984504 (+   389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.984532 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:37.985014 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:21:38.966183 (+981169us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:38.966241 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.966345 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.157332 (+1190987us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:40.382675 (+225343us) raft_consensus.cc:1685] finished
0709 12:21:40.382708 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.595651 (+212943us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2242,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":87,"spinlock_wait_cycles":283661056,"thread_start_us":980723,"threads_started":1,"wal-append.queue_time_us":1426320}
I20260709 12:21:40.697741  3349 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.004s
I20260709 12:21:40.704277  3878 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:40.704932  3878 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:40.705451  3878 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:40.708048 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.708712  3901 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:40.639804 27737 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:40.692376 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.497359  3783 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.685564  3299 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:40.713368  3783 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:40.713896 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:40.714726  3299 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:40.717778  3299 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:40.717851 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.720099  3299 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.722622  3299 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:40.700101  3424 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.716892  3913 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:40.627586 27738 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:40.822758  3201 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:40.823391  3201 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:21:40.824347 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.812494  3874 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:40.588266 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 295887us
I20260709 12:21:40.824991  3201 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.839493  3201 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:40.825212  3874 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.843583  3841 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.845216  3874 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:40.845389  3841 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:40.847975  3651 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:40.850446 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.850957 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.857729 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.590284 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3870) took 4155 ms. Trace:
W20260709 12:21:40.865038 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.742439 27582 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:40.698014 28331 rpcz_store.cc:276] 0709 12:21:38.123284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.123518 (+   234us) service_pool.cc:224] Handling call
0709 12:21:38.123905 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.358271 (+2234366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.651241 (+292970us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54005eabd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:40.697057 (+ 45816us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5054633344}
I20260709 12:21:40.734414 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.877620 27581 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:40.878839 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.519646  3819 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:40.879565 27703 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.25.202.62:43715, user_credentials={real_user=slave}} blocked reactor thread for 89856.9us
I20260709 12:21:40.749790 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.881476 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:40.882164 27580 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:40.882579  3878 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.884235 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:40.889148 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.893596 27482 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:21:40.774955 27588 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:40.605971 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1499) took 2619 ms. Trace:
I20260709 12:21:40.775198 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:40.747483  3725 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:40.911679 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.912539 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.913004 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.913308 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:40.913828 27583 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:40.914531 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.914849 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.915589 27582 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
W20260709 12:21:40.915880 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.006s	sys 0.026s
I20260709 12:21:40.916286  3903 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.921073 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.922000 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.922863 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.923826 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.924867 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:40.927690 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71145.4us
I20260709 12:21:40.928417  3818 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:40.928982 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:40.929762 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.970775  3845 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.971510  3845 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.971812  3818 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:21:40.971909 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54323.6us
I20260709 12:21:40.972325  3818 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:40.972728 27553 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39856 dropped due to backpressure: service queue is full with 50 items
I20260709 12:21:40.902486 27661 rpcz_store.cc:276] 0709 12:21:37.986208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.470486 (+484278us) service_pool.cc:224] Handling call
0709 12:21:38.471193 (+   707us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:38.471906 (+   713us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:38.471934 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:38.472406 (+   472us) log.cc:844] Serialized 26 byte log entry
0709 12:21:38.903078 (+430672us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:38.903134 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.903231 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.407421 (+1504190us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:40.605226 (+197805us) raft_consensus.cc:1685] finished
0709 12:21:40.605272 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.605712 (+   440us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":151284608,"thread_start_us":430215,"threads_started":1,"wal-append.queue_time_us":427373}
I20260709 12:21:40.973733  3818 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.937316 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.859381 27584 rpcz_store.cc:276] 0709 12:21:36.434293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.132587 (+698294us) service_pool.cc:224] Handling call
0709 12:21:37.133117 (+   530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.133439 (+   322us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:37.274711 (+141272us) log.cc:844] Serialized 14 byte log entry
0709 12:21:40.168143 (+2893432us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b1800468340. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:21:40.589137 (+420994us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:40.589280 (+   143us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:40.589440 (+   160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.589623 (+   183us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.589956 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":281029,"prepare.queue_time_us":138828,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":15472,"raft.queue_time_us":319861,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":278,"spinlock_wait_cycles":94004480,"thread_start_us":174220,"threads_started":1,"wal-append.queue_time_us":3170034}
I20260709 12:21:40.923061  3910 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:40.943595 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.976118 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.745774  3672 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:40.741583  3552 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:40.953284 27580 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:40.977469 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:40.953866 27581 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:40.955701  3004 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.557s	user 0.003s	sys 0.019s
I20260709 12:21:40.977975  3903 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.959451  3630 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:40.980230 27731 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:21:40.973048 27553 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39856 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3884)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3898)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3894)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3893)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1531)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1532)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1535)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1537)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1538)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1539)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1542)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3879)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3880)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3886)
I20260709 12:21:40.981346  3819 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:40.981829  3819 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:40.982241  3819 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:40.969357 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.007s	sys 0.026s
W20260709 12:21:40.983747 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 631294us
I20260709 12:21:40.986223  3723 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:40.988824 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3872) took 3289 ms (client timeout 1793 ms). Trace:
W20260709 12:21:40.989285 27589 rpcz_store.cc:269] 0709 12:21:37.699751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.759179 (+ 59428us) service_pool.cc:224] Handling call
0709 12:21:40.988433 (+3229254us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54000bc7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:40.988624 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5843939072}
I20260709 12:21:40.990597 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.939905 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:40.991040 27585 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:40.992470 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:40.993057 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.993423 27581 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:40.992967 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3880) took 3238 ms. Trace:
I20260709 12:21:40.994215 27585 rpcz_store.cc:276] 0709 12:21:37.754433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.935152 (+180719us) service_pool.cc:224] Handling call
0709 12:21:40.990439 (+3055287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.991499 (+  1060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.992727 (+  1228us) spinlock_profiling.cc:243] Waited 3.03 s on lock 0x7b54000bc7d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:40.992899 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6975275904}
I20260709 12:21:40.992736 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:40.995704 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:40.995954 27663 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:21:40.996429  3911 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:40.996603 27585 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:40.996853 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:40.997354 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3720) took 4785 ms. Trace:
I20260709 12:21:40.997978  3911 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:21:40.998199  3791 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.998603  3911 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:40.998838  3791 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.000205  3911 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:40.994995 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.997785 27663 rpcz_store.cc:276] 0709 12:21:36.211701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.244660 (+ 32959us) service_pool.cc:224] Handling call
0709 12:21:36.273155 (+ 28495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.996352 (+4723197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.997061 (+   709us) spinlock_profiling.cc:243] Waited 4.72 s on lock 0x7b5400291154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:40.997283 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10859788672}
I20260709 12:21:41.003274 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:41.005859  3004 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:41.006018 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:41.006520  3696 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:41.006534  3004 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:41.007815 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:41.008482 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.009227 27582 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:40.982048 27553 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39856 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3884)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3898)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3894)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3893)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1531)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1532)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1535)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1537)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1538)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1539)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1542)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3874)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3879)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3880)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3886)
I20260709 12:21:41.011849 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.012387  3903 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:41.012619 27584 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:41.013628 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.013725 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.014562 27584 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:21:41.015381 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.015795  3818 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:41.008697  3004 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.020247  3903 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:41.021543  3791 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.009913  3552 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:40.974205  3845 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:41.011337 27735 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:41.011379  3862 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:41.025360 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:41.010309 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.028318 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.028813 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.029914 27583 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:41.030714 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:40.994585 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.031167 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:41.025218 27589 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:41.035457 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:41.024266 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50705.4us
I20260709 12:21:41.032718 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:41.037261 27583 raft_consensus.cc:2376] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24. Re-sending same reply.
I20260709 12:21:41.045140 27584 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:21:41.045902 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.046269 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.047951 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:41.048614 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.047115 27584 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:41.043023 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.055054 27735 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:41.055377 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.055550  3791 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:41.056079  3791 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:41.056540  3791 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:41.072537 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:41.078823 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:41.077173 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.078629 27738 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:41.082461 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.078617  3723 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.006s
I20260709 12:21:41.085598 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.086503 27657 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:21:41.086426  3878 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.079047  3856 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:41.089131 27658 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:21:41.091035 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033345
I20260709 12:21:41.092578 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.067163 27581 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:41.098405 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:41.098435 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65854.7us
I20260709 12:21:41.099586 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.100417 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:41.110332 27664 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:41.111147 27659 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:41.111120 27660 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:21:41.113627 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3849) took 4878 ms. Trace:
I20260709 12:21:41.101684 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:41.108186 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.114159 27586 rpcz_store.cc:276] 0709 12:21:36.228148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.231387 (+  3239us) service_pool.cc:224] Handling call
0709 12:21:36.259371 (+ 27984us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.482380 (+223009us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4c00427200. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98614a20 00007f0e98625a29 00007f0e985eb644 00007f0e98612f2d 00007f0e98612b93 00007f0e9a34318b 00007f0e9a343cb5 00007f0e9a33ef30 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416
0709 12:21:37.931258 (+1448878us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b5400202720. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c606f 00007f0e9a33ef60 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62
0709 12:21:40.687386 (+2756128us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b58006a55d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:21:41.081966 (+394580us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:41.082012 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.093356 (+ 11344us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:41.093422 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.093521 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.106419 (+ 12898us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3202,"raft.queue_time_us":15529,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":2771,"spinlock_wait_cycles":4997610368}
I20260709 12:21:41.118708 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.121085 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.125360 27589 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:41.139791 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.143687 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.140460 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.122933  3349 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:41.147616 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:41.149055 27585 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:41.167691 27589 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:41.150427 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.150933  3299 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:41.152582  3791 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.155577  3201 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:41.164781  3219 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.695s	user 0.012s	sys 0.003s
I20260709 12:21:41.165407 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:41.173523 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.180557 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:41.182473 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:41.182642  3723 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:41.186020 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:21:41.187124 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.187804 27581 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:21:41.188370 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.188800 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.189576 27581 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:41.190446 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.190199  3299 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.175808 27586 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:41.145268  3826 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:41.191298 27581 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:41.192842 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:41.193600 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:41.194802 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119893us
I20260709 12:21:41.204599 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.201138 27587 raft_consensus.cc:2376] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27. Re-sending same reply.
I20260709 12:21:41.187626  3646 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:41.208385 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67296.3us
I20260709 12:21:41.210000 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.211853 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.212447 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.213264 27583 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:21:41.223019 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.223989 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:41.230935 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.222210 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:41.232159 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.229565 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:41.229099 27584 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:21:41.235184 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 157788us
W20260709 12:21:41.236460 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.237416 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:41.239465 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:41.240370 28328 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:41.203503  3201 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.243635  3201 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.244741 27482 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:41.235267  3299 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:41.252612 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:41.256755 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.262120 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.192061  3219 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.268824 27585 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 15.
I20260709 12:21:41.270255 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.272846 27581 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:41.273938 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.274611 27582 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
W20260709 12:21:41.277894 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.282953 27586 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:41.292183 27584 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 26 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:21:41.266706  3643 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:41.211476  3826 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.294159 28329 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:41.299522 28331 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:41.310397 27587 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:41.327618 27581 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:41.330667 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.264771  3723 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:21:41.296469  3643 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:41.337209 27582 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:41.338341 27586 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:41.329803  3630 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.012s
I20260709 12:21:41.347692  2667 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:21:41.324656  3791 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.334846 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:41.340965 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1532) took 1049 ms. Trace:
I20260709 12:21:41.341576 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3892) took 1060 ms. Trace:
I20260709 12:21:41.339284 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3893) took 1052 ms. Trace:
I20260709 12:21:41.342877 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1531) took 1053 ms. Trace:
I20260709 12:21:41.345623  3819 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.352140 27586 rpcz_store.cc:276] 0709 12:21:40.289983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.293018 (+1003035us) service_pool.cc:224] Handling call
0709 12:21:41.333359 (+ 40341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.338793 (+  5434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.338992 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.352471 27582 rpcz_store.cc:276] 0709 12:21:40.289708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.285654 (+995946us) service_pool.cc:224] Handling call
0709 12:21:41.333554 (+ 47900us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.342543 (+  8989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.342806 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.353983 27581 rpcz_store.cc:276] 0709 12:21:40.281469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.280401 (+998932us) service_pool.cc:224] Handling call
0709 12:21:41.325093 (+ 44692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.328021 (+  2928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.329751 (+  1730us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.364585 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.352398 27585 rpcz_store.cc:276] 0709 12:21:40.275028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.285172 (+1010144us) service_pool.cc:224] Handling call
0709 12:21:41.331874 (+ 46702us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.335424 (+  3550us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.335461 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.335513 (+    52us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:41.335546 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.335609 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.335713 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.339087  3643 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:41.359634 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:41.370860 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.375098 27729 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:41.376390  3856 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.377166  3856 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.355427  3878 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.372884  2667 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:41.383841 27580 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:41.385102  3819 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:41.385640  3940 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.282s	user 0.002s	sys 0.002s
I20260709 12:21:41.364367  3646 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:41.398538 27587 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:41.400472 27581 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:21:41.401539 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3645) took 4066 ms. Trace:
I20260709 12:21:41.403136  3201 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.409369 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:41.412967 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1533) took 1122 ms. Trace:
I20260709 12:21:41.413543 27587 rpcz_store.cc:276] 0709 12:21:40.290241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.332902 (+1042661us) service_pool.cc:224] Handling call
0709 12:21:41.379539 (+ 46637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.412599 (+ 33060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.412890 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:41.421715 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:41.426688 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.433857 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:41.406105 27733 rpcz_store.cc:276] 0709 12:21:37.335210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.336107 (+   897us) service_pool.cc:224] Handling call
0709 12:21:37.336479 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.336902 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.336953 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:37.337497 (+   544us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.689720 (+3352223us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b1800432e80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:21:40.958972 (+269252us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:40.959024 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.965392 (+  6368us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:41.384846 (+419454us) spinlock_profiling.cc:243] Waited 9.83 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:41.401048 (+ 16202us) raft_consensus.cc:1685] finished
0709 12:21:41.401078 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.401307 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":47772,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":56361,"spinlock_wait_cycles":25559040,"thread_start_us":214013,"threads_started":1,"wal-append.queue_time_us":3703438}
I20260709 12:21:41.438931  3709 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:41.439558  3709 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:41.446512 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87052us
W20260709 12:21:41.445307 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 118278us
I20260709 12:21:41.440022  3709 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:41.450606 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:41.452946 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:41.454763 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.454479  3754 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:21:41.455533  3819 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.456087 27587 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:41.457180  3552 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:21:41.460724 27480 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 226) took 38960 ms (client timeout 15000 ms). Trace:
I20260709 12:21:41.461853 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1536) took 1168 ms. Trace:
I20260709 12:21:41.418692  3878 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.466519 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:41.470675 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1534) took 1180 ms. Trace:
W20260709 12:21:41.451915 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1925513110002157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260709 12:21:41.477062 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1268513110001095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
W20260709 12:21:41.477805 27552 connection.cc:256] RPC call timeout handler was delayed by -1.116051311000092s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20260709 12:21:41.478200 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:41.478308 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1106513110000833s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20260709 12:21:41.478941 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0674513110002408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20260709 12:21:41.473764 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 174268us
I20260709 12:21:41.474967 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1535) took 1181 ms. Trace:
I20260709 12:21:41.473120 27583 rpcz_store.cc:276] 0709 12:21:40.293420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.460920 (+1167500us) service_pool.cc:224] Handling call
0709 12:21:41.461282 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.461495 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.461523 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.461587 (+    64us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:41.461617 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.461674 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.461778 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.433333  3201 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.471870 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:41.479773 27587 rpcz_store.cc:276] 0709 12:21:40.293113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.435700 (+1142587us) service_pool.cc:224] Handling call
0709 12:21:41.455576 (+ 19876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.474764 (+ 19188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.474923 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.482733 27586 rpcz_store.cc:276] 0709 12:21:40.292589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.417686 (+1125097us) service_pool.cc:224] Handling call
0709 12:21:41.443141 (+ 25455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.470448 (+ 27307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.470467 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.470499 (+    32us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:41.470518 (+    19us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.470561 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.470634 (+    73us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:41.477432  3725 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.529s	user 0.012s	sys 0.001s
I20260709 12:21:41.493624 27585 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:21:41.499636 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1537) took 1206 ms. Trace:
I20260709 12:21:41.500026 27585 rpcz_store.cc:276] 0709 12:21:40.293690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.465397 (+1171707us) service_pool.cc:224] Handling call
0709 12:21:41.493317 (+ 27920us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.499245 (+  5928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.499576 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.504012 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3900) took 1213 ms. Trace:
I20260709 12:21:41.480599  3646 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:41.504429 27582 rpcz_store.cc:276] 0709 12:21:40.290758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.403976 (+1113218us) service_pool.cc:224] Handling call
0709 12:21:41.465165 (+ 61189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.473063 (+  7898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.473101 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.473158 (+    57us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:41.483438 (+ 10280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.483513 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.503831 (+ 20318us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b54001fe5d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:41.503956 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56412032}
I20260709 12:21:41.509418 27583 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:41.511468  3862 raft_consensus.cc:2764] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:21:41.502528  3791 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:41.515693  3791 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:41.515796  3726 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.516201  3791 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:41.516441  3726 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.518705  3726 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.510210  3646 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.520876  3646 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:41.521602 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 184891us
I20260709 12:21:41.510427  3819 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.529407  3201 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.530249  3917 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:21:41.483126  3911 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:41.536343 27588 raft_consensus.cc:2410] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:21:41.544041 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1538) took 1217 ms. Trace:
I20260709 12:21:41.544484 27583 rpcz_store.cc:276] 0709 12:21:40.293995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.480492 (+1186497us) service_pool.cc:224] Handling call
0709 12:21:41.508297 (+ 27805us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.510687 (+  2390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.510920 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.542725 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:41.548923 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77152us
I20260709 12:21:41.550891 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 3897) took 1265 ms. Trace:
I20260709 12:21:41.555491 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.555998 27587 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:41.551348 27580 rpcz_store.cc:276] 0709 12:21:40.285204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.994630 (+709426us) service_pool.cc:224] Handling call
0709 12:21:41.550699 (+556069us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:41.550837 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":877216512}
I20260709 12:21:41.563396 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:41.563670 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62239.4us
I20260709 12:21:41.564405 27662 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:41.564982 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:41.566583  3935 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:41.567643  3935 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:41.557456  3917 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.571890  3917 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.572858 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1541) took 1277 ms. Trace:
I20260709 12:21:41.574201  3917 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.574561 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:41.573424 27586 rpcz_store.cc:276] 0709 12:21:40.294975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.498537 (+1203562us) service_pool.cc:224] Handling call
0709 12:21:41.572191 (+ 73654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.572472 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.572503 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.572565 (+    62us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:41.572596 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.572652 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.572785 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:41.579717 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 109673us
W20260709 12:21:41.581946 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.576007 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:41.542006  3911 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:41.587906 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1539) took 1293 ms. Trace:
I20260709 12:21:41.588534 27587 rpcz_store.cc:276] 0709 12:21:40.294308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.486765 (+1192457us) service_pool.cc:224] Handling call
0709 12:21:41.555827 (+ 69062us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.560805 (+  4978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.587829 (+ 27024us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.583025 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.591085  3831 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.591775 27658 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:41.591913  3831 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.593694 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1530) took 1304 ms. Trace:
I20260709 12:21:41.594185 27588 rpcz_store.cc:276] 0709 12:21:40.289421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.994014 (+704593us) service_pool.cc:224] Handling call
0709 12:21:41.593444 (+599430us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54000bdbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:41.593621 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172511488}
I20260709 12:21:41.597833  3856 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.598523  3219 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.598913 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.599211  3911 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:41.601882 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3874) took 1266 ms. Trace:
I20260709 12:21:41.605998  3826 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.585964 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.602409 27582 rpcz_store.cc:276] 0709 12:21:40.335435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.561109 (+1225674us) service_pool.cc:224] Handling call
0709 12:21:41.600802 (+ 39693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.601130 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.601172 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.601256 (+    84us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:41.601587 (+   331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.601687 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.601817 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:41.608940 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55452.8us
I20260709 12:21:41.615383 27588 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 15. (term mismatch)
I20260709 12:21:41.615823 27588 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 14
I20260709 12:21:41.576308  3917 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:41.621403 27580 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:21:41.626412 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.616034 27588 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:21:41.621978 27580 pending_rounds.cc:85] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:41.630921 27580 pending_rounds.cc:107] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:21:41.643607  3349 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.629678 27588 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:41.645170 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:41.631228 27580 raft_consensus.cc:2889] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:41.640746 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:41.644400  3349 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:41.630251  3818 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.614s	user 0.007s	sys 0.007s
I20260709 12:21:41.653530 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.656044 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:41.622236  3917 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:41.657918 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:41.666034 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.666271 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:41.667218 27663 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:41.668946 27588 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:21:41.670190 27588 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:41.670629 27585 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:21:41.671907 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3886) took 1071 ms. Trace:
I20260709 12:21:41.672536 27580 rpcz_store.cc:276] 0709 12:21:40.600623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.588610 (+987987us) service_pool.cc:224] Handling call
0709 12:21:41.620923 (+ 32313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.671636 (+ 50713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.671843 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.671254 27585 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:41.577203  3709 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.673671 27585 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:21:41.676726 27585 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:41.618333  3219 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:21:41.678119 27587 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 12:21:41.681208 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.660157  3917 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:41.662350  3935 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:41.662993  3723 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:21:41.473065 27480 rpcz_store.cc:269] 0709 12:21:02.499930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.500191 (+   261us) service_pool.cc:224] Handling call
0709 12:21:41.460431 (+38960240us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:32.007660 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:32.008378 (+   718us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:32.008448 (+    70us) write_op.cc:435] Acquired schema lock
0709 12:21:32.008477 (+    29us) tablet.cc:662] Decoding operations
0709 12:21:32.009465 (+   988us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:32.009557 (+    92us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:32.009589 (+    32us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:21:32.010276 (+   687us) tablet.cc:701] Row locks acquired
0709 12:21:32.010326 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:21:32.010491 (+   165us) write_op.cc:270] Start()
0709 12:21:32.010639 (+   148us) write_op.cc:276] Timestamp: P: 1783599692010425 usec, L: 0
0709 12:21:32.010673 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 12:21:32.028976 (+ 18303us) log.cc:844] Serialized 2901 byte log entry
0709 12:21:32.430911 (+401935us) op_driver.cc:464] REPLICATION: finished
0709 12:21:32.631293 (+200382us) write_op.cc:301] APPLY: starting
0709 12:21:32.641065 (+  9772us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:32.834389 (+193324us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:32.834519 (+   130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:33.190908 (+356389us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:33.191293 (+   385us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=14,mrs_lookups=0
0709 12:21:33.191409 (+   116us) write_op.cc:312] APPLY: finished
0709 12:21:33.191883 (+   474us) log.cc:844] Serialized 73 byte log entry
0709 12:21:33.201000 (+  9117us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:33.207838 (+  6838us) write_op.cc:454] Released schema lock
0709 12:21:33.218248 (+ 10410us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:33.239856 (+ 21608us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:33.239889 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23368,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":156706,"apply.run_cpu_time_us":15311,"apply.run_wall_time_us":624665,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33387,"num_ops":7,"prepare.queue_time_us":165152,"prepare.run_cpu_time_us":12532,"prepare.run_wall_time_us":148266,"raft.queue_time_us":108560,"raft.run_cpu_time_us":5062,"raft.run_wall_time_us":52350,"replication_time_us":420043,"thread_start_us":362772,"threads_started":4,"wal-append.queue_time_us":176979,"wal-append.run_cpu_time_us":12503,"wal-append.run_wall_time_us":1998324}]]}
I20260709 12:21:41.685608  3672 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:41.706037 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:41.710402 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 134558us
I20260709 12:21:41.718515 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:41.701102  3935 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:41.702606  3878 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:41.717937  3841 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.872s	user 0.015s	sys 0.001s
I20260709 12:21:41.740514 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:41.742399 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:41.742372 28334 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:41.708683  3791 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.745226 28332 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:41.748172 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:41.762883 27589 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.566s	user 0.005s	sys 0.005s
I20260709 12:21:41.763607 27589 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:41.764546 27581 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:41.742506  3349 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.767407 27658 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:41.756390  3841 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:41.766855  3004 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:41.767896  3878 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:41.771883  3349 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:41.790068  3841 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:41.789148  3004 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:41.828372 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 172445us
I20260709 12:21:41.798432  3878 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.824247  3826 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.828574  3841 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:41.860132 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:41.851378  3004 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:41.880070 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:41.880862 28332 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:21:41.885020 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 166690us
I20260709 12:21:41.887583 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:41.889976 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.895642 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:41.897006 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.897799 27731 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:41.898702 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:41.899926 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 242104us
W20260709 12:21:41.895190 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 330351us
I20260709 12:21:41.904836 28332 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:41.905100 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:41.869836  3709 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:41.910744 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.869015  3552 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:41.915864  3947 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:21:41.917703  3961 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:41.918195  3961 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.918560  3961 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:41.859089  3878 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:41.928251 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.876148  3831 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:41.934140 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:41.948559 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.950840 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:41.951895 27738 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:41.949261 27729 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:41.956180 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:41.887759  3826 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:41.971475 28331 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:41.930739  3630 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:21:41.959690  3552 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.983563  3552 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:41.984401  3552 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:41.997606 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:41.998600 27738 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:42.010519 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.015434 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:42.015731 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 197373us
I20260709 12:21:41.966827  3831 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:41.886925  3935 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:41.985472  3791 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.020145 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.023461  3956 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:21:42.024902  3643 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.000s
I20260709 12:21:42.025326  3643 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.027446  3952 raft_consensus.cc:2764] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
W20260709 12:21:42.025668  3646 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.504s	user 0.009s	sys 0.001s
W20260709 12:21:42.035274 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 212361us
I20260709 12:21:42.035562  3643 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.036203 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.036463 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1542) took 1741 ms. Trace:
I20260709 12:21:42.037612 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.036881 27584 rpcz_store.cc:276] 0709 12:21:40.295244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.488103 (+1192859us) service_pool.cc:224] Handling call
0709 12:21:41.488589 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:41.501365 (+ 12776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.501395 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:41.545908 (+ 44513us) log.cc:844] Serialized 26 byte log entry
0709 12:21:41.811217 (+265309us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:41.811278 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.811413 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.036241 (+224828us) raft_consensus.cc:1685] finished
0709 12:21:42.036266 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.036409 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3069,"raft.queue_time_us":26449,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":215395,"threads_started":1,"wal-append.queue_time_us":336474}
W20260709 12:21:42.032013 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 108547us
W20260709 12:21:42.032850 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.040127 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.032805 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 127757us
W20260709 12:21:42.043874 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88137.7us
W20260709 12:21:42.044448 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.044879 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.044497 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.048010 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.049432 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.056181 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:42.058921 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.041976  3954 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:42.060879 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:42.041882  3862 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:42.063226 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.040951  3646 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.064782 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:42.068444  3672 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:42.070531  3962 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:42.074055 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.077368  3956 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:21:42.061296  2667 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.666s	user 0.006s	sys 0.005s
I20260709 12:21:42.080209 27657 raft_consensus.cc:2410] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:21:42.078461  3954 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.095765 27663 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:42.096578 27731 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: [29.15-29.15]
I20260709 12:21:42.097313 27658 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:42.100747  2667 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:42.103017  3954 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:42.104257 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.105837 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.106940  3958 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:42.107630  3959 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:42.108605  3818 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:42.109030  3818 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.109393  3818 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:42.041435  3917 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.112951 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.113502 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.114494 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.115099  3299 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:42.115625  3299 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:42.116917  3299 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.118634  3299 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:42.123592  3723 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:42.124321  3723 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.124750  3723 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:42.128170  3953 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:42.128692  3953 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:42.129801  3953 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:42.135444 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64557.6us
I20260709 12:21:42.137017 27586 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:42.137326  3826 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.137893 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3880) took 1800 ms. Trace:
I20260709 12:21:42.138370 27586 rpcz_store.cc:276] 0709 12:21:40.337336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.580944 (+1243608us) service_pool.cc:224] Handling call
0709 12:21:42.136488 (+555544us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:21:42.136758 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.137562 (+   804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.137801 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11650816}
I20260709 12:21:42.139748  3962 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:42.139659 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81561us
I20260709 12:21:42.140434  3962 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.140902  3962 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:42.141080 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.142282  3935 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:42.142628 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:42.142714  3935 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:42.143139  3819 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.143636  3935 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:42.143850 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3873) took 1822 ms. Trace:
I20260709 12:21:42.144305 27585 rpcz_store.cc:276] 0709 12:21:40.318165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.506973 (+1188808us) service_pool.cc:224] Handling call
0709 12:21:41.586496 (+ 79523us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.683950 (+ 97454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.140490 (+456540us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b54000bd1d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:42.140670 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178983168}
I20260709 12:21:42.146884 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3879) took 1809 ms. Trace:
I20260709 12:21:42.147287 27583 rpcz_store.cc:276] 0709 12:21:40.337048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.582505 (+1245457us) service_pool.cc:224] Handling call
0709 12:21:42.146010 (+563505us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:21:42.146239 (+   229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.146496 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.146530 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:42.146595 (+    65us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:42.146630 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.146693 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.146822 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3608448}
I20260709 12:21:42.149458  3646 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.150334 27660 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:42.115752  2667 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:42.152927  3630 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:42.153048 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.153417  3630 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:42.153862  3630 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:42.154197  3903 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:42.154835  3903 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:42.101438  3831 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:21:42.155798  3831 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:42.156328  3831 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:21:42.156250  3903 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.159986  3791 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:21:42.160147 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51023.7us
I20260709 12:21:42.160480  3791 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.160975  3791 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:42.162133 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63743.7us
I20260709 12:21:42.166426 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:42.172116 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.110692 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49800.7us
I20260709 12:21:42.159379  3935 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.171804 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.161242  3862 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:42.180047 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:42.185652  3963 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:21:42.187587  3953 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:42.183696  3862 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.191598  3862 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:42.195881 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.197369 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:42.198431  3965 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:42.199273  3965 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:42.199800  3965 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:42.200865  3965 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.180799 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:42.208670 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58119.9us
W20260709 12:21:42.209456 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 135467us
I20260709 12:21:42.215080  2667 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:42.219110 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.225147  2667 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:42.225950 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1543) took 1930 ms. Trace:
I20260709 12:21:42.226326 27581 rpcz_store.cc:276] 0709 12:21:40.295632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.490357 (+1194725us) service_pool.cc:224] Handling call
0709 12:21:41.764296 (+273939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.765060 (+   764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.225769 (+460709us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540011bf50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:42.225897 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628887936}
W20260709 12:21:42.232182 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60302.4us
I20260709 12:21:42.232954 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.231722  3958 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:42.235942  3845 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:21:42.248838  3948 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:42.250048  3862 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.204262  3646 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:21:42.252312  3862 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.253904  3646 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:42.254521  3646 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.255019  3646 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:42.204424  3791 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:42.278216 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79140.4us
W20260709 12:21:42.280053 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 174327us
I20260709 12:21:42.240507  3219 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:21:42.276845 27475 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.243s	user 0.000s	sys 0.000s
I20260709 12:21:42.230109  3709 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:21:42.226441  3696 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:21:42.223832  3952 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
W20260709 12:21:42.322548 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 154313us
I20260709 12:21:42.263182  3917 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:42.345428 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81772.6us
I20260709 12:21:42.346212  3672 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:42.346784  3982 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:42.346843  3672 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.347652  3982 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.350096  3982 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.350097  3672 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:42.258555  3961 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.352056 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:42.356026 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 108747us
I20260709 12:21:42.337164  3349 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:42.371155 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55078.7us
I20260709 12:21:42.372848 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.315219  3954 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.378732 27660 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: [27.16-27.16]
W20260709 12:21:42.380510 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 68571us
I20260709 12:21:42.382434  3791 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.383422  3961 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.375941  3004 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.384487  3791 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:42.384977  3791 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:42.386199  3004 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.385910  3791 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.392418 28334 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:42.359598  3723 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.392643  3672 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:42.407339 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.370033  3349 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.370177  3962 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.415994  3754 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:42.416445  3754 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:42.395424  3630 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.421146  3630 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:42.411774 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:42.427819  3954 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.429785  3962 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:42.432169  3911 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.832s	user 0.013s	sys 0.000s
I20260709 12:21:42.432622  3911 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:42.443959 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77880us
I20260709 12:21:42.405910  3004 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:21:42.451016 27657 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:42.452664 27659 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:42.453285 27658 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.15-28.15]
I20260709 12:21:42.454308 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.454511 27661 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:42.441987  3965 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.436411  3911 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.458954  3723 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.459364  3299 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:42.462802 27656 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:42.467554 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:42.470343 27581 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:42.452749  3845 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:21:42.473338 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 239052us
I20260709 12:21:42.443665  3953 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 12:21:42.480717 27586 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:21:42.482542 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:42.487087 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.478258  3972 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
W20260709 12:21:42.486730 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144637us
I20260709 12:21:42.491735 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:42.486815 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:42.489912 27583 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:42.485407 27586 pending_rounds.cc:85] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:42.495187 27586 pending_rounds.cc:107] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:21:42.495566 27586 raft_consensus.cc:2889] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:42.499498 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:42.462167  3831 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.486931  3954 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:21:42.507278 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.508752 27738 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:42.503727 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:42.506592 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:42.513149 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49132.9us
W20260709 12:21:42.515293 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.506209  3970 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:21:42.515977 27662 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:42.473316  3965 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.519695  3983 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.531627 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.533947 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.494329  3903 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:21:42.510277  3845 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:42.539103 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 125643us
W20260709 12:21:42.518054  3841 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.000s
I20260709 12:21:42.541052 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:42.544215 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.510474  3954 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:42.537667  3971 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:21:42.519953  3990 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:42.559980 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.563935 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.550794  3954 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:42.566884 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:42.574821 27627 leader_election.cc:336] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.617s (SENT)
I20260709 12:21:42.576661 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.579054 27737 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:42.551599  3962 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:42.586850 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.587167  2667 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:42.587949  2667 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.588653 27730 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:42.588780  2667 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.564478  3990 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.584280  3962 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.597738  3962 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:42.546326  3723 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:21:42.541815  3983 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.549750  3956 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:21:42.605101 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.549625  3841 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.608764 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:42.609257  2667 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.599565  3985 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:42.565173  3201 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:21:42.620388  3976 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:42.620836  3976 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.621236  3976 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:42.623454  3988 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:21:42.624681  3967 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.625229  3967 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:42.627578  3986 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
W20260709 12:21:42.628768  3992 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:21:42.630237  3987 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:21:42.633064 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:42.633726 27663 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:42.635926  3349 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.636359 27660 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 18. (term mismatch)
I20260709 12:21:42.636746 27660 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 17
I20260709 12:21:42.636992 27660 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:21:42.637285 27660 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:42.637025  3845 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.638167  2667 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:42.638509  2667 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:42.629524 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.537s	user 0.046s	sys 0.003s Metrics: {"bytes_written":5821,"delete_count":0,"lbm_write_time_us":14386,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":225}
I20260709 12:21:42.600986  3819 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:42.641320 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:42.588037  3862 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:21:42.616556 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.650820 28332 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:21:42.639180  2667 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.595670  3990 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:42.580356  3994 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:21:42.612324  3985 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.637048  3956 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:42.661022 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:42.655364 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.662089 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53382.7us
I20260709 12:21:42.660849 27660 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:42.663014  2667 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:42.663064 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:42.665122 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78366.7us
I20260709 12:21:42.651160  3819 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.668674 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:42.671746 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.672813 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:42.655216  3349 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:42.678249 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:42.682560 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 165099us
I20260709 12:21:42.665252  3696 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:42.684604 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.664927  3646 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.699640  3954 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.700199  3646 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.701496 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.701766  3954 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.702358  3954 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:42.702723  3954 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:42.702833 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.703130  3954 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:42.703665 27655 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:42.704427  3981 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:42.704937  3981 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:42.705415  3981 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:42.692818 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.316117
I20260709 12:21:42.665557  3985 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:42.666627  3956 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:42.697190 27660 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:42.678714  3349 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.678839  3819 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:42.716661 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 120350us
W20260709 12:21:42.717554 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48741.2us
I20260709 12:21:42.709923 27731 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:42.719274 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:42.724954 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.725183 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.725723 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.727993 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.683717  3967 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:42.695976  3696 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:42.716835  3672 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:42.734880 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:42.736649 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.707360  3831 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.723783  3349 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:42.750519 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.757925 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.762547 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.763533  3993 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:42.764617 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.765023  3993 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:42.736851  3973 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:42.760538 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:42.738279  3630 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:21:42.759109  3967 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:42.785668 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.750559  3862 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:21:42.794813 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58247.8us
I20260709 12:21:42.797039  3982 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:42.797462  3983 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.797477  3982 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:42.798466  3969 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.459s	user 0.003s	sys 0.001s
I20260709 12:21:42.798838  3982 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.800218  3982 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:42.799453  3878 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.856s	user 0.013s	sys 0.000s
I20260709 12:21:42.808403  3878 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.810761  3878 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.811481 27729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:42.815466  3997 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:21:42.815491  3878 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.816378  3878 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.818073  3965 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.818997  3878 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.821002  3935 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.828423  3696 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.831650  3841 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:21:42.781680  3201 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:42.778872  3973 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.782398  3952 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:42.816068  3997 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:42.845054  3997 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.846343  3997 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:21:42.848109 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62548.3us
I20260709 12:21:42.837514 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.830143  3696 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:42.838574 27736 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:42.839668 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:42.849713 27585 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:21:42.847491  3987 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:42.805851  3967 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:42.873572 27729 raft_consensus.cc:606] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:42.793687  3993 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:21:42.793910  3959 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:21:42.848150  3973 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:42.883694 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148950us
I20260709 12:21:42.872833  3862 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.874266  3646 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:42.891829  3646 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:42.892277  3646 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:42.887997  4013 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:21:42.786980  3956 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:42.882777  3818 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.773s	user 0.010s	sys 0.003s
I20260709 12:21:42.879295  3983 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:42.911512  3818 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.914171  3956 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:42.903970  4013 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:42.915630 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:42.916179 27584 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:42.916774  3981 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.918627  3976 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.936111 27581 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:42.903189  3862 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.958195  3862 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:42.941596 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.954102 27582 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:21:42.960511 27582 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 16
I20260709 12:21:42.960762 27582 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:21:42.960989 27582 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:42.962076 27581 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:42.933629 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:42.939360 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:42.940909  3976 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.941639  3818 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:42.941830  3219 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:42.942109 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:42.945618 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:42.947513 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1254) took 1028 ms. Trace:
I20260709 12:21:42.980087  3219 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:42.950722 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.971788 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.981354 28329 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:42.973402 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 136009us
I20260709 12:21:42.982661 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:42.982177  3219 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:42.969559  4010 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:21:42.952930 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:42.953495  3646 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.986737  3646 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:42.954388 28328 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:21:42.987521 27655 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:42.955075 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:42.988811 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.981381 27656 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:42.967845 27580 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:21:42.978860 27658 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:42.972745 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.948729  3819 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.973690 27729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:42.998539 27729 raft_consensus.cc:606] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:42.972270 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:42.991578 27737 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:42.965687 27585 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:21:42.990296 28332 rpcz_store.cc:276] 0709 12:21:41.924059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.924311 (+   252us) service_pool.cc:224] Handling call
0709 12:21:41.924749 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:42.653719 (+728970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.947178 (+293459us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b540024a254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:42.947418 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1669189888}
I20260709 12:21:43.001056 27584 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:21:43.005324 27585 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
W20260709 12:21:42.971177  3989 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.451s	user 0.003s	sys 0.000s
I20260709 12:21:42.937587  4013 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:42.986680  3349 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:42.980501  3981 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:43.009876 27656 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:21:43.002199  3980 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:42.992143  3219 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:43.013513 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:43.018033 27589 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:43.002602  3952 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:43.021570 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:43.026408 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122685us
I20260709 12:21:43.030174 27580 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:42.927170  3985 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.040591 28330 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:43.045879  3990 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.051796 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:43.053218 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:42.972474  3962 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:42.979501  3983 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:43.069227 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.041719  3299 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:21:43.016927  3993 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.073740 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:43.017050  3980 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:43.076944 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 94366.1us
I20260709 12:21:43.084105 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:43.084233 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 326413us
I20260709 12:21:43.082942 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.085070 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.086618 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.089872 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:43.090637  3819 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.012075  3997 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:21:43.091223 27663 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:43.092613 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.018802  3841 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.060s
I20260709 12:21:43.025563  3973 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.097554 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:43.110800 27656 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:43.097867 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.098664  3819 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:43.100795  3990 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.107863  3299 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:21:43.084199  3953 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.009s	sys 0.012s
W20260709 12:21:43.118976 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:43.119555 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.120811 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:43.123221 28337 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:43.083845  3980 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:43.126127  4014 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:43.128387  4014 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:43.128938  4014 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:43.131062  3985 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.116454  3976 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:21:43.138536  3962 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.131767 27580 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:43.108781  3841 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:43.143601  3841 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.144865  3983 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.144618  3841 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.080755  3954 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.134624  2667 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:43.148171  2667 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:43.148579  3973 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.148625  2667 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:43.149312  3754 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.149744  2667 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.150650  3754 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:43.150964  2667 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:43.155635  3349 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:43.159675 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74658us
I20260709 12:21:43.160025  3977 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:43.160543  3977 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:43.160944  3977 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:43.161309 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.084956  3993 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.163955  3962 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:43.165530  3993 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:43.166224 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.167898  3962 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:43.168373 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:43.169027  3985 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:21:43.170449 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72693.5us
I20260709 12:21:43.171288 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.172276  3646 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:21:43.172529 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:43.159744  3818 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:43.173297  3818 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:43.173688  3818 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:43.120493  3299 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.176244  3299 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.177531  3299 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:43.125524 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:43.123483  3953 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.010s	sys 0.013s
I20260709 12:21:43.180799  4013 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.186713 27656 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:43.187541  4023 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:21:43.187834 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54761.9us
I20260709 12:21:43.188544  3878 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:43.190550 28331 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:21:43.190701 28329 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
W20260709 12:21:43.220908 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.221166 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.222545 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:43.134133 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.148499  3954 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:43.197221 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.212440 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:43.216782 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:43.230049 27583 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:21:43.198268  3831 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:43.231699 27655 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:21:43.217423  3646 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.013s
W20260709 12:21:43.235618 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 101615us
I20260709 12:21:43.235594 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.239523 27661 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:21:43.239029 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.244068 27588 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:43.244174 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.197713  3985 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:21:43.219563  3962 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:43.232295  3004 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:43.257445 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.313845 27657 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:21:43.314626 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:43.345965 27658 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:43.233274  3990 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:21:43.312654 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72448.6us
I20260709 12:21:43.217206  3973 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:21:43.355700 27664 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:43.357265  2667 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:43.240149  3956 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:43.243251  3696 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.112s
I20260709 12:21:43.351570  4011 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:21:43.352160  4020 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:43.243476  3954 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:43.370559 27580 pending_rounds.cc:85] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:43.352618  4013 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:43.370695 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.354604 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.364634 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:43.355412 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.357399 27661 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:43.346835 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.362190 27660 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:43.239596  3993 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:21:43.356987 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 99639.8us
I20260709 12:21:43.370961 27580 pending_rounds.cc:107] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:21:43.238107  3997 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:21:43.372540 27580 raft_consensus.cc:2889] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:43.374678 27581 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 21: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:43.344859  3004 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.381057  3831 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:43.381279  3980 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.382249  3993 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.384382 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.387379 27661 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:43.388589 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.311738  3982 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:21:43.392166 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.370627  3954 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.365538  3956 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:43.397439 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.400260 28329 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:21:43.350358  3962 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:43.400985 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.400162 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:43.432823 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60141us
I20260709 12:21:43.391057  3831 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:43.441514 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.386602  3997 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.387835  3004 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:43.440883 27585 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:43.442129 27588 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:43.385381  4020 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:43.442451 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45094us
I20260709 12:21:43.373399  4014 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.445427 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.438061 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.402729  4013 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:43.446372  3982 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 12:21:43.457285 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:43.510001 27589 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:43.463951 27588 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:43.464170 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.511175 27585 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:21:43.465070 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:43.512292 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.512730 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:43.514942  3819 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:43.516199  3984 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:43.516891 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:43.480248  4027 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:21:43.518122 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.521133 27589 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:43.488408  3994 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:43.490998  3818 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:43.454838  4028 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:21:43.432957  3956 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:43.495582  3985 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:43.496649  3997 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:43.500942 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55620.3us
I20260709 12:21:43.580751 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.456416 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.509006 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.481730  3948 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:43.519240 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:43.523942 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58970.1us
I20260709 12:21:43.582378 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.577302  3004 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.402506  3954 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:43.597352 27582 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:43.606271  3004 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:43.607663 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.611099 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:43.612655 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:43.615581  3552 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:43.616142  3552 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.616369  3299 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:43.617728  3952 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:43.618188  3952 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:43.619300 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:43.620163 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
W20260709 12:21:43.576643 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 451163us
I20260709 12:21:43.583827 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.578366 27586 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:21:43.590503 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 400843us
I20260709 12:21:43.622992 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:43.623915 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:43.540288 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.571791  3643 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:43.587517  3959 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.594529 27583 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:43.595566 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.587646  3646 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:43.588591  3993 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:43.625515 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.638845 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:43.585378  3349 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.591001 27584 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:43.641640 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:43.642052 27584 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:43.537456  3994 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:43.586557  3948 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.630447 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:43.634171 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.631968  3981 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:21:43.635525 27734 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:43.636018 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:43.585124  3977 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.627492  3983 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:43.639528 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:43.397923  3979 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.241s
W20260709 12:21:43.640949 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45466.1us
I20260709 12:21:43.594465  3982 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:43.642781  2667 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:43.633307  3985 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:43.457579  4020 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.643756  3349 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.647188  3004 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:43.650147 27731 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:43.651032  4014 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.650830 27584 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.652213  3219 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:43.590796  3958 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:21:43.641269  3959 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.659410  4023 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:43.721438  4023 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:43.648684 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:43.676654 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:43.667603  3961 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:43.747754 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:43.748557 27585 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
W20260709 12:21:43.679872 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 317023us
I20260709 12:21:43.709826 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:43.667807 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:43.752066 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:43.752315 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84605.2us
I20260709 12:21:43.715313 27662 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:43.671619  3646 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:43.718858 27584 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:43.720767  3959 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.717953  3349 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.715729  3219 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.722317 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:43.759734 27734 raft_consensus.cc:606] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:43.719841  3994 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.735833 27583 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:43.752942 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:43.753407 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 76874.6us
I20260709 12:21:43.747754 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:43.733203  3630 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.704577  4038 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:21:43.757064 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 83307.1us
I20260709 12:21:43.765614 27586 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:43.767436 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:43.773419 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:43.747999  3961 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:43.749454  4023 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:43.756613  4013 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:43.778961 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:43.786098 27585 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:21:43.762311  4034 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:43.759866  4014 raft_consensus.cc:2764] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:21:43.763712  3219 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:43.801412 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.793098 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:43.770599  3646 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:43.756501  3952 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.788805  3990 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:43.819715  3961 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.821220 27731 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:43.778236  3630 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.875471  4039 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.890981  4039 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.822293  3956 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.824148  3831 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:43.795143  3643 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:21:43.893361  4039 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:43.793092  3985 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:43.826926  3841 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:43.893694  4023 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:43.833494  3977 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.894986  4023 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:43.896234 27587 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:43.804133  3976 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:21:43.838294  4013 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:21:43.897469 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96247.8us
I20260709 12:21:43.897924  4013 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:43.848526 27589 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:43.849004  4003 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:43.851466  3980 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:43.788517  3994 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:43.788064  3818 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.894717  4034 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:43.862788 27583 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:43.847150 27588 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:43.876513 27585 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:43.884280 27584 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:21:43.882488  3819 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:21:43.833124  4035 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:21:43.894627  3917 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:43.800266  4038 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:43.905387 27656 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:43.914777 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:43.899376  3954 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:43.921319 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.897877  3962 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:43.900612  4003 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:43.913856 27584 pending_rounds.cc:85] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 14
I20260709 12:21:44.003363 27584 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:21:43.965394  3961 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:21:43.979367 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 205014us
I20260709 12:21:43.831173  4043 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260709 12:21:43.900694  3990 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:43.898902  4013 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.014777 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1265) took 1011 ms. Trace:
I20260709 12:21:43.993536 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:43.994626 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:43.995536  4039 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:21:44.015273 28330 rpcz_store.cc:276] 0709 12:21:43.002998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.003427 (+   429us) service_pool.cc:224] Handling call
0709 12:21:44.014596 (+1011169us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:44.014722 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24438144}
I20260709 12:21:43.996620 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:43.915709  3630 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:44.000335 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85664.2us
I20260709 12:21:44.003813 27584 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:44.022567 27584 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:21:44.022825 27584 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:43.893384  3956 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.011022 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:43.997682  3917 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:44.019563 27736 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:21:44.020102 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.027462 27730 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:44.019210 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.001905  3818 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:44.022311 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.028723 27735 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:44.028887 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:44.010998 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 218033us
I20260709 12:21:44.023656 27729 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:44.032261 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:44.034840 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.039436 27581 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:21:44.041594 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:44.042866 27588 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:44.044220 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.044337 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.048768 27589 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:44.050148 27731 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:44.057315 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:44.061523 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.064289 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:44.065584 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6657768049999504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20260709 12:21:44.067528 27729 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:44.013757  4044 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
W20260709 12:21:44.068430 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6279768049998893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260709 12:21:44.041563  3878 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
W20260709 12:21:44.070890 27552 connection.cc:256] RPC call timeout handler was delayed by -1.593776804999834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16110000000000002s for a response
W20260709 12:21:44.085052 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5433768049999799s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
W20260709 12:21:44.085439 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4713768049998635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20260709 12:21:44.085757 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4335768049998023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1789s for a response
W20260709 12:21:44.086040 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4245768050000152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20260709 12:21:44.086313 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3534768049999002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
W20260709 12:21:44.086607 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3354768049998711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20260709 12:21:44.086653 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3795) took 1623 ms. Trace:
W20260709 12:21:44.086972 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2589768049999748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
W20260709 12:21:44.087535 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2292768049999268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20260709 12:21:44.024134  3979 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.042s
W20260709 12:21:44.088150 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1590768049998132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
I20260709 12:21:44.088711 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:44.088743 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1536768049998045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
W20260709 12:21:44.089510 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1212768049999795s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2136s for a response
I20260709 12:21:44.087299 27663 rpcz_store.cc:276] 0709 12:21:42.447452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.643199 (+195747us) service_pool.cc:224] Handling call
0709 12:21:42.643588 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.091894 (+448306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.070135 (+978241us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:44.070298 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1011709824}
I20260709 12:21:44.091354 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.091909 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:44.114408 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.115724 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:44.067938  3819 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:44.209932 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.177026 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.044554  3630 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:21:44.061467  3845 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:44.064006  3956 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.105s
W20260709 12:21:44.240774 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 216416us
W20260709 12:21:44.128271  4037 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:21:44.250892 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:44.256134 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 164306us
W20260709 12:21:44.258021 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 166775us
I20260709 12:21:44.258737 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.118544  3831 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.260416 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.262809  3956 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:44.246060  4039 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:44.267755 27589 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:21:44.267685  4039 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.271658  4039 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.273267  4039 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:44.235381  3958 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.116761 27585 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:44.178884 27734 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:44.283099 27586 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
W20260709 12:21:44.283607 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 220594us
I20260709 12:21:44.071116  4043 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.183s
I20260709 12:21:44.245728  4030 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:44.245541  4048 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.091056  4045 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.194s
I20260709 12:21:44.240185  3845 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.257782  4046 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:21:44.275609  3630 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:44.126907  3696 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:44.260049  3952 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:44.308744 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.309445 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:44.309568 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.249482  3819 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.310865 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.311796 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.312940  3696 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:44.323843  3696 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.117345  4049 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.208s
I20260709 12:21:44.325922  3696 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.326119  4049 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:44.326478  4049 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.321179 27735 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:44.322770 27730 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:21:44.323331 27734 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:44.313428  3630 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:21:44.115553  3990 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.331681  3952 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:44.337950 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 80818.9us
I20260709 12:21:44.087568  4034 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.305740  4056 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:21:44.338543 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79885.8us
I20260709 12:21:44.300338  3831 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:44.346832 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:44.086215  4003 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:44.309005  3984 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.789s	user 0.011s	sys 0.000s
I20260709 12:21:44.351713 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:44.357265 27587 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.358060 27587 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:21:44.359194 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3921) took 1120 ms. Trace:
I20260709 12:21:44.359745 27580 rpcz_store.cc:276] 0709 12:21:43.238503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.370032 (+131529us) service_pool.cc:224] Handling call
0709 12:21:43.370387 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:43.385742 (+ 15355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:43.385773 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:43.386362 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:21:43.736380 (+350018us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:43.736432 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.736512 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:44.358897 (+622385us) raft_consensus.cc:1685] finished
0709 12:21:44.358929 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.359115 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":338177,"threads_started":1,"wal-append.queue_time_us":409600}
I20260709 12:21:44.345202 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.327414  4049 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.301800  4046 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.351456 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:44.352166 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:44.354701 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.371963 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.362025 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:44.335489  3954 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:44.374991 27586 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:44.379225 27588 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:44.379637 27588 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:44.380249 27589 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:44.111517  3917 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.291092  3958 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.343804  4056 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.377599  3984 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:21:44.387840  3997 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:44.401921  3958 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.401384  4046 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.402913  4045 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.091943  4038 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.395735 27588 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:44.411945 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 66793.2us
I20260709 12:21:44.343472  3952 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.414145 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.406761 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.367031  3988 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:21:44.437646 27588 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:44.330775  4048 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.403527  4049 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:44.349050  3696 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:21:44.398140  3973 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:21:44.410391  3997 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.450938 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.452044 27734 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:21:44.452893 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:44.457508 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 110758us
I20260709 12:21:44.458506 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:44.460578  4063 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:44.461290  4063 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.466243 27588 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 15. (term mismatch)
I20260709 12:21:44.466631 27588 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 14
I20260709 12:21:44.466876 27588 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 29.15
I20260709 12:21:44.466724 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:44.467209 27588 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:44.467432 27734 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:44.454002  3997 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:44.481873  3952 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:44.483027  3995 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.483594 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:44.470435  3841 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.003s
I20260709 12:21:44.486779 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.483666  4054 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:44.440577  4045 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.489400 27736 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:44.490897 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.398016  3845 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.509759 27732 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:44.494683 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:44.496883  3552 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.486120  4046 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:44.520830  3552 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:44.523689  3984 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:21:44.524096  3984 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.524513 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:44.525134 27732 raft_consensus.cc:606] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:44.524925  3984 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.526268  3984 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:44.526633 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:44.530994  3818 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:44.490258  3958 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.536983 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:44.541940  3958 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.512804  3841 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:44.512055  3831 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.532823  3973 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:21:44.552023 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57471.3us
I20260709 12:21:44.553128 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.557235 27732 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:44.539472  3845 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:44.555449 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.483649  3995 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.540560  3754 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.554055  3831 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.552758  4067 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:44.575291  3831 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:44.590231  3994 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.680s	user 0.007s	sys 0.002s
I20260709 12:21:44.591254 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.547771  4030 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:21:44.591984  3878 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.593394  3878 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.593748  4048 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.595927  3878 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.596591  3878 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:21:44.598138 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.599026  3818 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:44.599609 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:44.599707 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:44.578008  3958 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.600656 27580 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:44.601352  3958 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:44.602180  3985 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:21:44.602313 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.581717  4062 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:21:44.602630  3985 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.599725  3818 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:44.601933  3958 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.604010  4021 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:44.604646  3643 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:44.605464  4021 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:21:44.605932  3958 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.607064 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:44.607738 27588 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:44.607529  4020 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:21:44.593809 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 970692us
I20260709 12:21:44.477056  4056 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.580958  2667 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:44.550226  3841 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.617182  2667 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.622419 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:44.627135 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.634418  2667 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.638573 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:44.638484  3980 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:21:44.640353 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.510715  4054 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.641923 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.643381 27662 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:21:44.641284 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.646191 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:44.558339  3973 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:44.649747 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:44.651433 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.654047 27734 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:21:44.603215  3961 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.589s	user 0.006s	sys 0.003s
I20260709 12:21:44.651499 27582 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:21:44.607146  4021 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:44.603148  3985 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:44.657297  3818 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:21:44.671195 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.690189 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:44.686290 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.692054 27655 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
W20260709 12:21:44.660351 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.693403 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.694689 28332 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:44.696522 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:44.697489 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.698143 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.698921 27658 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
W20260709 12:21:44.686463 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59438.6us
I20260709 12:21:44.701504 28330 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
W20260709 12:21:44.701777 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.702340 28334 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:44.697551 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.662976 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.664513 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:44.707849 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.709856 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.711743  3643 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:44.713204  4055 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:44.713624  4055 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:44.714488  4055 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.715972  3948 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.717353  3948 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:44.725312 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.726348 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:44.734182  4054 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.642081  3841 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:44.736661 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.737196  4045 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.706504  4021 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:44.740309  4021 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.741581  4021 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:44.744393  3219 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.924s	user 0.010s	sys 0.000s
I20260709 12:21:44.744839  3219 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.747004  3219 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:44.748013  4023 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.853s	user 0.005s	sys 0.005s
I20260709 12:21:44.628822  3754 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.720311  3953 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:21:44.752417  4056 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:44.752530  3952 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.700920  3980 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:44.792443  3696 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:21:44.772038 27663 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:44.786527  3953 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:44.618058  4063 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.662042  3973 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:44.794152  3953 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.792479  3952 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.795379  3953 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:44.797238  3952 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.776834 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:44.782366 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.798288 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.799098 27655 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 21: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:44.778024 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.692883  3831 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.800396 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.663205 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:44.787096 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.789501  3982 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.791669 27660 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: [29.15-29.15]
I20260709 12:21:44.775298  3956 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.792807  3980 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:44.782822  4006 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.802541 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.807224 27655 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:21:44.802114 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.781256 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79569.3us
I20260709 12:21:44.809782 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.811270  3818 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:44.813265 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.815572  3818 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:44.817421 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.815080 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.819281  4071 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:44.823585  3818 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:21:44.826501 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 165215us
W20260709 12:21:44.828380 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.829032 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:44.829707 28330 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:44.829890  4023 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.808043  3696 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:44.804217  4054 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.830616  3696 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.831158  4054 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.832446  3754 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.833109  3696 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.830531  4023 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.827091 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.836925 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.829771 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:44.839947 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:44.841032 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.843106 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 180192us
I20260709 12:21:44.845072 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:44.846138 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 386971us
I20260709 12:21:44.845870 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:44.852514 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.853286 27664 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:44.855553 28332 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:44.860795 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.804005  3982 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.875612 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.873383 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.892256 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.895706 27658 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:44.893544 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.897614 27662 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:44.875849 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.899027 27731 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:44.893075 27661 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:21:44.869925  4078 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:44.869855  3990 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.891754 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.908211 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:44.910791 27663 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:44.852070  3754 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:44.806486  3956 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.957841 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.958537 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.953912 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.959188 27661 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:44.852483  3954 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:44.844944  3878 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:44.909584 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:44.955305 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82009.8us
I20260709 12:21:44.982594 27661 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 15. Current term is 16. Ops: []
I20260709 12:21:44.983594 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.873646  4020 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:44.809541  4006 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:44.910686  3979 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:21:44.956075 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47958.9us
I20260709 12:21:44.959576 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:44.808058  3831 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:44.994618  3956 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:44.994946  3831 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.870826  3552 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:44.995503  3643 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:45.004652  3982 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:44.981338  4049 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.001s
I20260709 12:21:44.908859 28330 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:44.852351  4063 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.108s
I20260709 12:21:44.879508  3958 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.011373 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:44.984851 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.883306  4023 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.909162 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:44.952093 28328 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:44.985962 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.020998 27731 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 21. (term mismatch)
I20260709 12:21:45.021137 27664 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:44.961520 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.021914 27658 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:44.990196 27729 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:45.023284 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.023552  3878 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:45.024152  3878 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:45.024622 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.025091  3878 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.025700 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.026809 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.029413  3819 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:45.029486  3958 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.031961  3958 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:44.990056 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:44.986749  3754 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:45.033932 27657 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:45.035104 27662 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:44.883905  3995 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:44.969450  4034 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:44.887039  3985 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.049660 27733 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:44.875438  3952 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.132s
I20260709 12:21:45.021395 27731 pending_rounds.cc:85] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 20
I20260709 12:21:45.053435 27731 pending_rounds.cc:107] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 31.21
I20260709 12:21:45.053717 27731 raft_consensus.cc:2889] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:45.036777 27729 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:44.951138  4067 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:21:45.038874 27704 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:45.043339 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53323.1us
I20260709 12:21:44.995529  3552 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.042395  4049 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:21:45.010219  3979 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:45.054926 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:44.950176  4032 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.085s
I20260709 12:21:45.071926  4032 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:45.072343  4032 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:45.073304  4032 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.074694  4032 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:45.075539  3982 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:45.055878  4034 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:45.078055  4034 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.056731 27735 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:44.909375 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:44.959067  4078 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:21:45.059942 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.052241  4020 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:44.989005  3954 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:45.010001  3841 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.100942  4056 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:21:45.102090  4056 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:45.102449  4056 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:45.102645  3841 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.102854  3976 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:45.064615  3552 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:45.068666  3995 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:21:45.068835 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:45.104079 27664 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:21:45.069718  3973 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.106155 28332 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:21:45.106328  3973 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.107884 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.109010 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.109714  3818 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:45.110069 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.110386  3818 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:45.070608  4003 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:45.100996  3754 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.070353  3979 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.115685  3979 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:45.116284  4006 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:45.117144 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.116272 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.118395  4020 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.053076  3983 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:21:45.119271  3973 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:45.111078  3818 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:45.063372  3952 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:45.120155  3952 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:45.120587  4020 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.121342 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:45.121443  3952 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.122423  3993 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:45.122643  3643 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:45.122810  3993 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:45.123363  3993 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:45.123308  3643 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.125767  3643 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.127723 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:45.129637  3643 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:21:45.130280 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.131522 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.131166 27738 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:45.133401  3917 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:45.135556 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.136279  3973 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:45.137877  4063 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.138712 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.138406  4063 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.139340 27730 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:45.139814  3973 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:21:45.140192 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.140344  3956 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:45.143260 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:44.909951  3990 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:45.085906  3997 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.594s	user 0.011s	sys 0.000s
I20260709 12:21:45.143882 27730 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:45.099845 27663 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:45.144302  3997 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.100361 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.087706 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:45.143039  3956 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.100054  3953 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.146600 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:45.163884 27730 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:45.147532  2667 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:45.147456 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:45.149488 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:45.165323 27731 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:45.086263 27655 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:45.166638 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:45.151049 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.151871 27729 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:45.168315 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:44.887763  3980 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.145699 27659 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:45.177501 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.180116 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.182185 27661 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:45.182821 27658 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:45.165267 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.174153 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:45.183462 27738 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:45.183681 27655 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:45.183462 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:45.185938 27730 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:45.190459 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.193265 27730 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:45.195582 27657 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:45.198383 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.172535  3953 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.204924 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.179503  3818 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:45.205754 27664 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:45.120185  4056 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.187925  4071 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:21:45.206506  3953 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:45.173172  4020 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:45.179446  3643 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:21:45.208204  3643 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:45.208585  3643 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:45.209389  3997 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.209618 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:45.209483  3643 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.187764  3962 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:45.211665  3962 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:45.212988  4055 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:45.212680  3962 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.202692 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:45.211055  3643 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:45.200546 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:45.161005  3983 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:45.204183 27730 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:45.200670 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:45.217970 27660 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:21:45.233197 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.233837 27738 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:45.207163  4056 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:45.241751 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:45.242375 27731 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:45.232203  4071 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:45.237756 27729 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:21:45.240880  4063 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:45.190879  4038 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:45.235507 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:45.200202  3831 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:45.243939 27730 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:45.192813 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:45.231462  3976 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.250774 27731 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
W20260709 12:21:45.275786 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.251641 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.231694  3953 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:45.247429 27475 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.202s	user 0.010s	sys 0.013s
I20260709 12:21:45.261145  3988 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.261669  4054 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:45.265213 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.268128 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:45.269124 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:45.269852 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:45.270972 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:45.271761  3956 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:45.274712 27729 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:45.241329  3990 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.283633 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:45.284641 27589 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:45.285264 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:45.286880 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:45.195576  3973 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:21:45.292327 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.376775 27664 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:45.377760 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:45.384130 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:45.167006  3954 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.294166 27730 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:21:45.295765 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:45.253198 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60438.6us
I20260709 12:21:45.402956 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:45.295650 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3212) took 1245 ms. Trace:
I20260709 12:21:45.403119 28335 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:45.399895 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:45.404505 27584 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
W20260709 12:21:45.367928 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74197.3us
I20260709 12:21:45.369195 28331 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:45.369612 27661 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:45.408262 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.369372 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:45.370126 28328 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
W20260709 12:21:45.371021 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102992us
I20260709 12:21:45.410122 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.403729 27736 rpcz_store.cc:276] 0709 12:21:44.028702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.029640 (+   938us) service_pool.cc:224] Handling call
0709 12:21:45.274186 (+1244546us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54001d47d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:45.274291 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916969344}
I20260709 12:21:45.249549  3997 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.374001 27586 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:45.252516  4071 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:45.292699 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:45.420167 27581 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
W20260709 12:21:45.389613 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.379220 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:45.392557 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.421818 27664 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:45.254271  4083 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.088s	user 0.002s	sys 0.000s
I20260709 12:21:45.412578 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.293962  4034 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:45.419193 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:45.295863  3961 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.430356  3754 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:45.435072 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:45.430962 27664 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:45.428206 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:45.445708  4034 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:45.458487  4034 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.437755  3984 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:45.462085 27729 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:21:45.432294 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:45.464208  3754 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:45.465055  3754 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:45.465396  3754 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.465715  3754 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:45.424693  4071 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:45.471431  3819 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:45.472011  3819 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:45.472457  3819 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:45.476652  3349 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.476958 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:45.477299  3349 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.480020  3349 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.489069  3552 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.424504  3818 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:45.370452  3976 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.444352  3977 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:21:45.445660  3961 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.362963  3953 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.495985  3841 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:45.537513  3841 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.537920  3841 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:45.430456  3997 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.540444  3953 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.542050  3997 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:45.505149 28333 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:45.463891 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:45.508993 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 73995.3us
W20260709 12:21:45.545673 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81886us
W20260709 12:21:45.546129 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:45.509394 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96911.3us
I20260709 12:21:45.547356 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:45.547817  3953 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:45.548722  3349 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:45.549181 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.549619 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:45.549775 27733 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:45.401734  4063 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:45.550778  4063 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.551250  4063 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:45.460067  3984 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:45.555727  4087 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.368s	user 0.001s	sys 0.000s
I20260709 12:21:45.558286  3984 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:45.443183 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 25 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:45.366060  3988 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.515733  3956 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:45.559314 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:45.559648  3956 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.560012  3956 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:45.550552  3953 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:45.501961  3754 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.560580  3988 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:45.562646 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:45.292099  4038 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.494966  3993 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.526274  4003 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.586920 27659 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:21:45.587173 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:45.620779 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:45.647204 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60108.1us
I20260709 12:21:45.536737  3818 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:45.569132  3349 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.639881 27733 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:45.641134 28330 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29.
I20260709 12:21:45.531526  3985 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.626665 27732 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:45.648067 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:45.588963 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:45.651783 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.658306 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:45.617062  3973 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:45.581138  3956 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.641212  3997 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:45.516366  3983 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.673629 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:45.750383 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.006s	sys 0.027s
W20260709 12:21:45.750845 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.006s	sys 0.027s
I20260709 12:21:45.756023 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.645735  4038 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.652906  4003 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.671147  3997 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.762596  3878 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:45.661655  3349 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:45.778082 28337 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.023s
I20260709 12:21:45.772939 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.755944  3985 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.791296 27586 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:45.786218 27658 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:45.800122 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:45.801249 27586 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:45.806399 27658 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:45.815762 27737 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:45.669202  3973 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:45.786785 28337 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.006s	sys 0.023s
I20260709 12:21:45.778904  3997 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:45.818069  3977 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:45.827204  3977 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.827759  3977 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:45.837198  3973 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:45.818445  3979 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.702s	user 0.008s	sys 0.004s
I20260709 12:21:45.756907  3953 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.885412 27733 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:21:45.873178  4055 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.660s	user 0.006s	sys 0.006s
I20260709 12:21:46.003988  3754 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.868571  4060 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.816908  3985 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.863101  4094 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:45.872759  3973 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:45.942031 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.874964  4093 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:45.820555  3818 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.890359  3552 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:45.997637 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:45.903568  3976 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.868806  3977 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:45.942047  3961 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:45.946314  3911 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.041080 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:21:46.140642 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.964489  3993 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.004251  3831 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:46.196269 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.008s	sys 0.008s
W20260709 12:21:45.888646 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.293665 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.240492 27583 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:21:46.296039 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.008s	sys 0.009s
I20260709 12:21:46.201067  3956 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.238847  4003 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.211640  4094 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:46.212607  3988 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.240372  3878 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:46.264945 27736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.056s	sys 0.069s
W20260709 12:21:46.362846 27736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.057s	sys 0.069s
W20260709 12:21:46.282235 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.363713 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.246942  4093 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:46.277401 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3101) took 1099 ms. Trace:
I20260709 12:21:46.270561  4038 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:46.245662 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.010s	sys 0.027s
I20260709 12:21:46.220764 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:46.367349 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.011s	sys 0.028s
W20260709 12:21:46.296823 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 707928us
W20260709 12:21:46.368022 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 709763us
I20260709 12:21:46.368688 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:46.369359 28337 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:46.366534 27660 rpcz_store.cc:276] 0709 12:21:45.178059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.178167 (+   108us) service_pool.cc:224] Handling call
0709 12:21:45.178436 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.232606 (+ 54170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.277017 (+1044411us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:46.277244 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90197248}
I20260709 12:21:46.261238  3973 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.211503  4060 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:46.324569 27735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3224) took 1814 ms (client timeout 1721 ms). Trace:
I20260709 12:21:46.390201  3552 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.395859  3552 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.263022  4110 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.325109  4094 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:46.397703  4110 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.400800  4060 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:46.395540 27735 rpcz_store.cc:269] 0709 12:21:44.473658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.473880 (+   222us) service_pool.cc:224] Handling call
0709 12:21:46.287645 (+1813765us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:46.287799 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1257475456}
I20260709 12:21:46.356706  4048 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.403615  4048 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.404189  4110 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.404740  4060 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.405210  4060 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.405934  4048 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.357199  3990 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.406662  3990 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.407873  4060 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:46.408959  3990 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.358417 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.410367 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.410995 27585 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:46.354866  3976 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.411872 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.411823  3976 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.413084 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.413646 27585 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:46.414017  3976 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.414470 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.359272 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:46.365368 27583 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:46.388593 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:46.353899  3956 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.322957  3961 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:46.316076 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.005s	sys 0.013s
I20260709 12:21:46.261977  3911 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.374473 27663 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:46.360692  3349 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:46.436414 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46653.9us
I20260709 12:21:46.461227 27584 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:46.462100 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:46.462852 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.007s	sys 0.014s
I20260709 12:21:46.463442 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.464560 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:46.464746 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:46.426748 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.465451 27583 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:46.415143 27585 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:46.469182 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:46.469673 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:46.470364 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:46.360805  4003 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:46.470952 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:46.415083  4095 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:21:46.473975 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:46.469933 28295 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:46.479912 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:46.481766 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 440786us
W20260709 12:21:46.416424  4096 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.527s	user 0.000s	sys 0.001s
I20260709 12:21:46.389236  3754 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:46.488530 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.489848 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:46.491523 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:46.491964 27583 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:21:46.492223 28334 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:46.493988 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:46.483366 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:46.487609 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.494705 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.495055 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:46.495379 27583 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:46.496096 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.493558 27733 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:46.496627 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:46.497764 27735 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:46.498904 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.499547 27583 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:46.501745 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:46.374296  3988 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.504386 27584 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
W20260709 12:21:46.437356 27630 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.562s (SENT) [suppressed 35 similar messages]
I20260709 12:21:46.505167 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:46.507781  4114 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:21:46.495146  4038 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:46.511888  4038 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.512497  3878 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.513468 27663 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:46.514128  4038 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.514699 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:46.515960 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:46.517954 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:46.518785  4071 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.052s	user 0.004s	sys 0.004s
I20260709 12:21:46.519115 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.523470 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3822) took 1155 ms. Trace:
I20260709 12:21:46.541308 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:21:46.523825 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3814) took 1249 ms. Trace:
I20260709 12:21:46.542709 27655 rpcz_store.cc:276] 0709 12:21:45.274886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.438174 (+163288us) service_pool.cc:224] Handling call
0709 12:21:45.438489 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.438822 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.438850 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:45.439290 (+   440us) log.cc:844] Serialized 26 byte log entry
0709 12:21:45.647181 (+207891us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:45.647227 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.647291 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.523643 (+876352us) raft_consensus.cc:1685] finished
0709 12:21:46.523662 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.523776 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16711,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"thread_start_us":193653,"threads_started":1,"wal-append.queue_time_us":371058}
W20260709 12:21:46.546120 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62858.9us
I20260709 12:21:46.523125 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1557) took 1305 ms. Trace:
W20260709 12:21:46.555390 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58838.9us
I20260709 12:21:46.462270  3954 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:21:46.618938  4105 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:46.548381 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:46.545538 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.624192 27655 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:21:46.625180 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.625937 27655 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:46.542840 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:46.521169  4099 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.303s	user 0.001s	sys 0.000s
I20260709 12:21:46.625700 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:46.520190  4112 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:21:46.541664 27658 rpcz_store.cc:276] 0709 12:21:45.368327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.807421 (+439094us) service_pool.cc:224] Handling call
0709 12:21:45.807739 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.808368 (+   629us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.808397 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:45.808904 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.068094 (+259190us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:46.068362 (+   268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.068636 (+   274us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.523233 (+454597us) raft_consensus.cc:1685] finished
0709 12:21:46.523252 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.523418 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":252370,"threads_started":1,"wal-append.queue_time_us":465097}
I20260709 12:21:46.649786 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.478634  4003 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:46.572438 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:46.489524  3349 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:46.572520  4034 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:46.617350  3911 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.554688  4060 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.572438  4038 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:46.619827 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.621766 27730 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:46.622294 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1452) took 1033 ms. Trace:
I20260709 12:21:46.687150 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:46.622898  3988 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:46.478375  3961 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.626919 27660 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:46.560160 27657 rpcz_store.cc:276] 0709 12:21:45.217303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.232061 (+ 14758us) service_pool.cc:224] Handling call
0709 12:21:45.239768 (+  7707us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.240274 (+   506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.240317 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:45.240918 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:21:45.752642 (+511724us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:45.752688 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.752763 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.522883 (+770120us) raft_consensus.cc:1685] finished
0709 12:21:46.522905 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.523070 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1406,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":8731,"thread_start_us":511339,"threads_started":1,"wal-append.queue_time_us":959167}
I20260709 12:21:46.482129  4108 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:46.523649  3993 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:21:46.519665  4102 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 12:21:46.522586  4110 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:46.632006 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.630438  4120 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:46.653548 27658 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:46.651392 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.656265 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:46.687752 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1583) took 1009 ms. Trace:
I20260709 12:21:46.701287 27585 rpcz_store.cc:276] 0709 12:21:45.678131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.619428 (+941297us) service_pool.cc:224] Handling call
0709 12:21:46.687691 (+ 68263us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:46.689165 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116885us
W20260709 12:21:46.647861  3819 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.175s	user 0.010s	sys 0.003s
I20260709 12:21:46.687222 27736 rpcz_store.cc:276] 0709 12:21:45.588966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.589183 (+   217us) service_pool.cc:224] Handling call
0709 12:21:45.647079 (+ 57896us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:45.649068 (+  1989us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.649113 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:45.664745 (+ 15632us) log.cc:844] Serialized 52 byte log entry
0709 12:21:46.363264 (+698519us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:46.391743 (+ 28479us) log.cc:844] Serialized 14 byte log entry
0709 12:21:46.391994 (+   251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.392100 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.622072 (+229972us) raft_consensus.cc:1685] finished
0709 12:21:46.622095 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.622236 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1605,"prepare.queue_time_us":28253,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":18174,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":3259,"spinlock_wait_cycles":50048,"thread_start_us":604717,"threads_started":1,"wal-append.queue_time_us":814108}
I20260709 12:21:46.653677  4003 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:46.519209  4071 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.651261  4117 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:46.698591 27655 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:46.693953  3911 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:46.630148  4119 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:46.502651 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 3.796s	user 0.123s	sys 0.046s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":27384,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":64597,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":20,"lbm_write_time_us":14716,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":10,"peak_mem_usage":290432,"reinsert_count":0,"thread_start_us":835420,"threads_started":3,"update_count":256,"wal-append.queue_time_us":149003,"wal-append.run_cpu_time_us":1897,"wal-append.run_wall_time_us":1002270}
I20260709 12:21:46.623885  4118 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:46.541617  3754 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.696003  3988 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:46.708923  3819 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.519506  4093 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.709913 27729 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 18: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:46.710963 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3825) took 1186 ms. Trace:
I20260709 12:21:46.484352  3956 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.711176 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:46.615567  4063 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.064s	user 0.012s	sys 0.001s
I20260709 12:21:46.712994 28332 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:46.713133 27730 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:46.715384 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:46.707141  3993 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:46.698072  3349 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.696707  4038 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:46.718093 27655 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:21:46.709551 27663 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:21:46.736128 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3824) took 1301 ms. Trace:
I20260709 12:21:46.736513 27663 rpcz_store.cc:276] 0709 12:21:45.435137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.708896 (+1273759us) service_pool.cc:224] Handling call
0709 12:21:46.709235 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:46.735844 (+ 26609us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.736069 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:46.722348 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96725.5us
I20260709 12:21:46.737578 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:46.723445 27658 rpcz_store.cc:276] 0709 12:21:45.524548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.710188 (+1185640us) service_pool.cc:224] Handling call
0709 12:21:46.710491 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:46.710674 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.710696 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:46.710738 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:46.710766 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.710817 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.710908 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:46.710673  3985 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.724587 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
W20260709 12:21:46.723950 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 250027us
I20260709 12:21:46.719293 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:46.725263 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:46.758201 27736 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:46.730587 27585 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 29
W20260709 12:21:46.731024 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.759657 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
W20260709 12:21:46.730772 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 293423us
I20260709 12:21:46.696503  4060 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:46.722438  3917 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.589s	user 0.014s	sys 0.000s
I20260709 12:21:46.694715  4034 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.715871  3977 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.737632 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3828) took 1066 ms. Trace:
I20260709 12:21:46.757481 27657 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 18.
I20260709 12:21:46.769129 27655 rpcz_store.cc:276] 0709 12:21:45.670765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.717530 (+1046765us) service_pool.cc:224] Handling call
0709 12:21:46.717866 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.737349 (+ 19483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.737567 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:46.721715 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 150933us
I20260709 12:21:46.728765  3845 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:46.732240  3911 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.731935  4120 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:21:46.735193  4109 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
I20260709 12:21:46.865748 27658 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:21:46.752573  4118 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:46.874794 27663 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:21:46.752291  3973 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.752835  3954 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:46.938788 27585 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:46.940125 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.949713 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:46.950515 27729 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:46.872938  4108 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:46.956621  4108 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.770251  4110 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:46.960798  4110 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.772131  4105 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.962013  4105 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.964753  4105 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:46.969272  3961 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:46.970144  3961 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:46.873543  3917 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 12:21:46.973062  3917 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.973699  3917 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:46.769328  3985 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:46.773034  4065 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:46.716652  4117 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.725162  3988 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:46.773195  3845 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:46.739459  3349 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:46.980070  4102 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20260709 12:21:47.078894 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.866698  4120 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:47.060107 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 285295us
I20260709 12:21:46.954941 27656 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:47.064383 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.732960  4063 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:47.070870 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 312881us
I20260709 12:21:46.736043  4094 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:47.072896 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 571247us
I20260709 12:21:47.014249 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:47.079615 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 21.
I20260709 12:21:46.931454  4132 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:21:47.096885 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:46.934279  4118 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.060777  3973 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:47.063449  3997 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.195s	user 0.013s	sys 0.000s
W20260709 12:21:47.097999 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.098744  4139 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:47.061287  4105 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.867291  3911 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.121069 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.121448 27729 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:47.122200 27585 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 33 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:21:47.093381  4125 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.079434  4065 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:47.093799  3953 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.094449  4021 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.101781  4132 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.130910 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:47.097364  4120 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:46.982628  3954 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.139070 27729 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:21:47.105854  4118 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:21:47.131635  3831 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.796s	user 0.011s	sys 0.000s
I20260709 12:21:47.122588  4139 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:47.125353  4071 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.148001 27585 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:21:47.152674 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3956) took 1486 ms. Trace:
I20260709 12:21:47.153263 27585 rpcz_store.cc:276] 0709 12:21:45.666512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.134644 (+1468132us) service_pool.cc:224] Handling call
0709 12:21:47.147674 (+ 13030us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.152277 (+  4603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.152592 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:47.142056  3841 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.604s	user 0.007s	sys 0.007s
W20260709 12:21:47.148490 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.009s	sys 0.011s
I20260709 12:21:47.085711  3985 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.106233  4136 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.054s
I20260709 12:21:47.145779  3845 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.096203  4038 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.138269  4125 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.099915  3993 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.125782  3973 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.126380  3997 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.128928  4105 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:45.457136 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:47.182587 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3958) took 1515 ms. Trace:
W20260709 12:21:47.157124 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.166222 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:47.211015 27487 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 22, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 22 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:47.274857 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.009s	sys 0.011s
I20260709 12:21:47.276150 27585 rpcz_store.cc:276] 0709 12:21:45.667084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.154703 (+1487619us) service_pool.cc:224] Handling call
0709 12:21:47.181883 (+ 27180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.182129 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.182167 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.182229 (+    62us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:47.182275 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.182347 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.182510 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:47.213665 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:47.170234 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64171.7us
I20260709 12:21:47.201805 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1448) took 2038 ms. Trace:
W20260709 12:21:47.276502 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.206853 27731 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:47.288324 27734 rpcz_store.cc:276] 0709 12:21:45.163039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.442846 (+279807us) service_pool.cc:224] Handling call
0709 12:21:45.443161 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.561773 (+118612us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:45.616999 (+ 55226us) log.cc:844] Serialized 14 byte log entry
0709 12:21:45.909797 (+292798us) log.cc:844] Serialized 14 byte log entry
0709 12:21:45.940309 (+ 30512us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:45.940379 (+    70us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:45.940450 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.201522 (+1261072us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:47.201573 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.201721 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30206,"prepare.queue_time_us":86192,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":5500,"raft.queue_time_us":37653,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7113,"spinlock_wait_cycles":288277248,"thread_start_us":261178,"threads_started":1,"wal-append.queue_time_us":814767}
I20260709 12:21:47.301126 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:47.148825  4141 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:47.135428  4119 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:47.312776 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:47.314280 27729 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:47.140918  4132 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:47.316373  4141 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.318564  4124 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:47.318852  4141 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.319121  4124 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.321261  4124 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.321780  4124 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.322203  4124 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.193281  4030 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:47.200973  4063 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.208567  3819 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.327606  4030 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:47.144550  4021 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.188565  4138 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.330076 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1574) took 1661 ms. Trace:
I20260709 12:21:47.205936  3954 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.207013  4094 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.198314  3911 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.208180  3754 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.204416  3956 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.208138  3988 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.364377  4063 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.151011  3831 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:47.370842 27585 rpcz_store.cc:276] 0709 12:21:45.668991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.287748 (+1618757us) service_pool.cc:224] Handling call
0709 12:21:47.329440 (+ 41692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.329660 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.329697 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.329754 (+    57us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:47.329799 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.329869 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.330006 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:47.389638  4063 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.390196  4148 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.292654  4055 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.390654  4030 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.391291  4055 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.292860  3917 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.391930  4063 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.392189  4030 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:47.395102  4125 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.390719  4148 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.397019  3985 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.397390  3997 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.315044  3845 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:47.397990  3349 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.373867  4138 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.376470 27729 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:47.399089  3973 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.206750  4003 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.202359  4147 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.170s
I20260709 12:21:47.143024  3953 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.401907  4147 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:47.402302  4147 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:47.167546  3841 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.403081  4147 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.201942  4118 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.383611  4132 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.371447  4034 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.374068  4137 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.012s
I20260709 12:21:47.387887  4060 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.388585  4119 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.204741  4065 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.382567  4124 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.417865 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1274) took 2151 ms. Trace:
I20260709 12:21:47.418342 28332 rpcz_store.cc:276] 0709 12:21:45.266691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.270310 (+  3619us) service_pool.cc:224] Handling call
0709 12:21:45.270581 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:46.731254 (+1460673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.417645 (+686391us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:47.417808 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3316748672}
I20260709 12:21:47.404269 28330 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:47.421352 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109116us
I20260709 12:21:47.397387  4148 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.376125  3962 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:21:47.442320 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.442899 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.443017 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.443647 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.443867 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.454842 27660 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:47.412930  3973 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:47.455751 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.183748  4139 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.458009 27655 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:47.455964 27587 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:47.457027 27658 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:47.460021 27661 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:47.457103  4105 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.462780 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.463064 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.463295  3973 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.463799 27655 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:47.464785 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.468696 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.469316 27655 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:21:47.474740 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184219us
I20260709 12:21:47.488751 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:47.441051  4147 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:47.489873 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.489964 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.488881 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.497227 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:21:47.497406 27658 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:47.494701 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:47.498879 28332 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:47.498107 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.499457 27660 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:47.444453  4003 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.506234 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.490029 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.488065 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:47.502202 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.511447 28335 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:47.458349  4094 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.475298  3953 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.510128 27661 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:47.459313  4020 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.464608  4136 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.515676 27660 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:47.590750 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.591885 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.592427 27660 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
W20260709 12:21:47.501266 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59563.6us
I20260709 12:21:47.518512 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3955) took 1852 ms. Trace:
I20260709 12:21:47.594858 27588 rpcz_store.cc:276] 0709 12:21:45.666224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.113437 (+1447213us) service_pool.cc:224] Handling call
0709 12:21:47.113772 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.114029 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:47.411998 (+297969us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.517764 (+105766us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.517816 (+    52us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:47.517875 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.518254 (+   379us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.518451 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34277,"prepare.queue_time_us":324796,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":22913,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":377710,"threads_started":2,"wal-append.queue_time_us":105959}
W20260709 12:21:47.539047  4146 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:21:47.542297  4163 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.543045  4108 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:47.463137  4120 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.548482  3917 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.500876  3985 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.613072  3985 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.613803 27581 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:21:47.614413 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3969) took 1129 ms. Trace:
I20260709 12:21:47.614732 27581 rpcz_store.cc:276] 0709 12:21:46.484592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.613207 (+1128615us) service_pool.cc:224] Handling call
0709 12:21:47.613507 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.614144 (+   637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.614352 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:47.549292 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47160.9us
W20260709 12:21:47.624285 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3074) took 2988 ms (client timeout 2171 ms). Trace:
W20260709 12:21:47.624711 27656 rpcz_store.cc:269] 0709 12:21:44.635474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.640069 (+  4595us) service_pool.cc:224] Handling call
0709 12:21:47.624088 (+2984019us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b54001e5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:47.624188 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4902235008}
I20260709 12:21:47.625916 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.626523 27656 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:47.565425  4094 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.629330  4094 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:47.549561 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61675us
I20260709 12:21:47.631218 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:47.632588  3917 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.632300 27487 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 20 to 21, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:47.634068 27487 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:47.634717  3917 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.550295  3980 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:21:47.552047  4154 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:21:47.552228  4141 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:47.455309  4124 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:21:47.564235 27655 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:21:47.565476 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1577) took 1894 ms. Trace:
I20260709 12:21:47.646348 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:47.652673 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 285840us
I20260709 12:21:47.599375 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.490608  3954 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
W20260709 12:21:47.544198  4155 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:21:47.606783 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.607283 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3968) took 1123 ms. Trace:
I20260709 12:21:47.608513 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:47.564294  4162 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.671183  4162 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.639022  3980 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.564447  4160 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.680492  4160 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:47.681005 27553 connection.cc:256] RPC call timeout handler was delayed by -1.2241487060000509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1537s for a response
I20260709 12:21:47.681777 27553 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29.
I20260709 12:21:47.682044 27553 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:47.682713  4160 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.597630  3962 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:47.683303  3962 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.683715  3962 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:47.684216  4133 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.684798  4133 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.686861  4133 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.515309  4105 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.093s
I20260709 12:21:47.661600 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3965) took 1178 ms. Trace:
I20260709 12:21:47.687454  4094 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.653051 27585 rpcz_store.cc:276] 0709 12:21:45.670917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.390088 (+1719171us) service_pool.cc:224] Handling call
0709 12:21:47.390444 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.390661 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:47.443069 (+ 52408us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.564871 (+121802us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.565066 (+   195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.565110 (+    44us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:47.565149 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.565234 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.565402 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42800,"prepare.queue_time_us":48541,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":11599,"raft.queue_time_us":1733,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":15562,"spinlock_wait_cycles":5839616,"thread_start_us":173375,"threads_started":2}
I20260709 12:21:47.662393 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.688035  4094 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.689512 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.607775  4138 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.615126  3985 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.664810 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:47.665447 28332 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:47.696133 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.662573 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.663983  3841 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.664024  3917 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:47.669685 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:47.664139  4141 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:47.701686 28331 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23.
I20260709 12:21:47.670464  3956 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:47.673970 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:47.674224 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65801.8us
I20260709 12:21:47.674623 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:47.585227  4110 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.690215  3980 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:47.676177  4162 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.679507  4156 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.679804 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:47.662850 27661 raft_consensus.cc:2368] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:21:47.688398  4105 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29.
I20260709 12:21:47.692111 28335 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:21:47.611542  4003 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:21:47.697146 27660 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:47.610351  4108 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:47.706951  3983 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.200s	user 0.009s	sys 0.047s
I20260709 12:21:47.737566 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.604947  4136 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.703029 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.690658  4133 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.707640 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.748061  4141 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.748837 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.749912 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.750967 27701 leader_election.cc:400] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:21:47.607553  4163 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.706772 28334 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:21:47.710513 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.752931 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.753461  4163 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.754259  4163 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:47.754674  4163 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.755571  4160 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:47.756043  4160 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.756415  4160 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:47.708779  4118 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.690809  4060 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:47.728634 28328 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:47.724311  4110 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.700810  3917 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.741037  4156 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.761479  3917 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:47.767534  4132 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:47.739544  3983 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.233s	user 0.009s	sys 0.049s
I20260709 12:21:47.770319 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:47.760411  4060 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.771763  4060 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:47.764448 27661 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 18: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:47.700639  4138 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:47.713308 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.753563 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:47.601740  4020 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.706465  3956 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.751271 27701 leader_election.cc:403] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:47.784829 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.769017  4148 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:47.779245 27658 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:47.769065 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.779583 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:47.785995 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72855.4us
I20260709 12:21:47.755084  4163 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:21:47.790009 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.794255 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:47.795701 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.792809 27660 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:47.765121  3973 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.792505 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.801339 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.802073 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:47.764158  4156 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.801621 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.759213  3845 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:47.765863  4120 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.811851  3953 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:21:47.812487  3953 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:47.813828  4020 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.813436  3953 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.814666  4094 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.790072  4153 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:47.798398 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.815629  4153 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.815896 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 17.
I20260709 12:21:47.816084  4153 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:47.817013 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.818209 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:47.792323 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.793548  4003 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.819749  4003 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.821902 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:47.822316  4003 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.823549  4157 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.824096  4157 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.826381  4157 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.826087  3980 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.793291  3988 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:21:47.827833 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.828553 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:47.828284  3917 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:47.829599 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.830188 27663 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:47.748116  4133 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.834658  4133 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.835435 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.836584 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.837770 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.838358 27585 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:47.839756  3985 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:47.840219  3985 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.840608  3985 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:47.798190  3983 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:47.843741  3983 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:47.843555  3976 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:47.844172  3983 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:47.844374  3976 consensus_queue.cc:1243] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 29.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:21:47.845413  3976 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:47.846838  3976 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:47.847677 27656 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:47.848052 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:47.848277  3845 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.848719  3845 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:47.851529  4162 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:47.853205  3984 raft_consensus.cc:993] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:47.853735  3984 raft_consensus.cc:1081] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:47.855031 27581 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:47.856441 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3962) took 2188 ms. Trace:
I20260709 12:21:47.856849 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.856817 27589 rpcz_store.cc:276] 0709 12:21:45.668478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.191834 (+1523356us) service_pool.cc:224] Handling call
0709 12:21:47.192264 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.323543 (+131279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.323577 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.323625 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:47.323658 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.856207 (+532549us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540006f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:47.856253 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.856383 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301392128}
I20260709 12:21:47.860026 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.860734 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:47.861784 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.861900  3917 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:47.862402 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:47.865159  4160 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:47.868263 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3738) took 1398 ms (client timeout 1518 ms). Trace:
W20260709 12:21:47.868742 27730 rpcz_store.cc:269] 0709 12:21:46.469399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.470120 (+   721us) service_pool.cc:224] Handling call
0709 12:21:47.868072 (+1397952us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:47.868197 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346664192}
I20260709 12:21:47.875903 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1280) took 1581 ms. Trace:
I20260709 12:21:47.876363 28330 rpcz_store.cc:276] 0709 12:21:46.294746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.294969 (+   223us) service_pool.cc:224] Handling call
0709 12:21:47.875681 (+1580712us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:47.875810 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2160162176}
I20260709 12:21:47.804523 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.782009  3841 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:21:47.806969  4089 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.944s	user 0.008s	sys 0.010s
I20260709 12:21:47.807224  4150 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:47.785683  4138 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:47.790659  4148 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:47.784169  3962 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:47.880239 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:47.880751 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76327.8us
I20260709 12:21:47.884117 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:47.919701 28334 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:47.920787 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.921383 28330 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:21:47.890406 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.923797 27658 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:47.925244 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:47.926892 27658 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 22: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:47.892733 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.839198 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.908113 27553 leader_election.cc:400] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23.
I20260709 12:21:47.929927 27553 leader_election.cc:403] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:47.930039 27585 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:47.931732 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:47.929291 27659 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:47.936122 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:47.935142 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.937225 27658 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:47.891993 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.789489  3956 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:47.915901 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:47.916791 28335 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:21:47.938002 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:47.905690  4157 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:47.894156  3841 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.895608  4124 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.941125 27589 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:47.950655 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:47.953770 27702 connection.cc:256] RPC call timeout handler was delayed by -1.2580487310001445s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20260709 12:21:47.889453 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.943894 28328 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:47.921787 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:47.909844  4020 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:47.949244  3961 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:47.952732 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:47.964589  4020 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.946409 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:47.968925 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.965682 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.971181 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.941650  4094 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:21:47.932063  4150 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:47.922818  4160 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:47.966581 27663 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:47.916528  4093 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:21:47.952628 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:47.972492 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:47.993285 27589 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:47.949615  3841 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:47.947813  4157 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.989440 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:47.948848  4133 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.023s
I20260709 12:21:47.943404  4030 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:21:47.993780 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.994324 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.005944 27659 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:47.950654  3349 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:21:47.968514  4020 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:47.966777 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.009578 27584 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:21:48.010589 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.997117 27663 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:47.944728  4156 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
W20260709 12:21:48.003415 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50844.2us
I20260709 12:21:48.022524 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.991221  3984 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.993296  4065 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:47.995929  4120 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:48.023254 27584 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:47.997705  3976 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:47.995352  4003 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:48.047132 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 157787us
I20260709 12:21:48.034343 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.048851 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:47.998239  4157 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:48.046168 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:48.049240 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.050743 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.051653 27589 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:48.053807 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.005373  4030 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:48.054968 27589 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:48.007205  4160 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:48.057471 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.009807  4038 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:47.988051  4163 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.024835  4065 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.966190  4124 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.034070  3349 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:48.030898  3818 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:48.068092 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.062510 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.033435  4120 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.071169 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:48.066839 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.091728 27584 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:48.047086  3976 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:48.047421  4003 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.989915  4148 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.064929  4030 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:21:48.086788 27589 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:48.081687  3980 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.064100  4160 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:48.088910 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 136264us
I20260709 12:21:48.064294  4038 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:48.030507  3984 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.079567  4065 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:47.998123  4138 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.047593  3973 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:21:48.053733  4093 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:21:48.099666  4003 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:48.084046  4049 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:48.104753 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:48.102380 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.094667  4120 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:48.096638  3976 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:48.117976 27589 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:21:48.083254 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 136920us
I20260709 12:21:48.144968 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:48.099059 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 177329us
I20260709 12:21:48.111850  4160 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:48.086257  3985 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.138036 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.150031 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.125047 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:48.112278  4038 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:48.147430 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:48.086243  3818 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.151439 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.159358 27663 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:48.160027  4178 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:48.145539  3962 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:48.161521  4179 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:21:48.163602 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1453) took 1693 ms. Trace:
I20260709 12:21:48.163978 27734 rpcz_store.cc:276] 0709 12:21:46.470678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.303954 (+833276us) service_pool.cc:224] Handling call
0709 12:21:47.304362 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.559216 (+254854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.559503 (+   287us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:47.563469 (+  3966us) log.cc:844] Serialized 26 byte log entry
0709 12:21:47.999645 (+436176us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:47.999705 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.999806 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:48.163384 (+163578us) raft_consensus.cc:1685] finished
0709 12:21:48.163405 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.163545 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76187,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":552414,"threads_started":2,"wal-append.queue_time_us":499487}
I20260709 12:21:48.167932 27663 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:48.130947  4049 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:48.157711  3845 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.151937 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:48.168771 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.153086 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.176723 27663 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:48.089188  4150 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.116540  4060 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.137508  3988 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:21:48.178874 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.177284 27589 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:48.182608 27584 raft_consensus.cc:2410] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:21:48.180548 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:48.174562  4174 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:21:48.180176  3962 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:48.211585 27630 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:48.211836  3962 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:48.212875  4156 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:48.213244  4156 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.213754  4156 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:48.213127  3962 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.174960  4049 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:21:48.188468 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:48.191133  4180 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:21:48.192025  4181 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:48.200446  3985 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.191053 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.206161  4185 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:48.209288  4060 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.222721 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:48.228047 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:48.228703 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48251.9us
I20260709 12:21:48.232990  3983 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.236193  3952 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:48.242401  4187 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:48.244295 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:48.244583 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3227) took 3615 ms (client timeout 1662 ms). Trace:
I20260709 12:21:48.247143  4136 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.247617  4193 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:48.248116  3988 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:48.249217  3979 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:48.254191  3961 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:48.254827  4055 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.257416  4148 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:48.262668  4163 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.148697  3349 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:48.268447 27662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1555) took 3167 ms (client timeout 2275 ms). Trace:
I20260709 12:21:48.271631 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.280337 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3732) took 3190 ms. Trace:
I20260709 12:21:48.257231  4030 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.282058 27585 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:48.288431 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.281057 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3971) took 2998 ms. Trace:
I20260709 12:21:48.272090 27663 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:48.317412 27580 rpcz_store.cc:276] 0709 12:21:45.282234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.363473 (+ 81239us) service_pool.cc:224] Handling call
0709 12:21:45.363926 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.364391 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.364430 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:45.365050 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:21:47.878048 (+2512998us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b180047b780. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:21:48.027117 (+149069us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:48.027189 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.027351 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:48.280728 (+253377us) raft_consensus.cc:1685] finished
0709 12:21:48.280747 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.280995 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":674,"raft.queue_time_us":31622,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6684,"spinlock_wait_cycles":177291520,"thread_start_us":148808,"threads_started":1,"wal-append.queue_time_us":2675867}
W20260709 12:21:48.285753 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 3963) took 2617 ms (client timeout 1518 ms). Trace:
I20260709 12:21:48.320086  4094 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23.
I20260709 12:21:48.288362 27660 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:21:48.323422 27586 rpcz_store.cc:269] 0709 12:21:45.668724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.673309 (+  4585us) service_pool.cc:224] Handling call
0709 12:21:48.285564 (+2612255us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:48.285676 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268738304}
I20260709 12:21:48.274204 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:48.288502 27732 rpcz_store.cc:269] 0709 12:21:44.629337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.630319 (+   982us) service_pool.cc:224] Handling call
0709 12:21:48.244378 (+3614059us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54001d3dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:48.244488 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1131881344}
I20260709 12:21:48.320502 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.274956  4003 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.286911  4187 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.327625 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.328584  4003 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.328836  4187 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.329931 27656 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
W20260709 12:21:48.289944 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67372.2us
I20260709 12:21:48.331244 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:48.332191  4187 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:48.325745 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.313488 27731 rpcz_store.cc:276] 0709 12:21:45.090099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.295518 (+205419us) service_pool.cc:224] Handling call
0709 12:21:45.295796 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.206514 (+1910718us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b540004bda0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:21:47.291314 (+ 84800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.280083 (+988769us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:48.280275 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2087413760}
I20260709 12:21:48.334405 27586 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:21:48.190554  4153 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.336875 27586 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
W20260709 12:21:48.337589 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49235.9us
I20260709 12:21:48.308841 27581 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:21:48.297016 27659 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:48.292610 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1560) took 1936 ms. Trace:
W20260709 12:21:48.282500  4147 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.781s	user 0.011s	sys 0.000s
I20260709 12:21:48.285069 27589 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:48.354291 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.319911 27663 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.17-21.17]
I20260709 12:21:48.349196 27585 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:48.294471 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.359057 27584 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:48.308638 27662 rpcz_store.cc:269] 0709 12:21:45.100839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.116896 (+ 16057us) service_pool.cc:224] Handling call
0709 12:21:48.268183 (+3151287us) spinlock_profiling.cc:243] Waited 2.66 s on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:48.268359 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6108086144}
I20260709 12:21:48.354527 27657 rpcz_store.cc:276] 0709 12:21:46.355864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.777131 (+421267us) service_pool.cc:224] Handling call
0709 12:21:46.936533 (+159402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.315600 (+379067us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:47.352167 (+ 36567us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.407473 (+ 55306us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.407721 (+   248us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.407784 (+    63us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:47.407856 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.622925 (+215069us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b1800302f80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:48.291485 (+668560us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b1800302f80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:21:48.292409 (+   924us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b5400211454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:48.292451 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.292561 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27763,"prepare.queue_time_us":73582,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":1107,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":5740,"spinlock_wait_cycles":936912384,"thread_start_us":90704,"threads_started":2,"wal-append.queue_time_us":166584}
I20260709 12:21:48.357497 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:48.308501  4182 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:21:48.289618  3985 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:48.366850 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.375887 27589 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:48.272209  3976 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.381801 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:48.294920  3988 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.291415  4194 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.393061 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:48.385531 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.295213  3979 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:48.369656  4163 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:48.362994  4003 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:48.373665  4193 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:48.372539  4126 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:48.394783 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.397075 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:48.356335  3961 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:21:48.401121  4198 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:48.402182  4198 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:48.393121  3988 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:48.397560  4194 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.396296  4185 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:48.276153  3841 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.291052  4160 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.390653  3911 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:48.371712  4055 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:21:48.396299  4003 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.369377  4013 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
W20260709 12:21:48.386459 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 241556us
I20260709 12:21:48.393019 27584 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:21:48.396559  3979 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:48.392446  4060 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 12:21:48.441224 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3737) took 1972 ms. Trace:
I20260709 12:21:48.407310  4194 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.410101  4184 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:21:48.445447 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48453.7us
I20260709 12:21:48.411659  4163 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:48.414700  4193 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:48.435741  4003 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:48.449903 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.451416 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.451889 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.454407 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.455029 27735 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:48.455149 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.455845 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:48.411904  4126 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.436882  4020 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.463752 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.464335 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:21:48.443560 27738 rpcz_store.cc:276] 0709 12:21:46.469122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.471039 (+  1917us) service_pool.cc:224] Handling call
0709 12:21:46.471298 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:46.471550 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:46.498404 (+ 26854us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.255341 (+1756937us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b18002d5bc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:48.266206 (+ 10865us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b18002d5bc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:21:48.389587 (+123381us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:48.389669 (+    82us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:48.389737 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.389853 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.441143 (+ 51290us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5455,"prepare.queue_time_us":34272,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":4450,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":7927,"spinlock_wait_cycles":12642944,"thread_start_us":309537,"threads_started":2,"wal-append.queue_time_us":209203}
I20260709 12:21:48.449118  3911 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:48.468945 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1451) took 2919 ms (client timeout 2849 ms). Trace:
I20260709 12:21:48.471153 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.472098 27735 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:48.471771 27737 rpcz_store.cc:269] 0709 12:21:45.549181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.549358 (+   177us) service_pool.cc:224] Handling call
0709 12:21:48.468781 (+2919423us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:48.468882 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":610519296}
I20260709 12:21:48.459214  3845 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.483496 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.489813  4193 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.495433 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.498574 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.499029 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.499905 27734 raft_consensus.cc:2376] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25. Re-sending same reply.
I20260709 12:21:48.492156 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.501243 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.503594 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.470649  4055 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:48.459354  4185 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:21:48.460570  3956 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.514180 27729 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:48.515637 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.516229 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.471558  4157 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.520289 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.522553 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.526051 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:48.527144  4055 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:48.528411 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.529901 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.527979 27735 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:21:48.510011  3976 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.531113 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.533900 27581 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:48.534870 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.536814 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:48.513936  3845 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:48.537873 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.538460 27589 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:21:48.515899  4030 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.542989 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.543632 27731 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:48.544368 28333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1278) took 3243 ms (client timeout 1557 ms). Trace:
I20260709 12:21:48.544646 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.546411 27737 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:48.547111 27735 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
W20260709 12:21:48.548059  4203 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:21:48.548331  4020 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:48.549455  4201 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:21:48.550839 27655 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:21:48.551368 27730 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:21:48.552414  4204 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:48.552561 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.506361  4163 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.553644 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.554235 27730 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:48.559422  4137 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.559953  4137 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.561513  4120 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.523352  4108 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.563493  3841 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.565440  4049 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:48.545714 28333 rpcz_store.cc:269] 0709 12:21:45.296552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.297507 (+   955us) service_pool.cc:224] Handling call
0709 12:21:48.539716 (+3242209us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:48.539817 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62828800}
I20260709 12:21:48.569092 28330 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 16.
I20260709 12:21:48.562682 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:48.525355  4153 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.577695 27729 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:21:48.580581 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:48.581732  4141 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.003s	sys 0.011s
I20260709 12:21:48.581732 27589 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:48.589439 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.592010 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.592604 27581 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:48.594118 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.597515 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.599545 27737 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:21:48.574496  3845 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:48.601550 27734 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
W20260709 12:21:48.588912  4141 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.004s	sys 0.011s
I20260709 12:21:48.610528  4065 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.627259 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:48.640472 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.642122 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.643047 27729 raft_consensus.cc:2368] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:21:48.643841 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.644567 27589 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:48.645526 27729 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:48.617024  4049 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.618924  3956 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.620849  4120 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.621701  4021 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.625624  4157 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.652423 27580 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:48.653147 27584 raft_consensus.cc:476] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Not starting forced leader election -- already a leader
I20260709 12:21:48.582207  3988 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.546085  3911 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:48.628454  4164 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.992s	user 0.004s	sys 0.000s
I20260709 12:21:48.627944  4156 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.629648 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.632092  3643 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:48.632014  3845 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.637553 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:48.611760 27581 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:48.651147 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:48.665697 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:48.599119  4160 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.654661  4020 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:48.659848 28335 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:21:48.660373 27586 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:48.660326 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.652941  4185 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.668855 27732 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:48.671473 27737 raft_consensus.cc:2376] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26. Re-sending same reply.
I20260709 12:21:48.671204  4020 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.667981  4021 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:48.628396  4038 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.671605 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:48.676229 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86605.6us
I20260709 12:21:48.673070  4185 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.680092  3961 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:48.670926  3990 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:48.682336 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:48.677971  4020 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:48.687896 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3837) took 1149 ms. Trace:
I20260709 12:21:48.688380 27655 rpcz_store.cc:276] 0709 12:21:47.538475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.662129 (+123654us) service_pool.cc:224] Handling call
0709 12:21:48.687685 (+1025556us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b540005fd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:48.687832 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1982083456}
I20260709 12:21:48.683236 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.669608  3952 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.007s
I20260709 12:21:48.684327  4141 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:48.653366 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.685007  3990 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:48.692451  4141 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.693487  3990 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.694746  4141 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.684474  3961 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:48.663477  3979 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.686141  4124 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.696013 27730 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:48.687685  4185 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.691892  4137 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.683313  4021 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:48.697335  4183 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:48.698102 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.691780 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.701365 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.702383  4195 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.703281  3552 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.719485 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.704779 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.720778 27583 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:48.705472 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.723610 27586 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:21:48.706104 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:48.706715 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.699203 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.697633  4021 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.708479 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.728080 27737 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:48.687144 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:48.713095 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.732710 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:48.734090 27655 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:48.734464 27583 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:48.734851 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.734059 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.725481 27589 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:48.735297 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:48.736342 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.736217 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.736634 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.737221 27583 raft_consensus.cc:2376] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23. Re-sending same reply.
I20260709 12:21:48.715934 27729 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:48.738027  4137 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:48.738415 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.738623  4137 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:48.738996 27660 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:48.739766  4137 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.743592 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.743822 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.744621 27656 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:48.744187 27657 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:48.745059 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.680927  3988 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.750012 27583 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:48.751257 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:48.751076 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.752631 27583 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 18.
I20260709 12:21:48.753008 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.753928 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.754089 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:48.745576 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.760898 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.761624 27589 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:48.764734 27660 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:48.767036 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.768867 27581 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:48.770519 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.744169  3552 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:21:48.743916  4094 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:21:48.773352 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.763574  4153 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:48.776654  4209 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:21:48.778333 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.778299  4003 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.779877 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:48.780529 27585 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:48.781607 27585 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:48.768720  3841 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:21:48.783113  3841 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.783511  3841 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:48.783635 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.784332 27730 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:48.785145 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.783614 27585 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:21:48.786943  4183 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:48.771109 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:48.798655  4020 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.789551 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.793802  4094 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:48.727929  4195 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.798990  3552 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:48.788094  4065 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.801034 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:48.822378  4148 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.001s
I20260709 12:21:48.802732  3977 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:48.803704  4133 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:48.841075  3552 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:48.804487  4199 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:48.804792 27730 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:48.841892  4199 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:48.805719  4003 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.809134  3984 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:48.810496 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.845095 27732 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:48.846002 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.846611 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:48.810467 27657 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:21:48.848003  4020 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:48.848130 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.849360 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.849912 27657 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:48.811111 27586 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:48.771653 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:48.851758 27581 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:21:48.852384  4090 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 3.557s	user 0.018s	sys 0.006s
I20260709 12:21:48.811638 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:48.824515 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:48.853868 27660 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:48.854787 27655 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:48.774695  4153 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:48.778240  4214 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:21:48.831521 27662 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:48.834053 27656 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:48.842329  3552 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.748227  4158 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:21:48.801153  3349 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:48.858026 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:48.884563 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:48.897684 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:48.887359 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:48.901513 27656 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:48.893941 27657 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:48.894076 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:48.905341 28295 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:48.911125 27660 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:48.886118  4153 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:48.916335 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.917897 27732 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:48.887130  4020 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:48.842839  4199 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:48.898649  4156 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.687654 27587 rpcz_store.cc:276] 0709 12:21:46.483564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.458988 (+975424us) service_pool.cc:224] Handling call
0709 12:21:47.459229 (+   241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.459488 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:47.507867 (+ 48379us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.661222 (+153355us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.661279 (+    57us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:47.661325 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.661400 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.661542 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48429,"prepare.queue_time_us":48662,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":22671,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":10713,"thread_start_us":130664,"threads_started":1}
I20260709 12:21:48.932365  4188 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:48.945309  4038 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:48.918069  4196 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:48.951103  4139 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:48.951900  4195 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:48.954196  4218 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:48.960702 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:48.948015  4158 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.949000  4199 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:48.969453 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:21:48.978099 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 138050us
I20260709 12:21:48.949570  4188 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:48.963601  3990 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.986094 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:48.949154  4094 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:48.987906 28328 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: [21.17-21.17]
I20260709 12:21:48.989289 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:48.994261 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:48.995263 27732 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:48.997165  4020 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:48.997442  4188 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:48.999199 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0070044179999513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260709 12:21:49.000305 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:48.978412  4158 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.002542 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.003077  4117 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.003299 27734 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:49.003906  4158 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:49.004251 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.004007  4117 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:49.005120  4117 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:49.005414 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.006163 27734 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:49.006265  4105 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:49.006678  4105 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:49.006902 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.007529  4105 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.008111 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.008705 27734 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
I20260709 12:21:49.008811  4105 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:49.001595 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.010532 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.011904 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.012261 27733 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:49.012328 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.012941 27552 leader_election.cc:400] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:21:49.013275 27552 leader_election.cc:403] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:49.013368  3841 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.015072 28335 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:49.015237 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:48.963904  4196 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.037999 27732 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:49.015862  4117 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:48.994446  3990 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.056674  4117 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:49.017010  4231 raft_consensus.cc:2764] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:21:49.058534  4117 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:49.060325  4228 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:49.018770  4196 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:49.062649  3878 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:49.080758  4158 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.086894 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.061885  4233 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:21:49.059875  4230 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:21:49.092592 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.093266 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.093874 27581 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:49.066324  4094 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 12:21:49.105690  4231 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:49.124763  4065 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:49.125434  4065 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:49.134696  4094 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:21:49.134989  4235 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:21:49.135519  4094 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:49.135869  4094 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:49.135901  4038 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:49.136548  4038 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:49.136751  4094 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.090562  4158 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.138043  4094 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:49.137643  4038 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.140957  3977 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.141825  3977 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:49.142685 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 397181us
I20260709 12:21:49.144112  3977 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:47.664752 27588 rpcz_store.cc:276] 0709 12:21:46.484332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.596425 (+1112093us) service_pool.cc:224] Handling call
0709 12:21:47.596716 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.606946 (+ 10230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.606979 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.607019 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:47.607050 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.607109 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.607227 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3661,"raft.queue_time_us":42130,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":13990}
I20260709 12:21:49.144742  3988 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:49.145298  3988 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.145614  4231 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:49.146185  3988 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.146586  4231 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.148541 27738 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:49.149669 27738 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:49.149999 27738 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.150437  4003 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:49.150707 27738 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.169988 27738 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:49.151840  4054 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:49.170545  4054 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.171370  4054 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.172619  4054 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:49.144068 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.152697  3841 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:49.153167 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:49.174595 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.158355  3818 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:49.162767  3917 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:21:49.150193 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 325836us
I20260709 12:21:49.126582  4065 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.168300  4158 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:49.141649  4153 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:49.168990 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76525.4us
I20260709 12:21:49.169093 27737 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:49.151620  4003 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:49.178506 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.198325 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:49.199823 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.200621 27588 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:49.201715 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:49.203099 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:49.178603 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:49.181178 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.185554 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.193748  3983 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:49.195757  4067 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.182305 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.209638 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.210155 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.206817 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.207113 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:49.207664 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 192493us
I20260709 12:21:49.176853  3917 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:49.214221 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:49.220026 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:49.236593 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51141.9us
I20260709 12:21:49.237423 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.196759  3911 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:49.241928 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.244439 27588 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:49.245085 27589 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 19: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:49.245718 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.197023  4003 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.248950 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.257068 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:49.206276  3845 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:49.233543  4199 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:49.271517 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.278981 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:49.272578 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:49.233870  4065 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:49.243636  3984 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:49.247745  3911 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.240592  3961 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:49.294945  4198 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.312302  4198 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.233283  4197 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:21:49.319255 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:49.272851 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.244060  4163 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:49.274133 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:49.332132 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.332391 27585 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:49.290868 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.261329  4230 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:49.247525  4034 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:49.239097  4219 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:49.244204  4183 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:21:49.288590 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:49.310971  4239 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:49.335615  4239 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.336474  4117 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.337551 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:49.337973  4239 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:49.310497  4144 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:49.338618  4117 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:49.338954  4144 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:49.340042  4144 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.317631  3984 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.343494  3984 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.348238  3979 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.296816  3911 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:49.349783 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:49.350014  4110 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.351014 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.292451  4158 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:49.355347 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3852) took 1567 ms. Trace:
I20260709 12:21:49.355991  4163 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.355855 27659 rpcz_store.cc:276] 0709 12:21:47.788134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.354580 (+566446us) service_pool.cc:224] Handling call
0709 12:21:48.354980 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.365308 (+ 10328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.365339 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:48.372411 (+  7072us) log.cc:844] Serialized 26 byte log entry
0709 12:21:48.449137 (+ 76726us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:48.467932 (+ 18795us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.468141 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.689852 (+221711us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b18003770e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:49.354916 (+665064us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:49.354991 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.355057 (+    66us) raft_consensus.cc:1685] finished
0709 12:21:49.355078 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.355248 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":20211,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":4372,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":11708,"spinlock_wait_cycles":28432512,"thread_start_us":38552,"threads_started":1,"wal-append.queue_time_us":131434}
I20260709 12:21:49.358120  4003 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.359758 27553 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:49.360090 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3117) took 1898 ms. Trace:
I20260709 12:21:49.360530 27664 rpcz_store.cc:276] 0709 12:21:47.461338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.461485 (+   147us) service_pool.cc:224] Handling call
0709 12:21:49.359902 (+1898417us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:49.360008 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3443394816}
I20260709 12:21:49.361198 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.360123 27553 leader_election.cc:403] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:21:49.362071 28335 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:21:49.363010 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.362877  4163 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.363667 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.364732 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.237445  4067 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.293185  3841 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:21:49.283811  3845 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.372645 28333 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:49.295127  4021 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:49.376528 28334 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:49.377483 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.385979 28337 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:49.297283  4199 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.384405 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.400063 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.311156  4207 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:49.403019 27737 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:49.427652 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.429960 27589 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:21:49.317219  4198 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.331586  3961 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:49.437026 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.377077  4140 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:49.379568  4206 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:21:49.458209  4236 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:21:49.459056  3845 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.607785  4055 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:49.608346  4055 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.460999  4188 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:49.601696 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.488826 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.612442 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.613205 28333 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:49.491907  4183 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:49.443176  4021 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:49.496058  4199 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.500020  4038 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:49.506556 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 295537us
I20260709 12:21:49.619647 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.516595 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 3978) took 1785 ms. Trace:
I20260709 12:21:49.620625 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:49.620563 27580 rpcz_store.cc:276] 0709 12:21:47.731041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.329762 (+598721us) service_pool.cc:224] Handling call
0709 12:21:48.330058 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:48.652947 (+322889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.516208 (+863261us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54001fead4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:49.516474 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":735360384}
I20260709 12:21:49.526561  3990 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:49.541424  4215 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.838s	user 0.004s	sys 0.000s
W20260709 12:21:49.542289 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.379316  4219 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.543290 27737 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:21:49.633185 27656 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:21:49.545265 27661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3125) took 1834 ms (client timeout 2159 ms). Trace:
W20260709 12:21:49.633829 27661 rpcz_store.cc:269] 0709 12:21:47.711209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.737282 (+ 26073us) service_pool.cc:224] Handling call
0709 12:21:49.545017 (+1807735us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:49.545192 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59553024}
I20260709 12:21:49.558651  4126 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.564703 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3854) took 1766 ms. Trace:
I20260709 12:21:49.637038 27663 rpcz_store.cc:276] 0709 12:21:47.798334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.357998 (+559664us) service_pool.cc:224] Handling call
0709 12:21:48.386530 (+ 28532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.421875 (+ 35345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.421932 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:48.435528 (+ 13596us) log.cc:844] Serialized 26 byte log entry
0709 12:21:48.512753 (+ 77225us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:48.512793 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.564330 (+1051537us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:49.564378 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.564425 (+    47us) raft_consensus.cc:1685] finished
0709 12:21:49.564438 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.564643 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26230,"raft.queue_time_us":41042,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":4741,"spinlock_wait_cycles":5401856,"thread_start_us":54168,"threads_started":1,"wal-append.queue_time_us":144442,"wal-append.run_cpu_time_us":1724,"wal-append.run_wall_time_us":1005674}
I20260709 12:21:49.567067  3961 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.578289 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:39856 (request call id 4004) took 1281 ms. Trace:
I20260709 12:21:49.640748 27584 rpcz_store.cc:276] 0709 12:21:48.309838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.364015 (+ 54177us) service_pool.cc:224] Handling call
0709 12:21:49.578095 (+1214080us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54001fead4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a33fa3d 00007f0e9f30fcd0 00007f0e98894254 00007f0e988941e2 00007f0e98894111 00007f0e9889408c 00007f0e98893342 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:49.578187 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508419328}
W20260709 12:21:49.579198  4169 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.614s	user 0.004s	sys 0.000s
I20260709 12:21:49.579967  4148 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:49.582854  4067 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:49.587895  4153 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:49.589007 27730 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
W20260709 12:21:49.589608  4105 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.580s	user 0.005s	sys 0.011s
I20260709 12:21:49.590359 27734 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:49.372148  3917 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.654974  3917 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:49.655555  4148 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:21:49.594144 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.597638 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.659603 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 16.
I20260709 12:21:49.598956 28336 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:49.599696 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.599879 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.601668 27738 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:21:49.602022 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.602807 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.602942 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3741) took 1152 ms. Trace:
I20260709 12:21:49.661626 27731 rpcz_store.cc:276] 0709 12:21:48.451023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.451153 (+   130us) service_pool.cc:224] Handling call
0709 12:21:49.602704 (+1151551us) spinlock_profiling.cc:243] Waited 91.4 ms on lock 0x7b54001d1fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:49.602863 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210312576}
I20260709 12:21:49.590370 27732 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:49.606149  4003 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:49.606794 27588 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:21:49.609472  4055 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.427353  3841 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:49.624629 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:49.670650 27664 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:49.623963 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.625625 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:49.626705 27658 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:49.672658 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.672966 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.673898 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:49.674633 28334 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:21:49.628778 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.628736 27660 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:21:49.629786 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.643074  3911 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.645031 27487 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:49.680958 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81168.8us
W20260709 12:21:49.681085 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57199.9us
I20260709 12:21:49.681740 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:49.644794  4118 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.650566 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.651425 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.592466  4139 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:21:49.665505 28337 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:49.667281 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.668474 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.673291 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:49.682013 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.684023 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:49.687875 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.684170 28334 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: []
I20260709 12:21:49.690995 28329 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:49.680083  4196 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.591s	user 0.010s	sys 0.000s
I20260709 12:21:49.702998 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3744) took 1248 ms. Trace:
I20260709 12:21:49.703197 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.703469 27736 rpcz_store.cc:276] 0709 12:21:48.454387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.454515 (+   128us) service_pool.cc:224] Handling call
0709 12:21:48.463943 (+  9428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.524310 (+ 60367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.524359 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:48.524861 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:21:48.652051 (+127190us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:48.727519 (+ 75468us) log.cc:844] Serialized 14 byte log entry
0709 12:21:49.141837 (+414318us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b180043bdc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:21:49.142505 (+   668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.340696 (+198191us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54000a64d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:49.340764 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.540716 (+199952us) spinlock_profiling.cc:243] Waited 7.65 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:49.702791 (+162075us) raft_consensus.cc:1685] finished
0709 12:21:49.702811 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.702943 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42961,"prepare.queue_time_us":101482,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":425829,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":11931,"spinlock_wait_cycles":148536704,"thread_start_us":128713,"threads_started":2,"wal-append.queue_time_us":140686}
I20260709 12:21:49.707803 27729 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:49.709666 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.711566 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.701478 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:49.713057 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.713284  4186 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:49.714111  4193 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:21:49.714426 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.714830 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.715191 27663 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:49.689595  4126 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.715765 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:49.715814  4126 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.716765  3988 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:49.715196  4193 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:49.717070  4198 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:49.717357  3988 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.717581  4198 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.718076  4198 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:49.718662 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.718426  3988 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.723649 27663 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:49.724097  3988 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:49.724174  3983 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:49.724829  3983 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.725203  3983 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:49.719601  4049 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:49.727576 27663 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:49.721089  4186 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:49.728410  4197 consensus_queue.cc:1175] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 30 status { last_received { term: 30 index: 21 } last_committed_idx: 21 last_received_current_leader { term: 30 index: 21 } }
I20260709 12:21:49.729146  4186 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.730093  4049 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:49.721772 27657 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:49.721871  4054 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:49.723042 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:49.720234  4184 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:49.732556  4184 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:49.733456  4184 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.736922  4049 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:49.737324  4049 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.737641  4049 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:49.740999 27663 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:49.742919 28333 raft_consensus.cc:2410] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 26 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:21:49.743454  4021 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.748343 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.686126  4067 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:49.685788  4158 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.685628  3984 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:49.764424 27735 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:49.685931  4139 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:49.756326 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.755231 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:49.758116 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:49.758782 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.759065 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.769944 27658 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 29 because replica has last-logged OpId of term: 27 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260709 12:21:49.760241 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:49.747416  4124 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:49.761050  4193 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:21:49.773700 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 346695us
I20260709 12:21:49.761626  3979 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.762118 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1578) took 4091 ms. Trace:
I20260709 12:21:49.762614  4020 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:49.763304  4158 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.706295  3990 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:49.769315 27663 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 16: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:49.771596 27658 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:21:49.769347 27660 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:49.772586 27664 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:49.706631  4055 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:49.778082 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:49.778157 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.797245 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:49.794066 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.689436  4118 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.692334  4048 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.083s
I20260709 12:21:49.780952 27582 rpcz_store.cc:276] 0709 12:21:45.671157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.417170 (+1746013us) service_pool.cc:224] Handling call
0709 12:21:47.417582 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.417996 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.418020 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:47.438068 (+ 20048us) log.cc:844] Serialized 26 byte log entry
0709 12:21:47.730434 (+292366us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:47.730488 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.730576 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.568378 (+1837802us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:49.754878 (+186500us) raft_consensus.cc:1685] finished
0709 12:21:49.754899 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.762050 (+  7151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19510,"raft.queue_time_us":19703,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":12172,"spinlock_wait_cycles":8790784,"thread_start_us":234132,"threads_started":1,"wal-append.queue_time_us":460339}
I20260709 12:21:49.809907 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:49.820128 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:49.820267 27664 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 21. (term mismatch)
I20260709 12:21:49.821363 27664 pending_rounds.cc:85] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 20
I20260709 12:21:49.821595 27664 pending_rounds.cc:107] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 27.21
I20260709 12:21:49.821991 27664 raft_consensus.cc:2889] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:49.822892 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:49.825268 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.805516  4124 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:49.747644  4196 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.766464  4219 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.840974 27582 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:21:49.828024  4228 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:49.845235  3911 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:49.825682  4094 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
W20260709 12:21:49.848959 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.829330  4151 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:49.851179  4151 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:49.851658  4151 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:49.823923  4183 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:49.803858  4150 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.107s	user 0.010s	sys 0.011s
W20260709 12:21:49.856449 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 565640us
I20260709 12:21:49.801436  3979 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.764591  4148 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.828274  4121 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:49.853296  4255 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:49.870018 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:49.808128  4049 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:49.885975 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.806456  4020 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.909474 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:49.914839 27736 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
W20260709 12:21:49.867213  4150 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.010s	sys 0.011s
W20260709 12:21:49.907508 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 272045us
W20260709 12:21:49.913435 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43506.3us
I20260709 12:21:49.814643  3841 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:49.966131  4156 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:21:49.975260 28292 connection.cc:256] RPC call timeout handler was delayed by -1.3879478339999878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
W20260709 12:21:49.963277 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.981150 28331 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:49.869939  4094 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:49.980561 28292 connection.cc:256] RPC call timeout handler was delayed by -1.3714537169998948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20260709 12:21:49.983338  4049 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:49.984158  4020 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.883944  4030 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.104s
I20260709 12:21:49.988773  4030 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:49.988628  4094 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.989202  4030 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:49.990067  4030 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:49.990931  4094 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:49.988037 28292 connection.cc:256] RPC call timeout handler was delayed by -1.215031387999943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20260709 12:21:49.987205  4185 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:50.008143  4185 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.009018 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.010051  4228 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:50.010432  4185 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:49.991182  4030 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:50.012059 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.012547  4158 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:50.014458 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.015022 27584 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:49.992785  3979 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:50.016605 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:50.017267 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:21:50.018518 27729 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:21:49.996523  4156 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:50.019610  4156 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:50.019968  4156 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:50.020782  4156 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.021351 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.021968  4156 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:49.962545  3818 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:21:50.023578  3818 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:50.024016  3818 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:50.024205 27730 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:21:50.024817  3818 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.000808  4198 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.001628 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:50.002978  4245 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.301s	user 0.002s	sys 0.000s
I20260709 12:21:50.059446 27583 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:21:50.062209 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:50.062772 27583 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:50.003118 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:50.064440 27732 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:50.004119  4260 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:21:50.089201 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.027591 27487 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 23. New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:50.090587 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.097546 27583 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 20.
I20260709 12:21:49.874007  4134 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.163s
I20260709 12:21:49.968129  4148 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:49.972438  4140 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:50.118885  3917 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:50.036648  4207 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:50.068396  3818 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:50.068296  4153 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:21:50.121977  4162 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.121714  4207 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.122525  4162 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.121532 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:50.075758 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:50.124074  4207 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:50.079653 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.080003  3983 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.084133 27589 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:21:50.086124  4151 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.089404  4198 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:49.979036 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69627us
I20260709 12:21:49.872793  4118 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.116029 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:50.142735 27732 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:49.871650  4141 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.160s
I20260709 12:21:49.873848  4183 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.130963 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.119383 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:49.988391  4196 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:49.912863  4121 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.163686 27659 raft_consensus.cc:2368] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24.
I20260709 12:21:50.147123 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.164672 27580 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:50.149685  4139 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.157058 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.161558 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3257) took 1512 ms. Trace:
I20260709 12:21:50.168799 27552 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19.
I20260709 12:21:50.168943 27734 rpcz_store.cc:276] 0709 12:21:48.649104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.009331 (+360227us) service_pool.cc:224] Handling call
0709 12:21:49.590122 (+580791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.654378 (+ 64256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.161300 (+506922us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54001d47d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:50.161481 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1334979456}
I20260709 12:21:50.125008  4162 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.170318 27582 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:50.170532 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:50.140923  3990 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:50.121578  3917 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.179203 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:50.173699 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:50.187060 27729 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:50.178402 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:50.188016 27738 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:50.188977 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:50.190594 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:50.192476 27655 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:50.193373 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:50.173908 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:50.197914 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:50.206197 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:50.208200 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:50.211758 27735 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:21:50.213485 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.214412 27580 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:21:50.214334 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:50.166498  4206 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:50.218698 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.176056 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.220294 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:50.223729 27735 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:21:50.221890 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.172755  4038 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:50.226904 27734 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:21:50.223788 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:50.192796  4030 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:50.193094  4139 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:21:50.289597  4183 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:50.293082  4121 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.294224 28336 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:21:50.303447 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.002s	sys 0.036s
W20260709 12:21:50.304638 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.002s	sys 0.036s
I20260709 12:21:50.302377 28336 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:50.215893  4094 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:21:50.307749 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 484421us
I20260709 12:21:50.227262  4133 raft_consensus.cc:2764] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:21:50.192375  4140 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.105s
I20260709 12:21:50.227962  4206 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:50.213557  4003 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:50.263947  4038 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:50.316165 27731 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.013s	sys 0.034s
W20260709 12:21:50.316572 27731 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.013s	sys 0.034s
I20260709 12:21:50.317015 27580 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:21:50.216715  4231 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:50.228371  3980 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:50.319175 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:50.298409  4030 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:50.228973  3990 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:50.263748  4034 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:21:50.308478 28336 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.358757 27580 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:50.343582 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:50.405469 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86364.8us
W20260709 12:21:50.409147 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.410672 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:50.313824  4186 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.013s	sys 0.026s
I20260709 12:21:50.313588  4197 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:50.335696  4021 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.402842  3980 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.356348  4003 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:50.424535 28336 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:21:50.424747  3983 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:50.337862  4196 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.376577  4030 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:50.405120  4020 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.433602  4048 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:21:50.419881  4186 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.013s	sys 0.026s
I20260709 12:21:50.429235  4197 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.514804 28328 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:21:50.517092 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:50.450215  3917 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.454907  4134 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.197715 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:50.425397  4021 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.431941  3980 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:50.455139 27487 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 29, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 29 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:50.458990 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:50.460186  4055 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:50.428846  4183 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:50.464453 27582 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:21:50.469954  4253 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.817s	user 0.007s	sys 0.000s
I20260709 12:21:50.471875 27732 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 12:21:50.474002 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.416188  4158 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:21:50.476190  4252 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.763s	user 0.005s	sys 0.000s
I20260709 12:21:50.480865  4003 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:50.483119  4266 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:50.422953 27580 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:21:50.494673 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.498248 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.498708 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.507592 28332 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:21:50.496861 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:50.621236 28337 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:50.507586 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.339069  4156 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.094s
I20260709 12:21:50.427711  4034 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 25.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:21:50.448828  4257 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:50.519057 28333 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:21:50.405870  3990 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:50.462353 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.634054 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:50.510218  4219 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.024s	sys 0.059s
W20260709 12:21:50.530756 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71851.4us
I20260709 12:21:50.539623  4186 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:50.516106  4140 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:50.524322  4199 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.872s	user 0.039s	sys 0.049s
I20260709 12:21:50.513505  4133 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:50.600464 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.617197 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.641927  4262 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:21:50.530274  4048 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:50.634644 27582 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:50.509838  4196 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.513006  4020 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.612897 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.536758  4151 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:50.537218 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.692247 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:50.667966  4144 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.034s	sys 0.043s
I20260709 12:21:50.624233 27580 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
W20260709 12:21:50.641482  4219 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.024s	sys 0.060s
W20260709 12:21:50.642155  3962 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.009s	sys 0.016s
I20260709 12:21:50.535885  4134 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.678233  4265 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:50.688258 27582 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19.
W20260709 12:21:50.659471  4199 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.041s	sys 0.054s
W20260709 12:21:50.653460 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.676008  3552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.024s	sys 0.027s
W20260709 12:21:50.745779  3552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.024s	sys 0.027s
I20260709 12:21:50.746058 27733 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:50.695744  3988 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:50.696223  4121 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:50.747365 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3264) took 1752 ms. Trace:
I20260709 12:21:50.696386  3984 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.748020  3984 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.750293  3984 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:50.678860  4048 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.750911  4048 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:50.752115 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.752844 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:50.753293  4270 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:21:50.753545 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.754125 27582 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:50.754967 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.698299  4020 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:50.699894  4271 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:21:50.756137  4134 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:50.704802  4196 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.708262  4021 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:50.712482  4030 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.669077  4186 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:50.757853  4186 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.758780  4030 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.746803  3988 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:50.681835  3818 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.004s
W20260709 12:21:50.732806 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.761307  3988 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.539942  4183 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:50.761912 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.762246 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 3996) took 1564 ms. Trace:
I20260709 12:21:50.747812 27733 rpcz_store.cc:276] 0709 12:21:48.995232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.351811 (+356579us) service_pool.cc:224] Handling call
0709 12:21:49.352098 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.746516 (+1394418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.747089 (+   573us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:50.747270 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3205280640}
I20260709 12:21:50.763265  4183 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.756688  4265 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:50.737879 27580 pending_rounds.cc:85] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:50.764084 27580 pending_rounds.cc:107] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 29.16
I20260709 12:21:50.764364 27580 raft_consensus.cc:2889] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:50.765354 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:50.765571 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.753479 27588 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:21:50.765901 27580 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:50.765610 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:50.762650 27586 rpcz_store.cc:276] 0709 12:21:49.197801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.203909 (+  6108us) service_pool.cc:224] Handling call
0709 12:21:49.204280 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:49.204708 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:49.236805 (+ 32097us) log.cc:844] Serialized 14 byte log entry
0709 12:21:49.625517 (+388712us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:49.625950 (+   433us) log.cc:844] Serialized 26 byte log entry
0709 12:21:49.626078 (+   128us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:49.626108 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.626169 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:50.303809 (+677640us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:50.761969 (+458160us) raft_consensus.cc:1685] finished
0709 12:21:50.762073 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.762196 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":33182,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":8875,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":38982,"spinlock_wait_cycles":82110720,"thread_start_us":388300,"threads_started":1,"wal-append.queue_time_us":475415}
I20260709 12:21:50.769093 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.655426  4198 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.076s
W20260709 12:21:50.729301  4268 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:21:50.786026 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:50.740265  3962 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.010s	sys 0.016s
W20260709 12:21:50.737978  4144 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.034s	sys 0.044s
I20260709 12:21:50.788727 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:50.798969 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:50.790364 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:50.800110 28334 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:50.800024 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.787823 27580 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:50.803059 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:50.793998 27704 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19.
I20260709 12:21:50.785449 28337 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:50.804121 27704 leader_election.cc:403] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:50.799366 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:50.669261  4140 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.677743  4133 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:50.819067 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.023s	sys 0.063s
I20260709 12:21:50.811893 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:50.812366 27580 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:21:50.826309 27580 pending_rounds.cc:85] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:50.826562 27580 pending_rounds.cc:107] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:21:50.826960 27580 raft_consensus.cc:2889] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:50.826644 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.023s	sys 0.063s
I20260709 12:21:50.830871 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:50.831561 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.834457 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.792506  4105 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:50.844558 27581 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.019s	sys 0.008s
W20260709 12:21:50.847126 27581 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.019s	sys 0.008s
W20260709 12:21:50.808590  4243 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.011s	sys 0.025s
I20260709 12:21:50.794386  4267 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:50.818702  4020 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:50.832636  4134 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:21:50.850097 27731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.005s	sys 0.012s
I20260709 12:21:50.785035  4030 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:21:50.785401  3988 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:21:50.823730  4199 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:21:50.859592 27586 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:50.860783 27589 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 12:21:50.863487 27583 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.007s	sys 0.032s
I20260709 12:21:50.863358 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:50.895864 27583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.007s	sys 0.032s
I20260709 12:21:50.823450  4021 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:50.865434  4197 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.867748 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.870641  4133 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:50.848404  3990 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:50.859607  4105 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.872537 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1606) took 1218 ms. Trace:
I20260709 12:21:50.885885  4020 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.870950  3980 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.807248  3984 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
W20260709 12:21:50.882768  4094 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.566s	user 0.005s	sys 0.004s
I20260709 12:21:50.884166 27582 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:21:50.832481  4257 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:21:50.887305 27731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.005s	sys 0.012s
I20260709 12:21:50.847460 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:50.925853 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81510.5us
W20260709 12:21:50.926672 27628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.000s	sys 0.036s
W20260709 12:21:50.927104 27628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.000s	sys 0.036s
I20260709 12:21:50.890815 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:50.927948 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.027s	sys 0.036s
W20260709 12:21:50.928375 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.027s	sys 0.036s
W20260709 12:21:50.929020 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 585513us
I20260709 12:21:50.920486  4020 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:50.940841 27655 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:21:50.924113  4094 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:50.943168 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:50.946090 27655 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 16.
I20260709 12:21:50.863682  4265 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:50.925001  4048 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.951588 27585 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:50.945115 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:50.963768 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:50.964948 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.965155 27658 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:21:50.965270 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.965600 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.948464 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:50.966984 27664 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:50.935019  4119 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:50.830595  4219 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:21:50.885248  4140 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.832158  4163 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:21:50.866855  4267 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.900527 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:50.863300  4243 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.011s	sys 0.025s
I20260709 12:21:50.845749  4121 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260709 12:21:50.988121  4119 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:51.020393  4119 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:51.022177  4048 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:50.945863  4158 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:21:50.983793  3984 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.885993  4134 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:50.906687  4148 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.775s	user 0.008s	sys 0.004s
I20260709 12:21:50.907533  4038 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:50.911917  4133 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.898108  4021 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.922111  3990 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 12:21:50.970312 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.031906 28330 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:50.954804  4199 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19.
I20260709 12:21:50.942615  3917 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:21:50.938781  4257 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:50.981417 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.004141 28336 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:21:51.013695  4183 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:50.950764  4206 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.612s	user 0.011s	sys 0.000s
I20260709 12:21:51.008241 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.901247 27587 rpcz_store.cc:276] 0709 12:21:49.653964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.654604 (+   640us) service_pool.cc:224] Handling call
0709 12:21:49.654913 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:49.655236 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.655270 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:49.667841 (+ 12571us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.198253 (+530412us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:50.198291 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.198357 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:50.872295 (+673938us) raft_consensus.cc:1685] finished
0709 12:21:50.872316 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.872476 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3421,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6411,"thread_start_us":465259,"threads_started":1,"wal-append.queue_time_us":902969}
I20260709 12:21:51.016456 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:50.946762 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:51.110661 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.112164 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:21:50.896811 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.004s	sys 0.016s
W20260709 12:21:51.126242 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 179426us
W20260709 12:21:51.126976 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.004s	sys 0.016s
I20260709 12:21:51.127841 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:51.004915 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:50.971836 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:51.108528 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.011s	sys 0.011s
I20260709 12:21:51.132359 27656 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:50.968647 27663 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:21:51.113952 27662 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:21:50.897056 27584 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.871s	user 0.005s	sys 0.018s
I20260709 12:21:51.133278 27663 pending_rounds.cc:85] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 15
I20260709 12:21:50.944512 27582 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:21:51.133929 27584 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.005s	sys 0.018s
I20260709 12:21:51.015057 27628 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:51.136938 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 121818us
I20260709 12:21:51.137198 27582 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:50.978150 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3774) took 1184 ms. Trace:
W20260709 12:21:50.982317 27737 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.008s	sys 0.013s
W20260709 12:21:51.118709 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:51.140139 27737 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.009s	sys 0.014s
I20260709 12:21:51.142858 27587 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:21:51.132591 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.887s	user 0.012s	sys 0.012s
W20260709 12:21:51.146663 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 900808us
I20260709 12:21:51.148100  3917 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:51.148200 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:51.148828  3917 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:51.149389  3917 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:50.984822  3980 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.024137  3984 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.151263  3980 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.152194  3980 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.152828  4030 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.034369  4021 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:51.153865  4030 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.036924 28336 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 13
W20260709 12:21:51.025532  3841 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.013s	sys 0.018s
I20260709 12:21:51.155730  3990 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:21:51.156260  3841 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.017s	sys 0.018s
W20260709 12:21:51.116766 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 216173us
I20260709 12:21:51.156870  3990 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:51.155791 28336 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:21:51.158927 28336 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.133932 27663 pending_rounds.cc:107] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 21.16
W20260709 12:21:50.981240  3961 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.000s	sys 0.020s
I20260709 12:21:51.159546 27663 raft_consensus.cc:2889] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.143457  3818 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.160096  3961 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.150s	user 0.000s	sys 0.020s
I20260709 12:21:51.161154  3980 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.161679 27657 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:51.139807 27731 rpcz_store.cc:276] 0709 12:21:49.793638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.793772 (+   134us) service_pool.cc:224] Handling call
0709 12:21:49.794078 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.794326 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:50.317075 (+522749us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.977668 (+660593us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.977748 (+    80us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:50.977810 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.977906 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.978075 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":901167,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":562,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":1174415,"threads_started":2,"wal-append.queue_time_us":718686}
I20260709 12:21:51.160902  3818 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.168603  3818 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.166013  4030 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.957926  4105 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:50.958945  4118 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:21:51.257126  4118 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:51.257889  4118 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:50.904661  4126 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.028472  4134 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:51.009123  4162 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:51.264199  4162 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.026121  4156 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.268435  4156 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:51.269743 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.047484  4183 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.273957  4183 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.031858  4197 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:51.048373  4206 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.278311  4206 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.052408  4257 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.280905 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:51.281158 27662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3129) took 1570 ms (client timeout 2047 ms). Trace:
W20260709 12:21:51.281729 27662 rpcz_store.cc:269] 0709 12:21:49.710960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.711352 (+   392us) service_pool.cc:224] Handling call
0709 12:21:51.280847 (+1569495us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:51.281046 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3222923008}
I20260709 12:21:51.070839  4265 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:51.283244 27586 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:51.283438  4198 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.283825 27586 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:51.284296  4198 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.018788  4267 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:51.285370 27586 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.286921 27586 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:51.287912 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4026) took 1258 ms. Trace:
I20260709 12:21:51.049042  4038 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:51.035956  4133 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.235472  4121 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.235996 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.237336  4199 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:51.155172  3984 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.244808 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.149425 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:51.252801  3818 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:51.307600  3818 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.308146  3818 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:21:51.274863 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 127065us
I20260709 12:21:51.288396 27585 rpcz_store.cc:276] 0709 12:21:50.029120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.216671 (+187551us) service_pool.cc:224] Handling call
0709 12:21:50.256788 (+ 40117us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.969745 (+712957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.279186 (+309441us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:51.287676 (+  8490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1597267712}
I20260709 12:21:51.301972  4121 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.306809 27587 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:51.314577  4283 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.340919  4158 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.341174  4105 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.341791  4158 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.342698  4287 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:51.341944  4105 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.320147  4198 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.351562 28329 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
W20260709 12:21:51.354359 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 216184us
I20260709 12:21:51.343490  4287 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.336266  4162 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.352730 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:51.348165  4158 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.317423  4283 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.346005  4183 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:51.349967  4156 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.385900 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:51.391666 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 235528us
W20260709 12:21:51.420698 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:51.426709  4003 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.004s	sys 0.025s
W20260709 12:21:51.436326 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.386602  4199 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.413640  4198 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.443079  4231 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:21:51.443745  4144 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.019s	sys 0.035s
W20260709 12:21:51.444069  3962 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.013s	sys 0.037s
W20260709 12:21:51.459717  3962 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.014s	sys 0.037s
W20260709 12:21:51.462798 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 469921us
I20260709 12:21:51.463702  4105 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.464080  4126 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.464933  4126 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.466895  4280 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:21:51.468343 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1299) took 1355 ms. Trace:
I20260709 12:21:51.468837 28336 rpcz_store.cc:276] 0709 12:21:50.112779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.112925 (+   146us) service_pool.cc:224] Handling call
0709 12:21:50.121715 (+  8790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.986854 (+865139us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b5400520920. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34ad87 00007f0e9a34b470 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:21:51.159811 (+172957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.467900 (+308089us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:51.468233 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400940672}
I20260709 12:21:51.469552  4280 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:21:51.443099 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.006s	sys 0.013s
I20260709 12:21:51.459313  4289 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:51.472546  4289 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.473068  4133 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:51.447459 27659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.003s	sys 0.015s
W20260709 12:21:51.473908 27659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.004s	sys 0.016s
I20260709 12:21:51.474486  4199 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.474943 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3145) took 1055 ms. Trace:
W20260709 12:21:51.448071 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 220218us
I20260709 12:21:51.444077  4141 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:51.443768 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3778) took 1265 ms. Trace:
I20260709 12:21:51.443011 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4014) took 1418 ms. Trace:
W20260709 12:21:51.453187 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.013s	sys 0.011s
I20260709 12:21:51.456327  4156 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:51.456758  4198 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.388334  4118 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:51.444319 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.031s	sys 0.025s
W20260709 12:21:51.437779  4003 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.004s	sys 0.026s
I20260709 12:21:51.462860 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:51.457408  4290 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:21:51.471707 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.006s	sys 0.013s
W20260709 12:21:51.488817 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.015s	sys 0.012s
W20260709 12:21:51.489595 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:51.494977 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.031s	sys 0.025s
I20260709 12:21:51.475354 27659 rpcz_store.cc:276] 0709 12:21:50.419448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.419591 (+   143us) service_pool.cc:224] Handling call
0709 12:21:50.434249 (+ 14658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.434462 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:21:50.920004 (+485542us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.474412 (+554408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.474537 (+   125us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:51.474628 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.474715 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.474864 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":469705,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":2810,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":1039159,"threads_started":2}
W20260709 12:21:51.490694  4144 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.019s	sys 0.035s
I20260709 12:21:51.475102  4289 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.432165  4183 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:51.571797 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:51.576510 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 154017us
W20260709 12:21:51.580955  4186 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.028s	sys 0.045s
W20260709 12:21:51.581533  4186 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.028s	sys 0.045s
I20260709 12:21:51.582286  4228 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.583302  4228 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.586679  4228 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.589239 27586 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260709 12:21:51.481350 27584 rpcz_store.cc:276] 0709 12:21:50.024849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.025000 (+   151us) service_pool.cc:224] Handling call
0709 12:21:50.025246 (+   246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.025420 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:51.134870 (+1109450us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.404747 (+269877us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.414430 (+  9683us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:51.428702 (+ 14272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.438173 (+  9471us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.442947 (+  4774us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2247,"prepare.queue_time_us":924553,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":144,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":21474,"thread_start_us":1361459,"threads_started":2}
W20260709 12:21:51.561764 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 699983us
I20260709 12:21:51.480873 27737 rpcz_store.cc:276] 0709 12:21:50.177958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.181396 (+  3438us) service_pool.cc:224] Handling call
0709 12:21:50.215025 (+ 33629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.215371 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:51.165293 (+949922us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.440775 (+275482us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.443156 (+  2381us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:51.443277 (+   121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.443407 (+   130us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.443630 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12245,"prepare.queue_time_us":480491,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":31,"raft.queue_time_us":20330,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":269,"thread_start_us":1139404,"threads_started":2,"wal-append.queue_time_us":305567}
I20260709 12:21:51.589901 27586 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 14
I20260709 12:21:51.599140 27586 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260709 12:21:51.599584 27586 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.608927  4197 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.610354  4020 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.665s	user 0.018s	sys 0.002s
I20260709 12:21:51.611354  4141 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:51.611718 28332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.001s	sys 0.022s
I20260709 12:21:51.611992  4141 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:51.619565 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:51.620090 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3135) took 1878 ms. Trace:
I20260709 12:21:51.609858  4197 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.588104  4126 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.620620 27657 rpcz_store.cc:276] 0709 12:21:49.742658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.743020 (+   362us) service_pool.cc:224] Handling call
0709 12:21:49.743392 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.162354 (+1418962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.619693 (+457339us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:51.620006 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3260186496}
I20260709 12:21:51.600890 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:51.626791 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76881.6us
W20260709 12:21:51.630831 27583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.012s	sys 0.052s
W20260709 12:21:51.631476 27583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.012s	sys 0.052s
I20260709 12:21:51.632483 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4027) took 1603 ms. Trace:
W20260709 12:21:51.633246 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.633626  4162 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:51.634227  4162 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:51.600174  4199 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.635537  4199 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.636266  4034 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:51.636463 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 20. New cstate: current_term: 20 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:51.638475  4199 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.644161  4126 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:51.643617 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 11 to 12, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 12 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:21:51.647462 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 317672us
I20260709 12:21:51.635771  4162 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.601305  4287 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.647245 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:21:51.650377  4281 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:21:51.651543 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1387) took 2003 ms (client timeout 1693 ms). Trace:
I20260709 12:21:51.651907  4021 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.632844 27583 rpcz_store.cc:276] 0709 12:21:50.029357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.289405 (+260048us) service_pool.cc:224] Handling call
0709 12:21:50.289732 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.289954 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:50.896412 (+606458us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.631943 (+735531us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.632039 (+    96us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:51.632139 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.632241 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.632430 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":660412,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"thread_start_us":1341206,"threads_started":2,"wal-append.queue_time_us":670255}
I20260709 12:21:51.653555  4183 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.573081  3984 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.706315  4003 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.706274  3984 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.707101  4003 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.708802  3984 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.655644  4257 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.659433 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81040.6us
I20260709 12:21:51.709820  4257 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.711253 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:51.712291  4257 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.712890  4257 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:21:51.714119 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.717316 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:51.667672 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:51.608116 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:51.574014  4121 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.677783  4283 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:51.667140 28335 rpcz_store.cc:269] 0709 12:21:49.598242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.601425 (+  3183us) service_pool.cc:224] Handling call
0709 12:21:51.587407 (+1985982us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400962654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:51.601481 (+ 14074us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3555968896}
I20260709 12:21:51.725389  4283 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:51.679067 27580 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:21:51.726017 27580 pending_rounds.cc:85] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:21:51.726373 27580 pending_rounds.cc:107] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:21:51.715704  4148 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 12:21:51.718648  3984 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:51.680043  4299 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.728925  4299 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.709381  4003 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.627193  3917 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.726140  4283 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.732635  3917 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.723126  4118 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:51.733894  4283 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.610904  4020 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.682226 27487 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:51.684180 27582 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:51.735714 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:51.736791 27582 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:21:51.685336 27655 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:51.740231 27487 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:51.743242 27662 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:21:51.685613  4021 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.686008 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1612) took 1051 ms. Trace:
I20260709 12:21:51.693549  4198 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.698179 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1611) took 1572 ms. Trace:
I20260709 12:21:51.698927  4303 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.703722 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:51.704088 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 239) took 85312 ms (client timeout 15000 ms). Trace:
W20260709 12:21:51.704493  4243 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.004s	sys 0.023s
I20260709 12:21:51.704572  4267 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.714838 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:51.648550 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:51.718127 27586 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:51.719818  4199 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.720319  4163 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:51.721364 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 157264us
I20260709 12:21:51.721976  4219 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:51.724135 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:51.726729 27580 raft_consensus.cc:2889] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.766760 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:51.747682 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81804.4us
I20260709 12:21:51.662559  4197 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.742367 27487 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 20 to 23, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:51.759796 27587 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:51.678742  4141 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:51.756361 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 291354us
I20260709 12:21:51.758463 27588 rpcz_store.cc:276] 0709 12:21:50.125847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.819701 (+693854us) service_pool.cc:224] Handling call
0709 12:21:50.820008 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.820349 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.820376 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.848539 (+ 28163us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.697837 (+849298us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:51.697905 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.697991 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.698029 (+    38us) raft_consensus.cc:1685] finished
0709 12:21:51.698040 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.698131 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":6142,"thread_start_us":671606,"threads_started":1,"wal-append.queue_time_us":689631}
I20260709 12:21:51.778062 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1610) took 1751 ms. Trace:
I20260709 12:21:51.746025 27589 rpcz_store.cc:276] 0709 12:21:50.634555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.105817 (+471262us) service_pool.cc:224] Handling call
0709 12:21:51.106408 (+   591us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:51.106996 (+   588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.107020 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.108337 (+  1317us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.429035 (+320698us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:21:51.429101 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.429208 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.685678 (+256470us) raft_consensus.cc:1685] finished
0709 12:21:51.685699 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.685837 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33878,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":320089,"threads_started":1,"wal-append.queue_time_us":570886}
W20260709 12:21:51.777755  4243 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.005s	sys 0.024s
I20260709 12:21:51.761063  4133 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:51.788472 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 64502.8us
I20260709 12:21:51.791505 27586 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:51.770440  3917 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.768774  4299 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.801213  4021 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:21:51.801358 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:51.802414 27588 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:51.803570  4121 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:51.804504 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.804785  4240 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:51.805962  4219 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:51.806217 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.806455  4219 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.806744 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:51.807842 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 93132.1us
I20260709 12:21:51.808054  4021 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:51.808557  4021 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.808763 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.809244 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.809598 28337 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:51.810765 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:51.811005 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 162623us
I20260709 12:21:51.812450  4199 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.806951  4219 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:51.808970  4021 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:51.804382 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:51.817512  4308 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:21:51.820322 27658 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:21:51.821456 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.822718 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.823745 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:51.824477 27656 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:51.824513  4140 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:51.825045  4267 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.766600  4303 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.828244  4303 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.828923 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.823563  4287 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:51.807885 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:51.836124 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.781879 27581 rpcz_store.cc:276] 0709 12:21:50.026761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.027429 (+   668us) service_pool.cc:224] Handling call
0709 12:21:50.027679 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.027855 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:21:50.847712 (+819857us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.588785 (+741073us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.607773 (+ 18988us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:21:51.709612 (+101839us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x7b18004739e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:51.727957 (+ 18345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.760297 (+ 32340us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.777987 (+ 17690us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":922334,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":484,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":38357,"spinlock_wait_cycles":129104128,"thread_start_us":1429066,"threads_started":2,"wal-append.queue_time_us":752413}
I20260709 12:21:51.838356 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:51.838814  4038 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:51.839218  4038 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.839371  4141 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:51.839666  4038 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:51.841215  4119 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.820s	user 0.007s	sys 0.014s
I20260709 12:21:51.841800  4119 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.823814 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.788340  3984 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:51.843858  4268 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:21:51.845050 27656 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:51.845544  3984 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:51.846004 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.847390 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.847934 27656 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:51.848380 27656 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19.
I20260709 12:21:51.848326  4240 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.849152  4240 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.839692 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:51.851970 27663 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:21:51.853250 27663 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:21:51.853638 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4043) took 1351 ms. Trace:
I20260709 12:21:51.854017 27580 rpcz_store.cc:276] 0709 12:21:50.502154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.860708 (+358554us) service_pool.cc:224] Handling call
0709 12:21:50.860991 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.791697 (+930706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.853349 (+ 61652us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:51.853581 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1879756032}
I20260709 12:21:51.854560 27663 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:21:51.855545 28331 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:51.857970 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:51.839394 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:51.837468 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:51.843914 27661 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:51.790829  4163 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.831792 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:51.860627 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:51.847098  4228 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.857551 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.897s (SENT) [suppressed 24 similar messages]
W20260709 12:21:51.889663 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81930.3us
I20260709 12:21:51.891695 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:51.894518 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57089.4us
I20260709 12:21:51.899531 27656 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:21:51.894388  4186 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.905148 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.880105  4148 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.865697  3917 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:51.866017  4133 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.921993 27487 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 14 to 17, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 17 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:51.900971  4240 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.904960  4126 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:51.914482  4162 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:51.916004  4231 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.906237  4197 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:51.940223 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.917773  4020 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:51.929571  3818 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.621s	user 0.011s	sys 0.004s
I20260709 12:21:51.927946  4243 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:51.945606 27487 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 19 to 22, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 22 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:21:51.953007 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:51.966734 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48162.2us
I20260709 12:21:51.970161  3988 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.100s
W20260709 12:21:51.953356  4134 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.538s	user 0.015s	sys 0.000s
I20260709 12:21:51.975519 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:51.914867  4283 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:21:51.944738  4162 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.923300  4228 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.909940  4186 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.954850  4197 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.954203  4126 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:52.006783  4126 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:52.010480 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:51.984961 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 192775us
W20260709 12:21:51.994100  4319 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.182s	user 0.002s	sys 0.000s
I20260709 12:21:51.994509  3552 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:21:51.996505 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.997360 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:51.998716 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.001585  4228 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.002283 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.005440  4186 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:51.981922 27657 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:51.972803  4243 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:51.986707  4162 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:52.011300  4197 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:51.967976  3984 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:21:51.954339  4140 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:21:52.048647  4140 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:52.049083  4140 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:52.050239  4140 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.055759 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:52.056460  4265 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.701s	user 0.007s	sys 0.007s
I20260709 12:21:52.056542  4330 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:52.057171  4330 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:52.058758 27660 raft_consensus.cc:2410] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:21:52.028918 27658 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:51.975831  4231 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.060748 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1586) took 1098 ms. Trace:
I20260709 12:21:52.061184 27660 rpcz_store.cc:276] 0709 12:21:50.961937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.963416 (+  1479us) service_pool.cc:224] Handling call
0709 12:21:52.060431 (+1097015us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:52.060679 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2515227776}
I20260709 12:21:52.061743  4231 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.061968  4038 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.065045  4330 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:52.066452  4325 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.003s
I20260709 12:21:52.069087  4141 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:52.069515  4141 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:52.070557  4141 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.028782 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.011047  4119 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.034536 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:51.976115  3818 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:51.967046  4148 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.118669 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:21:51.760532 27482 rpcz_store.cc:269] 0709 12:20:26.351677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:26.352104 (+   427us) service_pool.cc:224] Handling call
0709 12:21:51.648113 (+85296009us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:41.895115 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:41.895870 (+   755us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:41.895919 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:21:41.895938 (+    19us) tablet.cc:662] Decoding operations
0709 12:21:41.896738 (+   800us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:41.896982 (+   244us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:41.897021 (+    39us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:21:41.897429 (+   408us) tablet.cc:701] Row locks acquired
0709 12:21:41.897471 (+    42us) write_op.cc:260] PREPARE: finished
0709 12:21:41.897649 (+   178us) write_op.cc:270] Start()
0709 12:21:41.897799 (+   150us) write_op.cc:276] Timestamp: P: 1783599701897577 usec, L: 0
0709 12:21:41.897846 (+    47us) op_driver.cc:348] REPLICATION: starting
0709 12:21:41.898526 (+   680us) log.cc:844] Serialized 1308 byte log entry
0709 12:21:42.375297 (+476771us) op_driver.cc:464] REPLICATION: finished
0709 12:21:42.592024 (+216727us) write_op.cc:301] APPLY: starting
0709 12:21:42.604932 (+ 12908us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:42.949787 (+344855us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:42.949916 (+   129us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:42.952739 (+  2823us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:42.953026 (+   287us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=12,mrs_lookups=0
0709 12:21:42.953085 (+    59us) write_op.cc:312] APPLY: finished
0709 12:21:42.953434 (+   349us) log.cc:844] Serialized 41 byte log entry
0709 12:21:43.008509 (+ 55075us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:43.012160 (+  3651us) write_op.cc:454] Released schema lock
0709 12:21:43.030587 (+ 18427us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:43.097077 (+ 66490us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:43.097120 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6815298,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":167938,"apply.run_cpu_time_us":12405,"apply.run_wall_time_us":491660,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16667,"cfile_init":1,"lbm_read_time_us":523,"lbm_reads_lt_1ms":4,"num_ops":3,"prepare.queue_time_us":304819,"prepare.run_cpu_time_us":10300,"prepare.run_wall_time_us":34657,"raft.queue_time_us":153300,"raft.run_cpu_time_us":16112,"raft.run_wall_time_us":154813,"replication_time_us":453531,"thread_start_us":375819,"threads_started":4,"wal-append.queue_time_us":72139,"wal-append.run_cpu_time_us":25793,"wal-append.run_wall_time_us":2006223}]]}
I20260709 12:21:52.043747 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.044427 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:52.044859 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.046124 27732 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:52.047710 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.066501  4330 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:52.073385 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.025317  4198 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:52.022830  4121 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.144574  4121 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.145253  4141 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:52.080292 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.023578  3552 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:52.147830 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:52.148233  3552 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:52.148424 28330 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:51.983156  4134 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.110523 27660 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:21:52.151615 27731 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:52.152942  4134 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:52.153872 28328 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:52.153985  4299 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:52.154533  3988 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:21:52.155409  4321 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:21:52.115090 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.156737 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.157038 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1406) took 1149 ms. Trace:
I20260709 12:21:52.157523 27580 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:52.157416 28328 rpcz_store.cc:276] 0709 12:21:51.008076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.008499 (+   423us) service_pool.cc:224] Handling call
0709 12:21:51.010894 (+  2395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:52.154320 (+1143426us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.156777 (+  2457us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:52.156981 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2627749632}
I20260709 12:21:52.158458 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.159142 27580 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:51.986728  4199 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:52.127588 27737 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:52.129102  3818 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.129405  4148 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.129614  4163 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:52.135047 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.135419  4140 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:52.135560  4133 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.135919 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.136214  4243 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.136691 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.139508 27656 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:21:52.140779  4219 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.143533 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:52.154882  4299 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.122011 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.159777 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.111519 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:52.161670 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 126660us
W20260709 12:21:52.165989 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 370750us
I20260709 12:21:52.170605 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:52.178990 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:52.190037 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.180953 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.191000 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:52.191334 27657 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:21:52.184307 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:52.202956 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.195796 27580 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:52.203024 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:52.208253  4310 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:21:52.208611  4162 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.209386  4219 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:52.209426 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.209933  4126 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:21:52.210165 27737 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:52.210889  4162 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:52.211524 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84925.9us
I20260709 12:21:52.213294  4267 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:52.213889  4267 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:52.214807  4267 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.215703  4121 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.180176  4199 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.216990 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.218672 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.219375 27586 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
W20260709 12:21:52.219672 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 108267us
I20260709 12:21:52.200155  4243 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:52.224594 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.203843 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.228094 27659 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:52.228696 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.228904 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.229525 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:52.230026 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.230377 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.230581 27586 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:52.231077 27582 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:52.232599  4197 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.233063  4198 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.235741  4316 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:52.238097 27737 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:52.228012  4038 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.244604 28330 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:52.243793 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:52.251853 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4035) took 1106 ms. Trace:
I20260709 12:21:52.252276 27583 rpcz_store.cc:276] 0709 12:21:51.145410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.743569 (+598159us) service_pool.cc:224] Handling call
0709 12:21:51.764036 (+ 20467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:51.794803 (+ 30767us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.794827 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.863219 (+ 68392us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.976408 (+113189us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:51.976470 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.976581 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:52.251637 (+275056us) raft_consensus.cc:1685] finished
0709 12:21:52.251660 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.251796 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17048,"raft.queue_time_us":4414,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":10864,"thread_start_us":56837,"threads_started":1,"wal-append.queue_time_us":224238}
I20260709 12:21:52.259569 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:52.269400 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.271641 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:52.274178 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55206.2us
I20260709 12:21:52.267586  4162 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:52.286578 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:52.291077 27586 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:52.267568  4121 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.284266 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.297703 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:52.265815  3917 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.302153 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.276712  4198 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.300206  4162 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.283792  3818 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:21:52.302173  4118 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.568s	user 0.007s	sys 0.006s
I20260709 12:21:52.281284  4038 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:52.315981 27582 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:21:52.302835  4265 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.272078  4199 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.314766  4290 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.351208  4290 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.351608 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.340513  4133 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:52.353144  4133 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.353444  4290 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.355518  4265 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:52.358289  4332 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:21:52.358425  4265 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.317713  4162 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:52.359114  4038 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:52.360328  4328 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:21:52.361563  4038 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.365427  4199 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.367255 27662 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:21:52.367547 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:52.368031 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81704us
I20260709 12:21:52.368309 27735 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27.
I20260709 12:21:52.370862 27664 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
W20260709 12:21:52.373155 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.329255  4299 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.269982  4183 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:21:52.319798 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.319551 28329 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:21:52.339676 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:52.377434 28332 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:52.347218 27659 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:21:52.348026 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.348376 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.349769  4198 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.317960  3917 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:52.391626 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:52.391934  3917 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:52.392201 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.392939 27732 raft_consensus.cc:2368] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25.
I20260709 12:21:52.393021  3917 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.393842 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.394259  4299 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:52.394424 27731 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:52.394712  4299 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.315824  3841 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:52.395186  4299 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:52.395509  3841 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:52.377521 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.396241 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.384326 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:52.371215  4021 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.508s	user 0.007s	sys 0.003s
W20260709 12:21:52.388497  4338 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:21:52.388909  4198 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.389739 27580 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:21:52.372541  3818 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:52.390913 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:52.391023  4183 consensus_queue.cc:1243] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 25.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:21:52.412741 28329 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:52.404132 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:52.397765 27584 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:52.403532  3841 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:52.347249  4197 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.390441 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.323493  4121 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.384387  4133 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:52.412716 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6662438870000642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
W20260709 12:21:52.424603 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2162438870000187s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20470000000000002s for a response
I20260709 12:21:52.415758 27664 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:52.410730  4021 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.422186 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.417165 27737 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:52.427804 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.433132 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.433167 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.439014 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.439577 27735 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:52.440728 27661 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:52.422258  4105 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.444816 27655 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:52.446458 27731 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:52.450485 27584 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:52.436458  4183 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.457835 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.457857 27657 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:21:52.460443 27662 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:52.457801 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:52.462824 27737 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:52.436571  3988 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.467780 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:52.464890 27552 leader_election.cc:400] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24.
I20260709 12:21:52.466553 27733 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:52.468456 27552 leader_election.cc:403] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:52.469678 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.433851  4199 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:21:52.486967  4330 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:21:52.488667  4346 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:21:52.491722 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.494354  3552 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.495605 28328 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 16. Current term is 17. Ops: []
I20260709 12:21:52.496908 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.498497 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.491748 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.472277  4105 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:52.505853 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.475383 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.474833 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.512499 27584 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:52.476760 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.502372 27657 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:52.501040 27656 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:21:52.499158 27583 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:52.511679 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:52.516599 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.513782 27737 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:21:52.508275  4343 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:21:52.519474 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.519639 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.514204  4038 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24.
I20260709 12:21:52.486373 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.518296 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.524974 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:21:52.496829  3552 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:52.548910 28332 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:52.527143 27738 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
W20260709 12:21:52.525789 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53624.4us
I20260709 12:21:52.546617 27589 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:52.528404 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:52.530287 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.529332 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.532279 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.536979 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.554101 28329 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
W20260709 12:21:52.537261 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98283.7us
I20260709 12:21:52.537856 27585 pending_rounds.cc:85] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:52.555116 27585 pending_rounds.cc:107] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:21:52.554155 27733 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:52.555621 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:52.466761  4183 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.556537 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:52.556643 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.557933 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.558028 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:52.558867  4183 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:52.543570 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126008us
I20260709 12:21:52.560032  4186 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:52.560647  4186 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:52.561668  4219 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:52.561599  4186 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.550118 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.503358  4267 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:52.538899  4021 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.545140 27582 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:21:52.555658 27585 raft_consensus.cc:2889] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:52.566884 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.569550 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.563027 27661 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:21:52.571159 27730 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:52.573469 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.581609 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.500960  3988 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.574466 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.588959 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:52.581943 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.553438  4280 raft_consensus.cc:2764] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:21:52.605643 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.606225 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:52.594300 27729 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:52.604628 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.570761  4267 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.572583  4038 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:21:52.622418 27701 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 18. Current term is 19.
I20260709 12:21:52.609493 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:52.610435 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50719.9us
I20260709 12:21:52.610536 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.624164 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:52.612257  4105 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.612852 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:52.616084  4118 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:21:52.637182 27584 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:52.599922 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.641846 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.640694 27734 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:21:52.644152 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:52.524114  3818 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.646380  4038 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:52.647658 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.647470  4038 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.648209 27730 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:52.648875 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:52.649937 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:52.650946  4105 raft_consensus.cc:2764] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
W20260709 12:21:52.651081 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46923.2us
I20260709 12:21:52.652052  4121 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:52.652439 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.652666  4121 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:52.617997  4243 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:52.656080  4206 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:21:52.659076 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.659482  3984 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:52.659777  4219 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:52.661046 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.661603 27736 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:21:52.662129  4348 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 12:21:52.662750 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:52.644680  4118 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:52.652837 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.655234 27734 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:21:52.677039 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:52.622241  4267 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:52.680065 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.683547 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.685585 27660 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:52.686913 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:52.687825 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69382.6us
I20260709 12:21:52.678570  4021 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:52.655227  4243 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.684573 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.694607 27734 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:21:52.695168 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:52.696425 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:52.696535 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.711583 27730 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:21:52.713349 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.713912 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.715047 28329 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:21:52.716903 27731 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:21:52.659134  4206 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:21:52.723904 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:52.725400 27734 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:52.697606 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.725718 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:52.686327  4219 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:52.686646  3984 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:21:52.686376  3988 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.730645 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.689805  4119 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:21:52.732735 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:52.729705 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.735183 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 19.
W20260709 12:21:52.739037 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.739773  4183 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:52.742645  4198 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:52.742846 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:52.725086  4257 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:21:52.746745  4257 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:52.747167 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:52.748093  4352 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:52.748421  4265 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:52.748608  4352 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.748951  4265 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:52.749177  4352 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:52.750069  4265 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.743611 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:52.752652 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:52.744095 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.753786 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.744498 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:52.754812 27655 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:52.755178 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.745281 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:52.758337 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109548us
I20260709 12:21:52.774400 28332 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:52.754227 27662 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:52.774642 27738 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:21:52.760560 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.778838 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:52.781352 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85060.6us
I20260709 12:21:52.787844 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.788748 27660 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:52.798746 27580 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:52.798933 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:52.694334  4118 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.720813  3917 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:21:52.757270  4316 raft_consensus.cc:2764] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:21:52.757664  4134 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:52.808849 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:52.811651 28329 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:52.737955  3984 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:52.813853 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56121.6us
I20260709 12:21:52.732918  4303 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:21:52.799727 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:52.822460 27656 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:52.803917  4333 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:52.810135  4228 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:52.837847 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:52.831947  4333 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.825726  4265 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:52.826268  4283 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:52.849592 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:52.834995  4257 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:52.820327  4299 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.847066  4333 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:52.873818  3984 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.877322  4021 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:21:52.879750  4162 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.000s
W20260709 12:21:52.880103  4140 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.000s
I20260709 12:21:52.882222  4183 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:52.882306 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:52.883157 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57175.2us
W20260709 12:21:52.883836 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.889904 27733 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:21:52.896215 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 113914us
I20260709 12:21:52.897063 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:52.890455 27734 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:52.850835  4133 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:52.851002  4283 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:52.843135  4134 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:21:52.855685  4118 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:52.898698 27732 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:52.846225  4228 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.913822 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:52.894131  3841 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:52.961899 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:52.948050 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45570.3us
W20260709 12:21:52.926198 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 172610us
I20260709 12:21:52.982510 27585 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:21:52.953374 27655 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
W20260709 12:21:52.949874 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:52.994861 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81131.1us
W20260709 12:21:52.949757  4337 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.558s	user 0.004s	sys 0.000s
I20260709 12:21:52.911693  4283 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:52.898865  4358 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:53.011996 28337 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
W20260709 12:21:52.945853  4355 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:52.917078  4290 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:52.943562  4183 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:53.024361  4183 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.025034  4183 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:53.025648  3841 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:52.894605  4162 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:52.961344  4243 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:53.012734 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115769us
I20260709 12:21:53.036449 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.003074  4163 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:52.919628  4267 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.033241 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:52.918768  4330 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.095s
I20260709 12:21:53.053467 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.057255 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:52.965941  4003 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
W20260709 12:21:53.060220 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 235727us
I20260709 12:21:53.062305 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:53.048470  4197 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:53.067641 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:53.069154 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.051857  3552 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 12:21:53.072724 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.020833  4352 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.078378 27552 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27.
I20260709 12:21:53.077001 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:53.078855 27552 leader_election.cc:403] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:53.067905  4197 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:53.078073  4351 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:53.081895  4351 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:53.082226  4228 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:53.082080  4197 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.083256  4197 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:53.065564  4257 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:53.074241  4290 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:53.076392  3917 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:53.095517 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.095924 27552 leader_election.cc:400] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25.
W20260709 12:21:53.095108 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93328.4us
I20260709 12:21:53.148831 27552 leader_election.cc:403] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:21:53.149112 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:53.149937 28329 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:53.100404 27586 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:21:53.150930 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.108505  4196 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:53.119231  4299 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:53.095834  4290 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.125708  4352 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.082329  4351 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:53.242276 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91496.1us
W20260709 12:21:53.231340 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.243016  3818 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25.
I20260709 12:21:53.095335  4228 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27.
I20260709 12:21:53.248262 27586 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
W20260709 12:21:53.255045 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 178279us
I20260709 12:21:53.251461 27586 pending_rounds.cc:85] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:53.258361 27586 pending_rounds.cc:107] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:21:53.258716 27586 raft_consensus.cc:2889] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:53.231666  3841 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:21:53.261401 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.257308 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1618) took 1056 ms. Trace:
I20260709 12:21:53.231979  4290 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:21:53.263720 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.289925 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.263777 27587 rpcz_store.cc:276] 0709 12:21:52.201137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.963838 (+762701us) service_pool.cc:224] Handling call
0709 12:21:52.964267 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:52.964608 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:21:53.007575 (+ 42967us) log.cc:844] Serialized 14 byte log entry
0709 12:21:53.256851 (+249276us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.256919 (+    68us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:21:53.256987 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.257078 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.257238 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":173550,"prepare.queue_time_us":97488,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":2496,"spinlock_wait_cycles":1799552,"thread_start_us":284053,"threads_started":2}
W20260709 12:21:53.306052 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 311687us
I20260709 12:21:53.308195 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:53.247881  4351 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:53.300350 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1621) took 1055 ms. Trace:
W20260709 12:21:53.313476 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 219690us
I20260709 12:21:53.297075 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:53.320578 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:53.321477 27586 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:53.312578 27585 rpcz_store.cc:276] 0709 12:21:52.244618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.002494 (+757876us) service_pool.cc:224] Handling call
0709 12:21:53.014368 (+ 11874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.014620 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:21:53.094463 (+ 79843us) log.cc:844] Serialized 14 byte log entry
0709 12:21:53.299735 (+205272us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.299814 (+    79us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:21:53.299988 (+   174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.300089 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.300255 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1273,"prepare.queue_time_us":279985,"raft.queue_time_us":135,"thread_start_us":262756,"threads_started":2}
I20260709 12:21:53.323652 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.325582 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:53.287299  4163 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:53.254577  4243 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.063s
I20260709 12:21:53.311422  3917 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:53.338970 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:53.351824 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 273468us
I20260709 12:21:53.358598 28329 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:21:53.323657  4105 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:53.335117  4257 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:53.372049  4105 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.374351  4316 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:53.374572  4105 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:53.374872  4316 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:53.372467  4257 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.344897  4198 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:53.371062  4163 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.371102 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:53.363077 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:53.359812  3818 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:21:53.352675  4381 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:21:53.398257  4333 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.503s	user 0.008s	sys 0.000s
I20260709 12:21:53.369055  3917 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:21:53.390858  3818 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:53.404541  3818 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.405987  3818 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:53.403460  4198 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:53.407641  4198 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:53.408105  4198 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:21:53.409098  4367 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.315s	user 0.002s	sys 0.000s
W20260709 12:21:53.415220 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61891.4us
I20260709 12:21:53.417621 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.419924 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:53.428644  4384 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:53.434455  3841 raft_consensus.cc:2764] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260709 12:21:53.435344  4186 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:53.435830  4186 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.436230  4186 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:53.439666  4283 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.441071 27589 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:53.442386  4283 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:53.398787  4333 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.444692 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4056) took 1162 ms. Trace:
I20260709 12:21:53.445174 27589 rpcz_store.cc:276] 0709 12:21:52.282709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.065693 (+782984us) service_pool.cc:224] Handling call
0709 12:21:53.071288 (+  5595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.441579 (+370291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.444386 (+  2807us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54000bd1d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:53.444620 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":849804032}
I20260709 12:21:53.444869  4333 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.381850  4257 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:21:53.449522 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111818us
W20260709 12:21:53.399307  4370 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:53.457823 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.456357 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:53.461387 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:53.448575  4055 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:53.467005 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3156) took 1006 ms. Trace:
W20260709 12:21:53.457628  4388 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:21:53.455125  4283 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:53.467041  4126 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:53.469478 27661 rpcz_store.cc:276] 0709 12:21:52.461211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.461489 (+   278us) service_pool.cc:224] Handling call
0709 12:21:53.466829 (+1005340us) spinlock_profiling.cc:243] Waited 92.7 ms on lock 0x7b5400344754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:53.466944 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213168896}
I20260709 12:21:53.485544 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.486418 27732 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:53.463806  4105 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
W20260709 12:21:53.493709 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 122769us
I20260709 12:21:53.494583 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:53.497148  3552 raft_consensus.cc:2764] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20260709 12:21:53.502688  4186 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.504405 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.505473 27660 raft_consensus.cc:2410] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:21:53.508752 27658 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:21:53.511201 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.513602 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.467099  3917 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:21:53.524737 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4078) took 1345 ms. Trace:
I20260709 12:21:53.527900 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.529333 27663 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:53.530665 27661 raft_consensus.cc:2410] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:21:53.491796  4126 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:53.487638  4333 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:21:53.525359 27582 rpcz_store.cc:276] 0709 12:21:52.179186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.872242 (+693056us) service_pool.cc:224] Handling call
0709 12:21:52.872626 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.873015 (+   389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.873048 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:52.873948 (+   900us) log.cc:844] Serialized 26 byte log entry
0709 12:21:53.093754 (+219806us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:53.093796 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.093861 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.524417 (+430556us) raft_consensus.cc:1685] finished
0709 12:21:53.524450 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.524655 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":528,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":97,"thread_start_us":213211,"threads_started":1,"wal-append.queue_time_us":373587}
W20260709 12:21:53.574103 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111867us
I20260709 12:21:53.575232 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:53.563961  4183 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.538s	user 0.012s	sys 0.002s
I20260709 12:21:53.585583 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.586560 27663 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:21:53.492640  4316 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:53.588167 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.597539  3818 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:53.605365  4265 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.754s	user 0.008s	sys 0.005s
I20260709 12:21:53.584442  4183 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.618835 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4057) took 1570 ms. Trace:
I20260709 12:21:53.622252  4400 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:53.624539 27588 rpcz_store.cc:276] 0709 12:21:52.048296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.370267 (+321971us) service_pool.cc:224] Handling call
0709 12:21:52.370535 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:52.902739 (+532204us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:21:52.902775 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:52.902818 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:52.902854 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.618458 (+715604us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54002bacd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:53.618527 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.618732 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2401,"raft.queue_time_us":2839,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":4922,"spinlock_wait_cycles":1217390464}
I20260709 12:21:53.583266  4351 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.651790 27656 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:21:53.627410  4400 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:53.663539 27588 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:53.642231  4316 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:53.624562  3818 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.619748  4126 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.653137  4403 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:21:53.711736  4403 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:53.712210  4403 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:53.701200  4290 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.702513  4400 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:53.707526 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86851.2us
W20260709 12:21:53.710247 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 135133us
I20260709 12:21:53.678143 27659 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:53.713651 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.717489 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.718750 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.720278 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.721048 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:53.722478 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.725097 27656 raft_consensus.cc:2368] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:21:53.726052 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.728797  3818 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:53.730121  4198 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.714741  4402 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.731845  4402 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.732307 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.733037 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:21:53.733572  4267 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.734440  4402 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:53.736608  4351 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.737317 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.737955  4333 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:53.738545  4333 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.740878  4333 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.743117 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4058) took 1694 ms. Trace:
I20260709 12:21:53.743660 27581 rpcz_store.cc:276] 0709 12:21:52.048613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.529519 (+480906us) service_pool.cc:224] Handling call
0709 12:21:53.031413 (+501894us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:53.049309 (+ 17896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.049346 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:53.070826 (+ 21480us) log.cc:844] Serialized 26 byte log entry
0709 12:21:53.322166 (+251340us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:53.322215 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.322298 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.498405 (+176107us) raft_consensus.cc:1685] finished
0709 12:21:53.498431 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.617662 (+119231us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:21:53.742939 (+125277us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b540011b550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:53.743058 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9291,"raft.queue_time_us":9464,"spinlock_wait_cycles":1160385536,"thread_start_us":250952,"threads_started":1,"wal-append.queue_time_us":310291}
I20260709 12:21:53.746085 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:53.746695 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.746852 27581 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:53.747728 27658 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:53.743115 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:53.748940  4373 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.449s	user 0.008s	sys 0.000s
I20260709 12:21:53.750300 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4074) took 1618 ms. Trace:
W20260709 12:21:53.751484  4353 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.019s	user 0.002s	sys 0.002s
I20260709 12:21:53.750712 27580 rpcz_store.cc:276] 0709 12:21:52.132101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.802083 (+669982us) service_pool.cc:224] Handling call
0709 12:21:52.802588 (+   505us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.826178 (+ 23590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.826227 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:52.844277 (+ 18050us) log.cc:844] Serialized 26 byte log entry
0709 12:21:53.063493 (+219216us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:53.314958 (+251465us) log.cc:844] Serialized 14 byte log entry
0709 12:21:53.315189 (+   231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.315259 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.748313 (+433054us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:53.749980 (+  1667us) raft_consensus.cc:1685] finished
0709 12:21:53.750098 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.750239 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11844,"prepare.queue_time_us":241427,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":28582,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":6995,"spinlock_wait_cycles":34035584,"thread_start_us":336790,"threads_started":2,"wal-append.queue_time_us":333075}
I20260709 12:21:53.753369  4400 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:53.753539 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:53.754266 27580 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:53.748843 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.757552 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.758843 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.755661 27662 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:53.714133 27663 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 17.
I20260709 12:21:53.764037  4186 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.715456  4404 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:53.767449 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.768661 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.766510 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:53.772321 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.773397 27663 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:53.763533  4290 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:53.778139 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:53.781512 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.782071 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.782335 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:53.797102 27658 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:53.798347 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.785441 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:53.804572 27730 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:21:53.805655 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.806010 27732 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:53.807202 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.799732 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:53.810076 27729 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:53.769311  3917 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.811339 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:53.811586 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.812868 27734 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:53.813910 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
I20260709 12:21:53.815057 27734 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:21:53.815575 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:53.815678 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.799448  4267 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:53.816269 27738 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:53.818591 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.819509 27658 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:21:53.806563 27736 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:53.801195  4055 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:53.828526  4055 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.828974  4055 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:53.829221  4399 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:53.779733  4206 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.832711 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:53.832628 27732 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:21:53.846078 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:53.886744 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:53.817267  4267 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.818078  4408 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.817678  4352 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:53.895524 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:53.875005 27730 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
W20260709 12:21:53.906646 28292 connection.cc:256] RPC call timeout handler was delayed by -2.0349354080001376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260709 12:21:53.907102 28292 connection.cc:256] RPC call timeout handler was delayed by -1.9656354080000256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20260709 12:21:53.771301  4404 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:53.908918 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103282us
W20260709 12:21:53.907573 28292 connection.cc:256] RPC call timeout handler was delayed by -1.8198354080000172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
I20260709 12:21:53.908447  4243 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:53.911281 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4086) took 1345 ms. Trace:
I20260709 12:21:53.914173 27580 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:21:53.831588  4118 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.073s
I20260709 12:21:53.886348  4351 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:53.918049 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:53.902599  4352 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:53.921128 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.916661 27585 rpcz_store.cc:276] 0709 12:21:52.565633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.323123 (+757490us) service_pool.cc:224] Handling call
0709 12:21:53.345147 (+ 22024us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.345430 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:53.759749 (+414319us) log.cc:844] Serialized 14 byte log entry
0709 12:21:53.908132 (+148383us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.908191 (+    59us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:53.908240 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.911093 (+  2853us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.911224 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":413923,"prepare.queue_time_us":414642,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"thread_start_us":147773,"threads_started":1}
I20260709 12:21:53.921351 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.837745  4399 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:21:53.932081 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:53.853505  4219 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.062s
I20260709 12:21:53.885344  4257 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.841877  3917 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.952404 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:53.954519 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4089) took 1067 ms. Trace:
I20260709 12:21:53.940325 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:53.916931  4411 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:53.954998 27585 rpcz_store.cc:276] 0709 12:21:52.887514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.953601 (+1066087us) service_pool.cc:224] Handling call
0709 12:21:53.953924 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.954108 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.954136 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.954176 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:53.954207 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.954263 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.954462 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:53.968458 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.968403 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:53.942476  4352 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:21:53.984145 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.951074  4148 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:54.004999 27663 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:21:54.008724 27657 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:53.903680  4206 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:53.951521  4421 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:21:53.942273  4351 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:53.989303  4402 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:21:53.957157  4316 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:21:54.006875  4399 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:21:54.006124  4148 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:54.021446  3818 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:54.126469 27736 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
W20260709 12:21:54.124130  4412 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.204s	user 0.005s	sys 0.004s
I20260709 12:21:54.176960  4148 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:54.178792  4148 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:54.180474  4393 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.171s
I20260709 12:21:54.210557 27581 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:54.162344  3818 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.222102 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4088) took 1335 ms. Trace:
W20260709 12:21:54.214804  4403 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.502s	user 0.007s	sys 0.007s
I20260709 12:21:54.222535 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4091) took 1070 ms. Trace:
I20260709 12:21:54.228154 27581 rpcz_store.cc:276] 0709 12:21:53.151813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.217939 (+1066126us) service_pool.cc:224] Handling call
0709 12:21:54.218274 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.222003 (+  3729us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.222032 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.222078 (+    46us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:54.222112 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.222173 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.222288 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:54.222782 27588 rpcz_store.cc:276] 0709 12:21:52.886394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.704080 (+817686us) service_pool.cc:224] Handling call
0709 12:21:53.714700 (+ 10620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.714927 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:53.774475 (+ 59548us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.221799 (+447324us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.221849 (+    50us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:54.221888 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.221948 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.222051 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59213,"prepare.queue_time_us":64652,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":444880,"threads_started":1}
W20260709 12:21:54.245147 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 323176us
W20260709 12:21:54.268934 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 347623us
I20260709 12:21:54.227270  3818 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:54.286746  4404 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:54.299150 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.302845 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.303373 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.308459 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.310941  4399 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:21:54.311633  4399 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:54.277900  4267 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.316381 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:54.317466  4206 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:54.317850 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:54.318238 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:54.318624  4428 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:54.318677 27659 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:54.327862 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4090) took 1331 ms. Trace:
W20260709 12:21:54.365541 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72754.7us
I20260709 12:21:54.373109 27656 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:21:54.373828 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.289835  4257 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.378286 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.386806 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.387482 28328 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:54.377606 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4085) took 1812 ms. Trace:
I20260709 12:21:54.388902 27587 rpcz_store.cc:276] 0709 12:21:52.565279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.310916 (+745637us) service_pool.cc:224] Handling call
0709 12:21:53.323205 (+ 12289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.323443 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:53.759570 (+436127us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.330362 (+570792us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.371968 (+ 41606us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:54.376861 (+  4893us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.376951 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.377547 (+   596us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":533507,"prepare.queue_time_us":436410,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":120432,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":101990,"thread_start_us":568703,"threads_started":2,"wal-append.queue_time_us":523304}
I20260709 12:21:53.986435  4411 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.398897 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.399118 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:54.432519 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.432762  3917 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:54.433554 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:54.434250 28328 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:21:54.434475  4414 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:54.431293 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1623) took 1809 ms. Trace:
I20260709 12:21:54.435230  4183 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:54.436347  4426 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:21:54.436623  3818 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.319259  4428 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:54.439206  4428 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.439275  4351 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:53.905335  4408 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:54.440282 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.441756  4408 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.235507  4403 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.435632 27582 rpcz_store.cc:276] 0709 12:21:52.622346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.558177 (+935831us) service_pool.cc:224] Handling call
0709 12:21:53.558654 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.558982 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:53.774658 (+215676us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.410287 (+635629us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.430985 (+ 20698us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:54.431038 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.431104 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.431235 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":222792,"prepare.queue_time_us":236663,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":68945,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":61,"thread_start_us":627379,"threads_started":2}
I20260709 12:21:54.443274 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.444132 27582 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:21:54.444331  4403 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:54.431962  4038 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:54.445068 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:54.445698  4419 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
W20260709 12:21:54.447580  4431 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
I20260709 12:21:54.447712  4038 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:54.447950  4162 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.448370  4038 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:54.450806  4038 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:54.445756 27582 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:54.451838 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.453176 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.453329  4316 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:21:54.454550 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55747.7us
W20260709 12:21:54.456076  4425 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:21:54.457278 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:54.457647  4420 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:21:54.461158 27587 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:21:54.458159 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:54.461314 27660 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:21:54.464564 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1255) took 1917 ms. Trace:
I20260709 12:21:54.464952 28337 rpcz_store.cc:276] 0709 12:21:52.547487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.547622 (+   135us) service_pool.cc:224] Handling call
0709 12:21:52.549317 (+  1695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.019189 (+469872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.464362 (+1445173us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:54.464509 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1063549824}
I20260709 12:21:54.466620 27582 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:54.377264  4421 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.365375  4404 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:54.469370 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:54.472128 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.491570 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.441773 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:54.434540 27662 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:54.489573 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.495023 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.497510 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.433645 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:54.489413 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4063) took 1918 ms. Trace:
I20260709 12:21:54.500260 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:21:54.402832  4415 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 12:21:54.499130 27737 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:54.500209 27586 rpcz_store.cc:276] 0709 12:21:52.570997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.343962 (+772965us) service_pool.cc:224] Handling call
0709 12:21:53.364049 (+ 20087us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.364296 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:21:53.762800 (+398504us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.489079 (+726279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.489133 (+    54us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:54.489176 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.489238 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.489357 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":402732,"prepare.queue_time_us":399311,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":46,"raft.queue_time_us":66060,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":65,"thread_start_us":721214,"threads_started":2,"wal-append.queue_time_us":616356}
I20260709 12:21:54.502492 27730 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:54.503037 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.503837 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.507980 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.508599 28328 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:54.504215 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.504472 28335 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:54.503885 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.375218  4402 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:54.505441 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.509871 28335 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:54.513780 28337 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:54.523870 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.524380 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4093) took 1354 ms. Trace:
W20260709 12:21:54.519831 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 209293us
I20260709 12:21:54.537643 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:54.538569 27729 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:54.312682  4399 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.507453  4283 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:54.545780 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.506062  4403 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:54.539569 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:54.506188  4408 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:54.567916 27736 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:54.513342  4404 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:54.544740 28335 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.598233 27586 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:21:54.592991 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 319787us
W20260709 12:21:54.621742 27589 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.012s	sys 0.017s
W20260709 12:21:54.622107 27589 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.012s	sys 0.017s
I20260709 12:21:54.540671  3552 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:54.616079 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.571400 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:54.619230 27731 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:21:54.622785 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4065) took 2034 ms. Trace:
I20260709 12:21:54.628031 28337 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:21:54.567263  4038 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:54.633400 28335 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:54.568778  4403 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:54.629279 27738 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.003s	sys 0.029s
W20260709 12:21:54.640417 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69115.6us
I20260709 12:21:54.506407  4421 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.590121  4243 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:21:54.632905 27589 rpcz_store.cc:276] 0709 12:21:52.588573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.448289 (+859716us) service_pool.cc:224] Handling call
0709 12:21:53.448662 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.448895 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:53.764028 (+315133us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.622429 (+858401us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.622493 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:54.622545 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.622609 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.622726 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":314845,"prepare.queue_time_us":314992,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":166902,"spinlock_wait_cycles":16199424,"thread_start_us":857902,"threads_started":2,"wal-append.queue_time_us":409866}
W20260709 12:21:54.621556  4400 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.868s	user 0.014s	sys 0.000s
I20260709 12:21:54.592257  4021 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:21:54.642868 27738 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.004s	sys 0.029s
I20260709 12:21:54.640731 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.646380 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:54.653920 27582 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
I20260709 12:21:54.654376 27582 pending_rounds.cc:85] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:21:54.654639 27582 pending_rounds.cc:107] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 29.16
I20260709 12:21:54.654950 27582 raft_consensus.cc:2889] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:54.655637  4424 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.365s	user 0.003s	sys 0.006s
I20260709 12:21:54.659912  4404 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.328904 27585 rpcz_store.cc:276] 0709 12:21:52.996353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.963899 (+967546us) service_pool.cc:224] Handling call
0709 12:21:53.964266 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.964608 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:54.004105 (+ 39497us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.327133 (+323028us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.327408 (+   275us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.327478 (+    70us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:54.327546 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.327643 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.327801 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3838,"prepare.queue_time_us":41856,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":3745,"raft.queue_time_us":24031,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":41312,"thread_start_us":285011,"threads_started":1}
I20260709 12:21:54.698673  4240 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:54.649787  4351 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.700395  4351 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.701306  4240 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:54.701954  4240 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:54.702868  4351 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:54.667759  4393 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.669848  4021 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:54.704902  4393 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.674997  4402 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:54.680080  4352 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.001s
I20260709 12:21:54.680357  4411 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:54.680662  4055 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.851s	user 0.013s	sys 0.000s
I20260709 12:21:54.689793  4421 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:54.691259 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.724972 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.643743  4403 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:54.725718 28332 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:21:54.724313  4430 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260709 12:21:54.692917 27659 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:21:54.670642 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.728111 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:54.729041 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 226573us
I20260709 12:21:54.730032 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:54.730682  4438 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:21:54.731200  4438 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:54.731648  4438 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:54.731906 27664 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:21:54.643100  4434 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:21:54.708537 28335 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
W20260709 12:21:54.711997 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 170846us
I20260709 12:21:54.714102 27729 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:54.630735  3552 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:54.708577 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.720044 27736 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:21:54.720585 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79944.6us
I20260709 12:21:54.720086  4038 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:54.738008 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:54.724216  4055 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.741981 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.718145 27580 pending_rounds.cc:85] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 12
I20260709 12:21:54.708881 27585 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:21:54.707124  4393 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:54.746735 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:54.751421 27732 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:21:54.729305 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:54.734278 28328 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:54.752787 27659 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:54.720018  4126 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:54.754560 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.731648  4423 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:54.724084  4400 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:54.736778 28336 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:21:54.743278 27580 pending_rounds.cc:107] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:21:54.763953 27580 raft_consensus.cc:2889] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:54.708348 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4057) took 2329 ms. Trace:
I20260709 12:21:54.765321 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:54.748147 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.750115 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.769763 27585 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:54.735857 28335 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 15
I20260709 12:21:54.772807 28335 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:21:54.762616 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:54.763260 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 395170us
I20260709 12:21:54.761902 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.774094 28335 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:54.772817 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60919.4us
I20260709 12:21:54.749877  4434 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:54.754500  4401 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:54.734831  4352 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:54.785032 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.775305 27729 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:21:54.756060  4351 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:21:54.787840 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49945.6us
I20260709 12:21:54.784065 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.765986 27583 rpcz_store.cc:276] 0709 12:21:52.379024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.065979 (+686955us) service_pool.cc:224] Handling call
0709 12:21:53.074881 (+  8902us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.075106 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:53.392703 (+317597us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.698078 (+1305375us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b18004a5d20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287
0709 12:21:54.698221 (+   143us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.698357 (+   136us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:54.698425 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.708135 (+  9710us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.708277 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1293717,"prepare.queue_time_us":249112,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":14214,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":821409,"spinlock_wait_cycles":3439872,"thread_start_us":325300,"threads_started":3,"wal-append.queue_time_us":172476,"wal-append.run_cpu_time_us":1567,"wal-append.run_wall_time_us":1044659}
I20260709 12:21:54.760332  4393 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:54.790789  4421 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:54.790979 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:54.791185 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.791890  3917 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:21:54.792446 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:54.775934  4021 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:54.763155  4400 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:54.801926 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 319918us
I20260709 12:21:54.804615 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:54.805492 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:54.806447 28335 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:54.776772  4196 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:54.755764  4121 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:21:54.860258 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.860622 27662 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:21:54.866662 27583 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 28 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
W20260709 12:21:54.860671 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52760us
I20260709 12:21:54.758430  4038 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.795955  4408 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:54.787425  4401 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:54.767673  4126 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:54.870704 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.875381 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.778589  3818 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.087s
I20260709 12:21:54.789943  4351 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.800105  4183 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.074s
W20260709 12:21:54.874346 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 83436.2us
I20260709 12:21:54.888962 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:54.808969  4400 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:54.869019 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:54.859660  4186 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:54.769560 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:54.860203  4196 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.887035 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:54.894928 27583 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:21:54.807232  4162 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.078s
I20260709 12:21:54.878522  4393 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.875551  4401 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:54.796357  3552 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:54.870360  4447 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:21:54.806222  4243 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.083s
I20260709 12:21:54.911648 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:54.899712 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93601.7us
I20260709 12:21:54.912734 27729 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:54.913057 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:54.914409 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:54.902258 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3940) took 1097 ms. Trace:
I20260709 12:21:54.905892 27737 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:21:54.883104  4407 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.887593  4351 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:54.900028  4436 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
W20260709 12:21:54.903307  4148 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.724s	user 0.014s	sys 0.000s
I20260709 12:21:54.917377 27660 rpcz_store.cc:276] 0709 12:21:53.827513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.466706 (+639193us) service_pool.cc:224] Handling call
0709 12:21:54.467140 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.467404 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.467433 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.467940 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.743862 (+275922us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:54.743920 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.744020 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.902032 (+158012us) raft_consensus.cc:1685] finished
0709 12:21:54.902053 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.902190 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":275458,"threads_started":1,"wal-append.queue_time_us":408903}
I20260709 12:21:54.871713  4163 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:21:54.923637 27583 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:21:54.924973 27701 leader_election.cc:400] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:21:54.955937 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.956267 27701 leader_election.cc:403] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:54.957263 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.899927  4422 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:54.896891  4196 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:54.905208  4448 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:54.885772  4206 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:21:54.899593  4411 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.958691 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:54.911258 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:54.908505  4428 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:54.895287  4021 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
W20260709 12:21:55.056332 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 261563us
I20260709 12:21:54.912145  4447 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:55.051802 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.052794 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:55.054816 27736 raft_consensus.cc:606] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:55.062872 27736 raft_consensus.cc:614] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:54.922470  4186 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:21:55.056107 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144947us
I20260709 12:21:54.953709  4378 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:54.899993  4445 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.295s
I20260709 12:21:55.067461 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3947) took 1121 ms. Trace:
I20260709 12:21:55.069340 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:55.069828 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 201651us
I20260709 12:21:54.911405  4393 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.071998 27660 rpcz_store.cc:276] 0709 12:21:53.946063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.050837 (+1104774us) service_pool.cc:224] Handling call
0709 12:21:55.066793 (+ 15956us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.067030 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.067067 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.067123 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:55.067167 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.067237 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.067395 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:54.950492  4421 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.115s
W20260709 12:21:55.084460 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 275684us
I20260709 12:21:55.055670  4198 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:55.095845 27660 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:21:54.950363  4407 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.066134  4447 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:55.068456  4378 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:55.070995  3984 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:55.101742 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3824) took 1108 ms. Trace:
I20260709 12:21:55.112880 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4096) took 1378 ms. Trace:
I20260709 12:21:55.079658  4333 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:55.114406 27587 rpcz_store.cc:276] 0709 12:21:53.734874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.463665 (+728791us) service_pool.cc:224] Handling call
0709 12:21:54.463976 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.464169 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:54.766605 (+302436us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.112546 (+345941us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.112596 (+    50us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:55.112635 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.112696 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.112820 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2098,"prepare.queue_time_us":1913,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"thread_start_us":625208,"threads_started":2}
I20260709 12:21:55.059491  4428 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.114305 27733 rpcz_store.cc:276] 0709 12:21:53.992857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.993025 (+   168us) service_pool.cc:224] Handling call
0709 12:21:54.018715 (+ 25690us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.019172 (+   457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.019214 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.019756 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.397011 (+377255us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:54.397167 (+   156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.397393 (+   226us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.101451 (+704058us) raft_consensus.cc:1685] finished
0709 12:21:55.101482 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.101660 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":7088,"thread_start_us":256008,"threads_started":1,"wal-append.queue_time_us":495381}
W20260709 12:21:55.120838 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76488.1us
I20260709 12:21:54.952255  4434 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.122848 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3822) took 1131 ms. Trace:
I20260709 12:21:55.064947  4411 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.121737 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.102331  4055 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.126904  4440 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.128407  4186 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:55.129426 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.124249 27738 rpcz_store.cc:276] 0709 12:21:53.992018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.992184 (+   166us) service_pool.cc:224] Handling call
0709 12:21:54.020150 (+ 27966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.020396 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:54.652250 (+631854us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.122496 (+470246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.122553 (+    57us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:55.122598 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.122669 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.122792 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70377,"prepare.queue_time_us":409891,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":7495,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":8247,"thread_start_us":998435,"threads_started":3,"wal-append.queue_time_us":412141}
I20260709 12:21:55.129962  4393 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:55.130823  4455 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:21:55.131654  4449 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:21:55.136986 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4076) took 2958 ms. Trace:
I20260709 12:21:55.140234 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 3924) took 1431 ms. Trace:
I20260709 12:21:55.140532 27660 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:55.144385  4403 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.140291  4126 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:55.147310 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90218.3us
W20260709 12:21:55.176087 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.148509 28336 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:21:55.149255  4283 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.577s	user 0.007s	sys 0.001s
I20260709 12:21:55.149510  4438 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.118252  4378 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:55.157079 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:21:55.157958 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 237363us
I20260709 12:21:55.158288 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3937) took 1387 ms. Trace:
I20260709 12:21:55.161285 27583 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:21:55.120932  4352 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.169720 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.170432 27587 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:55.174314 27656 rpcz_store.cc:276] 0709 12:21:53.708688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.725813 (+ 17125us) service_pool.cc:224] Handling call
0709 12:21:55.140055 (+1414242us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:55.140177 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312046464}
I20260709 12:21:55.187106 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.174150 27584 rpcz_store.cc:276] 0709 12:21:52.178728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.862585 (+683857us) service_pool.cc:224] Handling call
0709 12:21:52.862939 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.002814 (+139875us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:53.215787 (+212973us) log.cc:844] Serialized 14 byte log entry
0709 12:21:53.302772 (+ 86985us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.302837 (+    65us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:53.302881 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.759075 (+456194us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b1800469660. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9e21c58f 00007f0e9725b571 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd
0709 12:21:54.179963 (+420888us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b1800469660. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:21:55.136669 (+956706us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54000bc7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:55.136726 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.136886 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4718,"prepare.queue_time_us":256311,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":252,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":8031,"spinlock_wait_cycles":349201920,"thread_start_us":283110,"threads_started":2,"wal-append.queue_time_us":224117,"wal-append.run_cpu_time_us":1077,"wal-append.run_wall_time_us":1002328}
I20260709 12:21:55.168372  3917 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:55.181154 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.190917 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.181043 27663 rpcz_store.cc:276] 0709 12:21:53.771160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.450171 (+679011us) service_pool.cc:224] Handling call
0709 12:21:54.450583 (+   412us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:54.451129 (+   546us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:54.761087 (+309958us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.063116 (+302029us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:55.082548 (+ 19432us) log.cc:844] Serialized 52 byte log entry
0709 12:21:55.082739 (+   191us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:55.083304 (+   565us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.157966 (+ 74662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.158039 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.158086 (+    47us) raft_consensus.cc:1685] finished
0709 12:21:55.158099 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.158234 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82902,"prepare.queue_time_us":433525,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":5999,"raft.queue_time_us":90322,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":12464,"thread_start_us":520114,"threads_started":2,"wal-append.queue_time_us":360347}
I20260709 12:21:55.193040 27656 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 10
I20260709 12:21:55.197901 27656 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:21:55.198309 27656 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:55.195034 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.201428 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:55.178663  4333 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:55.202744 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:55.188063  4283 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:55.220621 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.189847  4038 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.182482  4199 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:21:55.200470  4055 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:21:55.224493 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.225888 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:55.184046  4445 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:21:55.206593  4474 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.207749  3984 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:21:55.168617  4440 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.235013 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4094) took 2064 ms. Trace:
I20260709 12:21:55.231786 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3935) took 1497 ms. Trace:
I20260709 12:21:55.245854 27588 rpcz_store.cc:276] 0709 12:21:53.170146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.241204 (+1071058us) service_pool.cc:224] Handling call
0709 12:21:54.256865 (+ 15661us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.506566 (+249701us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.506606 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.537479 (+ 30873us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.804033 (+266554us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:21:54.804305 (+   272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.147638 (+343333us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:55.147688 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.234637 (+ 86949us) raft_consensus.cc:1685] finished
0709 12:21:55.234665 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.234941 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31934,"raft.queue_time_us":18880,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":13237,"spinlock_wait_cycles":488699648,"thread_start_us":175838,"threads_started":1,"wal-append.queue_time_us":364730}
W20260709 12:21:55.246603 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51688.6us
I20260709 12:21:55.239305 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:55.212791  4198 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:55.214973  4352 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.243026 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3945) took 1320 ms. Trace:
I20260709 12:21:55.207969  4448 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:21:55.249680 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.250946 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.225464  4330 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:21:55.246423 27658 rpcz_store.cc:276] 0709 12:21:53.765549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.449236 (+683687us) service_pool.cc:224] Handling call
0709 12:21:54.449547 (+   311us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.449785 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.449824 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.450315 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.754387 (+304072us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:54.754424 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.754492 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.231535 (+477043us) raft_consensus.cc:1685] finished
0709 12:21:55.231557 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.231714 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":303710,"threads_started":1,"wal-append.queue_time_us":463679}
I20260709 12:21:55.271452 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:55.278582 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:55.241679  4333 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:55.244840  4257 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:55.255373 27657 rpcz_store.cc:276] 0709 12:21:53.922342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.684843 (+762501us) service_pool.cc:224] Handling call
0709 12:21:54.685264 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.685544 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:54.888177 (+202633us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.225016 (+336839us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.242556 (+ 17540us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.242632 (+    76us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:55.242692 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.242798 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.242936 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24948,"prepare.queue_time_us":221681,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":13084,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":9110,"thread_start_us":491219,"threads_started":2}
I20260709 12:21:55.254901  4198 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:55.236042  4411 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:55.313143 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 63594.8us
I20260709 12:21:55.319092 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.320883 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:55.310493 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84682.8us
I20260709 12:21:55.257565  4121 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:21:55.326545 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86021.1us
I20260709 12:21:55.257126  4475 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:55.278373  3984 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:55.235651  4196 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:55.329766 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 127126us
I20260709 12:21:55.325997  4428 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.330417 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.332628 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.334610 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.308969  4399 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.336771 28332 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:21:55.268646  4352 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.282378  4479 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:55.352823 28328 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:55.327803  4404 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:55.353813 27587 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:21:55.312458  4198 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:55.356624 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.356768 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.325807  4267 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:55.371362 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48104.6us
I20260709 12:21:55.337555  4121 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:55.365522 27663 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:55.335441  4283 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.337330  4448 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:21:55.244462  4474 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.333835  4475 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:55.340302  4411 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:21:55.396708  4473 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:21:55.397578  4457 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:21:55.401610 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:55.402570 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88560.4us
W20260709 12:21:55.450410 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 179111us
I20260709 12:21:55.256283  4472 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.114s
I20260709 12:21:55.406658 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:55.359923  4407 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.376773  4428 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.459566  4486 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.010s
W20260709 12:21:55.460206  4465 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:21:55.459790  4121 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.484223  4427 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:21:55.484973  4427 consensus_queue.cc:1243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 21.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:21:55.485605  4427 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.486038  4427 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.486310  4352 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.487306  3552 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:55.455127  4038 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.455636  4438 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:55.487718 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.488009  4038 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:55.489007  4240 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.786s	user 0.012s	sys 0.002s
I20260709 12:21:55.489435  4407 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:55.489503  4240 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.490023  4407 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:55.490237  4038 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.490960  4407 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.492401  4407 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:21:55.492477  4482 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:55.493531  4482 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:55.493919  4482 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:21:55.461303  4461 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:21:55.449183  4265 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:21:55.495640 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4100) took 1721 ms. Trace:
I20260709 12:21:55.497093 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3832) took 1494 ms. Trace:
W20260709 12:21:55.462487  4478 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:55.376811  4421 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:55.497676  4448 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:55.498313  4421 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.500422  4421 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.500301  4448 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.465492  4475 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:55.449328  4404 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:21:55.467517  4485 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:55.377429  4267 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.515497  4485 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:55.516489  4485 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.517107  4267 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.517401  4440 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.517771  4485 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:55.479665  4428 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.441018  4283 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.501891 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4071) took 1567 ms. Trace:
W20260709 12:21:55.522022 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51880.7us
I20260709 12:21:55.467945 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.359531  4399 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.496130 27580 rpcz_store.cc:276] 0709 12:21:53.774130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.645774 (+871644us) service_pool.cc:224] Handling call
0709 12:21:54.717922 (+ 72148us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.782695 (+ 64773us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.782745 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.783204 (+   459us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.075446 (+292242us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:55.075486 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.075565 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.495397 (+419832us) raft_consensus.cc:1685] finished
0709 12:21:55.495421 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.495576 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":197867,"threads_started":1,"wal-append.queue_time_us":379931}
I20260709 12:21:55.441160  4403 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.529295 27656 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:55.505522 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:55.505190 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 98639.4us
I20260709 12:21:55.402877  4479 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.531951  4479 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.532902 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.533186 28335 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:21:55.531945  4440 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:21:55.513648  4351 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.000s
I20260709 12:21:55.534253 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.534580  4351 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.535228 28335 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:21:55.536203 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.536816  4351 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.537653 27658 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:55.513790  4376 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:55.538343  4376 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:55.539387  4376 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.540789  4376 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:55.521102 27589 rpcz_store.cc:276] 0709 12:21:53.934138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.650266 (+716128us) service_pool.cc:224] Handling call
0709 12:21:54.659859 (+  9593us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.740975 (+ 81116us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.741018 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.766985 (+ 25967us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.083641 (+316656us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:55.083706 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.083807 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.501699 (+417892us) raft_consensus.cc:1685] finished
0709 12:21:55.501718 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.501838 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":102729,"raft.queue_time_us":78039,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":22829,"thread_start_us":282750,"threads_started":1,"wal-append.queue_time_us":419284}
I20260709 12:21:55.541263 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:55.541823 28335 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:55.468161  4471 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:55.497488 27731 rpcz_store.cc:276] 0709 12:21:54.002631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.637614 (+634983us) service_pool.cc:224] Handling call
0709 12:21:54.649947 (+ 12333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.732112 (+ 82165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.732142 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.732630 (+   488us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.953533 (+220903us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:55.044220 (+ 90687us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.044474 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.044553 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.489229 (+444676us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b180032d880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:21:55.496811 (+  7582us) raft_consensus.cc:1685] finished
0709 12:21:55.496914 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.497036 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81784,"prepare.queue_time_us":90052,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":82554,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":209,"spinlock_wait_cycles":7897472,"thread_start_us":220518,"threads_started":1,"wal-append.queue_time_us":569164}
I20260709 12:21:55.547108 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3946) took 1614 ms. Trace:
I20260709 12:21:55.547523 27659 rpcz_store.cc:276] 0709 12:21:53.932352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.760334 (+827982us) service_pool.cc:224] Handling call
0709 12:21:54.760690 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.789600 (+ 28910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.789641 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.806622 (+ 16981us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.161033 (+354411us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:21:55.161095 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.161206 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.546866 (+385660us) raft_consensus.cc:1685] finished
0709 12:21:55.546889 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.547041 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15192,"raft.queue_time_us":15404,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":7235,"thread_start_us":278407,"threads_started":1,"wal-append.queue_time_us":456745}
I20260709 12:21:55.549435 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:55.550014 27659 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:55.523556 27587 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:55.474856  4470 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:55.570498 27589 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:55.521134  4421 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:55.572664 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.579955 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:55.581383 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47213.7us
I20260709 12:21:55.494776  4482 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:55.584201 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61082us
W20260709 12:21:55.586580 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81483.5us
W20260709 12:21:55.587205 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.587625 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.589473 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:21:55.553062  4163 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:21:55.571815  4428 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.597105 28335 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:55.597303 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:55.600279 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.600821 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.601310  4474 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.603127 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.603832 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.585961  4403 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:21:55.591360  4408 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.708s	user 0.012s	sys 0.000s
I20260709 12:21:55.609014  4408 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:55.610029 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1484) took 1802 ms. Trace:
I20260709 12:21:55.610420 27734 rpcz_store.cc:276] 0709 12:21:53.807619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.833932 (+ 26313us) service_pool.cc:224] Handling call
0709 12:21:53.834276 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:53.843792 (+  9516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.843823 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:53.885475 (+ 41652us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.019776 (+134301us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:54.020033 (+   257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.020208 (+   175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.608816 (+1588608us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b24002c5988. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:21:55.609682 (+   866us) raft_consensus.cc:1685] finished
0709 12:21:55.609705 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.609966 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12293,"raft.queue_time_us":244,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":45549,"spinlock_wait_cycles":84714240,"thread_start_us":86536,"threads_started":1,"wal-append.queue_time_us":332008}
I20260709 12:21:55.606947 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.612154  4479 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.612648 27659 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:55.599525  4240 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.615067  4198 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.616861  4399 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.596660  4423 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:55.617913  4423 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.618376  4196 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.618376  4198 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.620080  4423 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.570248  4470 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.533329  4489 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.071s
I20260709 12:21:55.623612  4470 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.624388 27736 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:55.627487  4482 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:55.628201  4443 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:21:55.628677  4447 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.000s
I20260709 12:21:55.629256  4447 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.629837  3841 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:21:55.596776  4481 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:21:55.603971  4378 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.631343 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.632869 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:55.633432 28333 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:55.633731 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:55.633893  4469 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.635066  4469 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.637048 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1627) took 1337 ms. Trace:
I20260709 12:21:55.637369  4469 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:55.638659  4422 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.591s	user 0.008s	sys 0.000s
I20260709 12:21:55.640735  4489 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:54.524830 27581 rpcz_store.cc:276] 0709 12:21:53.169810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.230039 (+1060229us) service_pool.cc:224] Handling call
0709 12:21:54.230428 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.230673 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:54.246267 (+ 15594us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.523629 (+277362us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.524087 (+   458us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:54.524134 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.524203 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.524321 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16067,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":23897,"thread_start_us":253097,"threads_started":1}
I20260709 12:21:55.628635  4443 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:55.637393 27583 rpcz_store.cc:276] 0709 12:21:54.299796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.047864 (+748068us) service_pool.cc:224] Handling call
0709 12:21:55.061918 (+ 14054us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.182968 (+121050us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.636805 (+453837us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:55.636994 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205063424}
I20260709 12:21:55.645609 27656 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:55.650444 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:55.649080 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.651742 27731 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:55.603847  4490 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:21:55.596892  4333 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.652645 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:55.663717 27581 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:21:55.633698  4378 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:55.651480  4422 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:55.680441 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:55.681365 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:55.683246 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.686096 27736 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:55.689683 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:55.692870 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.693187 27735 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:55.659466  4490 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:55.704967 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:55.708956 27656 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:21:55.706321 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.695294 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:55.717903 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 129360us
I20260709 12:21:55.720481 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:55.722968 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 134601us
I20260709 12:21:55.723780 27581 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:55.728562 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:55.730692  4399 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:55.731559  4399 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:55.732870  4399 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.685637  4381 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:55.687119  4423 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:55.742863  4494 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:55.752323 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:55.690888  4493 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:55.741658 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.671010  4333 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.751986 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:55.755643  4423 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.742457  4404 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:55.720346  4427 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.757508  4038 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:55.753961  4378 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.758486 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4085) took 1037 ms. Trace:
W20260709 12:21:55.762717  4492 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:21:55.765348 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3951) took 1051 ms. Trace:
I20260709 12:21:55.836124 27660 rpcz_store.cc:276] 0709 12:21:54.714150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.097108 (+382958us) service_pool.cc:224] Handling call
0709 12:21:55.140180 (+ 43072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.175066 (+ 34886us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.765111 (+590045us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b5400041d50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:55.765281 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82310016}
I20260709 12:21:55.769408 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:55.776878 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:55.837646 27731 raft_consensus.cc:606] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:55.837852 27583 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:21:55.838163 27731 raft_consensus.cc:614] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:55.778240 28331 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:55.780463  4475 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.782335 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3941) took 1939 ms. Trace:
I20260709 12:21:55.841152 27662 rpcz_store.cc:276] 0709 12:21:53.843221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.506063 (+662842us) service_pool.cc:224] Handling call
0709 12:21:54.541593 (+ 35530us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.868631 (+327038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.781995 (+913364us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:55.782260 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616760192}
I20260709 12:21:55.786369  4162 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.843580  4162 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.846192  4162 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.846678 28334 raft_consensus.cc:2410] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:21:55.791647  4490 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.793787 27663 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: [27.20-27.20]
W20260709 12:21:55.800004  4396 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 2.209s	user 0.005s	sys 0.007s
I20260709 12:21:55.801017  4503 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:55.865391  4503 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:55.866132  4503 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:55.753773  4494 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.763162 28333 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:55.805239  4381 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.867825 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:55.869117 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:55.869396  4423 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:55.870371 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.871006 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.804296 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.872255 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.872867 28330 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:55.873315 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.795058 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:55.877475 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1595) took 2366 ms (client timeout 2137 ms). Trace:
I20260709 12:21:55.877616 27655 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 18. Current term is 19. Ops: []
W20260709 12:21:55.877936 27664 rpcz_store.cc:269] 0709 12:21:53.510734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.510894 (+   160us) service_pool.cc:224] Handling call
0709 12:21:55.877244 (+2366350us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:55.877380 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2805483904}
I20260709 12:21:55.881879 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:55.881901  4381 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:55.806886 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.808332 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:55.812819 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.834215 27580 rpcz_store.cc:276] 0709 12:21:54.721583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.532271 (+810688us) service_pool.cc:224] Handling call
0709 12:21:55.534347 (+  2076us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.537052 (+  2705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.537277 (+   225us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.538725 (+  1448us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.654513 (+115788us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:55.654563 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.654637 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.758259 (+103622us) raft_consensus.cc:1685] finished
0709 12:21:55.758283 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.758423 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2028,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":114642,"threads_started":1,"wal-append.queue_time_us":217338}
I20260709 12:21:55.835265 27581 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:55.786998  4421 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:55.838907 28331 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:55.854339 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:55.855149 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.894631 27660 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:21:55.813891  4400 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.915s	user 0.005s	sys 0.007s
I20260709 12:21:55.859719 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.896216 27662 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 17.
W20260709 12:21:55.903051 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96252.5us
I20260709 12:21:55.903956 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.904212 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:55.910585 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:55.911384 27580 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:55.813743  4404 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:55.917735 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:55.923864 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.924099 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:55.924660 27660 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:55.931618 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:55.933552 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:55.856510  4283 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:55.916054 28331 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:55.936030 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194433us
I20260709 12:21:55.957396 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.957208 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:55.967406 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:55.917310  4489 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:55.974395 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:55.975685 28331 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:55.975523 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:55.976366 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:55.907603  4381 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.841712  4427 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.089s
I20260709 12:21:55.813414  4493 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.842483  4378 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.943369  3917 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.901808  3552 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:21:55.944149  4055 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:21:56.094792 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 127492us
I20260709 12:21:56.100003 27583 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:55.954336  4481 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:21:55.928035  4404 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:55.980764  4196 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:56.146921 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4076) took 1703 ms. Trace:
I20260709 12:21:56.147465 27584 rpcz_store.cc:276] 0709 12:21:54.443594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.188299 (+744705us) service_pool.cc:224] Handling call
0709 12:21:55.205540 (+ 17241us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.271714 (+ 66174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.271745 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.279078 (+  7333us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.469164 (+190086us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:55.469282 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.469459 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.762304 (+292845us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b240013ac18. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:21:56.128649 (+366345us) raft_consensus.cc:1685] finished
0709 12:21:56.128670 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.144160 (+ 15490us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3536,"raft.queue_time_us":72747,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":26897,"spinlock_wait_cycles":36390144,"thread_start_us":199637,"threads_started":2,"wal-append.queue_time_us":168376}
I20260709 12:21:56.106387  4501 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:56.107839  4427 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:55.802942  4443 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:55.924191  4403 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:55.955183  4408 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:55.945025  4358 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:55.908277  4400 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:56.061707  4376 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.000s
W20260709 12:21:56.116583 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.959126  4162 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:56.127970 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 154156us
I20260709 12:21:55.934641  4421 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:56.099404  4407 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.606s	user 0.009s	sys 0.002s
I20260709 12:21:55.961978  4490 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:56.099718  4381 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.129357  3917 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.161008  4503 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.281826 27588 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:56.146013  4411 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:56.286525 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4098) took 2518 ms. Trace:
W20260709 12:21:56.111779  4316 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.008s	sys 0.024s
I20260709 12:21:56.109280  4283 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:21:56.129452  3984 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.775s	user 0.010s	sys 0.001s
I20260709 12:21:56.173381 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.363611 27586 rpcz_store.cc:276] 0709 12:21:53.767921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.621664 (+853743us) service_pool.cc:224] Handling call
0709 12:21:54.644636 (+ 22972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.644824 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:54.722114 (+ 77290us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.917000 (+1194886us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b18002ef540. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:21:56.286136 (+369136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.286197 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:56.286249 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.286329 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.286465 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":77044,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":8040,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":5781,"spinlock_wait_cycles":29055872,"thread_start_us":368908,"threads_started":1,"wal-append.queue_time_us":1625972}
W20260709 12:21:56.361380 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 404010us
W20260709 12:21:56.099090  4401 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.148s	user 0.014s	sys 0.000s
W20260709 12:21:56.366400  4500 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.450s	user 0.001s	sys 0.000s
I20260709 12:21:56.395682 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.396481 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.397047 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.397722 27586 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:56.398669 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1429) took 1081 ms. Trace:
I20260709 12:21:56.146507  4267 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.362922 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.164208  4020 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:56.365918  4316 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.008s	sys 0.024s
I20260709 12:21:56.156162  4055 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:56.157253 27580 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:21:56.397746 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:56.333770  4403 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:56.409214 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46358.9us
I20260709 12:21:56.424860 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:56.426301 28335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.100s	sys 0.147s
W20260709 12:21:56.206487 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.426702 28335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.100s	sys 0.148s
I20260709 12:21:56.264907 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.292143  4163 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:56.385520  4283 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.352155  4475 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.286648  4494 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:56.275213  4498 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.520s	user 0.001s	sys 0.000s
I20260709 12:21:56.164799  4501 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.170168  4196 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:55.981572  4474 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:56.399214 28336 rpcz_store.cc:276] 0709 12:21:55.317588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.317758 (+   170us) service_pool.cc:224] Handling call
0709 12:21:55.318187 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.524520 (+206333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.524553 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.525162 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.702328 (+177166us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:55.702373 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.703421 (+  1048us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.398414 (+694993us) raft_consensus.cc:1685] finished
0709 12:21:56.398440 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.398586 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17013,"raft.queue_time_us":121602,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":19393,"thread_start_us":327032,"threads_started":2,"wal-append.queue_time_us":290561}
I20260709 12:21:55.955077  4479 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.460s
I20260709 12:21:56.428678 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.397663  3984 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:21:56.299160  4441 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:56.452901 27586 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260709 12:21:56.443184  4489 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:56.432792 27731 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:21:56.437074 27580 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:56.457672 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4089) took 1134 ms. Trace:
I20260709 12:21:56.381503  4162 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.458153 27580 rpcz_store.cc:276] 0709 12:21:55.323765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.436497 (+1112732us) service_pool.cc:224] Handling call
0709 12:21:56.436852 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.457328 (+ 20476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.457590 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.442899 27583 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
I20260709 12:21:56.168545  4408 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.443612 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.444962 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:56.431586  4283 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:56.462195 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1630) took 1233 ms. Trace:
I20260709 12:21:56.432082  4403 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.174078  4358 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.426185  4493 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:56.446204 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 271632us
I20260709 12:21:56.462618 27583 rpcz_store.cc:276] 0709 12:21:55.228869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.434586 (+1205717us) service_pool.cc:224] Handling call
0709 12:21:56.442715 (+  8129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.461911 (+ 19196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.462133 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.463701 28331 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 20. (term mismatch)
W20260709 12:21:56.450767  4463 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.191s	user 0.003s	sys 0.003s
I20260709 12:21:56.413592  4267 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.445047  4523 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 12:21:56.395682  4401 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.444761  4411 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:56.417639  4020 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.510548 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3183) took 1193 ms. Trace:
I20260709 12:21:56.511101 27658 rpcz_store.cc:276] 0709 12:21:55.316675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.356341 (+ 39666us) service_pool.cc:224] Handling call
0709 12:21:56.484765 (+1128424us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:56.510466 (+ 25701us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400629504}
I20260709 12:21:56.478688 28331 pending_rounds.cc:85] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 19
I20260709 12:21:56.467274  4162 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.475247  4055 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:56.468230  4515 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:56.502915 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.507560 27729 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:21:56.470846  4504 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:21:56.513433 28331 pending_rounds.cc:107] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 27.20
I20260709 12:21:56.518560  4493 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:56.504269  4485 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.983s	user 0.012s	sys 0.000s
I20260709 12:21:56.435876  4407 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:21:56.452466  4475 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:56.443387  4163 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.477373  4408 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.527532 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:56.544106 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1435) took 1009 ms. Trace:
I20260709 12:21:56.544250 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.527889 28331 raft_consensus.cc:2889] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:56.528295 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:56.470141  4413 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.238s	user 0.002s	sys 0.005s
W20260709 12:21:56.499177  4519 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:21:56.531584 27585 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:56.464846  4400 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.471843  4474 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:56.467682  4428 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:56.467531  4511 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:21:56.487025  4378 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.471611  4479 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.475392  4358 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:56.480469  4267 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:56.527101  4514 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:21:56.550040 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1425) took 2030 ms. Trace:
I20260709 12:21:56.544780 28332 rpcz_store.cc:276] 0709 12:21:55.534987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.535139 (+   152us) service_pool.cc:224] Handling call
0709 12:21:55.535470 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.535803 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.535833 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.536302 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.583235 (+ 46933us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:55.583283 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.583396 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.543756 (+960360us) raft_consensus.cc:1685] finished
0709 12:21:56.543785 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.544021 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":84341,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":46557,"threads_started":1,"wal-append.queue_time_us":142576}
I20260709 12:21:56.581452  4196 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.582875  4020 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.582962  4404 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:56.585055 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 156528us
I20260709 12:21:56.586812 28331 rpcz_store.cc:276] 0709 12:21:54.519433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.519607 (+   174us) service_pool.cc:224] Handling call
0709 12:21:54.541919 (+ 22312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.778148 (+1236229us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b54005d4e20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:21:56.549758 (+771610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.549984 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68836096}
W20260709 12:21:56.584719  4495 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260709 12:21:56.597684 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:56.584342  4428 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.586331  4479 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.589346  4267 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:56.600734  4510 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:56.602185  4163 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:56.604781 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1604) took 1240 ms. Trace:
I20260709 12:21:56.605468  4501 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.605861  4381 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:56.606518  4521 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:21:56.607582  4403 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.608144 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1602) took 1246 ms. Trace:
I20260709 12:21:56.609293  4532 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:21:56.577576  4474 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.610919  4401 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.548905  4475 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.619894  4421 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.620409  4490 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:56.621078  4482 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.993s	user 0.007s	sys 0.006s
I20260709 12:21:56.622036  4404 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.535640  4493 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.627983  4530 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.663568  4428 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.651703  4267 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.663687  4530 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.636889  4283 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.666080  4267 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.637890 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1591) took 3136 ms. Trace:
I20260709 12:21:56.667217  4283 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.667760  4474 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.648541  4532 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:56.631242 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1436) took 1095 ms. Trace:
W20260709 12:21:56.643019 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45480.1us
I20260709 12:21:56.642514 27663 rpcz_store.cc:276] 0709 12:21:55.364702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.931908 (+567206us) service_pool.cc:224] Handling call
0709 12:21:55.943567 (+ 11659us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.943936 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.943965 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.960834 (+ 16869us) log.cc:844] Serialized 26 byte log entry
0709 12:21:56.407385 (+446551us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:56.407434 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.407521 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.604606 (+197085us) raft_consensus.cc:1685] finished
0709 12:21:56.604624 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.604726 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82626,"raft.queue_time_us":10461,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":14777,"thread_start_us":280256,"threads_started":1,"wal-append.queue_time_us":491598}
I20260709 12:21:56.668828  4532 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.643862 27659 rpcz_store.cc:276] 0709 12:21:55.362028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.613398 (+251370us) service_pool.cc:224] Handling call
0709 12:21:55.613749 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:55.614178 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.614214 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:55.614880 (+   666us) log.cc:844] Serialized 52 byte log entry
0709 12:21:55.904088 (+289208us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:55.904123 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.904184 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.607900 (+703716us) raft_consensus.cc:1685] finished
0709 12:21:56.607924 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.608085 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3348,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":288797,"threads_started":1,"wal-append.queue_time_us":561535}
W20260709 12:21:56.646145 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 117982us
W20260709 12:21:56.637084 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.672516  4475 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:56.672864 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.660629  4490 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.676683  4490 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:56.677358  4475 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:56.657667 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:56.678051  4475 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.679723 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:56.680138  4475 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.638377  4489 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:56.661962  4479 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.680327 27586 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:56.677400  4283 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:56.655563 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.682435  4283 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.686434 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.688169 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.688232 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:56.668494 28334 rpcz_store.cc:276] 0709 12:21:55.535692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.535833 (+   141us) service_pool.cc:224] Handling call
0709 12:21:56.631017 (+1095184us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:56.631181 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":711395584}
I20260709 12:21:56.667490 27655 rpcz_store.cc:276] 0709 12:21:53.501573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.502333 (+   760us) service_pool.cc:224] Handling call
0709 12:21:55.877389 (+2375056us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.878209 (+   820us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.637645 (+759436us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b54000c2450. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:56.637778 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5127372672}
I20260709 12:21:56.691082 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.691761 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:56.693717 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:56.694720 27655 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:56.696470 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.699626 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.700423 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:56.697309 27658 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:21:56.705013 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.707965 27659 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:56.709545 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.709734 27586 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:56.711133 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.712579 27586 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:56.713361 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.703457  4479 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:56.711369  4474 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:56.715703  4479 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.716199  4479 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:56.717146 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:56.667011  4530 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.718887 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.651371  4531 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:56.720392 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.720553 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.721836 28335 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:21:56.722110  4538 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:56.722579  4538 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.672365  4126 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:21:56.722967  4538 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:56.723543  4126 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:56.723961  4126 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:56.724805  4126 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.726058  4126 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:56.708194  4489 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:21:56.727797  4489 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.729622  4333 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:56.733925  4474 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:56.721051 27662 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:56.736127 27552 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:56.736521 27552 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:56.728189  4489 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:56.739830 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.744089 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.745139 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.720250  4531 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.749680 27655 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:21:56.748875 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:56.752130 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.753813 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.754635 27586 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:21:56.761260 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1631) took 1426 ms. Trace:
I20260709 12:21:56.761757 27587 rpcz_store.cc:276] 0709 12:21:55.335223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.395137 (+ 59914us) service_pool.cc:224] Handling call
0709 12:21:56.761024 (+1365887us) spinlock_profiling.cc:243] Waited 72.9 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:56.761155 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167723904}
I20260709 12:21:56.763227 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.763995 27587 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:21:56.765022 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.765666 27587 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:21:56.766906 27587 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:56.745203  4475 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:21:56.769276 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:56.772084 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1632) took 1185 ms. Trace:
W20260709 12:21:56.773636 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 144140us
I20260709 12:21:56.773733 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:56.750790  4531 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:56.770138 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:56.758806  4468 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:56.759130  4482 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.787751 27586 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:21:56.786460 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.780876 27587 rpcz_store.cc:276] 0709 12:21:55.587192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.766340 (+1179148us) service_pool.cc:224] Handling call
0709 12:21:56.766688 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.771764 (+  5076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.772011 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.792135 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4092) took 1164 ms. Trace:
I20260709 12:21:56.792840 27586 rpcz_store.cc:276] 0709 12:21:55.627689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.785176 (+1157487us) service_pool.cc:224] Handling call
0709 12:21:56.785570 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.791123 (+  5553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.792078 (+   955us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.793499 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:56.782233  4267 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:56.800007 27586 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
I20260709 12:21:56.782058  4421 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:56.800724 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1633) took 1074 ms. Trace:
I20260709 12:21:56.786044  4148 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:21:56.788583  4468 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:56.796280  4351 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.010s	sys 0.007s
W20260709 12:21:56.794003  4481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.000s	sys 0.010s
W20260709 12:21:56.807560  4481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.000s	sys 0.010s
I20260709 12:21:56.807978  4481 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:56.773563  4470 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.025s
I20260709 12:21:56.774874  4404 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:21:56.796571  4534 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.803072 27586 rpcz_store.cc:276] 0709 12:21:55.726112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.793902 (+1067790us) service_pool.cc:224] Handling call
0709 12:21:56.799830 (+  5928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.800440 (+   610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.800665 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.796083  4490 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:56.796581  4399 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:56.822062  4196 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:21:56.823071  4196 consensus_queue.cc:1243] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 25.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:21:56.824903  4441 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:56.825399  4441 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:56.806794  4468 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:56.800508  4533 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:21:56.828737  4493 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.795694  4482 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.830930  4482 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.806097  4421 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.830925  4445 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:21:56.832017  4445 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:56.832862  4445 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.833669  4534 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.834079  4445 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:56.834611  4441 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.835948  4534 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.836141 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:56.836361 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.836716  4441 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.837075 28337 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:56.838073 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.786923  4503 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.018s
W20260709 12:21:56.841856 27630 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.702s (SENT) [suppressed 5 similar messages]
I20260709 12:21:56.833248  4421 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.839120 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3842) took 1159 ms. Trace:
I20260709 12:21:56.845662 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:56.848834 27734 rpcz_store.cc:276] 0709 12:21:55.679694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.679827 (+   133us) service_pool.cc:224] Handling call
0709 12:21:55.693448 (+ 13621us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:55.693835 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.693880 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:55.720162 (+ 26282us) log.cc:844] Serialized 52 byte log entry
0709 12:21:55.909159 (+188997us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:21:56.838480 (+929321us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.838730 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.838836 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.838889 (+    53us) raft_consensus.cc:1685] finished
0709 12:21:56.838908 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.839057 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":932857,"raft.queue_time_us":3766,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":15590,"thread_start_us":139879,"threads_started":1,"wal-append.queue_time_us":705263}
I20260709 12:21:56.855154  4532 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:56.858359  4532 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.858695  4532 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:56.859213  4538 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.863009 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:56.810087  4351 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.010s	sys 0.007s
I20260709 12:21:56.812410  4283 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.868419  4378 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:56.870299  4378 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.871912 27735 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:56.872193  4283 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:56.872843 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.873116  4316 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:56.872944  4283 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.873560 28331 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:21:56.873602  4316 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:56.874130 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:56.872434  4378 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:56.875355 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:56.875648 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.875684  4378 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:56.876432 27737 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:21:56.876677  4378 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:56.879310 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:56.879700 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.879968 27735 raft_consensus.cc:606] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:56.879366 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.881266 28331 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:21:56.872272  3841 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:56.886878 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.888442 27737 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:21:56.882347 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:56.894083 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.894953 27737 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:56.896515 27734 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:21:56.871083  3917 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.875161  4283 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.900341 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.901896 27734 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260709 12:21:56.874696  4316 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.901628  4538 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:56.857921  4489 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.922041  4490 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:56.903724  4481 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.898953  3988 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:56.895748  4427 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:21:56.904758  4378 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:21:56.905905 27737 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:21:56.908198  4479 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:56.917132 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.885056  3841 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:56.903802  4055 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:21:56.936231 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:56.924798 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 129923us
I20260709 12:21:56.926662 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.937994 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:56.929941 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.936371 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.925887 27737 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:56.940657 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:56.941766 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:56.945011 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:56.948201  3841 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:56.951454 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:56.955658 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:56.959738 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.965462 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:56.966236 27735 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:56.927776  4427 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.968927 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.969360 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:56.964229 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:56.974339 27659 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:56.976009 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.969882 27730 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:21:56.970975 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:56.971094 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.972473 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:56.937100  4316 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:56.995514 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.935191  4538 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:56.943050  4470 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.942641  4399 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:56.997617 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:56.925266  3988 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:56.950093  4283 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:56.949708  4534 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:21:56.958489  4546 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:56.996477 27737 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:56.997617 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.037070 27662 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:56.971966  4441 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:56.959124  4475 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:21:56.977485  4427 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:57.021373 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.044871 27663 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
W20260709 12:21:56.978936  4448 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.010s	sys 0.020s
I20260709 12:21:57.024498 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4100) took 1283 ms. Trace:
I20260709 12:21:57.027097 27659 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:57.047251 27586 rpcz_store.cc:276] 0709 12:21:55.741202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.812613 (+1071411us) service_pool.cc:224] Handling call
0709 12:21:56.866293 (+ 53680us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.866543 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:56.886735 (+ 20192us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.024118 (+137383us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.024178 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:57.024220 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.024284 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.024422 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25658,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":162,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":137005,"threads_started":1}
I20260709 12:21:57.048439 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:56.990815  4512 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:56.990646  4428 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:21:57.054136 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4087) took 1825 ms (client timeout 1702 ms). Trace:
I20260709 12:21:57.054289  4428 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.000699 27658 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:21:57.046109 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 204275us
W20260709 12:21:57.054595 27588 rpcz_store.cc:269] 0709 12:21:55.229181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.248457 (+ 19276us) service_pool.cc:224] Handling call
0709 12:21:57.053921 (+1805464us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540011b554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:57.054073 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2306240384}
I20260709 12:21:57.049052 27733 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:21:57.021672  4400 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.050782 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:57.050019  4448 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.010s	sys 0.024s
I20260709 12:21:57.021142  4283 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.080756 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.011891  4538 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.043941  4475 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.043613  4532 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.053392 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.081739 27658 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:21:57.049986  4534 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.021446  4470 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.047160  4403 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.061811  4428 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:57.063746 28332 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:21:57.063983 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.065675 27584 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
W20260709 12:21:57.065735 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.079119 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.076128 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.052119 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:21:57.092018 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.083443 27737 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: [29.17-29.17]
W20260709 12:21:57.083443 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 110435us
I20260709 12:21:57.082934 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.044517  4531 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.021486  4399 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.084374  4283 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:57.079917  4055 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.087425 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.094700 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.086696  4546 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:57.098052 27733 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:21:57.098919 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.084308  4401 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.086508  4475 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:57.101768 27655 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:21:57.101791 27662 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:57.082736  4400 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.065260  4468 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.100734  4378 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:21:57.119693  4378 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:57.111508 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:57.102156  4546 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.113857  4531 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.105613  4401 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:57.115748  4506 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.141s	user 0.003s	sys 0.003s
I20260709 12:21:57.111222 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.100500  4481 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:21:57.102507  4422 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:21:57.090433  4534 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.124907 27662 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 18.
I20260709 12:21:57.131909 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:57.127586 27588 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:57.129236  3988 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.131902  4441 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.130259  4532 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.132823 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.091924  4403 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.134418 27662 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:57.133745  4422 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:57.104044  4055 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.137795 27662 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:57.123680  4378 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:57.125375  4546 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:21:57.134018  4548 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:21:57.131630  4479 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.150619  4403 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.151918 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3843) took 1471 ms. Trace:
I20260709 12:21:57.151934 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4107) took 2290 ms. Trace:
I20260709 12:21:57.148042 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.154253 27589 rpcz_store.cc:276] 0709 12:21:54.861656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.544351 (+682695us) service_pool.cc:224] Handling call
0709 12:21:55.544636 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.571226 (+ 26590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.151698 (+1580472us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:57.151876 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38459648}
I20260709 12:21:57.152760 27732 rpcz_store.cc:276] 0709 12:21:55.680348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.680516 (+   168us) service_pool.cc:224] Handling call
0709 12:21:55.680944 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.687474 (+  6530us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:21:56.835372 (+1147898us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b180031c180. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:21:57.036146 (+200774us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.101310 (+ 65164us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.101779 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.101939 (+   160us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:57.101968 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.102028 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.151603 (+ 49575us) raft_consensus.cc:1685] finished
0709 12:21:57.151699 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.151855 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38255,"prepare.queue_time_us":1150182,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":10630,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":12558,"spinlock_wait_cycles":3345280,"thread_start_us":247024,"threads_started":2,"wal-append.queue_time_us":99074}
I20260709 12:21:57.156322 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4072) took 3172 ms. Trace:
I20260709 12:21:57.156694 27582 rpcz_store.cc:276] 0709 12:21:53.983745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.656676 (+672931us) service_pool.cc:224] Handling call
0709 12:21:54.657040 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.721250 (+ 64210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.721286 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.721823 (+   537us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.645760 (+923937us) spinlock_profiling.cc:243] Waited 7.23 ms on lock 0x7b1800432640. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:21:55.757287 (+111527us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:55.757329 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.757871 (+   542us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.115178 (+1357307us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:57.156141 (+ 40963us) raft_consensus.cc:1685] finished
0709 12:21:57.156161 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.156270 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93889,"raft.queue_time_us":79583,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":17857,"spinlock_wait_cycles":719534720,"thread_start_us":111293,"threads_started":1,"wal-append.queue_time_us":1109950}
I20260709 12:21:57.158394 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.137526  4126 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.015s
I20260709 12:21:57.163975 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:57.168136 27735 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:21:57.137333  4441 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.177577 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.187341  4441 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.165973  4534 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:57.191854  4553 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:21:57.193360 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4097) took 1554 ms. Trace:
I20260709 12:21:57.193852 27587 rpcz_store.cc:276] 0709 12:21:55.638962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.791975 (+1153013us) service_pool.cc:224] Handling call
0709 12:21:56.801313 (+  9338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:56.801558 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:56.801575 (+    17us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:56.810555 (+  8980us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.013330 (+202775us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:57.013390 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.013487 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.193039 (+179552us) raft_consensus.cc:1685] finished
0709 12:21:57.193071 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.193282 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20784,"raft.queue_time_us":88,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3362,"thread_start_us":144763,"threads_started":1,"wal-append.queue_time_us":235989}
I20260709 12:21:57.199563  4401 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.200392 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.201861 27587 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:21:57.203212 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.204164 27587 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:21:57.209328 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:57.210579 27587 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:57.208226 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.212741 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:57.214591 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.215541  3988 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:57.216128  3988 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.216761  4121 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.217387  4121 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.219482 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4090) took 1819 ms. Trace:
I20260709 12:21:57.216609  3988 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:57.220124 27580 rpcz_store.cc:276] 0709 12:21:55.400091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.463466 (+1063375us) service_pool.cc:224] Handling call
0709 12:21:56.505826 (+ 42360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.683149 (+177323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:56.683181 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.683222 (+    41us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:56.683250 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.219184 (+535934us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540011ba54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:57.219238 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.219417 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407328640}
I20260709 12:21:57.221817 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.224428  4121 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.227406 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1625) took 2928 ms. Trace:
I20260709 12:21:57.228009 27585 rpcz_store.cc:276] 0709 12:21:54.299260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.800195 (+500935us) service_pool.cc:224] Handling call
0709 12:21:54.800604 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.549370 (+1748766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.227115 (+677745us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54000bdbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:57.227304 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3976566528}
I20260709 12:21:57.229799 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.161154  4563 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.230692 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:57.231276  4563 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:57.232882  4557 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:21:57.183523  4381 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:21:57.235648  4563 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.207125  4493 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.236943  4493 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:57.238694 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.164834  4538 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.240550  4198 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.252082  4400 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.252353 27660 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:21:57.242821 28336 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:21:57.244184  4333 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.253690  4333 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:57.252084 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3187) took 1713 ms (client timeout 1838 ms). Trace:
I20260709 12:21:57.256155  4333 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:57.256090 27656 rpcz_store.cc:269] 0709 12:21:55.539142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.579493 (+ 40351us) service_pool.cc:224] Handling call
0709 12:21:57.251895 (+1672402us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:57.252024 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148699392}
I20260709 12:21:57.256785  4333 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.257300  4333 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.257977  4447 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.240684  4470 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.244357  4468 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.181564  4422 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.260478 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.260941  4333 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.261313 27583 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:21:57.261595  4333 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.262179  4333 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.262488 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.264788 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.265944 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.266911 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.267524  4402 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:57.267962  4402 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.268151 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.246449  4399 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.268810  4468 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:21:57.269006  4399 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:57.216609  4520 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:57.269577  4399 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.269937  4520 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:57.270081  4399 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:57.270951  4520 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.272167  4520 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:57.247682 28328 pending_rounds.cc:85] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 16
I20260709 12:21:57.273269 28328 pending_rounds.cc:107] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:21:57.273571 28328 raft_consensus.cc:2889] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:57.247988  4467 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.248281 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:57.275256  4516 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.940s	user 0.002s	sys 0.000s
I20260709 12:21:57.275764 28333 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:57.251292  4198 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.243347 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.269147  4240 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:21:57.279917 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.280369  4240 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:57.281232  4240 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.282531  4240 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:57.280396  4470 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:57.283833  4470 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.284200  4470 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:57.269412  4468 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:57.285838  4468 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.245808  4055 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.274705  4467 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.288698  4411 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.289294  4411 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.289875  4467 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.272193 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.290860 27588 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:57.295931 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.268388  4402 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:57.222576 27580 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
W20260709 12:21:57.249742 27701 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.654s (SENT)
I20260709 12:21:57.305830 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:57.306149 27585 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:21:57.312666 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4111) took 1542 ms. Trace:
I20260709 12:21:57.313113 27585 rpcz_store.cc:276] 0709 12:21:55.770176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.305212 (+1535036us) service_pool.cc:224] Handling call
0709 12:21:57.305819 (+   607us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.312304 (+  6485us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.312595 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:57.309513 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.317461 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4110) took 1547 ms. Trace:
W20260709 12:21:57.318230 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68530.8us
I20260709 12:21:57.317962 27580 rpcz_store.cc:276] 0709 12:21:55.769922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.298726 (+1528804us) service_pool.cc:224] Handling call
0709 12:21:57.316891 (+ 18165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.317086 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.317120 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.317185 (+    65us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:57.317236 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.317299 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.317401 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.321601 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.322685 27589 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:57.309995 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.323881 27587 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:21:57.298095  4493 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.330463 27588 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:57.332039 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.336710 27580 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:21:57.337575 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4113) took 1566 ms. Trace:
I20260709 12:21:57.337965 27580 rpcz_store.cc:276] 0709 12:21:55.770652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.319932 (+1549280us) service_pool.cc:224] Handling call
0709 12:21:57.336418 (+ 16486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.337265 (+   847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.337509 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.341575 27587 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:21:57.349745 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.344309 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.369479 28335 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:21:57.367944 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.332847  4121 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:21:57.279857  4428 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.346885  4481 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:57.370725  4121 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:57.371063 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4114) took 1596 ms. Trace:
I20260709 12:21:57.371300  4481 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.371552 27589 rpcz_store.cc:276] 0709 12:21:55.774528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.333407 (+1558879us) service_pool.cc:224] Handling call
0709 12:21:57.370518 (+ 37111us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.370714 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.370739 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.370787 (+    48us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:57.370836 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.370893 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.371008 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.372251  4428 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.371109  4121 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.375907  4121 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:57.375969  4283 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.327646  4148 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:21:57.378971 27587 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:57.392563  4558 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:57.393477  4283 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.280134  4198 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.320890  4475 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.380476 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.398236 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.403705 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.404798 27732 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:57.405735 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.407474 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.347136  4411 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.412823 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.327702  4404 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.072s
I20260709 12:21:57.410262 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.410542 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.323622  4399 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.321806  4381 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:21:57.416735 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.417320 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.417611 27734 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:57.417936 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:57.414749 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:57.420120 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.420964 27734 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:57.395525  4148 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.397866  4558 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.427232 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.427996 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.424502 27585 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:21:57.423019 27655 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:21:57.427469 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.451923 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:57.413602  4535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.005s	sys 0.024s
I20260709 12:21:57.431947 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.454769 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.431156 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.427129  4162 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:57.456924  4162 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.457410  4162 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:57.457302  3988 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.451392  4148 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:21:57.459985  4562 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.246s	user 0.001s	sys 0.000s
I20260709 12:21:57.423569  4020 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.467854  4020 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.469594  4121 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.453302  4558 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:57.471731  4121 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.472529 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.468349  3988 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.476104 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4122) took 1699 ms. Trace:
I20260709 12:21:57.476557 27581 rpcz_store.cc:276] 0709 12:21:55.776599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.473666 (+1697067us) service_pool.cc:224] Handling call
0709 12:21:57.475580 (+  1914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.475761 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.475791 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.475837 (+    46us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:21:57.475870 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.475941 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.476049 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.475780  4427 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.412393  4267 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:57.423296  4196 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:57.482527 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.493688 27735 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:57.495265 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.489675 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.490557 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:57.496968 27581 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 28 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
W20260709 12:21:57.454208  4535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.005s	sys 0.025s
I20260709 12:21:57.491887 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.466809  4378 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.431985  4333 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:57.506883 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.471366  4020 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.503674 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.505362 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4101) took 1697 ms. Trace:
I20260709 12:21:57.481808  4404 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:57.516573 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4109) took 1747 ms. Trace:
I20260709 12:21:57.516733  4427 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.517022 27588 rpcz_store.cc:276] 0709 12:21:55.769650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.291738 (+1522088us) service_pool.cc:224] Handling call
0709 12:21:57.330232 (+ 38494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.333324 (+  3092us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.516371 (+183047us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b54000bd6d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:57.516515 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87534336}
I20260709 12:21:57.511549 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.520411 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4108) took 2657 ms. Trace:
I20260709 12:21:57.520782 27584 rpcz_store.cc:276] 0709 12:21:54.863258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.430901 (+1567643us) service_pool.cc:224] Handling call
0709 12:21:57.058670 (+627769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.090793 (+ 32123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.519740 (+428947us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b540011b554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:57.520250 (+   510us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b540011b550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:57.520359 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1442247936}
I20260709 12:21:57.520900  4546 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.522404  4402 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.510689 27581 rpcz_store.cc:276] 0709 12:21:55.807805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.504457 (+1696652us) service_pool.cc:224] Handling call
0709 12:21:57.504743 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.504953 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.504990 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.505046 (+    56us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:57.505090 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.505159 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.505284 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:57.515072  4570 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:21:57.524495  4399 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.525465 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4103) took 1667 ms. Trace:
I20260709 12:21:57.526758  4443 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:57.493443  4196 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.527570  4196 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:57.529064 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.530063 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.530359 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.530934 27735 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:57.530917 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.522437 27586 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:57.516420  4404 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.511364 28329 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: [32.22-32.22]
I20260709 12:21:57.534574 27584 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:57.535568 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4102) took 1704 ms. Trace:
I20260709 12:21:57.535954 27584 rpcz_store.cc:276] 0709 12:21:55.831562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.521998 (+1690436us) service_pool.cc:224] Handling call
0709 12:21:57.533489 (+ 11491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.535272 (+  1783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.535506 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.524601 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.541018 27737 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:57.541970 27584 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
W20260709 12:21:57.543481 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 436780us
I20260709 12:21:57.544343 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.525836 27581 rpcz_store.cc:276] 0709 12:21:55.858379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.524729 (+1666350us) service_pool.cc:224] Handling call
0709 12:21:57.524990 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.525157 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.525181 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.525222 (+    41us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:57.525253 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.525304 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.525411 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.547988 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.549268 27581 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 23. Current term is 24. Ops: []
I20260709 12:21:57.550467 27664 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:21:57.554512 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 244616us
W20260709 12:21:57.588798 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47201.9us
I20260709 12:21:57.550657 27731 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:57.557577 27581 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:21:57.566154 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.587862 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.605969 27584 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
W20260709 12:21:57.551026 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 113504us
I20260709 12:21:57.587466 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.585996 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:57.612308 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.577879 27582 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:21:57.569012 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:57.568632 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4112) took 1798 ms. Trace:
I20260709 12:21:57.618609 27581 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:57.620357 27588 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:57.618532 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:57.615238 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.503257  4148 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.624706 27589 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:57.634060 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:57.615926 27734 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:21:57.636663 27582 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:57.624723 27585 rpcz_store.cc:276] 0709 12:21:55.770399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.314050 (+1543651us) service_pool.cc:224] Handling call
0709 12:21:57.314418 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.455232 (+140814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.568301 (+113069us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:57.568558 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252680448}
I20260709 12:21:57.640995 27582 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:57.621482  3841 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:21:57.643165 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.010s	sys 0.014s
I20260709 12:21:57.584558  4475 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.584859  4546 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.572624  4488 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:57.645277 27588 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:57.574991  4489 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:57.646692 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.011s	sys 0.014s
I20260709 12:21:57.628171 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.647816 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:57.660384 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 116114us
I20260709 12:21:57.661381 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.665863 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4120) took 1889 ms. Trace:
W20260709 12:21:57.670699 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 133756us
I20260709 12:21:57.642210  3818 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:21:57.669972 27587 rpcz_store.cc:276] 0709 12:21:55.776129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.396057 (+1619928us) service_pool.cc:224] Handling call
0709 12:21:57.396466 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.396714 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:21:57.425713 (+ 28999us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.665473 (+239760us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.665537 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:57.665580 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.665659 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.665804 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28911,"prepare.queue_time_us":21407,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":19952,"raft.queue_time_us":22833,"thread_start_us":217308,"threads_started":1}
I20260709 12:21:57.621420  4576 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:57.670918 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.677227 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.634483  4020 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:21:57.656348  4432 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.626349  4162 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.687894  4493 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.016s
I20260709 12:21:57.694880  4535 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.010s
I20260709 12:21:57.705595  4162 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.706317  4283 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:57.706862  4558 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.662175  4488 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:57.681758 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:57.666663  4358 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:57.684756 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.661994  4411 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:57.621672  4376 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:21:57.627137  4333 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:21:57.666916  4267 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:57.677863  4421 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:57.716877 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.723922  3818 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:57.724382  3818 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.724773  3818 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:57.725250  4421 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.691265  4121 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:21:57.727593  4531 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:21:57.692168  4148 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.728119  4531 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.728600  4531 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:57.711486  4432 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.731405 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:57.731730  4432 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.732369  4422 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:21:57.732518 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:57.732860  4422 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.733333  4422 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:57.733387  4489 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:21:57.734048  4489 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:57.669951  4198 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:21:57.719460  4411 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.735662  4411 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:57.735265  4489 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.736681  4378 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.680439  4533 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:21:57.680131  4475 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:57.739259 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.729206  4421 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.719389 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:57.743279 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.676293  4576 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.745931 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.749342 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.669565  4468 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:21:57.760926 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.675776  4381 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:21:57.712361  4358 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.739753  4240 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:57.687736  4578 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:57.772294 27659 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:21:57.717744  4196 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.683446  4055 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
I20260709 12:21:57.778127 27658 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:57.786680 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.739399  4475 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.779475 27662 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:57.777335 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:57.782770 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96943.1us
I20260709 12:21:57.746675  4474 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:21:57.757284  4432 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:57.746413  4576 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:57.759806  4020 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:21:57.779628  4546 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.792761 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.803620 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4121) took 2027 ms. Trace:
I20260709 12:21:57.794183 27656 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:57.796586 27655 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:57.769280  4428 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:57.769424  4358 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:57.773876  4162 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:21:57.771893  4240 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:57.792008 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.804175 27580 rpcz_store.cc:276] 0709 12:21:55.776369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.418044 (+1641675us) service_pool.cc:224] Handling call
0709 12:21:57.452883 (+ 34839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.453081 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:57.646772 (+193691us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.803166 (+156394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.803240 (+    74us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:57.803299 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.803400 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.803550 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26058,"prepare.queue_time_us":35068,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"thread_start_us":290473,"threads_started":2,"wal-append.queue_time_us":127656}
I20260709 12:21:57.768498  3818 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.768887  4467 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:21:57.760769  4470 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.793244  4475 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:57.801928  4020 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.824337 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.770459  4481 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:21:57.796518  4402 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.800475  4283 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:21:57.826761 27657 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:21:57.824410 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.837767 27655 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:57.797447  4558 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.799134  4432 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.801563  4055 consensus_queue.cc:1243] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 31.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:21:57.810663 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.846887 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.847463 27663 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:57.807468  4428 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:57.812655  4240 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:57.845068 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.807296  4546 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.835067  4020 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:57.850196 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:57.848616 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55496.7us
I20260709 12:21:57.855634 27655 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:57.845693  4432 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:57.862049 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.862775 27656 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:21:57.862768 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101969us
I20260709 12:21:57.863688 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.864583 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.865435 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:57.857928  4574 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.203s	user 0.002s	sys 0.000s
I20260709 12:21:57.866590 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.849335  4428 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:57.873119 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:57.877014 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.878692 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.879932 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.881037 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.882205 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.882783  4590 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:57.883234  4590 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:57.868984  4447 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:57.884222  4590 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.899554  4590 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:57.901870  4531 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.887064  4488 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.904229  4482 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.923949 27736 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:21:57.888805 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 796740us
I20260709 12:21:57.927280 27736 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:57.905109 27662 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:21:57.889328  4493 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:21:57.889672  4559 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:21:57.930496 27658 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:57.892578  4448 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.893414 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:21:57.894376 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 621753us
I20260709 12:21:57.928438 27736 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.895565  4483 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:21:57.895975  4472 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:57.883276 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.908497 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.867093  4586 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:21:57.909454  4402 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:57.911752  4162 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:21:57.950970  4162 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:21:57.951395  4162 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:57.952333  4162 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.953612  4162 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:57.896165 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:57.896692 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.955734 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.912600 27582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60870
I20260709 12:21:57.957858 27582 raft_consensus.cc:606] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Received request to transfer leadership
I20260709 12:21:57.958191 27582 raft_consensus.cc:614] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:57.898172 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.958472 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.911808  3818 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.927671  4482 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.914052  4411 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.922878 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.975373 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.976012 28332 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:21:57.976987  4404 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.978664  4586 consensus_queue.cc:1243] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:21:57.971848  4592 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:57.887957  4196 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.851819  4055 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:21:57.980448 27588 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:21:57.980595  4055 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:21:57.980962  4055 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:57.946079  4472 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.981824  4055 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.983093  4055 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:57.983934  4198 raft_consensus.cc:993] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:57.984397  4198 raft_consensus.cc:1081] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:21:57.926294 27734 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:21:57.974157  4490 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.920310 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:57.987879 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3934) took 4254 ms. Trace:
I20260709 12:21:57.987697  4490 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.988212  4432 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:57.929296 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.928172 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:57.931978 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.881109  4474 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:21:57.934906 27736 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:58.005807 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.006429 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.006865 27734 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:21:57.949707  4402 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:57.930994  4546 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:57.897500  4422 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.931733  4559 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:57.962750 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54341us
I20260709 12:21:58.025490 27583 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:21:57.930399  4493 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:57.971103 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.970896 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:57.988607 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.033054 27656 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:57.928839 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:57.989063  4240 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.989833  4490 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:57.990567 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:57.991274 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.041772 27663 raft_consensus.cc:2376] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20. Re-sending same reply.
I20260709 12:21:58.042739 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:57.991823 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.994005 27657 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:57.994544  4055 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.996129  4475 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:57.999908  4531 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:58.000411  3841 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:21:57.938922 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:58.005983 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:21:58.067709 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 128867us
I20260709 12:21:58.068452 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:58.070816 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1281) took 1153 ms. Trace:
I20260709 12:21:58.006939 27737 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:58.072661 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:21:57.940474  4448 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.005791 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:57.947288 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.073540 27663 raft_consensus.cc:2376] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26. Re-sending same reply.
I20260709 12:21:58.005417 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:58.027266 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:58.077613 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:57.988272 27661 rpcz_store.cc:276] 0709 12:21:53.733656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.446190 (+712534us) service_pool.cc:224] Handling call
0709 12:21:55.878968 (+1432778us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.879490 (+   522us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.879536 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.880204 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:21:56.109725 (+229521us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:21:56.406564 (+296839us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.514122 (+107558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.514850 (+   728us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.274707 (+759857us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:21:57.278199 (+  3492us) raft_consensus.cc:1685] finished
0709 12:21:57.278320 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.987715 (+709395us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54000c2450. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:57.987823 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2649,"prepare.queue_time_us":354297,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":169801,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":10343,"spinlock_wait_cycles":3305663104,"thread_start_us":582181,"threads_started":3,"wal-append.queue_time_us":413677}
I20260709 12:21:58.035182 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:58.078727 27656 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:21:58.078855 27732 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260709 12:21:57.983264  4472 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.039683 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:58.065174 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:58.010437 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82330.6us
I20260709 12:21:58.037305 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.088018 27655 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:21:58.089284 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 83926.3us
I20260709 12:21:58.091763 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.084254 27658 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:21:58.085114 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59189.7us
W20260709 12:21:58.082504 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76591us
I20260709 12:21:58.079695 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.080343 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.071260 28331 rpcz_store.cc:276] 0709 12:21:56.917406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.921483 (+  4077us) service_pool.cc:224] Handling call
0709 12:21:56.928702 (+  7219us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.928934 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:21:56.954157 (+ 25223us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.994032 (+ 39875us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.528921 (+534889us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b4800032f58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:21:57.879447 (+350526us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x7b1800171ee0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:21:58.070488 (+191041us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.070539 (+    51us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:58.070580 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.070643 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.070753 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":412267,"prepare.queue_time_us":82914,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":80,"raft.queue_time_us":890701,"raft.run_cpu_time_us":2626,"raft.run_wall_time_us":570845,"spinlock_wait_cycles":28765824,"thread_start_us":190774,"threads_started":1}
I20260709 12:21:58.100142 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:58.100560 27656 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:58.100625 27733 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:21:58.079159 27662 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:58.103658 27658 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:21:58.104592 27662 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:58.031822  4559 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:58.102516 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:58.025518  4483 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:21:58.104852 27658 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:58.105451 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.109365 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.116762 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:58.117461 27587 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:58.110116 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.084180 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:58.107990 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.111402 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:58.106101 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:58.123708 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.074699  4283 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:58.124979 28337 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:21:58.067013  3841 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:58.141665 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.127924 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:21:58.142223 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77165.5us
I20260709 12:21:58.142972 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.143142 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.128458 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.144495 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.144953 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.145418 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.146020 27729 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:58.126468 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.132665 28329 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:21:58.138346 27659 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:21:58.146755 27731 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:21:58.147847 27659 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:21:58.128008 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.084941  4490 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:21:58.151937 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.148798 27664 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:21:58.146988 27580 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:21:58.153120 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.150820 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:58.081393  4561 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:58.101285  4592 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:21:58.072228  4572 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:21:58.076951  4198 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:21:58.192287 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.191200 27659 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:21:58.107880  4404 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:58.175889 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:58.193864 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4119) took 2418 ms. Trace:
I20260709 12:21:58.200220 27655 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260709 12:21:58.200421 27659 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:58.198338 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.124677  4467 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:58.202689 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.200996 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.192541 27585 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:58.203953 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103927us
I20260709 12:21:58.210086 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:58.134495  4546 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:21:58.136468  4472 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:58.224221 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:58.230001 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:58.208189 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.108392  4503 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.090s
I20260709 12:21:58.233510 27582 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:58.200125 27589 rpcz_store.cc:276] 0709 12:21:55.775878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.372763 (+1596885us) service_pool.cc:224] Handling call
0709 12:21:57.624533 (+251770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.633718 (+  9185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.193658 (+559940us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54002ba7d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:58.193797 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":578191744}
I20260709 12:21:58.092841  4594 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.120s
I20260709 12:21:58.242458 27580 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:21:58.142656  4565 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:58.155232  4597 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.143509  4399 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:58.163932  4490 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:58.196180  4299 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:58.246528 27581 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:58.108618  4428 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.094849  4055 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.190169  4333 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:58.198709  3988 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:21:58.235476 27585 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:21:58.300040  3988 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:58.300239 27585 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:58.300659  4569 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:21:58.301167  3988 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.302794  3988 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:58.162570  4493 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.304188  4493 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:58.304314  4448 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:58.305625  4411 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.305764 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:58.306140  4402 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:58.108414  4560 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.139s
I20260709 12:21:58.306608 27664 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:21:58.307387  4560 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:21:58.307793  4560 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:58.288545 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.308836 27733 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:58.125391  4564 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.136s
I20260709 12:21:58.282272 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:21:58.245162  4562 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:21:58.300700 27585 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:58.199429 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.236104  4593 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:21:58.312466 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:58.323801 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.327777 27580 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:21:58.328563 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:58.221607  4483 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:21:58.329305 28333 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:58.329938 27584 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:58.196826  4591 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:21:58.246892  4472 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.280647 27587 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:58.354120 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.290567  4399 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:58.353149 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.352253 27664 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:21:58.357033 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.358436 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.360306 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:58.361707 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.364073 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 188338us
I20260709 12:21:58.365550 27730 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:58.366308 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:58.367338 27659 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:58.370061 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.371457 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.322535  4569 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:58.380225 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.386243 27732 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:21:58.339162  4501 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:58.328990  4546 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:21:58.310287  4474 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.068s
I20260709 12:21:58.375540 27737 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:58.386803 27730 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:21:58.388444 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.389849 27588 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:58.250418  4447 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:21:58.252511  4489 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:21:58.260761  4490 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:58.339468  4593 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:58.345255  4407 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.354768  4411 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:21:58.310482  4592 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.083s
I20260709 12:21:58.411201 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.244201  4597 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.406574 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.374804  4594 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:58.374481  4503 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:58.441474  4503 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:58.442533  4503 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.403090  4489 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.443655  4503 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:58.439437  4411 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:58.416862  4572 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:21:58.416611  4240 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.445142  4411 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.426945 27732 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:58.447536 27732 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:58.448024  4472 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:58.448400 27732 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.449900 27732 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:58.392890  4546 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.451575  4546 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:21:58.452432  4594 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.453014  4594 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:21:58.456202 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49731.8us
I20260709 12:21:58.393329  4501 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.457736 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.443717  4489 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:58.399106  4432 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.488139 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.505342 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.487423 27736 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 15. (term mismatch)
I20260709 12:21:58.508929 27736 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 14
I20260709 12:21:58.509357 27736 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:21:58.509841 27736 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:58.512432 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:58.466481  4472 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:58.417507  4407 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.515334  4472 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.517014  4407 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.517940  4472 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.518924  4531 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:58.519667  4531 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:58.521965  4596 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260709 12:21:58.522326  4531 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:58.523061 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:58.523821 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4126) took 1051 ms. Trace:
I20260709 12:21:58.524999 27732 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:21:58.525357  4468 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:58.524226 27586 rpcz_store.cc:276] 0709 12:21:57.472887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.789727 (+316840us) service_pool.cc:224] Handling call
0709 12:21:57.806194 (+ 16467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.806663 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:21:57.829512 (+ 22849us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.061544 (+232032us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.062083 (+   539us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.062248 (+   165us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:21:58.062308 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.062411 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.523610 (+461199us) raft_consensus.cc:1685] finished
0709 12:21:58.523633 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.523763 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75421,"prepare.queue_time_us":22005,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":81716,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":23503,"spinlock_wait_cycles":1945984,"thread_start_us":156583,"threads_started":1,"wal-append.queue_time_us":306846}
I20260709 12:21:58.526435 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:58.527048 27586 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:21:58.529241 27659 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:58.464044  4572 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:58.472093  4501 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:58.531034 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:58.591617  4598 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:21:58.617080  4593 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.482569  4475 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:58.620689 27734 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.043s	sys 0.046s
W20260709 12:21:58.664976 27734 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.044s	sys 0.046s
I20260709 12:21:58.528155  4591 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:21:58.661196 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.656112 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.668838 27735 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
W20260709 12:21:58.675374 27731 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.001s
I20260709 12:21:58.676239 27731 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:21:58.537220  4299 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:58.684548 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:58.677237  4528 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:21:58.698705 27731 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [27.15-29.16]
I20260709 12:21:58.674220  4523 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:58.666788  4055 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:58.491374  4399 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.668740  4572 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:21:58.721306 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.722389 27731 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:58.723891 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:58.727339 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:58.731107  4198 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:21:58.736486 27589 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:21:58.738153 27589 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []
I20260709 12:21:58.745273  4561 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:21:58.745738  4561 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:21:58.746564 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
W20260709 12:21:58.710255  4283 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.000s
W20260709 12:21:58.713840  4020 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.849s	user 0.005s	sys 0.005s
I20260709 12:21:58.756247 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:21:58.670886  4569 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.765902 27585 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.717341  4428 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.779004 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:58.785679 27586 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:58.757625  4283 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:21:58.708725  4523 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.373090 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:58.757930  4198 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:58.760681  3818 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.826363 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:58.827041 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.827178 28330 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:58.828575 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:58.830199 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.831884 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.831029 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:21:58.787880  4407 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:58.842229 27738 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:21:58.844231 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:21:58.835144 27660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.046s	sys 0.076s
W20260709 12:21:58.858691 27660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.046s	sys 0.076s
I20260709 12:21:58.759087  4020 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.848133 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3979) took 1035 ms. Trace:
W20260709 12:21:58.817116  4378 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 31 current term: 31 is higher.
I20260709 12:21:58.850636 27585 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:58.853849 27589 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
W20260709 12:21:58.846035  4528 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:21:58.854805 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.840148 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:58.865428 27731 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:21:58.842204  4407 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:58.837638  4475 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:58.859277 27658 rpcz_store.cc:276] 0709 12:21:57.813173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.116057 (+302884us) service_pool.cc:224] Handling call
0709 12:21:58.116302 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.116487 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:21:58.358058 (+241571us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.847831 (+489773us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.847888 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:21:58.847932 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.847995 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.848084 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28177,"prepare.queue_time_us":287521,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":112,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":1774,"thread_start_us":639691,"threads_started":2}
I20260709 12:21:58.816077  3818 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.871505 27585 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:21:58.842535 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:58.875170 27661 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:58.866945  4467 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:21:58.872503  4407 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:58.842533  4561 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.16, Last appended by leader: 16, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.888604 27589 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
W20260709 12:21:58.884055 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 353024us
W20260709 12:21:58.882066 27553 thread.cc:641] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.009s	sys 0.007s
W20260709 12:21:58.897449 27553 thread.cc:608] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.010s	sys 0.008s
I20260709 12:21:58.895617 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:58.888944 27662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.000s	sys 0.015s
W20260709 12:21:58.942214 27662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.000s	sys 0.017s
I20260709 12:21:58.899580  4428 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:58.900847  4564 consensus_queue.cc:1175] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 18 status { last_received { term: 18 index: 16 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:58.905210  3818 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.906870  4561 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:21:58.923022  4474 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.943249  4589 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:21:58.946103  4020 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:58.892290  4467 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.934168  4482 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:21:58.947310 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51792us
I20260709 12:21:58.886170  3988 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:21:58.896049  4493 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.003s
I20260709 12:21:58.997085  3988 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:59.001101  4489 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.508s	user 0.002s	sys 0.008s
I20260709 12:21:59.001200  3988 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:58.877326  4475 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.001695  4489 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.007115 27581 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:58.973953  4474 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.012311  4474 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.012879  4474 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.013371  4474 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.014539  4475 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.957801 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.015130  4475 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:59.015641 28335 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:21:59.015725  4475 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:58.961009 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:59.016861 28337 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:59.018205  4475 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:58.930712  4376 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:21:59.019521  4376 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:59.019811  4162 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.065s	user 0.010s	sys 0.005s
I20260709 12:21:59.020165 27737 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:21:59.020609 27656 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 33 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:21:59.020448  4376 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.021166 27733 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:21:59.021490  4376 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:21:58.950004  4579 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.035705 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.036895 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:59.037734 28329 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:21:58.948529 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3978) took 1143 ms. Trace:
I20260709 12:21:58.880158  4198 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:58.957851  3818 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:59.068708 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.048727 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:59.054613 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:59.071617 27734 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:59.056082 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:59.059367 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:59.074047 27735 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:21:59.005410  4618 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.140s	user 0.000s	sys 0.002s
I20260709 12:21:59.040525 27662 rpcz_store.cc:276] 0709 12:21:57.805532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.111757 (+306225us) service_pool.cc:224] Handling call
0709 12:21:58.112090 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.112343 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:21:58.346150 (+233807us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.948193 (+602043us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.948249 (+    56us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:21:58.948295 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.948361 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.948474 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":247251,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":7509,"raft.queue_time_us":87,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":2785,"thread_start_us":805799,"threads_started":2,"wal-append.queue_time_us":625633}
I20260709 12:21:59.077119  4586 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:21:59.077524  4586 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:21:59.077569  4490 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.649s	user 0.007s	sys 0.004s
I20260709 12:21:59.078114  4490 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.078382  4586 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.079589 28334 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:59.068620  3818 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.082134 28337 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:59.082844 27730 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
W20260709 12:21:59.088792 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.033s	sys 0.017s
I20260709 12:21:59.093432 27735 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:21:59.094377 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:21:59.089190 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.033s	sys 0.017s
I20260709 12:21:59.066105 28329 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:21:59.110349 27736 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:21:59.041311  4594 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.588s	user 0.012s	sys 0.000s
I20260709 12:21:59.200516  4594 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:59.114153 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.205905 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.206416 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:59.095985 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:59.079424  4586 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:21:59.208046  4055 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.208832  4055 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:59.119163 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 227713us
W20260709 12:21:59.081482  4358 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.257s	user 0.011s	sys 0.002s
I20260709 12:21:59.212707  4358 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.214136  4594 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:58.955070  4428 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.215523  4055 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.217303  4428 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.217659  4591 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:21:59.217974  4428 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:59.218330  4591 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:59.138782 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:59.220021  4378 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:21:59.140954 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:59.220403  4333 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:21:59.220929  4378 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:21:59.143568  4411 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:21:59.221804 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.222388 27655 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:21:59.222769  4411 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:59.223289  4411 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.223676  4358 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:59.223881  4594 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:59.224298  4358 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:21:59.224531  4594 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:59.224771  4358 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:59.225448  4411 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.225939  4411 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:59.225790  4594 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.226521  4411 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.150075 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.005662  4467 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.227527 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:21:59.169672 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:21:59.253695 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84156.6us
I20260709 12:21:59.218981  4591 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:59.220515 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.004796  4597 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.280421 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.067255  4162 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.157033 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.023736  4579 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.001572  4445 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.111s
W20260709 12:21:59.263646 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 43519.7us
I20260709 12:21:59.281608 28329 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:21:59.284173 27664 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:21:59.285925 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:21:59.245237 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.280139  4358 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.292305  4162 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.293289 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.218736  4428 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.116231  4493 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:21:59.281126  4503 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.828s	user 0.010s	sys 0.003s
I20260709 12:21:59.303828  4623 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:21:59.304615  4489 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.306075 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:59.309538  4482 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:21:59.287232 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:21:59.381531 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94384.5us
W20260709 12:21:59.381587 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75586.4us
I20260709 12:21:59.079453  3988 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.222782  4598 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:21:59.228293  4501 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:59.302389 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.370735 27587 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: []
W20260709 12:21:59.375761 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.041s	sys 0.085s
W20260709 12:21:59.398526 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.042s	sys 0.086s
W20260709 12:21:59.328272  4559 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.186s	user 0.000s	sys 0.013s
W20260709 12:21:59.325377  4546 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.873s	user 0.009s	sys 0.002s
I20260709 12:21:59.114197  4523 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:21:59.333160  4576 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.523s	user 0.012s	sys 0.000s
I20260709 12:21:59.384101 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:21:59.362558  4594 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:21:59.343546  4597 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:59.406937 28330 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.009s	sys 0.043s
I20260709 12:21:59.338570  4579 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.311241  4474 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.367053  4547 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:21:59.345175  4490 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.393000  4598 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:59.340760 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.415696 27655 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:21:59.454936 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:59.412865 27662 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:21:59.455809 27663 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:21:59.408231 28330 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.010s	sys 0.043s
I20260709 12:21:59.394058  4121 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:21:59.447463 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:59.462888 27588 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:59.335000  4475 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:21:59.449914 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.467121 27664 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 31 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260709 12:21:59.406705  4482 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:59.407636  4358 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:21:59.475867 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4151) took 1377 ms. Trace:
I20260709 12:21:59.476261 27584 rpcz_store.cc:276] 0709 12:21:58.098424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.203550 (+105126us) service_pool.cc:224] Handling call
0709 12:21:58.203935 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.357033 (+153098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.475589 (+1118556us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:21:59.475807 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289184128}
I20260709 12:21:59.378170  4593 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:21:59.474938  4482 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.485482 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:21:59.486291 27737 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:21:59.487238 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:59.488340  4567 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.488890  4567 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.490939  4567 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.494817  4162 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.497069  3988 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.499819 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4134) took 1395 ms. Trace:
I20260709 12:21:59.500344 27664 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:21:59.501830 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.502585 27664 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:21:59.444103  4447 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:21:59.504361  4411 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.508380  4428 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.509816  4501 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:59.509963 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:59.519650 28332 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:21:59.407228  4628 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:21:59.500195 27582 rpcz_store.cc:276] 0709 12:21:58.103917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.142685 (+ 38768us) service_pool.cc:224] Handling call
0709 12:21:59.499627 (+1356942us) spinlock_profiling.cc:243] Waited 76.3 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:59.499762 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175450368}
I20260709 12:21:59.511101 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:59.526690 27586 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:59.379209  4493 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.512197 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1644) took 1202 ms. Trace:
I20260709 12:21:59.527411 27581 rpcz_store.cc:276] 0709 12:21:58.310022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.356761 (+ 46739us) service_pool.cc:224] Handling call
0709 12:21:59.512017 (+1155256us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:59.512143 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1335041280}
I20260709 12:21:59.504712 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.482903 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:21:59.467015 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:59.528913 27589 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:59.447005  4547 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:21:59.524190  4422 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:21:59.524781 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.043s	sys 0.081s
W20260709 12:21:59.528923 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46079.5us
I20260709 12:21:59.527209 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.569729 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:59.570299 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.406476  4559 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.443838  4576 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:21:59.570657 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.043s	sys 0.081s
W20260709 12:21:59.534631 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.413514  4546 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.590790 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:59.448761  4597 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.525872  4628 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.600960 27663 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:21:59.601773 27738 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:21:59.603161 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:59.605557 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 3964) took 1825 ms. Trace:
I20260709 12:21:59.513875  4447 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.404644  4560 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.612300 27657 rpcz_store.cc:276] 0709 12:21:57.780029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.909559 (+129530us) service_pool.cc:224] Handling call
0709 12:21:57.993823 (+ 84264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.082570 (+ 88747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.605360 (+1522790us) spinlock_profiling.cc:243] Waited 83.9 ms on lock 0x7b5400210550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:59.605501 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192960768}
I20260709 12:21:59.606053 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.518771  4121 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:21:59.608139  4470 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.601186  4598 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.618149  4121 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.618597  4598 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:59.614593  4590 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.614643  4503 raft_consensus.cc:2764] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:21:59.620246  4121 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.619975  4590 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.621502 27657 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:21:59.622493 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:59.622886  4590 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:59.623160 27657 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:21:59.630697  3818 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:21:59.636637  3818 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:59.637878 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.637396  3818 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.623922 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.589979  4501 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:59.595607  4411 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.646080 27729 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:21:59.567730  4632 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:59.475160  3841 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:21:59.650111 28335 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.033s	sys 0.040s
I20260709 12:21:59.614743  4447 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:59.606897  4567 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.609176  4523 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:21:59.601207  4623 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:21:59.659453 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.665539 28332 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:59.648140 27663 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 17. Current term is 18. Ops: [16.15-17.16]
I20260709 12:21:59.615451  4628 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:59.614634  4558 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:21:59.656555 28335 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.034s	sys 0.040s
I20260709 12:21:59.666175 27664 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:59.669565 28336 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:21:59.664928 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:59.655122  4632 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.680658  4488 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:21:59.666672  3841 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:21:59.683637  4121 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:21:59.684123  4561 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.725s	user 0.011s	sys 0.001s
W20260709 12:21:59.685652  4407 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.788s	user 0.010s	sys 0.000s
W20260709 12:21:59.686983 27583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4137) took 2770 ms (client timeout 3302 ms). Trace:
W20260709 12:21:59.723551 27583 rpcz_store.cc:269] 0709 12:21:56.916755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.262166 (+345411us) service_pool.cc:224] Handling call
0709 12:21:59.686735 (+2424569us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b5400186fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:59.686877 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1752834304}
I20260709 12:21:59.666065  4567 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.692258  4588 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:21:59.698315 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.700030 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.731837 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.733574 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:21:59.671675 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:21:59.701556  3818 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.723145 28329 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 19 because replica has last-logged OpId of term: 17 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
W20260709 12:21:59.734902 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63322.7us
I20260709 12:21:59.735241  3818 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:21:59.734400 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.019s	sys 0.077s
W20260709 12:21:59.739827 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.019s	sys 0.077s
I20260709 12:21:59.700217  4593 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:21:59.754520 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:59.758570 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.751680  3818 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:21:59.763279 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:59.759539 27664 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:21:59.764490 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:21:59.764963 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:59.765790 27664 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:21:59.769027 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:59.771957 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:21:59.721918  4376 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.698s	user 0.013s	sys 0.000s
I20260709 12:21:59.780365 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:21:59.781567 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:21:59.776670 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:59.772886 27582 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:21:59.779961 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.782572 28332 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:21:59.785473 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56840.7us
I20260709 12:21:59.787992 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:21:59.782588  3818 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.789341 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.790409 28337 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 31 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
W20260709 12:21:59.790630 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.792841 27585 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:21:59.670533  4623 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.730379  4470 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.708546  4632 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:21:59.658339  4411 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.794087 27657 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:21:59.795032 27663 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:21:59.730010  4196 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:59.730685  4443 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:21:59.654247  4547 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.803507 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.759449  4055 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.804023 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:21:59.709760  4121 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.779894  4333 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:21:59.775421  4590 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.807476  4376 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:21:59.808239  4629 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:21:59.757081  4593 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.782565  4162 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.801000  4589 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:21:59.820322  4468 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:21:59.824716 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:21:59.750871  4588 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:21:59.705842  4558 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.751192  4407 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.825973 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:59.897871 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98684.3us
W20260709 12:21:59.828148 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1613) took 1706 ms (client timeout 2156 ms). Trace:
I20260709 12:21:59.785986  4198 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:21:59.826797 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3887) took 1529 ms (client timeout 1996 ms). Trace:
I20260709 12:21:59.831079 27664 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:21:59.897951 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:21:59.899191 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87739.4us
W20260709 12:21:59.901028 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.798396 27735 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:21:59.786230  4569 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:21:59.900134 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.906307 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:59.913476 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:59.923478 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:21:59.785714  4422 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.952988 27731 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:21:59.955303 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:21:59.960645 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:21:59.966125  4636 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:21:59.966686 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:21:59.804625  4579 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.247s
I20260709 12:21:59.969269 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:21:59.969614 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.836056  4633 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:21:59.832077  4559 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:59.842552  4546 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.841200  4493 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.967798  4623 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:21:59.846473  4020 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.204s
I20260709 12:21:59.854502  4470 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:21:59.905256  4483 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:00.131187  4381 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.007s	sys 0.029s
I20260709 12:21:59.838551  4576 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:00.131438 27658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.029s	sys 0.046s
W20260709 12:22:00.194310 27658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.033s	sys 0.047s
I20260709 12:22:00.194202 27655 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:22:00.129974 27660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.007s	sys 0.041s
I20260709 12:21:59.951808  4196 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.195887 27737 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:22:00.050992 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.022s	sys 0.070s
W20260709 12:22:00.197539 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.025s	sys 0.070s
I20260709 12:21:59.953339  4055 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.198755 27661 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:22:00.076642 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.964888  4443 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:21:59.957352  4590 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.065727 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1641) took 1938 ms. Trace:
I20260709 12:21:59.976756  4593 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:00.201634 27587 rpcz_store.cc:276] 0709 12:21:58.127486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.740453 (+612967us) service_pool.cc:224] Handling call
0709 12:21:58.740795 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.404896 (+664101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.065434 (+660538us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:00.065653 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1435347712}
I20260709 12:22:00.203764 27657 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:00.130273 28330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.023s	sys 0.025s
W20260709 12:22:00.215977 28330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.023s	sys 0.029s
I20260709 12:22:00.216900 27730 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:00.204847 27738 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:22:00.205433  4569 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.089092  4561 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:00.206167  4475 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:00.084404 28334 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 18 index: 16. (term mismatch)
I20260709 12:21:59.832623  4358 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.089257  4447 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:00.207665 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 252474us
W20260709 12:22:00.206507 27733 rpcz_store.cc:269] 0709 12:21:58.297918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.318670 (+ 20752us) service_pool.cc:224] Handling call
0709 12:21:59.826588 (+1507918us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:59.826708 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1489464704}
W20260709 12:22:00.192624 27656 rpcz_store.cc:269] 0709 12:21:58.121865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.137542 (+ 15677us) service_pool.cc:224] Handling call
0709 12:21:59.827935 (+1690393us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:21:59.828085 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1887897472}
I20260709 12:22:00.208902 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.209497 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.210208 27734 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:21:59.974167  4547 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:00.210577 27660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.010s	sys 0.044s
I20260709 12:21:59.905978  4567 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.065202 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3890) took 1595 ms. Trace:
W20260709 12:22:00.193547  4381 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.009s	sys 0.031s
I20260709 12:22:00.131466 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.212685 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3879) took 2106 ms. Trace:
W20260709 12:22:00.276116 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 469576us
I20260709 12:22:00.052953  4598 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:00.276201 27729 rpcz_store.cc:276] 0709 12:21:58.106010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.151523 (+ 45513us) service_pool.cc:224] Handling call
0709 12:21:58.151864 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.651672 (+1499808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.212476 (+560804us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54000a5fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:00.212634 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3435755776}
I20260709 12:22:00.220150 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:00.194016  4422 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.216766 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1361) took 1317 ms. Trace:
I20260709 12:21:59.970342  4333 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:00.227484 27737 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:21:59.952118  4592 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.316s
I20260709 12:22:00.207479  4470 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:21:59.953557  4560 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:00.262907 28334 pending_rounds.cc:85] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 15
W20260709 12:22:00.130169  4482 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.006s	sys 0.030s
I20260709 12:22:00.192955  4468 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:00.266389 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.805325  4399 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.388s
I20260709 12:22:00.267805 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:00.270489 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.204931  4579 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:00.271641 27732 rpcz_store.cc:276] 0709 12:21:58.470379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.526163 (+ 55784us) service_pool.cc:224] Handling call
0709 12:21:58.526598 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.782643 (+256045us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:58.782960 (+   317us) log.cc:844] Serialized 14 byte log entry
0709 12:21:59.399089 (+616129us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.399161 (+    72us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:21:59.399224 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.064917 (+665693us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54000a41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:00.064973 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.065131 (+   158us) inbound_call.cc:177] Queueing success response
0709 12:22:00.123038 (+ 57907us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b1800201700. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
Metrics: {"prepare.queue_time_us":1762,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":506,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":1732,"spinlock_wait_cycles":592283264,"thread_start_us":615696,"threads_started":1,"wal-append.queue_time_us":861817}
W20260709 12:22:00.281849 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73058us
W20260709 12:22:00.288493 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68440.5us
I20260709 12:22:00.234038 27730 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:00.292805 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.286996 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.324545 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.325091 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.325865 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.326253 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.327167 27730 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:00.290714 28330 rpcz_store.cc:276] 0709 12:21:58.899020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.899475 (+   455us) service_pool.cc:224] Handling call
0709 12:21:58.899881 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.900165 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:21:59.456550 (+556385us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.216389 (+759839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.216453 (+    64us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:00.216509 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.216584 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.216711 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8995,"prepare.queue_time_us":886436,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4955,"raft.queue_time_us":6697,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":21405,"thread_start_us":1306373,"threads_started":2}
I20260709 12:22:00.304261 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.328382 27737 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:00.329442 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.330138 27737 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:00.330554 27737 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:00.307933 28334 pending_rounds.cc:107] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 17.16
I20260709 12:22:00.303563  4623 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:00.268651  4358 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.311434 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.193341  4483 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.192822  4162 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.271575  4475 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:00.244189  4576 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:22:00.321244 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3227) took 2385 ms. Trace:
I20260709 12:22:00.289201  4411 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:00.322763 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55030.1us
I20260709 12:22:00.333817  4481 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:00.294780  4638 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:22:00.325254 27729 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:00.331478 28334 raft_consensus.cc:2889] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:00.262439  4501 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.308480 27734 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
W20260709 12:22:00.313148  4642 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:22:00.266942  4527 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:00.266379  4561 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:00.310755  4482 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.007s	sys 0.034s
I20260709 12:22:00.378443 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.414492  4559 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:00.332144 27737 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.445369 27737 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:00.302021  4598 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.337332 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:00.398428 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.290583  4567 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:00.398749 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.339218 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.323621  4579 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:00.394677 27660 rpcz_store.cc:276] 0709 12:21:57.936257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.196674 (+260417us) service_pool.cc:224] Handling call
0709 12:21:58.236265 (+ 39591us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:58.281667 (+ 45402us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:21:58.859381 (+577714us) log.cc:844] Serialized 14 byte log entry
0709 12:21:59.378705 (+519324us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:59.409004 (+ 30299us) log.cc:844] Serialized 52 byte log entry
0709 12:21:59.409218 (+   214us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:00.271439 (+862221us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.275838 (+  4399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.275946 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.320936 (+ 44990us) raft_consensus.cc:1685] finished
0709 12:22:00.320968 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.321161 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20769,"prepare.queue_time_us":1023160,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":22931,"raft.run_cpu_time_us":461,"raft.run_wall_time_us":51279,"spinlock_wait_cycles":17154816,"thread_start_us":1888183,"threads_started":3,"wal-append.queue_time_us":771677}
W20260709 12:22:00.332463  4594 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.889s	user 0.011s	sys 0.000s
I20260709 12:22:00.334796  4623 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:00.460176  4576 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:00.463716  4623 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:00.463846  4576 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.464484  4576 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:00.467574 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:22:00.332280  4591 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.007s	user 0.013s	sys 0.000s
W20260709 12:22:00.468302  4648 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:22:00.468650  4591 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.470048 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3242) took 1179 ms. Trace:
I20260709 12:22:00.470520 27658 rpcz_store.cc:276] 0709 12:21:59.290337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.290516 (+   179us) service_pool.cc:224] Handling call
0709 12:21:59.305320 (+ 14804us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.305658 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.305689 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.621373 (+315684us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:21:59.621777 (+   404us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.194821 (+573044us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:00.208818 (+ 13997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.208907 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.469669 (+260762us) raft_consensus.cc:1685] finished
0709 12:22:00.469691 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.469962 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":796,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":48,"spinlock_wait_cycles":104671616,"thread_start_us":572676,"threads_started":1,"wal-append.queue_time_us":754185}
I20260709 12:22:00.390169  4407 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.472844  4407 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.473541  4407 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:00.428648 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 98094.3us
W20260709 12:22:00.477270  4645 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:22:00.479059  4591 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.325729  4468 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.483664  4162 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.484804  4468 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.402278  4481 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.485662  4481 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:00.486968  4650 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:00.460632  4592 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:00.492427  4407 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:00.493011  4592 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:00.494346  4592 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.495806  4558 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:00.498764  4632 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.639s	user 0.007s	sys 0.001s
I20260709 12:22:00.437983 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.415463  4653 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.283159  4546 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:22:00.443254 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.500276 27729 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:22:00.501314 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58327.5us
I20260709 12:22:00.427515  4651 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260709 12:22:00.502461 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.503130  4376 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.503561  4472 raft_consensus.cc:2764] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:22:00.312988  4020 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
I20260709 12:22:00.504379  4472 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:00.505843  4481 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.417328  4121 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.438696  4196 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:00.451311 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52999.8us
I20260709 12:22:00.508188 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:00.448709 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 111468us
I20260709 12:22:00.508689 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.509388 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.509635  4447 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.510331  4447 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:00.510550 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.471632 27732 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:00.511233  4447 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:00.511858  3818 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:22:00.460414  4567 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.512650  4411 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:00.513376  4411 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:00.514614  4567 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.514343  4411 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.479727  4475 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.480958  4579 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.515636  4523 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:00.481642  4055 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.516357  4523 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.517268 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.517010  4523 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.518221  4546 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:00.518852  4546 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:00.520035 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.516321  4579 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:00.521382 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.522541 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.438925  4501 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.525317  4501 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.525939 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1459) took 1491 ms. Trace:
I20260709 12:22:00.521010 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.529702  4358 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.531013 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.435771  4333 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.532096 27730 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:00.534837 27730 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:22:00.446087  4527 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.535080 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1505) took 1050 ms. Trace:
I20260709 12:22:00.535555 27735 rpcz_store.cc:276] 0709 12:21:59.484694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.487007 (+  2313us) service_pool.cc:224] Handling call
0709 12:22:00.534913 (+1047906us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:00.535022 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":684419968}
I20260709 12:22:00.536983  4527 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.537111  4482 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:22:00.538493 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.538726 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.440250  4590 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.452663  4559 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.319168  4470 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.095s
I20260709 12:22:00.432313 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:00.473019  4647 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:22:00.503703  4623 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.539295 27581 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.562479  4481 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.547535 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.577878  4569 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.545506 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:00.580076 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:00.580302 27660 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 24: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:00.540376 27586 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:00.539746 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.550308  4598 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.541437 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.545099 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:00.582695 27589 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.582657 27656 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 19.
I20260709 12:22:00.547015 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.583753 27734 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:00.584369 28330 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:22:00.568569  4572 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:00.585147  4572 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:00.567241  4588 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.586436  4588 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:00.586223  4572 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.568110 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:00.581660  4623 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.588307  4623 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:00.579401  4559 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.542989  4483 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.574806 27583 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
W20260709 12:22:00.573839  4641 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.448s	user 0.002s	sys 0.000s
I20260709 12:22:00.588778  4623 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:00.549136  4628 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.849s	user 0.005s	sys 0.006s
I20260709 12:22:00.592676  4597 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.593307  4628 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.592875  4623 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:00.594595 27584 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.598780  4422 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:00.603432 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.608755 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.609475 28336 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:00.499275  4632 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.508751  4561 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.503721  4376 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:00.575266 27535 auto_rebalancer.cc:893] Removing replace marker failed: Replica 3f90188458da46f6ba138d5f82c55b69 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:22:00.611944 27535 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 12:22:00.612253 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:22:00.613121  4376 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.614192  4535 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:00.614646  4535 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:00.615545  4535 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.616072  4632 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.616842  4535 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:00.617049 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1503) took 1133 ms. Trace:
I20260709 12:22:00.617447 27736 rpcz_store.cc:276] 0709 12:21:59.484104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.484868 (+   764us) service_pool.cc:224] Handling call
0709 12:21:59.834321 (+349453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.834563 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.834589 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.834640 (+    51us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:21:59.834689 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.834767 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.595449 (+760682us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b54002e81d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:00.595581 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":802900352}
I20260709 12:22:00.512022  4447 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:00.619701  4447 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:00.620585  4447 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.621820  4447 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:00.557881  4593 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.500341  4653 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.622833 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:00.627619  4598 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:00.628587 27733 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:00.631255 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:00.631954 28330 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:00.634575 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.636304 28328 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
W20260709 12:22:00.556164 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.638993  4598 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.639569  4598 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.523123 27738 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:00.645339  4628 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.562299  4443 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.647696  4472 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:22:00.648617  3841 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.918s	user 0.008s	sys 0.004s
I20260709 12:22:00.649571 28333 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:22:00.651217  4407 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.651823  4407 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.657852 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.659083 27584 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:22:00.520036  4546 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.526343 28335 rpcz_store.cc:276] 0709 12:21:59.034827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.081660 (+ 46833us) service_pool.cc:224] Handling call
0709 12:21:59.082106 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.082386 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:21:59.676121 (+593735us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.132365 (+456244us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.515286 (+382921us) spinlock_profiling.cc:243] Waited 75 ms on lock 0x7b1800224500. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:00.525135 (+  9849us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b1800224500. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:22:00.525356 (+   221us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.525568 (+   212us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.525621 (+    53us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:00.525699 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.525770 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.525883 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88195,"prepare.queue_time_us":727556,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":87,"raft.queue_time_us":479,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":80309,"spinlock_wait_cycles":318041344,"thread_start_us":1006765,"threads_started":2,"wal-append.queue_time_us":655994}
I20260709 12:22:00.676388 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.677148 28334 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:00.690146 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.695218  4482 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.696105  4481 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.696843  4595 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:00.697431  4595 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.698815  3818 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:00.699462  3818 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:00.696833  4481 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.700032  3818 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:00.699041  4482 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.715356 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.716231 27583 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.717965 27588 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:00.719439 27583 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:00.727990  4020 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:00.729229  4020 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:00.729848  4470 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:00.730294  4470 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:00.731191  4470 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.732514  4470 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:00.735224  4651 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:00.735826  4651 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:00.737027  4651 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.739209  4651 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:00.739290  3841 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:00.740931 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.740494  3841 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.745754 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.746407 28334 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
W20260709 12:22:00.748620 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.749562 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:00.761024 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 222385us
I20260709 12:22:00.762239 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:00.762684 27657 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.007s
I20260709 12:22:00.763132 27657 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:00.763300  4443 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:22:00.763876 27481 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:00.765375  4407 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.766124 27481 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:00.767606 27661 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.008s
I20260709 12:22:00.791803 27661 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:00.770296 28331 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:22:00.778225  3841 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.771451  4376 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:00.776468 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 130) took 31674 ms (client timeout 15000 ms). Trace:
W20260709 12:22:00.791999 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.799708 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.800606 28334 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:00.793625 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4001) took 1031 ms. Trace:
I20260709 12:22:00.802861 27661 rpcz_store.cc:276] 0709 12:21:59.762612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.787114 (+ 24502us) service_pool.cc:224] Handling call
0709 12:22:00.793536 (+1006422us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.768093  4481 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.770370  4623 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:00.805544 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1463) took 1769 ms (client timeout 1749 ms). Trace:
W20260709 12:22:00.806205 28331 rpcz_store.cc:269] 0709 12:21:59.036373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.039431 (+  3058us) service_pool.cc:224] Handling call
0709 12:22:00.805296 (+1765865us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:00.805452 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3852141952}
I20260709 12:22:00.808509 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.809748 28336 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:00.811108 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.794204 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.794667 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.787107 28330 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:22:00.797506 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.797402 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:00.782604 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 159832us
I20260709 12:22:00.770946  4572 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:22:00.817042 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 491196us
I20260709 12:22:00.825094 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.826154 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.822927 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:00.826705 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:00.827080 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.823596 27587 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:00.818975 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.816942 27584 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.772322  4593 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.832835 27587 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:00.833885 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:00.834321 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.834959 28333 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:00.828603 27588 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:22:00.830008 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.792860  4445 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.829831  4643 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:00.816638  4579 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:00.844010  4445 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.844373 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:00.844870  4579 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.845715 27583 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:00.835906 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:00.840243 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.848970 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.849337  3841 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:00.842466  3818 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.850595 27737 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:00.857877 27702 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:00.834481 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4002) took 1071 ms. Trace:
I20260709 12:22:00.858417 27702 leader_election.cc:403] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:22:00.859474 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.852115 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:00.854342 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.860347 27589 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:00.798849 27487 rpcz_store.cc:269] 0709 12:21:29.102213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.103476 (+  1263us) service_pool.cc:224] Handling call
0709 12:22:00.776330 (+31672854us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:52.456491 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:52.486689 (+ 30198us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:52.486746 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:21:52.486774 (+    28us) tablet.cc:662] Decoding operations
0709 12:21:52.526033 (+ 39259us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:52.526130 (+    97us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:52.526155 (+    25us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:21:52.527581 (+  1426us) tablet.cc:701] Row locks acquired
0709 12:21:52.527643 (+    62us) write_op.cc:260] PREPARE: finished
0709 12:21:52.527855 (+   212us) write_op.cc:270] Start()
0709 12:21:52.528020 (+   165us) write_op.cc:276] Timestamp: P: 1783599712527783 usec, L: 0
0709 12:21:52.528056 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 12:21:52.528788 (+   732us) log.cc:844] Serialized 6292 byte log entry
0709 12:21:53.570601 (+1041813us) op_driver.cc:464] REPLICATION: finished
0709 12:21:53.719532 (+148931us) write_op.cc:301] APPLY: starting
0709 12:21:53.719670 (+   138us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:54.451956 (+732286us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:54.452081 (+   125us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:55.144651 (+692570us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b1800240c40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9a0870ec 00007f0e9a0d3c0a 00007f0e9a0d34f7 00007f0e9a0d3490 00007f0e9a08af48 00007f0e9dd9f1bb 00007f0e9dde65d4 00007f0e9dc928da 00007f0e9dc953cb
0709 12:21:55.166602 (+ 21951us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:55.166964 (+   362us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:21:55.167032 (+    68us) write_op.cc:312] APPLY: finished
0709 12:21:55.239775 (+ 72743us) log.cc:844] Serialized 148 byte log entry
0709 12:21:55.404631 (+164856us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:55.454839 (+ 50208us) write_op.cc:454] Released schema lock
0709 12:21:55.462060 (+  7221us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:55.462515 (+   455us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:55.462552 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8117857,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":142722,"apply.run_cpu_time_us":47425,"apply.run_wall_time_us":1742972,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71007,"cfile_init":1,"lbm_read_time_us":592,"lbm_reads_lt_1ms":4,"num_ops":16,"prepare.queue_time_us":269672,"prepare.run_cpu_time_us":72159,"prepare.run_wall_time_us":314562,"raft.queue_time_us":247521,"raft.run_cpu_time_us":15898,"raft.run_wall_time_us":105834,"replication_time_us":1042326,"spinlock_wait_cycles":33168640,"thread_start_us":638405,"threads_started":5,"wal-append.queue_time_us":451897,"wal-append.run_cpu_time_us":24758,"wal-append.run_wall_time_us":2725614}]]}
W20260709 12:22:00.867475 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.858660 27663 rpcz_store.cc:276] 0709 12:21:59.762954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.898662 (+135708us) service_pool.cc:224] Handling call
0709 12:21:59.967736 (+ 69074us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.968205 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.968246 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.968876 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.333367 (+364491us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:00.812797 (+479430us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.834174 (+ 21377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.834256 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.834299 (+    43us) raft_consensus.cc:1685] finished
0709 12:22:00.834314 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.834429 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18671,"prepare.queue_time_us":80916,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":159010,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":105969,"thread_start_us":655291,"threads_started":2,"wal-append.queue_time_us":371601}
I20260709 12:22:00.860260 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.872099 27581 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:00.829474  4481 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:22:00.872773 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.874601 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.879211 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.780332  4020 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.847158  4579 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.837149  4590 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:00.916227 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67320.1us
I20260709 12:22:00.916750 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:00.917676 28330 raft_consensus.cc:2410] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:22:00.836149  4399 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:00.921147 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.921929 27733 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:00.923768 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.924489 27736 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:00.925697 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.927618 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.928202 27732 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:00.885128 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.929224 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.930359 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:00.886104  3818 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.884615  4445 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:00.876621  4547 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.588s	user 0.006s	sys 0.009s
I20260709 12:22:00.877271  4443 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:22:00.884994 28335 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:00.948499 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.948683 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.949306 27586 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.887722 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.950021 27581 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:00.884959 27481 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 21, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 21 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:00.891410  4428 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:00.859684  4527 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:00.816835  4376 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.951867 27733 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:00.953193 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.953822 27733 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:00.955013 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:00.956187 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:00.956904 27659 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.012s	sys 0.008s
I20260709 12:22:00.957053 28331 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:00.957255 27659 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.013s	sys 0.008s
I20260709 12:22:00.945564  4493 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.885377  4628 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:00.891999 27553 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:00.896250  4481 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:00.895799 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.896477 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:00.900794  4523 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.862843  4482 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:00.972340 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.911715  4656 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:00.945887  4422 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.848054  4643 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:00.871199  4593 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:00.994303  4643 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:00.932767 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:00.995553 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62901.8us
I20260709 12:22:00.996652 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.997042 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3236) took 1850 ms. Trace:
I20260709 12:22:00.931640  4590 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.997783 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:00.934775 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:00.998838 27582 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:00.999400  4590 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:00.999720 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.000399 27582 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:22:01.000844  4589 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:01.001266 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.001389  4589 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.935782 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:01.002563 27734 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:01.003371 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:00.937263 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.004235 27738 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:01.005095 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.005419  4020 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:01.006037 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.006892 27738 raft_consensus.cc:2410] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:22:00.970116  4653 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.938351 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.007364 27730 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:22:00.939347 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55565.9us
I20260709 12:22:01.007695  4422 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.008198 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:01.009810  4422 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:00.919754  4162 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:00.932492 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:00.956015  4607 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.012109 27589 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:00.877574  4591 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:00.933403 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:00.947576  3818 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.014102  3818 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.961548 27736 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:00.968752 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.972151 28334 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:00.908162  4489 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:22:01.018966 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.006s	sys 0.018s
I20260709 12:22:01.021273 27738 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
W20260709 12:22:01.021543 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.006s	sys 0.018s
I20260709 12:22:01.023556 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.024541 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:00.975035 27481 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 23 to 25, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 25 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:00.911851 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.028800 27737 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:00.979189 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.980914 27704 leader_election.cc:400] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:22:01.029657 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.030395 27704 leader_election.cc:403] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:22:00.982203 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.030606 27736 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:00.949538  3988 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:00.985761 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:00.982012  4482 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.967819  4445 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:00.987416  4481 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:00.988490  4656 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:00.994405  4593 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.934490 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.016247 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:01.066504 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.067509 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:01.068162  4656 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:01.068941 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52996.6us
I20260709 12:22:01.012545  4569 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:00.957589  4579 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.063s
I20260709 12:22:01.071641 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:00.978513  4055 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:01.018499  4527 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:22:01.018960  3818 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.034438 27733 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:01.035195 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.039445 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.086138 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:00.971184  4628 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:01.029716 27586 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:00.971499  4493 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:01.052420 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71672.1us
I20260709 12:22:00.997471 27659 rpcz_store.cc:276] 0709 12:21:59.146537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.146715 (+   178us) service_pool.cc:224] Handling call
0709 12:21:59.147201 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.645137 (+497936us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:00.133721 (+488584us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.428093 (+294372us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.957588 (+529495us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.958050 (+   462us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.961077 (+  3027us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:00.961132 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.961205 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.996814 (+ 35609us) raft_consensus.cc:1685] finished
0709 12:22:00.996835 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.996981 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21910,"prepare.queue_time_us":684984,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":3215,"raft.queue_time_us":754501,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":280,"thread_start_us":1794052,"threads_started":4,"wal-append.queue_time_us":699630}
I20260709 12:22:01.090898 27704 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:01.092020 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.058501  4547 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:01.060361  4589 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:01.063195  4660 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:01.064095  4445 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.066574  4593 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:01.017907  4658 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:00.983579  4598 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:01.120915  4445 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.073204 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:01.073344 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:01.070894 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:01.124455 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53674.9us
I20260709 12:22:01.126010 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:01.077677 27589 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:01.127455  4572 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:01.127609  4593 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.128247  4470 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:01.128925 27584 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:01.128912  4470 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:00.958264  4594 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.092s
I20260709 12:22:01.010986  4595 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:01.058507  4482 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.092175 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:01.133082 27735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.016s	sys 0.024s
I20260709 12:22:01.099128  4523 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.133613 27735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.016s	sys 0.024s
I20260709 12:22:01.019260  4607 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.049397  4588 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.049296  4640 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:01.073603  4162 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:01.073136  4590 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:22:01.098587  4628 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:01.133808 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:01.095762 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.131156 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:01.136814 27729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.019s	sys 0.023s
W20260709 12:22:01.167485 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 253871us
I20260709 12:22:01.128818  4572 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.170064 27481 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 26, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 26 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:01.173233 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:01.208813 28334 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:01.134150  4470 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:01.209146 27481 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 27 to 30, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 30 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:22:01.211148 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 80061us
I20260709 12:22:01.225296 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:01.138051  4445 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:22:01.134399  4358 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:01.209486 27729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.023s	sys 0.031s
W20260709 12:22:01.178907 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45171.7us
W20260709 12:22:01.229475 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 448219us
W20260709 12:22:01.232436 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.133123  4535 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
I20260709 12:22:01.210498  4572 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:01.176426  4623 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:01.232438 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:01.232867 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:01.243250 27481 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:01.209903  4656 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:01.208211  4594 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:01.251708  3841 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:01.265652  3841 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:01.230587  4445 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:01.266671 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.266530  3841 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.226516  4567 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:01.210987  4640 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:22:01.178331  4333 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.069s
I20260709 12:22:01.264772 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:01.165405  4523 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.270231 27587 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:01.270509 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.285231 27661 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:22:01.277179 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.169212  4588 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.279454 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.280160 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.288432 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.288463 27656 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:22:01.241309  4653 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:22:01.265868  4560 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:22:01.299472 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 407558us
I20260709 12:22:01.299114 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:01.288558  4594 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.243795  4358 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.308228 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.262755  4663 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.269505  4445 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:22:01.312880 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:01.313066 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:01.381222 28331 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:01.266017  4399 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:01.314457 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:01.384338 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 70005us
I20260709 12:22:01.285156  4662 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.344020 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.319950 28335 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:22:01.342682  4523 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.335988 27657 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:01.388617 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:01.345796 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:01.392948 27662 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:01.315006 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.396420 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:01.397642 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.361385 27587 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:01.321812 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3379) took 1371 ms. Trace:
I20260709 12:22:01.345366 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.399978 27664 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:01.399312 27729 rpcz_store.cc:276] 0709 12:21:59.951134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.536169 (+585035us) service_pool.cc:224] Handling call
0709 12:22:00.536549 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.555617 (+ 19068us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.555657 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.556165 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.231191 (+675026us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:01.231234 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.231348 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.321600 (+ 90252us) raft_consensus.cc:1685] finished
0709 12:22:01.321619 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.321739 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16414,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":674717,"threads_started":1,"wal-append.queue_time_us":754479}
I20260709 12:22:01.299301  4660 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:01.375900 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.337036  4588 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:01.402544 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:01.342705  4664 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:22:01.349839 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:01.314988 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.406911 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.407279 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:01.407851  4607 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.408005 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:01.391534  4593 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:01.408584 27658 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:01.408999  3988 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:22:01.307828  4407 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:01.409662  4407 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.410068  4407 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:01.411441  4660 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:01.412595  4594 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:01.378813  4598 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:01.345767  4640 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:01.413138  4598 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:01.413451  4640 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:01.387678  3818 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:01.336230  4659 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:01.343868  4482 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:01.340128  4560 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:01.339273  4493 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:01.419385 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:01.346519 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.420014 27657 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:01.291277  4376 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:01.412377  4660 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.424976 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:01.426402 27660 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:22:01.429162 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 327921us
I20260709 12:22:01.416385  4523 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:22:01.415448 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65740.9us
I20260709 12:22:01.417197 27663 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:01.391283  4547 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:01.424631 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:01.447697 27664 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:01.315233  4470 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.436882 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:01.414937  4659 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:01.452965 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:01.340013 27481 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 25, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 25 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:01.668097 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:01.670406 27664 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:01.627914 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:01.637719 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 249006us
W20260709 12:22:01.383210 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70384.1us
I20260709 12:22:01.649259 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:01.675910 27481 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 22, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 22 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:01.684939 27481 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 13 to 23, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 23 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:01.504484 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58848.5us
W20260709 12:22:01.501806 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.693174 27729 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:22:01.695222 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.695457 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:01.702832 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 436071us
I20260709 12:22:01.523923 28334 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:01.704185 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.511529 28331 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:22:01.532140  4020 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.526s	user 0.004s	sys 0.008s
I20260709 12:22:01.770033  4489 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:01.771982 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.773160 27586 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:01.550884  4358 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:01.775940 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.563577  4407 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.777784  4592 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.414860  4411 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:22:01.780373  4407 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.779901 27481 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:01.553476  4445 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.786046 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.503374  4523 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:01.402877  4546 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:22:01.510526  4547 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:01.836192  4547 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.837749  4468 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.544445  4560 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.838672  4468 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.566573  4572 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:01.547710  4576 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.083s	user 0.014s	sys 0.003s
I20260709 12:22:01.426169  4588 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.082s
I20260709 12:22:01.532603  4567 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:01.425277  4590 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:01.887638  4590 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:01.888223  4590 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:01.890714 28334 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:22:01.883074  4567 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:01.538949  4591 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:01.510692  4594 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.893857  4594 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:01.894644 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.895838  4468 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.414036  4598 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.571836  4628 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.414350  4640 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:01.547398  4643 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.552s	user 0.010s	sys 0.000s
I20260709 12:22:01.988217  4643 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.990880  4643 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.543390  4659 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.997018  4428 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.005s
I20260709 12:22:01.959607 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:01.951750  4628 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.993294 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:02.000316 27658 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.591s	user 0.003s	sys 0.020s
I20260709 12:22:01.403059  4662 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:01.543057  4665 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 12:22:01.735633 27731 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:22:01.754256 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.756575 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:01.758147 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.759392 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.759953  4475 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.761839  4055 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.762507  3841 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.766777  4678 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.776018 27657 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
I20260709 12:22:01.553434  4527 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:01.778801  4592 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.066947  4592 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.806330  4333 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:02.068126 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3380) took 2116 ms. Trace:
I20260709 12:22:02.068313  4333 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.807804  4589 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:01.808542  4682 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:01.810125  4162 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:22:02.070782  4527 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:02.071017  4399 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:22:02.071539  4162 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:02.072120  4162 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:02.069963  4682 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.073547  4682 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:02.073276  4162 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.074957  4162 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:01.811663 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:01.794893  4445 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:01.817799  4674 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:02.086709  4674 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:01.817962  4623 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.817843 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:01.819084 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:01.828598 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:01.829473 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:01.831027 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:01.833109  4523 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:01.803787 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 116927us
I20260709 12:22:01.782589 27481 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 22. New cstate: current_term: 22 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:01.862421 28328 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
W20260709 12:22:01.864802 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.868006 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.869017 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:01.820552  4535 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:01.880115  4576 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:01.898402 27736 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:01.930176  4594 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:01.937674  4590 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:01.942299 27662 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.014s
I20260709 12:22:01.947456 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:01.956372 28333 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 19. Current term is 20. Ops: []
W20260709 12:22:01.949961 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92216.9us
I20260709 12:22:01.883176  4572 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:01.950758 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 523612us
I20260709 12:22:01.983762 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:01.568215  4651 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.828s	user 0.006s	sys 0.011s
I20260709 12:22:01.851980  4489 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:02.117295 27584 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:02.121856  4489 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.999209 28332 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:02.123469  4445 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:02.124251  4445 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.126137 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.126698  4445 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:02.127194 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.002282 27658 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:02.128700 27583 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
W20260709 12:22:02.009420 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59611.4us
I20260709 12:22:01.952075  4567 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.129900 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.131752  4567 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.038620 27657 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 15
I20260709 12:22:02.132576 27657 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 29.16
I20260709 12:22:02.132926 27732 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:02.133275 27588 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:02.132932 27657 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:02.133934  4576 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.049517  4055 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.136634  4055 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.138154 27733 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:02.050347  4678 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.126958 27582 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:02.141510  4678 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.053830 28331 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:02.142169  4659 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:02.049878 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.063971  4591 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:02.139811 27730 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:02.143177  4591 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:02.144547  4591 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.064246  3841 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:01.811666  4658 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:02.069948  4589 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.152878  4589 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:02.069598  4333 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:02.068611  4662 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:02.155676 27587 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:02.083886 27734 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:02.156366 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3396) took 1208 ms. Trace:
I20260709 12:22:02.156857 27732 rpcz_store.cc:276] 0709 12:22:00.935052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.938135 (+  3083us) service_pool.cc:224] Handling call
0709 12:22:02.143102 (+1204967us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001d38d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:02.143291 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2586569088}
W20260709 12:22:02.158113 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3400) took 1221 ms (client timeout 1542 ms). Trace:
I20260709 12:22:02.159536 27738 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:02.167451 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.114465 27662 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:02.129870 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.129861 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.170125  4662 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:02.171762  4658 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:02.172230  4658 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:02.172380  4576 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:02.172936  4576 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:02.173354  4576 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:02.173657  4658 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.175308 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:02.176219  4535 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:02.176671  4535 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.177086  4535 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:02.178406  4428 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:02.179702  4527 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.180383  3818 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:02.180776  4428 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:02.181161  4428 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:02.181299  4527 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:02.182207  4428 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:02.084751 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 335078us
I20260709 12:22:02.085579  4560 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:02.087262  4674 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:02.114895 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 284141us
I20260709 12:22:02.117241 28336 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:02.119127  4590 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:02.119282  4628 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:02.111929  4623 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.120033 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:01.390249  4663 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.228744  4628 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:02.236769 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.225279 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:02.244187 27588 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:02.248479 27733 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:02.249565 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.250860  4376 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:02.042611  4475 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:02.048190 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.158687 27734 rpcz_store.cc:269] 0709 12:22:00.936838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.003118 (+ 66280us) service_pool.cc:224] Handling call
0709 12:22:02.157872 (+1154754us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:02.158055 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2050246400}
I20260709 12:22:02.081354  4598 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:02.192898 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.194839 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.199812 28332 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:02.206525 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.211133 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.207094  4055 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.068635 27735 rpcz_store.cc:276] 0709 12:21:59.951548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.536381 (+584833us) service_pool.cc:224] Handling call
0709 12:22:00.536715 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.553782 (+ 17067us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.553832 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.554490 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.134128 (+579638us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:01.134187 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.134309 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.067616 (+933307us) raft_consensus.cc:1685] finished
0709 12:22:02.067661 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.067901 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16615,"raft.queue_time_us":18141,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":99,"thread_start_us":579256,"threads_started":1,"wal-append.queue_time_us":735623}
I20260709 12:22:02.216563 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.217154  4594 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:02.218055 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:02.219465 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:02.221201  3841 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.221760 27584 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:02.272478 27738 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:02.205643 27481 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 21 to 22, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 22 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:22:02.228210 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 108318us
I20260709 12:22:02.275094 28330 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:02.233311  4663 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.251272  4376 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.268359 27730 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:02.225636 28336 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:02.230378  4162 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:02.277323 27581 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:02.285777  3841 raft_consensus.cc:2764] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:22:02.287004 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70619us
I20260709 12:22:02.191946  4682 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.288738 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.286547 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:02.278324  4376 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:02.291215  4055 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.282420  4162 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.299731 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:22:02.296236 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.796s (SENT) [suppressed 11 similar messages]
I20260709 12:22:02.306651 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.319167  4643 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:02.323055  4643 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:02.323712  4643 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:02.327654  4576 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.331444  3818 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:22:02.331738 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 141642us
I20260709 12:22:02.332341  3818 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:02.333580 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.333772  3818 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.341216  4475 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:02.306437  4428 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:02.352110  4640 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.353277  4640 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.315778 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:02.356839 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50289.8us
I20260709 12:22:02.357949 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:02.280457  4651 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:22:02.359925 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.278201  4628 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.360971 27734 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:02.367841  4567 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:02.368995  4567 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:02.372730  4055 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:02.378575 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:02.379631 28332 raft_consensus.cc:2410] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:22:02.386291  4588 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:22:02.386898  4588 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:02.392244  4535 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.395535 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.302059  4162 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:02.307178 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:02.401899 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 115535us
I20260709 12:22:02.403577  4662 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:02.405838  4658 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:02.408907  4623 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.360122  4651 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:02.411736  3841 raft_consensus.cc:2764] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
W20260709 12:22:02.412418 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78945.8us
I20260709 12:22:02.409945  4651 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:02.415508  4659 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.416030  4651 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.417410  4651 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:02.419099 28330 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:02.304620 27481 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 23 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 25 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:02.447533 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:02.449666 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:02.455412 27481 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 26, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 26 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:02.457577 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.456847  4623 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.478149  4482 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.478032  4588 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.478905  4482 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.479689  4588 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:22:02.478039  4623 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.481621  4482 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.482030  4623 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.482228  4558 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:02.482946  4558 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.485283 27733 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:02.485308  4558 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.487790 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:02.488621 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:02.490442 27736 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:22:02.464408 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.495013 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.495539 27586 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:22:02.505312 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55735.5us
I20260709 12:22:02.509488 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:02.512485 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.416046  4659 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.520035 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.520499 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:02.521306 27587 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:02.522022 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:02.529066 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.525910 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:02.499579  4592 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:02.506896  4475 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.534164 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.534705 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:02.511009  4640 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.518890 27481 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:02.539523  4055 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:02.544638  4055 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:02.514865  4535 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.549039 27588 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:02.551532  4592 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:02.554255 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.554867 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.532512  4572 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:02.556883 27730 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:02.554733  4653 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.535611  4558 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:02.562887 28334 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:02.571147 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 274918us
W20260709 12:22:02.567974  4698 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:22:02.558513  4475 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.580837  4475 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.582069 27581 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:02.576020 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.588744 27658 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:02.533064  4589 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.570683  4376 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.565421  4558 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.539911  4333 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.572065  4572 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:02.607903  4558 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:02.572670  4482 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:02.611989  4376 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.612859 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:02.628381 27657 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:02.638933 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.649525 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:02.650571 27657 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25.
I20260709 12:22:02.666224 27588 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:02.675472 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.689607 27587 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:02.694389 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.695564 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:02.701862 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.704176 27481 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 17 to 20, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 20 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:02.633260  4623 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.648587  4482 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:02.739661 27658 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
W20260709 12:22:02.731774 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 279968us
I20260709 12:22:02.654047  4590 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.596211  4653 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.708971  4376 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.741868 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:02.742224 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:02.638000  4535 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:02.751091 27656 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:22:02.721746 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.751577 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.730554  4678 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:02.763621 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:02.753934 27481 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 14 to 16, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 16 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:02.783506  4709 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.257s	user 0.000s	sys 0.001s
I20260709 12:22:02.783555  4678 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:02.788354  4678 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.793355  4653 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:02.794090  4682 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.797158 28330 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:02.812084  4482 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.819849  4576 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:02.796382  4674 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.604s	user 0.012s	sys 0.000s
I20260709 12:22:02.737666  4623 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.830008 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.766372  4659 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.831022 27657 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:02.839444 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:02.839609 28334 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:02.829859  4651 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:02.799757  4399 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:02.837620 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.847736 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.864981  4623 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.864529  4162 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.865582  4399 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:02.866279 27662 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:02.866781  4399 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.869537 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:02.870527 27657 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:02.872189  4589 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.798892  4590 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.873303 28328 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 20. (term mismatch)
I20260709 12:22:02.873725 28328 pending_rounds.cc:85] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 19
I20260709 12:22:02.873970 28328 pending_rounds.cc:107] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 31.20
I20260709 12:22:02.874205 28328 raft_consensus.cc:2889] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:02.875505  4598 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.001s
I20260709 12:22:02.766119  4376 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.875835 27481 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 19 to 26, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 26 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:02.876564  4493 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.884063  3841 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.888887 27587 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:22:02.802093  4535 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:02.891732 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126352us
I20260709 12:22:02.888991 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:02.891024  3841 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.767195  4055 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.912863 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:02.913692 28330 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:22:02.915354  4489 raft_consensus.cc:2764] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:22:02.916419  3841 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.894413  4493 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.917431 28331 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:02.919974  4376 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.920549 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.920863 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.921700 27661 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:22:02.922499  4333 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.923580 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.924423 27659 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:02.925673 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.928429 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.928750 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.929680 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.930039 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.930323 27660 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:02.930748 27656 raft_consensus.cc:2376] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24. Re-sending same reply.
I20260709 12:22:02.931211 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:02.931779 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.932129 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.933130 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:02.933279 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.933746 27663 raft_consensus.cc:2376] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24. Re-sending same reply.
I20260709 12:22:02.934010 27661 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:02.931836 27660 raft_consensus.cc:2376] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32. Re-sending same reply.
I20260709 12:22:02.934726 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:02.931753 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.936036 27656 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:02.936834 27663 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:02.940028  4527 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.758s	user 0.013s	sys 0.002s
I20260709 12:22:02.940092  4659 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:02.941278 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71858.9us
I20260709 12:22:02.940987  4659 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.944202 27657 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:22:02.949343 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.840545  4674 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.889724  4663 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:22:02.895696  4640 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.961175 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.962288 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:02.799178  4572 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.967312 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:02.971854 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.974856 28334 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:22:02.969945 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 278167us
I20260709 12:22:02.958477  4663 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:02.973834 27664 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:02.979475 27657 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:22:02.984444  3818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.005s	sys 0.011s
W20260709 12:22:02.984791  3818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.005s	sys 0.011s
W20260709 12:22:02.980458 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 341584us
I20260709 12:22:02.977190 27659 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 31 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 15.
W20260709 12:22:02.979450 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 327125us
I20260709 12:22:02.952912  4651 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:02.986208 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.979239 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:02.994478  4358 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.995195  4358 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:02.998256  4358 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:02.998979  4358 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:02.999531  4594 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:03.000128  4594 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.000568  4594 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:03.007936  4358 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:03.009667  4358 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.018463 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:03.021144 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61618.7us
I20260709 12:22:02.974263  4162 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.025964 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:02.993394  4489 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:22:03.027616 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.029006 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:03.029665 27655 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:03.029829 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:03.030380 27664 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:03.031416  4593 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.078s
I20260709 12:22:03.035763 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.036489 27660 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:03.029129 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:03.041446  4558 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:03.044080  4721 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:03.045109 27664 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:22:03.046933  4588 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.000s
I20260709 12:22:03.048823  4674 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:02.969290  4640 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:03.054174  4658 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.000s
I20260709 12:22:02.987778  4663 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:03.056102 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.060668  4659 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.061229  4590 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.061295  4651 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.062763  4535 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.063315  4333 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:03.064898  4527 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.003s
W20260709 12:22:03.069536 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.076109 27663 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
W20260709 12:22:03.077539 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90006.2us
W20260709 12:22:03.064499  4643 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.740s	user 0.013s	sys 0.000s
W20260709 12:22:03.084558 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.077315  3841 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.099463 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.105188 27736 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:03.100253 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.100909 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.101078 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.101730 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:03.109543 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.109566  3841 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.104169  4333 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.115605 27735 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:03.119791 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.107981  4643 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.114243  4682 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:22:03.114307  4674 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:03.119282  4333 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:03.116117  4590 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.117472  4588 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:03.139794  3841 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:03.148321 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.127298  4558 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:03.158036 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 199426us
I20260709 12:22:03.129925  4674 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.129751  4592 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.149199 27735 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:03.134119  4593 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:03.128551  4682 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:03.185382 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 92689.1us
I20260709 12:22:03.126730  4594 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.188125 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.188511 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:03.189082 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.189020 28336 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:03.190035 27733 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:03.180545  4674 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:03.195343 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.183576  4682 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:22:03.198824 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.137660  4590 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.204679 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:03.188526  4399 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:03.200762  3818 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:03.150882  4588 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.207583 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.192386  4593 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:03.199702  4558 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:22:03.219188  4640 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.239207 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.397531 27737 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:03.362924 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.364395 27729 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:03.398118 27731 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:22:03.372388  4594 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:03.238632  4728 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.372676  3841 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:03.373298  4588 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:03.375118 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 214343us
I20260709 12:22:03.364903  4663 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:03.376467 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 169934us
I20260709 12:22:03.427479 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.214389 27734 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:03.428591 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:03.383950 27733 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:03.387919 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.389889 27735 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:22:03.389878 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 191239us
I20260709 12:22:03.394471 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:03.392490  4674 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.400527  4399 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.400822 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.401995  4643 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.391079  4493 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.411862 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:03.438521 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:03.438897 27581 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:03.413101 27587 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:03.413928 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.396436  4558 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:03.440855  4558 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:03.442070 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:03.443019  4399 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.443691 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:03.443928 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.444346 27582 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:03.414716 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:03.447505  4594 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.447677 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:03.448405 28328 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:03.448639 28332 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:03.448217  4594 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.449625 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:03.450872  4594 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:03.451700 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:03.451872 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.452332 27584 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:03.452571 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.447631 27585 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:03.454025 28330 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:03.455184 28336 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:03.456511  4493 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:03.456826  4663 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:03.457686 27583 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:03.458803 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.459729 27657 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 19 because replica has last-logged OpId of term: 18 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 15.
I20260709 12:22:03.460976 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.461930 27586 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:03.462698  4724 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.463300  4724 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.463727 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.464886  4333 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.421949 27731 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:03.379063 27655 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:03.481420 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1314) took 1623 ms. Trace:
I20260709 12:22:03.433522 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.483476  4728 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.484607 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.483312 28337 rpcz_store.cc:276] 0709 12:22:01.857740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.858170 (+   430us) service_pool.cc:224] Handling call
0709 12:22:01.858613 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:02.231545 (+372932us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:02.741363 (+509818us) log.cc:844] Serialized 14 byte log entry
0709 12:22:03.073468 (+332105us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:03.107195 (+ 33727us) log.cc:844] Serialized 26 byte log entry
0709 12:22:03.107405 (+   210us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:03.107458 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.474755 (+367297us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400520dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:03.474807 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:03.474853 (+    46us) raft_consensus.cc:1685] finished
0709 12:22:03.481046 (+  6193us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:03.481216 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38079,"prepare.queue_time_us":518387,"prepare.run_cpu_time_us":4293,"prepare.run_wall_time_us":29191,"raft.queue_time_us":923,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":2561,"spinlock_wait_cycles":856139008,"thread_start_us":732103,"threads_started":2,"wal-append.queue_time_us":434068}
I20260709 12:22:03.484990 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.435523 28335 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:03.486264  4728 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:03.492491  4643 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:03.492977 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.494084 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.437644 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.498410 27582 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:03.420704 27732 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 20.
I20260709 12:22:03.466019  4674 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.495630 27589 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:03.467522  3841 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:03.467553  4399 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:03.468346 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.444036  4593 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.456847 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.400050 27736 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:03.474720 27581 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:03.475701 27584 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:03.479075 27701 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.591s (SENT)
I20260709 12:22:03.479228 27585 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:03.448120  4722 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:03.526062 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.481957 27737 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:03.438076 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:03.531819 27730 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
W20260709 12:22:03.501067 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.527560 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:03.466987 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90561.5us
I20260709 12:22:03.528306 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:03.538033 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:03.540015  4662 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:03.537837 28328 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:03.541095  4663 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.540011 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:03.541586  4663 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.543362 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:03.544515  4640 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:03.545028  4640 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:03.545432  4640 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:03.524825  4628 raft_consensus.cc:2764] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:22:03.536933 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.550835 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.526002  4399 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:03.531020  4489 raft_consensus.cc:2764] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:22:03.530858  4722 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:03.555189 27589 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:03.552596  4736 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:03.559337 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.535105  4527 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.548295  4591 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:03.526218  4376 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:22:03.552968  4493 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:03.554157  4662 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:03.556420  4399 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:03.561635  4728 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:03.562338  4722 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:03.596312  4591 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:03.611116  4728 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.607788  4658 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:03.637434 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94220.4us
I20260709 12:22:03.567387  4527 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.567366  4736 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.628746  4643 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:03.649497 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:03.650671 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 216550us
I20260709 12:22:03.648522 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3910) took 1196 ms. Trace:
I20260709 12:22:03.643338  4674 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.656808 27738 rpcz_store.cc:276] 0709 12:22:02.452192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.452374 (+   182us) service_pool.cc:224] Handling call
0709 12:22:02.504626 (+ 52252us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:02.555292 (+ 50666us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:22:02.910228 (+354936us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.933892 (+ 23664us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:03.280794 (+346902us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:03.491007 (+210213us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:22:03.491495 (+   488us) log.cc:844] Serialized 26 byte log entry
0709 12:22:03.491667 (+   172us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:03.491706 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.491773 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:03.648303 (+156530us) raft_consensus.cc:1685] finished
0709 12:22:03.648325 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:03.648463 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":231891,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":21562,"raft.run_cpu_time_us":4042,"raft.run_wall_time_us":363062,"spinlock_wait_cycles":62201984,"thread_start_us":657359,"threads_started":2,"wal-append.queue_time_us":458505}
I20260709 12:22:03.656395  4643 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:03.663839  4643 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:03.668962  4527 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.672391 28331 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:22:03.673889  4558 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.677444  4558 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.642014  4728 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:03.640543 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 202095us
W20260709 12:22:03.691821 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 196167us
W20260709 12:22:03.700671 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.710856 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:03.712809  4662 raft_consensus.cc:2764] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:22:03.719549  4589 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.720216  4589 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.723882  4589 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.727036  4722 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.729182  4663 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.729965  4722 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.731875  4736 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.733156  4333 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.738965  4590 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:03.743956  4591 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:03.745566 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102655us
I20260709 12:22:03.703780  4745 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:22:03.682016  4682 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.647444  4658 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.756412  4724 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:03.701558  4674 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.764529 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:03.765298 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.766723 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:03.767282  4678 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:03.767973 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:03.768636  4527 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:03.769847  4662 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:03.770569  4535 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:03.791909 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:03.794833 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:03.802557 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:03.804785 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.809556 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109669us
I20260709 12:22:03.786127  4558 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:22:03.815385 28331 raft_consensus.cc:2368] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
W20260709 12:22:03.818758 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126066us
I20260709 12:22:03.821882 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:03.832819 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:03.815798  4724 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:03.833513  4726 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.003s
I20260709 12:22:03.834172  4376 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:03.834131  4590 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:22:03.889619 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 121706us
W20260709 12:22:03.891728 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59017.2us
I20260709 12:22:03.875545  4592 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:22:03.892956 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 283089us
I20260709 12:22:03.883775  4590 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:03.894178 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95994.5us
W20260709 12:22:03.896579 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 102972us
I20260709 12:22:03.896852 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.900074 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:03.889256  4739 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:03.903199 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.907573 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.907843 27628 leader_election.cc:400] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:22:03.908310 27628 leader_election.cc:403] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:22:03.913182 28331 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:03.916034 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.916741  4376 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:03.917075 27582 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:22:03.917702  4376 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:03.908595 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.919883 27585 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:03.920198 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.921591 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:03.922036  4592 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:03.922617 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:03.923497  4725 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:03.923131  4592 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:03.924033  4725 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:03.924518 27738 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:22:03.924964  4725 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.925743  4572 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:03.926403  4572 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:03.926632  4333 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:03.927093  4333 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.927449  4572 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.930150 28328 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 20. Current term is 21. Ops: []
I20260709 12:22:03.930018  4643 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.932461 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:03.933070  4674 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:03.936538  4658 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.937238 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.937654 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.938827 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.939371 27730 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:03.939916 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:03.940292  4399 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.940658  4376 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:03.941565 27584 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:03.942721  4399 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:03.933123 27581 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:03.946436 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53590.8us
I20260709 12:22:03.895574  4590 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:03.947211 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.947430 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:03.955868 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:03.957630 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.958475 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:03.959781  3841 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:03.967724 27583 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:03.971474 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:03.927480  4333 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:03.987345 27657 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:03.983785 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:03.992282 27589 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:03.977171 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:03.985641 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:03.979146 27731 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:22:03.987010 27552 connection.cc:256] RPC call timeout handler was delayed by -1.6494113650001054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:22:03.997414 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5981113650000225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20260709 12:22:03.997633 27664 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:03.998065 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:04.003608 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.004036 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.004521 27655 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:03.980490 27587 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:03.999178 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:04.006453 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4811113649998333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
W20260709 12:22:04.006942 27552 connection.cc:256] RPC call timeout handler was delayed by -1.4271113650002007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
I20260709 12:22:04.011060 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.012045 27656 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:04.013810 27664 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:03.967612  4747 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:04.018090 27587 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:04.019032 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:04.014147 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3713113650001105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
W20260709 12:22:04.021193 27552 connection.cc:256] RPC call timeout handler was delayed by -1.2336113649998879s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1962s for a response
I20260709 12:22:04.019798 27660 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:03.971063  4739 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:04.014160 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.027040 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.011862 27661 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:04.014804 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:04.025313 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.030185 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.032877 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.038329 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.039499 27661 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:04.041146 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.016510  4535 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:22:04.041821 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 137025us
W20260709 12:22:04.041949 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0527113650000501s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
I20260709 12:22:04.044457 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.045329 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:04.045974 27730 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:04.046186 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.043040 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.045488 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:04.048513 27664 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:03.974872  4752 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:22:03.990393  4482 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:22:04.051739 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.057238 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.058205 27656 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:22:04.047040  4055 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:04.061594 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.064901 27587 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:04.068539  4399 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:04.044934  4722 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:04.071044  3818 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:03.977849  4527 consensus_queue.cc:1243] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 28.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:22:04.045002  4535 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.083506 27655 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:04.069128  4399 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:04.073506  4753 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:04.074717  4727 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:04.075448 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:04.075940  4749 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:22:04.078761  4674 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:22:04.079859  4739 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.054450  4764 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:04.047469  4736 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:22:04.097415 27731 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:04.085434  4535 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:04.105777 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.048676  4763 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.073s
I20260709 12:22:04.125308 27664 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:22:04.121516 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.122267 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.127168 27732 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 35 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:22:04.130328 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.131817 27586 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:04.132427  4766 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.135664  4745 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:04.136320  4745 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.105186  4767 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:04.139312  4767 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.107474  4768 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:22:04.139786  4767 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:04.140746  4768 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.112123  4753 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.141389  4768 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.132942 27586 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
W20260709 12:22:04.141780 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45722.8us
I20260709 12:22:04.150120 27730 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:04.089543  4722 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.155591 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:04.107019  4674 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:04.106208  4643 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.127930 27587 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:04.169135 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.169309 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.185936 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.186674 27732 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:04.186013 27730 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:22:04.142918  4762 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:22:04.194319 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:04.200417 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:04.178224 27581 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:04.188510 27655 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:04.203315 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.205789 27581 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 23: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:04.203768 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.209185 27730 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:04.202133 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:04.192433  4763 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:04.208129 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.212949 27731 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:04.218339 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.220741 27586 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:22:04.132982  4766 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.201499  4527 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:04.241586 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.244241 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.206880  4761 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.244887 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.248907 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.203181  4662 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.018s
I20260709 12:22:04.254809 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:04.223109  4640 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.677s	user 0.014s	sys 0.000s
I20260709 12:22:04.220546  4745 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.301999 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:04.286777 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.304800 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:04.287210 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:04.309115 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 124472us
W20260709 12:22:04.281473 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63885.2us
I20260709 12:22:04.312399 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.313022 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.315892 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.318806 28335 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:04.260085  4658 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:22:04.249780  4769 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:22:04.210922  4763 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:04.296825 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.299495 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:04.299767 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.322842 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.331421 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.332659 27658 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:04.291882 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:22:04.327965 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 122258us
I20260709 12:22:04.223172  4728 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.335458 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:04.335889 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:04.330710 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
W20260709 12:22:04.322026 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 119968us
I20260709 12:22:04.331918 27656 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:04.334887 27588 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:04.307494  4753 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.339541 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.345161 27658 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:04.353113 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.351238 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.355430 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.329919  4658 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:04.363767 27736 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:04.359829 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 118345us
I20260709 12:22:04.361605 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.365340 27659 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:04.306840  4527 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.366747 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.367156 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.305107  4640 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.373202 27581 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:04.374577 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.375211 27663 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:04.246433  4674 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:04.380357 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70256.3us
I20260709 12:22:04.381242 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:04.298533  4761 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.352855  4745 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:22:04.391727 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59840.5us
I20260709 12:22:04.396924 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.397717 27656 raft_consensus.cc:2376] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25. Re-sending same reply.
I20260709 12:22:04.405961 27663 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:04.409289 27662 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:04.372037  4753 raft_consensus.cc:2764] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:22:04.412351 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:04.387943  4773 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 12:22:04.386067  4770 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:22:04.418458 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.415076 27663 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:04.364506  4768 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.419092 27659 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:22:04.420125 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53460.8us
I20260709 12:22:04.419135 27656 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:04.384130  4779 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:04.383587  4785 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:04.423623 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:04.429554  4592 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:22:04.434717 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.421113  4376 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:22:04.451210 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115397us
I20260709 12:22:04.451597 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.429610  4722 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.452070 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:04.437753  4674 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:04.468037 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:04.481868 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58287.4us
I20260709 12:22:04.444916  4777 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:04.483758 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.436563  4726 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:22:04.453670  4376 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:04.496652 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49363.9us
W20260709 12:22:04.509181 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 167680us
I20260709 12:22:04.485800  4643 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:04.496042  4726 consensus_queue.cc:1243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 21.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:22:04.486477  4777 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:04.482358  4766 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.508195  4376 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:04.516703  4779 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:04.456514  4658 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.526476  4777 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:04.526482  4728 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:04.535068  4591 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:04.597012 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.533911  4779 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:04.609700 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 103302us
I20260709 12:22:04.586782  4527 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:04.588783  4640 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:04.606528  4774 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260709 12:22:04.615036 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 131441us
I20260709 12:22:04.617043 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.602047  4658 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:04.618984  4678 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:04.623757  4764 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:22:04.606071  4779 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:04.629525 27658 raft_consensus.cc:2368] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 23. Current term is 24.
I20260709 12:22:04.618425  4750 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:04.632193  4780 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:22:04.636837 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:04.654651 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 186738us
I20260709 12:22:04.656471 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:04.656915 28335 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:22:04.639863  4745 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.662757 27732 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:22:04.657742  4724 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.812s	user 0.013s	sys 0.000s
W20260709 12:22:04.662636  4767 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.002s
I20260709 12:22:04.670773  4767 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.671396  4726 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:04.671933  4726 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:04.672206 27657 raft_consensus.cc:2368] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26.
I20260709 12:22:04.673452  4767 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.674072  4724 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:04.675119  4777 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:04.676129  4590 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.701s	user 0.009s	sys 0.003s
W20260709 12:22:04.668426  4535 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.000s
I20260709 12:22:04.668525  4775 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:22:04.669544  3841 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:04.656111  4744 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:04.679980 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:04.681036 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.681030  4535 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.671548  4722 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 12:22:04.686343  4535 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.687896  4399 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.689718  4399 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:04.664510  4745 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.694128  4745 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.694748  4745 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:22:04.700663  4778 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.321s	user 0.002s	sys 0.000s
I20260709 12:22:04.664729  4643 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:22:04.704696  4763 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.706549  4763 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:04.708971  4333 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.697s	user 0.011s	sys 0.000s
I20260709 12:22:04.709434  4333 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.710533 27738 raft_consensus.cc:2368] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:22:04.712512  4333 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:04.682072  4590 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.715709 27734 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:04.717242 27737 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:04.718061 27734 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:22:04.679203  4726 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:04.722688  4765 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.580s	user 0.004s	sys 0.000s
W20260709 12:22:04.727711 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56437.2us
I20260709 12:22:04.728756  4678 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:04.729352  4678 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.729835 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:04.730844 27738 raft_consensus.cc:606] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.734251 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.735080 27580 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:22:04.736193 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.736841 27587 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:04.737172 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.737738 27583 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:04.738725  4750 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:04.740494  4783 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:04.740958  4783 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.741297  4783 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:04.742034 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.743551  4682 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:04.744509  4678 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:04.745021 27585 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:22:04.746061 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.746629 27582 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:04.729848 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:04.719286 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:04.734037  4772 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:04.695091  4745 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:04.770593 27587 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:04.757431 27729 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:04.774129 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:04.774214  4558 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:04.774482 27584 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:04.690990  4761 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:04.794073 27582 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
W20260709 12:22:04.809939 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 357929us
W20260709 12:22:04.814893 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 164903us
I20260709 12:22:04.815909 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.810971 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:04.759516  4772 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:04.819471 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 89690.6us
I20260709 12:22:04.785039  4558 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:04.818071 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:04.822081 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 308946us
I20260709 12:22:04.747490  4055 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:22:04.842527 27734 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:22:04.827104 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.845198 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.832680 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:04.821247  4785 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:04.797142  4747 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.033s
I20260709 12:22:04.822160  4682 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:04.819044  4590 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:04.850952 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.849669  4376 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.854291  4376 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:04.855957 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:04.849859  4643 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 12:22:04.857715 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:04.858325 27730 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:04.859478  4806 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:04.860112  4806 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:04.861315 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.861029  4806 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:04.868608 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 188724us
I20260709 12:22:04.863046 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:04.861917  4793 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:22:04.880628 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:04.843637  4678 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:04.882779 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:04.885797 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74945.8us
I20260709 12:22:04.886583 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.886982 27582 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:04.887303 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 360025us
I20260709 12:22:04.824532  4663 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:22:04.887368 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:04.830242  4744 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:04.890148  4784 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.890372  4807 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:04.890781  4786 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:04.892508 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:04.892777 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68946.5us
I20260709 12:22:04.893281 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:04.893719 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.894351 27657 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:22:04.894989 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.884392  4674 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:04.895450  4593 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:04.895648 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.896196  4763 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:04.896272 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:04.890213  4798 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:22:04.896883  4777 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:04.900075  4801 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:22:04.900914 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:04.901708  4795 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:04.902767  4793 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:22:04.903553 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52687.3us
I20260709 12:22:04.904140  4809 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:04.851070  4785 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.868006  4628 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:22:04.957331 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:04.906008 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:04.907446 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:22:04.909389 27730 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:04.848105  4775 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:04.908725 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:04.862886  4591 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:04.904314  4779 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:04.939833 27663 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:04.940691 28334 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:04.942215 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:04.970394 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64437.2us
I20260709 12:22:04.943506  3818 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:04.878417  4722 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:22:04.910179  4752 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:04.961845 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:04.926147  4761 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:04.927626  4786 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.948094  4593 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.988898 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3416) took 1053 ms. Trace:
I20260709 12:22:04.944403  4763 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.991146 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:04.989387 27736 rpcz_store.cc:276] 0709 12:22:03.935797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.938559 (+  2762us) service_pool.cc:224] Handling call
0709 12:22:04.988686 (+1050127us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:04.988842 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":975132544}
W20260709 12:22:04.977624 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68955.2us
I20260709 12:22:04.998716 28337 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:04.959077  4793 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:05.000054 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:05.006263 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.006449 27583 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:05.006894 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.007207 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.007551 27660 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:04.976361  4736 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:04.979372  4767 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:04.959597  4785 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:04.988729  4753 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:05.028023 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.024973 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:04.926493  4784 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.013792 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.030676 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.031097 27662 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:22:04.993865  4593 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:04.998700  4763 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:05.003409  4777 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:22:05.017318 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.017910 27661 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:22:05.013329 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.043648 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:05.055470 27664 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:05.055612 27730 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 24: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:05.026072  4482 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:22:05.046377  4793 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:04.910792  4772 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.028883  4739 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.993708  4786 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:05.032043 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:05.049571 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.976147  4663 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:22:04.962036  4726 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.081586  4628 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:05.090683  4628 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:05.091542  4726 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:05.083014  4739 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.093497 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1522) took 1104 ms. Trace:
I20260709 12:22:05.094180  4739 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.065635  4753 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:05.010036  4809 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:05.095413 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:05.096079 27657 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:05.053925 27734 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:05.010504  4747 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:22:05.058074 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.009935  4643 raft_consensus.cc:2764] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:22:04.991613  4795 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.044s
I20260709 12:22:05.024127  4761 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:04.993574  4727 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:22:05.128435  4727 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:05.128878  4727 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:05.129860  4727 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.066277 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:05.048076  4813 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:22:05.131645 27662 raft_consensus.cc:2376] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31. Re-sending same reply.
I20260709 12:22:05.068449 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.132753 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:04.954849  4558 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.071105 27655 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:05.071805  4808 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:22:05.072265  4817 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:22:05.073808  4784 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.047943  4775 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.048954  4766 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:22:05.083899 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51929.8us
I20260709 12:22:05.147673 27630 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:22:05.148077 27630 leader_election.cc:403] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:22:05.084419  4535 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:22:05.084571  4793 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:05.054150 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.094101 27729 rpcz_store.cc:276] 0709 12:22:03.989143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.989950 (+   807us) service_pool.cc:224] Handling call
0709 12:22:05.093286 (+1103336us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:05.093428 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1555010816}
I20260709 12:22:05.103292 27736 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:05.131244  4727 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:05.111675  4591 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:05.029340  4744 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.118120  4779 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.091570  4628 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.065412  4640 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:22:05.069092 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.187083 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.137303  4809 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:05.142642 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:05.131016  4789 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:22:05.152582 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95755.9us
I20260709 12:22:05.151540 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:05.192462 28328 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:05.111305 27734 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:05.177538 27657 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:05.179565 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.180390 27738 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:22:05.180799 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.185281 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.194319 27729 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:05.188001 28333 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:22:05.194713 27736 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:05.187529 27661 raft_consensus.cc:2376] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33. Re-sending same reply.
I20260709 12:22:05.142318  4808 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:05.196610 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45186.3us
I20260709 12:22:05.196076 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.210441 27663 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:22:05.211053 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:05.096127  4753 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.213951 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:05.216063 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.216684 27729 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:05.187453  4399 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.027s
W20260709 12:22:05.213394 27701 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.796s (SENT)
W20260709 12:22:05.219347 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.146062  4761 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.222959 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:05.223803 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:05.186165  4777 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:05.225862 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.226147 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:05.222221 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.192193  4789 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:05.191577  4784 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:22:05.230978 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.245332 27731 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:05.236948  4745 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.224916  4736 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:22:05.240360 27701 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.812s (SENT)
W20260709 12:22:05.241572  4824 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:22:05.209496  4766 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:05.258844 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.217650  4808 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:05.233675 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:05.254374 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.222584  4628 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:05.228495  4663 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:22:05.269315  4761 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.276501 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.278337  4763 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.222724  4796 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:05.217976  4591 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.282385  4789 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.218196  4779 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:05.283754 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.284300  4736 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.246742 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.290856 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:05.257517 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:05.319270 27663 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:05.266075 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1668) took 1128 ms. Trace:
I20260709 12:22:05.261200 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3431) took 1202 ms. Trace:
I20260709 12:22:05.258495 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3928) took 1118 ms. Trace:
I20260709 12:22:05.319880 27664 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:05.292204 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58612.7us
I20260709 12:22:05.302312 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4041) took 1018 ms. Trace:
I20260709 12:22:05.308025 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:05.321097 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:05.321561 27657 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:05.246392  4376 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:05.310840 27661 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:05.305992  4055 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.308478  4764 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:05.329960 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:05.309918  4807 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:05.319729 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:05.274633  4786 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.322547 27732 rpcz_store.cc:276] 0709 12:22:04.058434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.666327 (+607893us) service_pool.cc:224] Handling call
0709 12:22:04.684102 (+ 17775us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.727678 (+ 43576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.727733 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:04.728351 (+   618us) log.cc:844] Serialized 26 byte log entry
0709 12:22:04.920078 (+191727us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:04.920117 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.260788 (+340671us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b540017e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:05.260839 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.260890 (+    51us) raft_consensus.cc:1685] finished
0709 12:22:05.260904 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.261135 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99447040,"thread_start_us":172458,"threads_started":1,"wal-append.queue_time_us":185192}
I20260709 12:22:05.312341  4593 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.320151 27586 rpcz_store.cc:276] 0709 12:22:04.137756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.247608 (+109852us) service_pool.cc:224] Handling call
0709 12:22:05.265862 (+1018254us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:05.265987 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1019708288}
I20260709 12:22:05.321367 27658 rpcz_store.cc:276] 0709 12:22:04.284404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.352869 (+ 68465us) service_pool.cc:224] Handling call
0709 12:22:05.290774 (+937905us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b5400344754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:05.302244 (+ 11470us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":633868672}
I20260709 12:22:05.325268 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.341394 27731 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:05.193064  4783 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.320386 27737 rpcz_store.cc:276] 0709 12:22:04.139996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.719735 (+579739us) service_pool.cc:224] Handling call
0709 12:22:04.720059 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.720660 (+   601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.720766 (+   106us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:04.721997 (+  1231us) log.cc:844] Serialized 26 byte log entry
0709 12:22:04.912175 (+190178us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:05.257878 (+345703us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.258128 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.258207 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.258260 (+    53us) raft_consensus.cc:1685] finished
0709 12:22:05.258273 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.258438 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":340745,"prepare.queue_time_us":351432,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":35134,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3350,"thread_start_us":187542,"threads_started":1,"wal-append.queue_time_us":172019}
I20260709 12:22:05.343695 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:05.344241 27737 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:22:05.345413 27737 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:05.267058  4682 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:05.337631 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.325230  4736 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:05.338824 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c204963842941859d4313114fd92c93"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:05.319145  4663 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:05.347718 27736 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:22:05.350771 27732 raft_consensus.cc:606] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:05.348489 27657 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:05.333527  4761 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:05.334102  4764 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.331785  4789 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:05.335220  4728 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:05.351459 27658 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:05.383718  4803 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:22:05.384478 27658 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:22:05.386332 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:05.386901 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:05.387552  4779 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:05.388078  4779 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:05.388201  4535 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:05.388705  4535 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.389113  4535 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:05.389003  4779 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.373544 27657 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:05.391502  4399 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:05.392009  4750 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:22:05.351486 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.373706 27737 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:22:05.374357  4751 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:22:05.376201  4786 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.398902  4592 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:05.400770 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:05.401504  4592 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:05.401598 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:05.376871  4783 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.377592 27659 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:05.377946  4782 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.939s	user 0.004s	sys 0.002s
I20260709 12:22:05.259850  4809 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.313277  4785 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.389235  4784 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:05.354965 27731 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:05.406630 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1400) took 1306 ms. Trace:
I20260709 12:22:05.378615  3818 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:05.372579  4764 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:05.328615  4796 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.334580  4055 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.408169  3818 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:05.407037 28332 rpcz_store.cc:276] 0709 12:22:04.100513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.100678 (+   165us) service_pool.cc:224] Handling call
0709 12:22:04.101008 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.105752 (+  4744us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.105785 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:04.128822 (+ 23037us) log.cc:844] Serialized 26 byte log entry
0709 12:22:04.247513 (+118691us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:04.247561 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.247634 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.248099 (+1000465us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:05.403779 (+155680us) raft_consensus.cc:1685] finished
0709 12:22:05.403803 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.406569 (+  2766us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27025,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":4527488,"thread_start_us":47686,"threads_started":1,"wal-append.queue_time_us":177729}
W20260709 12:22:05.413998 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 173678us
I20260709 12:22:05.412705 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.417266 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:05.419978 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:05.438457 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:05.439843 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:05.438453 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:05.441327 27732 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:05.412187  4535 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.444798  4784 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:05.444065  4399 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:05.448846 28336 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:05.450968 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:05.354035  4777 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.455601 27729 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 20: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:05.372637  4766 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.373293  4376 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.045s
W20260709 12:22:05.462941 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.413751  4728 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:05.466004 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:05.467713 27656 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:22:05.448212  4572 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:05.442555  4482 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:05.455133  4592 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:05.461102  4751 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:05.491477  4830 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:05.492133  4830 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.448024  4752 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:22:05.476596  4572 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.516075 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:05.516779 28336 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:05.516270 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:22:05.522013 27586 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:05.518762  3818 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.488431  4793 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:05.519487 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:05.476989  4722 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:22:05.521354  4572 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:05.552211 27732 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:05.506363  4751 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:05.514676  4830 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.555886 27658 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:22:05.557696 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:05.530319  4482 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:05.559232  3818 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.514153  4643 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:22:05.536536  4784 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.548924  4763 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.580884 27658 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:22:05.559885  4593 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.577456  4842 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:05.593261 27584 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:22:05.593734  4796 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.587491  4376 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:05.499529  4663 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.594622  4376 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:05.516124  4728 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.548148  4793 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.574725  4830 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:05.596488 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66572.3us
W20260709 12:22:05.598963 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124612us
I20260709 12:22:05.602696 27552 leader_election.cc:400] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:22:05.603077 27552 leader_election.cc:403] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:22:05.594216  4535 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.583926  4643 consensus_queue.cc:1243] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 33.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:22:05.629441 27658 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:22:05.637739 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:05.639526 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:05.579146  4784 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:05.652014 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52183.1us
I20260709 12:22:05.655061  3818 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:22:05.581763  4482 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.657732 27658 pending_rounds.cc:85] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 15
I20260709 12:22:05.662508 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:05.662528 27658 pending_rounds.cc:107] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:22:05.663146 27658 raft_consensus.cc:2889] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:05.664167 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.663447  4832 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:22:05.629113  4830 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.686474  4591 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.667819  4055 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.668744  4786 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.652748  4376 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.717087  4775 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:05.763048 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.758561  4591 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:05.769793  4839 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:22:05.771567  4591 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:05.772068  4833 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 12:22:05.774242  4849 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:05.775970  4793 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.778503  4772 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:05.783504  4775 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:05.784977  4846 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:22:05.785401  4482 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.663595  4751 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.723922  4847 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 12:22:05.759541  4786 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.847903  4592 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:05.906392  4628 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.595s	user 0.007s	sys 0.005s
I20260709 12:22:05.909098  4786 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:05.913758  4841 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:22:05.922482 27657 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:22:05.924283 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1494) took 1032 ms. Trace:
I20260709 12:22:05.926487 27659 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:22:05.929029 28337 rpcz_store.cc:276] 0709 12:22:04.892100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.896094 (+  3994us) service_pool.cc:224] Handling call
0709 12:22:04.896388 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.001945 (+105557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.923948 (+922003us) spinlock_profiling.cc:243] Waited 99.8 ms on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:05.924175 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":229632256}
I20260709 12:22:05.904801  4768 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:22:05.926858  4727 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.746s	user 0.011s	sys 0.001s
I20260709 12:22:05.921144  4572 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:05.947690  4853 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.191s	user 0.000s	sys 0.003s
W20260709 12:22:05.953832 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.927176  4784 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:05.963682 27660 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
W20260709 12:22:05.964107  4852 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
W20260709 12:22:05.967453  4736 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.595s	user 0.004s	sys 0.008s
I20260709 12:22:05.967968  4736 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.969106 27661 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
W20260709 12:22:05.969619  4764 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.002s
I20260709 12:22:05.943454  4768 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:05.972152  4736 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:05.972031  4768 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:05.973414  4768 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:05.974853  4572 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:05.952594  4830 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:05.963690  4854 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
W20260709 12:22:05.982755  4789 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.602s	user 0.007s	sys 0.001s
I20260709 12:22:05.983772  4592 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.991094 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:05.970083  4764 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:05.997376 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.030519 27580 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:06.037812 27587 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:06.039269 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:06.041478 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 148941us
I20260709 12:22:06.042594  4761 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.043119  4761 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:06.044957  4860 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:22:06.049908  4803 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:06.050040 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:06.050621  4803 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.051524  4858 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:06.052006  4858 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.052379  4858 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:06.053004  4803 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:06.053601  4803 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:06.057896  4752 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:06.058374  4752 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.059791  4764 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.065431 27732 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:22:06.066671 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.067247 27737 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:06.075634 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.077072 27729 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:22:06.079773 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.081216 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.078965 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.082263 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.083042 27589 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:06.088737 27731 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:06.089166 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.103493 27580 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:22:06.105482 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.106346 27585 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:06.106551 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:06.106791 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.107080 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.107414 27586 raft_consensus.cc:2376] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21. Re-sending same reply.
I20260709 12:22:06.107774 27587 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:06.105630 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.106361 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.106839 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.116775 27730 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:06.059281  4752 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.119879  4752 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:06.120350 27585 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:06.121376 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.122344 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.123020 27585 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:06.108446  4764 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.123692 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.124069  4764 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.124855 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.125828 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.126121  4764 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.126809 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.127781 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.128847 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.129665 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.129841 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.130559 27587 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:06.131508 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.132635  4789 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:06.133448 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.134009 27586 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:06.135500  4775 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:06.141489 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:06.138775 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.138742 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.097069  4663 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:06.148396 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3308) took 1080 ms. Trace:
I20260709 12:22:06.138341 27587 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:06.148940 27662 rpcz_store.cc:276] 0709 12:22:05.067744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.784767 (+717023us) service_pool.cc:224] Handling call
0709 12:22:05.785204 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.785491 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.785528 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.786108 (+   580us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.035236 (+249128us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:06.035312 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.035431 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.148056 (+112625us) raft_consensus.cc:1685] finished
0709 12:22:06.148087 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.148336 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":248060,"threads_started":1,"wal-append.queue_time_us":320296}
I20260709 12:22:06.152609 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.154196 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.087972  4844 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.071s
I20260709 12:22:06.187773  4844 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:06.188153  4844 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.188985  4844 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.190717  4844 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:06.191385 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.192171  4789 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:06.192049  4761 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.155385 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.088157  4848 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:22:06.194522  4761 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.195883 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:06.150545  4808 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.840s	user 0.013s	sys 0.000s
I20260709 12:22:06.199743  4808 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.201059 28330 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:06.161578  4764 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:06.158896  4809 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:06.202502  4764 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:06.202692 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1405) took 1052 ms. Trace:
I20260709 12:22:06.202847  4809 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.203069 28330 rpcz_store.cc:276] 0709 12:22:05.150064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.151162 (+  1098us) service_pool.cc:224] Handling call
0709 12:22:05.177060 (+ 25898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.201417 (+1024357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.202469 (+  1052us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003ed0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:06.202634 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2354551552}
I20260709 12:22:06.203989  4772 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:06.202134  4808 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.205446  4809 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:06.206679  4863 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:22:06.208144 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 742189us
I20260709 12:22:06.208617  4845 raft_consensus.cc:993] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:22:06.209270  4845 raft_consensus.cc:1081] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:22:06.203507  4764 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.208654 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.215628 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.217897 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.220858  4761 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:06.221098  4728 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:22:06.221441  4772 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:06.223044 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1650) took 1101 ms. Trace:
I20260709 12:22:06.223563 27659 rpcz_store.cc:276] 0709 12:22:05.121295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.929768 (+808473us) service_pool.cc:224] Handling call
0709 12:22:05.953820 (+ 24052us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:05.959250 (+  5430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.959288 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:05.960000 (+   712us) log.cc:844] Serialized 52 byte log entry
0709 12:22:06.069264 (+109264us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:06.104290 (+ 35026us) log.cc:844] Serialized 14 byte log entry
0709 12:22:06.116314 (+ 12024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.116397 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.222796 (+106399us) raft_consensus.cc:1685] finished
0709 12:22:06.222825 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.222984 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11723,"prepare.queue_time_us":21288,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":6520,"raft.queue_time_us":25446,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":180,"thread_start_us":108851,"threads_started":1,"wal-append.queue_time_us":206566}
I20260709 12:22:06.231976 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.234243 27585 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:22:06.238566 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.258674 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:06.262009 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.263795 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.264506 27587 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:06.265413 27586 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 24. Current term is 25. Ops: []
W20260709 12:22:06.266834  4867 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260709 12:22:06.269954  4837 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.719s	user 0.001s	sys 0.001s
I20260709 12:22:06.270700 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4053) took 1265 ms. Trace:
I20260709 12:22:06.253463  4847 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:06.271135 27655 rpcz_store.cc:276] 0709 12:22:05.005090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.005193 (+   103us) service_pool.cc:224] Handling call
0709 12:22:06.270507 (+1265314us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b54000e4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:06.270610 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97408768}
I20260709 12:22:06.269941 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.272702  4798 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:06.273228  4845 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:06.274047 28331 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:06.275100 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:06.276266  4845 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:06.276652  4845 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:06.276686  4558 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:06.228745  4728 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:22:06.280109 27585 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:06.280397 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1652) took 1142 ms. Trace:
W20260709 12:22:06.275215  4851 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.324s	user 0.000s	sys 0.001s
I20260709 12:22:06.280778 27662 rpcz_store.cc:276] 0709 12:22:05.135344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.186843 (+1051499us) service_pool.cc:224] Handling call
0709 12:22:06.187230 (+   387us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:06.187644 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.187671 (+    27us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:06.188401 (+   730us) log.cc:844] Serialized 78 byte log entry
0709 12:22:06.223903 (+ 35502us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:06.243283 (+ 19380us) log.cc:844] Serialized 14 byte log entry
0709 12:22:06.247367 (+  4084us) log.cc:844] Serialized 14 byte log entry
0709 12:22:06.247606 (+   239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.247686 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.277643 (+ 29957us) raft_consensus.cc:1685] finished
0709 12:22:06.277661 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.277799 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":43333,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":79,"raft.queue_time_us":5651,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":8236,"spinlock_wait_cycles":10951552,"thread_start_us":34976,"threads_started":1,"wal-append.queue_time_us":61432}
I20260709 12:22:06.281441 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:06.277222  4845 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:06.284848 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.284673  4772 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:06.292131  4772 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:06.293015  4772 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:06.299064 27656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4059) took 1280 ms (client timeout 1500 ms). Trace:
W20260709 12:22:06.299536 27656 rpcz_store.cc:269] 0709 12:22:05.018684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.028199 (+  9515us) service_pool.cc:224] Handling call
0709 12:22:06.298823 (+1270624us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b5400210554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:06.298993 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":697272576}
I20260709 12:22:06.282506  4858 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:06.306334  4859 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:22:06.306954  4873 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:06.308077  4682 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:06.308477  4682 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:06.308552  4877 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:06.310026  4809 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:06.311270  4809 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:06.312093  4809 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.313289  4809 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:06.315243  4808 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:06.316114  4881 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:22:06.315840  4808 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.318886  4808 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.325059 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3300) took 1307 ms. Trace:
I20260709 12:22:06.325284 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4070) took 1258 ms. Trace:
I20260709 12:22:06.325501 27661 rpcz_store.cc:276] 0709 12:22:05.017593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.337986 (+320393us) service_pool.cc:224] Handling call
0709 12:22:05.349890 (+ 11904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.969604 (+619714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.324826 (+355222us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:06.324997 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1423573376}
I20260709 12:22:06.304064  4858 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:06.327847 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3299) took 1310 ms. Trace:
I20260709 12:22:06.328383  4874 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:22:06.328325 27663 rpcz_store.cc:276] 0709 12:22:05.017295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.335751 (+318456us) service_pool.cc:224] Handling call
0709 12:22:05.337753 (+  2002us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.459263 (+121510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.459302 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:05.459371 (+    69us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:05.459411 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.327627 (+868216us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:06.327675 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.327784 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264582528}
I20260709 12:22:06.325774 27658 rpcz_store.cc:276] 0709 12:22:05.066426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.643044 (+576618us) service_pool.cc:224] Handling call
0709 12:22:05.657438 (+ 14394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:05.681399 (+ 23961us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.681446 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:05.682190 (+   744us) log.cc:844] Serialized 52 byte log entry
0709 12:22:05.841563 (+159373us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:05.855893 (+ 14330us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.856627 (+   734us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.856920 (+   293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.857034 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.305582 (+448548us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:06.317662 (+ 12080us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b18002bef40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:06.325067 (+  7405us) raft_consensus.cc:1685] finished
0709 12:22:06.325094 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.325224 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":93700,"prepare.run_cpu_time_us":9168,"prepare.run_wall_time_us":41580,"raft.queue_time_us":394,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":50515,"spinlock_wait_cycles":51090048,"thread_start_us":76405,"threads_started":1,"wal-append.queue_time_us":276912}
I20260709 12:22:06.309478  4682 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.331001  4682 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:06.335676 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.336963 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:06.335065 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.350477 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.351141 27733 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:22:06.350991 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:06.354127 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.336777  4768 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.361176 27735 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:06.363896 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.369073 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.371170 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.373018 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.342424  4767 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:06.372042 27734 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:06.386251 27733 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:06.386802 27731 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:22:06.346558  4728 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:06.349915  4726 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:06.357717  4752 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:22:06.354192  4783 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.342209  4643 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:22:06.374428  4592 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:22:06.408471 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.410425 27730 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:22:06.353652  4798 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
W20260709 12:22:06.416656 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 131879us
I20260709 12:22:06.391592  4736 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.389026  4767 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:06.421756 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.376595  4857 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:22:06.385722  4748 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:22:06.377830  4768 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.430824 27663 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:22:06.431491 27663 pending_rounds.cc:85] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 16
I20260709 12:22:06.431790 27663 pending_rounds.cc:107] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:22:06.432056 27663 raft_consensus.cc:2889] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:06.388782  4628 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:22:06.433951 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.391505  4877 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:22:06.414716  4753 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.435251 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.437325 27658 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:06.437251 27664 raft_consensus.cc:2376] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31. Re-sending same reply.
I20260709 12:22:06.414072  4592 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:06.436605 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.436179 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.439613 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.441028 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.441980 27662 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:06.426771  4726 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:06.442909  3818 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:06.440387  4739 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:06.394816  4728 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.406215  4783 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.432245  4748 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:06.456723 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.428382  4848 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:22:06.454820  4857 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:06.428314  4055 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:22:06.461585  3818 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:06.429176  4736 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.452448  4739 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:06.442724  4592 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:06.447098  4726 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:06.467088 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.469630 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.456975  4748 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.474668 27662 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 19.
I20260709 12:22:06.461280  4834 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.458680  4643 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.012s
I20260709 12:22:06.445509  4753 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.486919 27664 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:06.464926  4857 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.532260  4857 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.534070  4857 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:06.538980 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74097.4us
I20260709 12:22:06.539305 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:06.490489  4887 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:22:06.490413  4726 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.541582 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:06.487973  4845 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.491055  4748 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.492101  4834 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.494478  4728 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.486140  4767 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.501283  4753 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.490433  4739 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.505456 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.506040 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.507766 27658 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:06.509027 27663 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
W20260709 12:22:06.511440  4880 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:22:06.511823  4736 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:06.512188  4883 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:22:06.513312 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.561889 27662 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:06.513581  4768 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.521626  4783 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.527583  4784 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.544986 28337 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:06.545015 27661 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:06.545781 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.546412  3841 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:06.546648 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.541081 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:06.549299 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.466161  4844 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.072s
I20260709 12:22:06.548082  4748 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:06.574038  4844 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:06.574239 27734 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:22:06.574501  4844 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.574607  4874 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:06.575438  4874 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:22:06.575344  4844 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.576627  4844 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:06.575440  4784 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.559221  4726 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:06.579897  4784 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.553654  4847 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:06.582554 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:06.583137 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:06.588793 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1402) took 1771 ms (client timeout 2177 ms). Trace:
I20260709 12:22:06.557101  4739 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:06.559732 27656 raft_consensus.cc:2368] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29.
I20260709 12:22:06.566376 27581 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 33 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:22:06.566201  3841 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:06.569237 28334 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:06.558878  4845 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:06.571501  4753 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:06.560998  4643 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
W20260709 12:22:06.584120 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1492) took 1692 ms (client timeout 1996 ms). Trace:
I20260709 12:22:06.593377 27730 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
W20260709 12:22:06.593609 28335 rpcz_store.cc:269] 0709 12:22:04.891403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.892257 (+   854us) service_pool.cc:224] Handling call
0709 12:22:06.583914 (+1691657us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:06.584036 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1658920832}
I20260709 12:22:06.594146 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.555359 27663 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:22:06.591785  3841 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:06.489004  4751 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.076s
W20260709 12:22:06.589211 28331 rpcz_store.cc:269] 0709 12:22:04.817675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.817817 (+   142us) service_pool.cc:224] Handling call
0709 12:22:06.588635 (+1770818us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540024a254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:06.588736 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2847129472}
I20260709 12:22:06.589499  3818 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:22:06.603421 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.604472 27734 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:06.609165 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.591290  4055 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:06.598477  4768 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:06.597934  4783 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.624009 27736 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:22:06.624835 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.629594 27734 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:22:06.627493  4055 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.632956 28329 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:06.634007  4055 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.633805  4783 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.635102 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.634884  3841 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.636185  4783 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.636920  3841 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.637024 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.638845 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.598030  4875 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.026s
W20260709 12:22:06.641772 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61308.1us
I20260709 12:22:06.623399  4843 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:06.641150  4768 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:06.647282  4843 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.670140  4843 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.671574  4843 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:06.676294 27583 raft_consensus.cc:1217] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Deduplicated request from leader. Original: 23.16->[25.17-26.18]   Dedup: 25.17->[26.18-26.18]
I20260709 12:22:06.676852 27583 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: [25.17-26.18]
I20260709 12:22:06.680868  4834 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.681721 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.682595 27656 raft_consensus.cc:2410] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 31 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:22:06.683401 27588 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:22:06.683606 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.703001 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.712576  4785 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.712935  4768 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.716665 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.718088 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.718851 27735 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:06.719590  4893 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
I20260709 12:22:06.720613  4893 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:06.720960  4893 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.721843  4893 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.723168  4893 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:06.732467  4830 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:06.732999  4830 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:06.733974  4830 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:06.736205 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.746665 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.748093 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.748690 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.749730 28335 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:22:06.743813 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.752871 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.756495 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.757119 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.757367 27585 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:06.758121 27735 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:06.759368 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.759609 27656 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:06.760241 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.760586  4847 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:22:06.761360 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.762421 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.762519  4847 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:06.762952 27658 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:06.764808 27582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7c204963842941859d4313114fd92c93"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:22:06.765484 27582 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:06.765858 27582 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:22:06.766109  4894 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:22:06.768749 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:06.748574  4785 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.769845  4785 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.771425  4728 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:06.771178  4592 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.772193  4785 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.772903 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.773484  4592 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:06.701625  4789 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.774121 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.774600  4789 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.770664 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.775293 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.776734 27662 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:06.779600 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.779953 27664 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:22:06.776846 27656 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:22:06.777575 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.780718 27733 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:06.782662 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.784816 28331 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:06.784878 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.780335 27584 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:06.794198 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:06.786028 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.777731  4798 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:22:06.807399  4785 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:06.808431 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.808790 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1344) took 1562 ms. Trace:
I20260709 12:22:06.809058 27737 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:06.809278 28329 rpcz_store.cc:276] 0709 12:22:05.246123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.260508 (+ 14385us) service_pool.cc:224] Handling call
0709 12:22:06.588741 (+1328233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.633424 (+ 44683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.808575 (+175151us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540024a250. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:06.808716 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3053902080}
I20260709 12:22:06.778251  4821 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:06.811410 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1530) took 1497 ms. Trace:
I20260709 12:22:06.811918 27736 rpcz_store.cc:276] 0709 12:22:05.317007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.359938 (+ 42931us) service_pool.cc:224] Handling call
0709 12:22:05.381021 (+ 21083us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.624502 (+1243481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.811136 (+186634us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:06.811348 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2857896320}
I20260709 12:22:06.812616 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:06.813320 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:06.814271 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.814476 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.807242 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.796644 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:06.816565 27589 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:22:06.817198 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.819532 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:06.820968 27656 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:06.754026  4830 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:06.825733 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.833850 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:06.833956 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:06.831673 27736 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:06.834137 27584 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:06.840711 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.842607 27734 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:22:06.843665 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.845372 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:06.776008  4834 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:22:06.852738 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.855095 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:22:06.857544 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.832823  4784 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.777050  4789 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:06.857344 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:06.824442  4785 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.763047  4847 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.840095  4535 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:06.847112 27736 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:06.877714 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:06.878412  4748 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:06.840322  4893 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:22:06.880482  4871 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.699s	user 0.000s	sys 0.004s
W20260709 12:22:06.881883  4870 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.653s	user 0.001s	sys 0.001s
I20260709 12:22:06.882516  4753 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:06.883865  4820 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:06.884054  4376 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:06.833233  4843 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:22:06.885936 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.775528  4848 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.074s
W20260709 12:22:07.030319  4900 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:06.928665 27585 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:06.849109  4590 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:06.932117 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44263.2us
I20260709 12:22:06.935532 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:06.938953 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:07.041332 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105879us
W20260709 12:22:07.041693 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 102848us
W20260709 12:22:06.948488 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 102701us
I20260709 12:22:06.958771  4905 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:06.932085  4535 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.872969  4798 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:22:06.929315 27582 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:06.849566  4834 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:06.962679 27658 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:06.981674 27583 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:22:07.009073 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107389us
I20260709 12:22:07.054505 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:06.887470  4784 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:06.926390  4796 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:07.008616  4753 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.052055  4905 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.007746  4752 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:07.070990 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:07.055097 27658 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:06.886226  4857 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.159s
I20260709 12:22:07.098145 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:07.047639  4727 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:07.048763  4798 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.053885  4535 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:07.055450  4767 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:07.057710  4834 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:07.063897  4906 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:22:07.064291  4728 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:07.060657  4847 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:07.096844 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 243207us
I20260709 12:22:07.100886 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3305) took 2051 ms. Trace:
I20260709 12:22:07.046072  4739 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.982462  4736 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:22:07.111549  3818 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:07.136997 27582 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:07.174811 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.046856  4843 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:07.044191  4748 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:22:06.960819  4785 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.068133  4751 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.241s
W20260709 12:22:07.310410 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 262107us
I20260709 12:22:07.305125  4798 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:07.097443  4752 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:07.308837 27657 rpcz_store.cc:276] 0709 12:22:05.049477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.456659 (+407182us) service_pool.cc:224] Handling call
0709 12:22:05.922212 (+465553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.928929 (+  6717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.100634 (+1171705us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54001e50d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:07.100796 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1069416704}
I20260709 12:22:07.269014  4821 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:07.273535 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:07.271315  4745 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:07.269747  4783 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:07.273033 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 199468us
I20260709 12:22:07.136036  4724 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.200s
I20260709 12:22:07.391551 27658 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:07.107607  4796 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.338130  4772 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:22:07.429548 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.441843 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:07.379487  4727 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:07.335734  4739 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:07.458178 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.798s (SENT) [suppressed 16 similar messages]
I20260709 12:22:07.452440 27701 leader_election.cc:400] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29.
I20260709 12:22:07.421177  4783 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.507059 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:07.507838 27734 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:22:07.508184 27734 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:07.515302 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:07.515820 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.497501 27658 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:22:07.506397 27701 leader_election.cc:403] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:07.518271 27658 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:22:07.455451  4727 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:07.510905 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 412792us
I20260709 12:22:07.505785  4751 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.558732 27734 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.585685 27734 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:07.487619  4909 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:07.505575  4736 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.594247 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:07.587888 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 278555us
I20260709 12:22:07.563393  4590 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:22:07.635911 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194162us
I20260709 12:22:07.469218  4752 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:07.637079 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.654287 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:07.654356 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.488114  4843 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.664283 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:07.643345  4901 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.507s	user 0.001s	sys 0.000s
W20260709 12:22:07.660928  4768 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.005s	sys 0.021s
I20260709 12:22:07.634981  4772 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:07.676205 27734 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:22:07.657840  4905 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:07.691205 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 232950us
I20260709 12:22:07.663198  4926 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:07.676344  4910 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.364s	user 0.001s	sys 0.001s
I20260709 12:22:07.610798  4751 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.668875  4753 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:07.686910  4768 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.005s	sys 0.021s
I20260709 12:22:07.660640  4923 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:07.665759  4784 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:07.710209 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:07.608000  4736 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:07.672320  4908 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.398s	user 0.001s	sys 0.000s
W20260709 12:22:07.697850  4912 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
W20260709 12:22:07.772372 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62272.1us
I20260709 12:22:07.705943  4905 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.776973 27656 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:07.782629 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3458) took 1186 ms. Trace:
I20260709 12:22:07.775976  4895 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.783213 27730 rpcz_store.cc:276] 0709 12:22:06.596573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.597068 (+   495us) service_pool.cc:224] Handling call
0709 12:22:06.597286 (+   218us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.602484 (+  5198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.602514 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.602976 (+   462us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.699690 (+ 96714us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:06.699756 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.699848 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.782243 (+1082395us) raft_consensus.cc:1685] finished
0709 12:22:07.782284 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.782547 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5377,"thread_start_us":58774,"threads_started":1,"wal-append.queue_time_us":386113}
I20260709 12:22:07.786345  4895 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.745352  4935 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:07.787595  4935 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.788096 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:07.788841  4895 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.789853  4935 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:07.790491  4935 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:22:07.791621  4914 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:22:07.793533  4939 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:07.794289  4939 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:07.801811 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:07.694556  4772 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:07.776881  4904 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.343s	user 0.000s	sys 0.001s
W20260709 12:22:07.816161 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 209690us
W20260709 12:22:07.782369  4916 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.332s	user 0.000s	sys 0.001s
I20260709 12:22:07.822775 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:07.800854  4924 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 12:22:07.799865  4895 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:07.796306  4923 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:22:07.840101  4940 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.847900  4785 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:07.848255  4940 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:07.852149 27661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4077) took 1422 ms (client timeout 1616 ms). Trace:
W20260709 12:22:07.852602 27661 rpcz_store.cc:269] 0709 12:22:06.429911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.430067 (+   156us) service_pool.cc:224] Handling call
0709 12:22:07.851966 (+1421899us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b540005fd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:07.852088 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31547648}
I20260709 12:22:07.813405  4921 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.884145  4751 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.885334  4873 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.885735  4727 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:07.885908  4873 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.888041 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4093) took 1277 ms. Trace:
W20260709 12:22:07.889096  4830 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.047s	user 0.009s	sys 0.001s
W20260709 12:22:07.889961  4907 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.374s	user 0.000s	sys 0.001s
I20260709 12:22:07.888448 27663 rpcz_store.cc:276] 0709 12:22:06.610697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.610883 (+   186us) service_pool.cc:224] Handling call
0709 12:22:06.613503 (+  2620us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:06.642054 (+ 28551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.642089 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:06.642592 (+   503us) log.cc:844] Serialized 52 byte log entry
0709 12:22:06.841024 (+198432us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:06.841074 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.887614 (+1046540us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b5400211e54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:07.887664 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.887716 (+    52us) raft_consensus.cc:1685] finished
0709 12:22:07.887731 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.887972 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1694,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6119,"spinlock_wait_cycles":55771392,"thread_start_us":186309,"threads_started":1,"wal-append.queue_time_us":456114}
I20260709 12:22:07.890964 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3317) took 1066 ms. Trace:
I20260709 12:22:07.891629 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3462) took 1256 ms. Trace:
I20260709 12:22:07.891906 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4247) took 1164 ms. Trace:
I20260709 12:22:07.800645  4905 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.892737 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1654) took 1153 ms. Trace:
I20260709 12:22:07.893307 27662 rpcz_store.cc:276] 0709 12:22:06.739811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.739940 (+   129us) service_pool.cc:224] Handling call
0709 12:22:07.892580 (+1152640us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:07.892685 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38392576}
I20260709 12:22:07.893675  4783 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.894409  4905 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:07.884964  4921 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.739126  4926 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:07.892412 27581 rpcz_store.cc:276] 0709 12:22:06.727253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.733523 (+  6270us) service_pool.cc:224] Handling call
0709 12:22:06.733903 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.734249 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.734282 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.734832 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.945167 (+210335us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:06.945208 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.945283 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.891715 (+946432us) raft_consensus.cc:1685] finished
0709 12:22:07.891735 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.891834 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":969,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":53,"thread_start_us":209898,"threads_started":1,"wal-append.queue_time_us":332809}
I20260709 12:22:07.868722 27661 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
W20260709 12:22:07.869660  4918 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:22:07.871706  4938 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:22:07.863883  4895 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.874907  4739 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.917057 27730 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:07.876145  3818 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:07.919626  3818 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:07.876441  4736 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:07.877948  4925 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
W20260709 12:22:07.878562  4879 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.449s	user 0.000s	sys 0.018s
I20260709 12:22:07.879432  4772 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:07.880103  4934 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.881395  4535 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.611s	user 0.007s	sys 0.002s
I20260709 12:22:07.899935 27656 rpcz_store.cc:276] 0709 12:22:06.824430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.824814 (+   384us) service_pool.cc:224] Handling call
0709 12:22:06.825120 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.781884 (+956764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.890725 (+108841us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54001e46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:07.890901 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2188512000}
I20260709 12:22:07.907063 27661 pending_rounds.cc:85] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 16
W20260709 12:22:07.856271  4857 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.019s	sys 0.011s
I20260709 12:22:07.892058 27738 rpcz_store.cc:276] 0709 12:22:06.634851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.704117 (+ 69266us) service_pool.cc:224] Handling call
0709 12:22:06.759316 (+ 55199us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.777443 (+ 18127us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.777474 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.777951 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.853243 (+ 75292us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:06.884412 (+ 31169us) log.cc:844] Serialized 14 byte log entry
0709 12:22:06.884612 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.884688 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.891419 (+1006731us) raft_consensus.cc:1685] finished
0709 12:22:07.891441 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.891567 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87932,"prepare.queue_time_us":154638,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":67,"raft.queue_time_us":28552,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":69388,"thread_start_us":74907,"threads_started":1,"wal-append.queue_time_us":659185}
W20260709 12:22:07.924597 27664 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.008s	sys 0.004s
I20260709 12:22:07.925798 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:07.937451 27664 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.008s	sys 0.004s
I20260709 12:22:07.937675 27729 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:22:07.932519 27555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.000s	sys 0.008s
W20260709 12:22:07.940552 27555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.000s	sys 0.008s
I20260709 12:22:07.934144 27661 pending_rounds.cc:107] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:22:07.942577 27661 raft_consensus.cc:2889] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:07.923588  4834 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.003s
W20260709 12:22:07.943665 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 279497us
I20260709 12:22:07.945389  3818 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.947069  3818 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:07.947376 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:07.916079  4830 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.935103  4924 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:22:07.936898  4857 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.019s	sys 0.012s
I20260709 12:22:07.931722  4938 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:07.955178  4938 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.940145  4934 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:07.961256  4934 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.968977  4921 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.969189  4895 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:07.971784  4923 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:22:07.948624  4753 consensus_queue.cc:1175] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 34 status { last_received { term: 33 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:07.986625 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3973) took 1053 ms. Trace:
I20260709 12:22:07.481941 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:07.989141 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.996210 27582 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260709 12:22:07.996622 27582 pending_rounds.cc:85] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:22:07.996682 27581 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:22:07.996968 27582 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:22:07.997238 27582 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:07.996030 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:07.933291  4784 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.987074 27736 rpcz_store.cc:276] 0709 12:22:06.933541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.933718 (+   177us) service_pool.cc:224] Handling call
0709 12:22:06.934003 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.934207 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:07.274158 (+339951us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.986079 (+711921us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.986248 (+   169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:07.986294 (+    46us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:07.986356 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.986428 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.986555 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":576409,"prepare.queue_time_us":128738,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":579539,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"thread_start_us":400746,"threads_started":2,"wal-append.queue_time_us":600011}
I20260709 12:22:08.004786 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 3967) took 1246 ms. Trace:
I20260709 12:22:08.005937  4873 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:08.015157 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.016839 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:08.007380  4925 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 12:22:08.019944 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 367061us
I20260709 12:22:08.018838 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:07.998438  4643 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.020941 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:08.024248 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.008139  4895 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.026149 27582 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:08.028102 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.012094  4857 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.029126 27582 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:07.949862  4834 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:08.031121 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 208572us
I20260709 12:22:08.017088  4724 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:07.998080  4830 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.034664 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.008932 27737 rpcz_store.cc:276] 0709 12:22:06.758597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.856122 (+ 97525us) service_pool.cc:224] Handling call
0709 12:22:06.856543 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.856792 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:22:07.321788 (+464996us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.004083 (+682295us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.004322 (+   239us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:08.004394 (+    72us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:22:08.004473 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.004567 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.004713 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":656680,"prepare.queue_time_us":207759,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":508415,"spinlock_wait_cycles":85773056,"thread_start_us":429170,"threads_started":2,"wal-append.queue_time_us":453329}
I20260709 12:22:08.053063  4830 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:08.019479  4909 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:08.037356 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3322) took 1136 ms. Trace:
I20260709 12:22:08.035640 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:08.038357 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3310) took 1297 ms. Trace:
I20260709 12:22:08.055584 27658 rpcz_store.cc:276] 0709 12:22:06.901489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.547100 (+645611us) service_pool.cc:224] Handling call
0709 12:22:07.596152 (+ 49052us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.596398 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:07.640203 (+ 43805us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.036740 (+396537us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:08.036800 (+    60us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:08.037031 (+   231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.037127 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.037290 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":133576,"prepare.queue_time_us":55559,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":42,"raft.queue_time_us":131627,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":266151,"threads_started":1,"wal-append.queue_time_us":402567}
I20260709 12:22:08.038915 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3447) took 1969 ms. Trace:
I20260709 12:22:08.030565  4857 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.058095 27732 rpcz_store.cc:276] 0709 12:22:06.069735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.069890 (+   155us) service_pool.cc:224] Handling call
0709 12:22:06.095764 (+ 25874us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.143604 (+ 47840us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.143656 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.149789 (+  6133us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.263551 (+113762us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:06.263596 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.263666 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.877743 (+1614077us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:08.038580 (+160837us) raft_consensus.cc:1685] finished
0709 12:22:08.038612 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.038844 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10577,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":9867,"spinlock_wait_cycles":3858304,"thread_start_us":96855,"threads_started":1,"wal-append.queue_time_us":224464}
I20260709 12:22:08.029315  4643 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.042387  4939 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:08.044531  4941 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:07.939335  4535 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.045846  4558 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:08.068323 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.069095 27656 raft_consensus.cc:2376] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33. Re-sending same reply.
I20260709 12:22:08.070328 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.046283  4674 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:08.047835  4926 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:08.048929  4872 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.871s	user 0.007s	sys 0.002s
I20260709 12:22:08.050004  4834 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.050601  4724 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.033712  4928 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:22:08.045267 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98006.1us
I20260709 12:22:08.082931 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:08.084669 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.086175 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.056118 27664 rpcz_store.cc:276] 0709 12:22:06.741277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.780801 (+ 39524us) service_pool.cc:224] Handling call
0709 12:22:06.781093 (+   292us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:06.781493 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.781523 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:06.814295 (+ 32772us) log.cc:844] Serialized 52 byte log entry
0709 12:22:06.944499 (+130204us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:07.045370 (+100871us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.938313 (+892943us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.038048 (+ 99735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.038114 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.038152 (+    38us) raft_consensus.cc:1685] finished
0709 12:22:08.038162 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.038296 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107520,"prepare.queue_time_us":15221,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":4387,"raft.queue_time_us":366404,"raft.run_cpu_time_us":518,"raft.run_wall_time_us":45971,"thread_start_us":1072922,"threads_started":3,"wal-append.queue_time_us":232153}
I20260709 12:22:08.011662  4784 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.067011 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:08.072031 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 273313us
I20260709 12:22:08.073530 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.089270  4939 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.089564 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3463) took 1414 ms. Trace:
I20260709 12:22:08.089839  4939 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.092671 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:08.093665 27738 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:22:08.092164  4939 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.066471  4873 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:08.071390  4783 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.095639  4873 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.096135  4873 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:08.095923  4783 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.099634 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.100252 27661 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:08.100876  4643 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.101377  4592 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:08.101805  4592 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.102164  4592 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:08.102697  4789 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:22:08.103768 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 689310us
I20260709 12:22:08.105095 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:08.106218  4591 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29.
W20260709 12:22:08.106518 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 85702.6us
I20260709 12:22:08.078416  4745 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.107278 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:08.107861  4745 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.108374  4857 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.108604 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:08.108912  4915 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:08.109388  4915 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:08.109413  4803 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:08.109962  4915 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:08.110180  4745 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.110325  4803 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:08.108385  4821 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:08.075135  4674 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.089897 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.113811 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.114596 27658 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:08.114876  4844 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:22:08.115466  4844 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.116466  4844 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.079862 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.118924 27662 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:08.119585 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:08.119897 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.120484  4905 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:08.121111 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.122474 27662 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:08.122727 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.057576 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:08.113220  4674 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.117290  4857 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:22:08.095835  4903 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.248s	user 0.000s	sys 0.003s
I20260709 12:22:08.112959 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:08.123381 27656 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:08.041193  4940 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.116671 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:08.122404 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1688) took 1436 ms. Trace:
I20260709 12:22:08.134078 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.137301 27583 rpcz_store.cc:276] 0709 12:22:06.685789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.698670 (+ 12881us) service_pool.cc:224] Handling call
0709 12:22:06.699069 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.053929 (+354860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.119313 (+1065384us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b540011bf54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:08.122348 (+  3035us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607698816}
I20260709 12:22:08.122440 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.036433  4772 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:22:08.032168  4895 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.089955 27735 rpcz_store.cc:276] 0709 12:22:06.675200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.819083 (+143883us) service_pool.cc:224] Handling call
0709 12:22:06.819461 (+   378us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:06.842327 (+ 22866us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.842360 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:06.874972 (+ 32612us) log.cc:844] Serialized 52 byte log entry
0709 12:22:07.111203 (+236231us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:07.322263 (+211060us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.979070 (+656807us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b18004cc780. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:07.995589 (+ 16519us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.078152 (+ 82563us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.078240 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.089311 (+ 11071us) raft_consensus.cc:1685] finished
0709 12:22:08.089338 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.089505 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84898,"prepare.queue_time_us":211469,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":9696,"raft.queue_time_us":765402,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":19034,"spinlock_wait_cycles":21100416,"thread_start_us":79991,"threads_started":1,"wal-append.queue_time_us":605518}
I20260709 12:22:08.140295  3818 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:08.127270 27661 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
W20260709 12:22:08.127573 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70075.2us
I20260709 12:22:08.135529 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.144245 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.145033 27583 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
W20260709 12:22:08.129669 27701 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.870s (SENT)
I20260709 12:22:08.121119  4905 consensus_queue.cc:1243] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 29.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:22:08.116734 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.132898 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.155982 27658 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:08.156526 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.141590 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.159818 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.134454 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.147225 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:08.162220 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.164088 27656 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:22:08.165577 27656 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:22:08.160818 27663 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:08.132874  4785 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:08.101640  4643 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:22:08.078593  4753 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.081s
I20260709 12:22:08.167101 27656 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:22:08.077342  4931 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.081s
I20260709 12:22:08.132155  4779 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
W20260709 12:22:08.172197 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59435.5us
I20260709 12:22:08.133900  4945 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:08.176738 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.178067 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.171877 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:08.175278 27657 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.036s	sys 0.070s
I20260709 12:22:08.182895 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:08.136435  4572 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.134078  4877 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:08.136029  4821 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.135277  4745 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:22:08.215189 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85558.4us
I20260709 12:22:08.177846 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.152145  4928 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:22:08.206733 27657 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.036s	sys 0.070s
I20260709 12:22:08.133541  4934 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:22:08.244683  4752 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:08.245137  4934 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 29.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:22:08.213043 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.216220 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.171362  4926 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:08.247309 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.251364 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:08.200354  4798 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.799s	user 0.012s	sys 0.001s
I20260709 12:22:08.160804  4940 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.224025 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.180166  4944 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.224957  4750 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:08.220167  4572 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.228574  4779 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:08.229070  4643 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:08.183763  4785 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:08.179881  4767 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:08.266428 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.212735  4820 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:08.178807  4830 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:08.252774  4893 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:08.249830  4926 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.232306  4784 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.202664  4857 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:08.232904  4753 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.233765  4724 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.236531  4895 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.215751  4909 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.215163  4877 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.238667  4591 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:08.279690  4893 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.212287  4333 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:08.241010 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.242061 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.243983  4905 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:08.302467 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.303229 27738 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:08.284781 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 162391us
I20260709 12:22:08.256880 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:08.220157  4745 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:08.286689  4558 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:08.256750  4748 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:08.258090  4628 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:08.265699  4947 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.266331  4802 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.274307  4643 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.282979  4857 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.278774  4830 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.281284  4926 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:08.299075 27661 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:08.302420  4893 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:08.264569  4944 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.231617  4821 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.340026 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.340641 27729 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:08.341428 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.341776  4915 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.342288  4821 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:08.342520 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.265120  4940 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.343825  4857 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.344874  4940 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.290341  4834 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:08.312525 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:08.258394  4798 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.346999 27659 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:22:08.348114  4798 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:08.310536  4333 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:08.310140  4779 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.005s
I20260709 12:22:08.348917 27737 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:22:08.307660  4745 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:08.350940 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:08.309011 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 162485us
I20260709 12:22:08.310343  4748 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.296837  4877 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:08.332427 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.279028  4767 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.329706  4535 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:08.343050 27736 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
W20260709 12:22:08.345956 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4063) took 3316 ms (client timeout 1870 ms). Trace:
I20260709 12:22:08.354964 27663 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:08.292865  4753 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.307871  4847 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.355697  4748 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:08.361095  4592 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.362383 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.362507 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:08.363193 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.366590 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.363754  4940 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:08.367189 27733 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:08.281849  4820 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.369419 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:08.356208 27660 rpcz_store.cc:269] 0709 12:22:05.019912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.068212 (+ 48300us) service_pool.cc:224] Handling call
0709 12:22:08.335821 (+3267609us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:08.335975 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909875456}
I20260709 12:22:08.381526 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.335548  4830 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:08.396646 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44493.3us
I20260709 12:22:08.401201 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.349867  4857 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:08.337934  4628 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.406905 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.406613 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.408440 27661 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:08.411026 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48614.1us
I20260709 12:22:08.398065 27733 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:08.364869  4333 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.415493  4858 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:08.415949  4858 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:08.416356  4858 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:08.416711  4944 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.419217 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.336661  4802 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.424602  4753 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.425339  4802 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.426530  4767 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:08.430540  4873 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.432119 27729 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:08.402382  4761 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:08.434136  4858 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.435756  4915 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:08.410750  4963 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.438076 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.438916  4592 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.439882  4802 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:08.440207 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.486688 27661 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:08.440974  4964 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:08.395012  4847 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.442657  4572 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.450609 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.331193  4947 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.454115 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.454589  4628 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.408025  4643 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.460225  4905 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:08.409354  4940 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:08.461114  4958 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:08.468603 27730 raft_consensus.cc:2410] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:08.474071 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:08.488669 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.490306 27663 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:08.475603  4779 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:22:08.498736 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.384476  4785 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.17, Last appended by leader: 17, Current term: 32, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.499969 27732 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:22:08.517936 27733 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:08.497121  4752 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.515822 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.530614 27733 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:08.516126 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.515388 27661 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:22:08.532667 27731 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:22:08.533246 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.533915 27661 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:08.534780 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.535400 27661 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:08.536242 27663 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:08.537632 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.535677 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.541574  3818 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:08.541927  3818 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.542244  3818 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:08.543135 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.491477  4802 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.487003  4963 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.517596  4964 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:08.546267 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.555183 27661 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:08.556543 27731 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:08.556973 27731 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:08.549322  4895 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:08.551098  4959 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:08.511021  4905 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.570520 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.575078 27733 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:22:08.518518  4748 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.539301  4752 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.556365  4767 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.557845  4964 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.581791 27663 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:22:08.577920 27731 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.586143 27731 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:08.571254  3818 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.591544 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.577545  4895 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.592370 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:08.593932 27661 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:08.584051  4857 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.597335  4858 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.597528  4752 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.597864  4893 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.598649  4895 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:08.601044  4905 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.602514 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.585888  4964 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:08.587373  4767 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.629879  4963 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.630645  4893 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.587316  4959 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:08.632009  4959 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.633049  4748 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.604357  4877 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.635694  4767 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.637073  4877 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.605631  3818 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.606413 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.608796 27731 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 12:22:08.578876  4965 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:08.639567 27731 pending_rounds.cc:85] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 13
I20260709 12:22:08.639727  4965 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.640232  4965 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:08.614295 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.616377 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.615540  4857 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.624424  4947 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.639887 27731 pending_rounds.cc:107] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:22:08.646931 27731 raft_consensus.cc:2889] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:08.643527 27661 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:08.643846  3818 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:08.644407 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:08.644743 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:08.577454  4926 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.645474 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:08.650745 27588 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:08.651213 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.660530  3818 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.659190 27661 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:08.664072 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.664837 27661 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:08.632004  4674 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:08.665963 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.666548 27661 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:08.664327 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.667946 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.671350  4802 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.646412  4820 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:08.672381  3818 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.675290  4947 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:08.677532  4796 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:08.685290  4929 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.697s	user 0.002s	sys 0.005s
I20260709 12:22:08.687740 27588 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:08.688400 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:08.693238 27663 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:08.694540 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.696538 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.696959 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.700419 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.701017 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.714591 27584 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:08.701366 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.703092 27580 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:22:08.715442 27661 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:08.646883  4905 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.712258 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.694278 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:08.750092 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.660145  4958 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:08.741533 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:08.685334  4857 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.743462  4905 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:08.750351 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56158.7us
I20260709 12:22:08.674819  4830 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.751262 27734 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:08.726071  4947 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.751245 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:08.770752 27661 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.770886 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:08.771237 27661 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:22:08.755263 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.773379  4779 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.773713  4796 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:08.725848  4751 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:08.756438 27660 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:08.774614  4751 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.760012 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.775111  4751 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:08.775560 27584 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:22:08.776432 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.777328 27584 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 28 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:22:08.755759 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:08.774189  4796 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:08.773898  4779 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.782845  4796 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:08.784368  4926 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.784511  4779 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.774699  4940 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.785406  4745 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.787724 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.785336  4940 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.788123  4947 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.759822  4958 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.793376  4958 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.794250 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:08.794531 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.762351 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.796571 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:08.797252  3818 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:08.797698  3818 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:08.798522  3818 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.800005  4674 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:08.800209 27584 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 16. (term mismatch)
I20260709 12:22:08.800432  4674 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.800658 27584 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:22:08.800932  4674 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:08.801015 27584 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:22:08.801471 27584 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:08.803682  4945 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:08.760314  4877 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:08.804951 27660 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:22:08.805428  4877 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:08.806537  4877 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.808665  4830 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.797941  4751 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.810609 27587 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:08.814241 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4087) took 2309 ms. Trace:
I20260709 12:22:08.826391 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:08.837205 27586 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:08.814592 27655 rpcz_store.cc:276] 0709 12:22:06.505239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.509604 (+  4365us) service_pool.cc:224] Handling call
0709 12:22:07.854028 (+1344424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.037997 (+183969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.038025 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:08.038474 (+   449us) log.cc:844] Serialized 26 byte log entry
0709 12:22:08.354508 (+316034us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:08.398951 (+ 44443us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.399241 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.399352 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.468254 (+ 68902us) raft_consensus.cc:1685] finished
0709 12:22:08.468273 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.814034 (+345761us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540005fd50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:08.814152 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":187471,"prepare.queue_time_us":35142,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":189142,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":8686,"spinlock_wait_cycles":3090896640,"thread_start_us":312454,"threads_started":1,"wal-append.queue_time_us":424106}
I20260709 12:22:08.864086 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:08.790928  4796 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:08.835815  4751 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:08.863839  4964 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:08.887920  4965 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.028352  4959 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.029088  4959 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.030176  4674 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.037454  4959 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:09.055797 27734 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 28 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:22:09.053563 27655 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
W20260709 12:22:09.054401 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.057097 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 212195us
I20260709 12:22:09.058598 27656 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:09.056628  4927 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:09.107170  4927 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.042495  4895 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.058490  4783 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:09.095280  4834 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.062958  4958 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:09.120596  4789 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:09.132356  4789 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.134966  4789 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.135408 27661 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [31.18-33.19]
I20260709 12:22:09.136673 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.195180 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.117477  4834 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.121243  4958 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.139119  4927 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.139966  4895 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.120425  4674 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.154670  4965 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.156111  4964 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.199770  4789 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.002s
W20260709 12:22:09.157109  4972 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:22:09.158563 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.159813 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.160706  4779 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:09.109660  4905 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.206240  4779 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:09.161283  4932 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.369s	user 0.004s	sys 0.000s
I20260709 12:22:09.161811  4938 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.163235 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1543) took 1031 ms. Trace:
I20260709 12:22:09.212800 27736 rpcz_store.cc:276] 0709 12:22:08.132182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.132630 (+   448us) service_pool.cc:224] Handling call
0709 12:22:09.163019 (+1030389us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:09.163159 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":428708736}
I20260709 12:22:09.207098  4958 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:09.167883 27658 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:09.176541  4796 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.177635 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.178534 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:09.185847 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.118521  4785 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.164853 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 3979) took 1004 ms. Trace:
I20260709 12:22:09.188855 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1538) took 1096 ms. Trace:
I20260709 12:22:09.195241 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4103) took 1012 ms. Trace:
I20260709 12:22:09.198184 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:09.142107 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:09.165020 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.258400 27737 rpcz_store.cc:276] 0709 12:22:08.091995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.111877 (+ 19882us) service_pool.cc:224] Handling call
0709 12:22:08.112225 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.355517 (+243292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.188465 (+832948us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540029fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:09.188773 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":543675264}
I20260709 12:22:09.219414 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.260285 27730 rpcz_store.cc:276] 0709 12:22:08.160226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.418950 (+258724us) service_pool.cc:224] Handling call
0709 12:22:09.164652 (+745702us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b54000a46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:09.164795 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111538048}
I20260709 12:22:09.262179 27582 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:09.262930 27661 rpcz_store.cc:276] 0709 12:22:08.182644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.134678 (+952034us) service_pool.cc:224] Handling call
0709 12:22:09.135107 (+   429us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:09.194961 (+ 59854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.195188 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:09.264796 27661 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 18
I20260709 12:22:09.266309 27589 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:09.254727 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4095) took 1074 ms. Trace:
I20260709 12:22:09.267402 27656 rpcz_store.cc:276] 0709 12:22:08.180617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.212768 (+ 32151us) service_pool.cc:224] Handling call
0709 12:22:09.254535 (+1041767us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b5400210554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:09.254664 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1868786432}
I20260709 12:22:09.269337 27656 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:22:09.270743 27656 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:09.265123 27661 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:22:09.275593 27661 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:09.240020 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3319) took 2398 ms. Trace:
I20260709 12:22:09.279851 27657 rpcz_store.cc:276] 0709 12:22:06.841444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.376344 (+534900us) service_pool.cc:224] Handling call
0709 12:22:07.395880 (+ 19536us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:08.220794 (+824914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.220842 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:08.239604 (+ 18762us) log.cc:844] Serialized 52 byte log entry
0709 12:22:08.338237 (+ 98633us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:08.338273 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.239418 (+901145us) spinlock_profiling.cc:243] Waited 79.8 ms on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:09.239491 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.239569 (+    78us) raft_consensus.cc:1685] finished
0709 12:22:09.239590 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.239936 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25258,"raft.queue_time_us":294242,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":18684,"spinlock_wait_cycles":183526144,"thread_start_us":810859,"threads_started":2,"wal-append.queue_time_us":175282}
I20260709 12:22:09.291498 27583 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:09.332782 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.335284 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.238765 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:09.340073 27735 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:09.340664 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.341544 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 163093us
W20260709 12:22:09.342140 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103442us
I20260709 12:22:09.350891 27587 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:09.343685 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:09.354962 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:09.355777 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:09.359283 27701 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:22:09.359593 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.359786 27701 leader_election.cc:403] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:09.361294 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.361996 27736 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:09.361418 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.362869 27737 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:09.255349  4958 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:09.253341  4745 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:09.238062  4725 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.387437 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.388911 27737 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:09.343911  4944 raft_consensus.cc:2764] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:22:09.394300 27736 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:09.375888 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:09.376984 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.374275 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:09.389649 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.392489 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.396296 27730 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:09.217921  4938 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.395870 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:09.355180  4848 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:22:09.364815  4591 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:09.416308 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.418190 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:09.367036  4055 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:09.336511  4785 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.421482 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:09.390583  4783 raft_consensus.cc:2764] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:22:09.434790 27730 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:22:09.424286 27584 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:09.435177 27730 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:09.429803 28337 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:09.429641 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.429363 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.436121 28336 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:09.436314 28331 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:09.437700 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:09.440066 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.440682 27735 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:09.448766 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:09.456378 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.404047  4935 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:09.397485  4745 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:09.457937 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:09.437747 27730 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.459467  3818 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.417907  4591 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.462469 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.461535  3818 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.484737 27730 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:22:09.485658  4980 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:22:09.433940  4535 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:09.492132 27736 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:22:09.484591 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.496315 27731 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:22:09.396883  4725 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.454828  4945 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:09.459955  4935 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.420200  4958 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.541229  4931 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:22:09.529747  4783 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:09.475265  4750 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:09.494719  4591 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:09.561106 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.560287 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.568688 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.530148  4628 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:09.571271 27735 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:09.575947 27729 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:09.582479 27729 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:09.535563  4779 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:09.576313  4535 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 12:22:09.578081  4725 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:09.580425 27734 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:09.583205 27729 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.560562  4935 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:09.482395  4745 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.584697  4628 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.464916 27737 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:09.594766 27734 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:09.625372 27734 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.625883 27589 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:09.626346 27734 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:09.591598  3818 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.595157 27729 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:09.632400 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3318) took 2799 ms. Trace:
I20260709 12:22:09.632994 27659 rpcz_store.cc:276] 0709 12:22:06.832923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.271624 (+438701us) service_pool.cc:224] Handling call
0709 12:22:08.336278 (+1064654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.347525 (+ 11247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.632124 (+1284599us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540022a950. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:09.632318 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2447636224}
I20260709 12:22:09.610028 27730 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:22:09.641836 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:09.591259 27731 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:09.642839 27659 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:22:09.642037 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 460417us
I20260709 12:22:09.645817 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:09.646428 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:09.611387  4628 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:09.657856  4761 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.658730  4761 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.596545  4958 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.616411  4927 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:22:09.670992 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:09.673308 27659 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:09.680974 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.681233 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.681473 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.682547 28331 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260709 12:22:09.682778 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.683763 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.684319 28334 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:09.685472 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.686407 28332 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:09.681946 28336 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
W20260709 12:22:09.695479 27662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.009s	sys 0.020s
W20260709 12:22:09.695927 27662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.009s	sys 0.020s
W20260709 12:22:09.696463 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50114.2us
I20260709 12:22:09.717943 27734 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:22:09.718376 27734 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 17
I20260709 12:22:09.718603 27734 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 31.18
I20260709 12:22:09.718837 27734 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:09.719139 27734 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 33.19
I20260709 12:22:09.720166 27734 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:09.719718 27729 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:09.756107 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:09.758559 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60413.3us
I20260709 12:22:09.759109 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:09.759720 28337 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:09.767009 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:09.767975 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.768769 27583 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
W20260709 12:22:09.722173  4905 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.002s	sys 0.029s
I20260709 12:22:09.790347 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.791607 27589 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:22:09.793886 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.794816 27583 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:22:09.796416  4982 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:22:09.796535 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.797343  4761 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.798990 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:09.800177 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:09.759239  4796 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:09.811524 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:09.812441 27583 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:09.802965 28337 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:09.829057 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.759647  4977 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:22:09.790572  4905 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.002s	sys 0.029s
W20260709 12:22:09.840394 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.034s	sys 0.046s
I20260709 12:22:09.843235 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:09.849068 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.034s	sys 0.046s
I20260709 12:22:09.844484 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:09.849984 28333 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 18.
I20260709 12:22:09.777874  4785 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.763715  4958 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:09.847703 27732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.019s	sys 0.063s
W20260709 12:22:09.891942 27732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.019s	sys 0.063s
I20260709 12:22:09.866955  4725 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:09.891050 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4244) took 1189 ms. Trace:
I20260709 12:22:09.893033  4725 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:09.892979 27581 rpcz_store.cc:276] 0709 12:22:08.701596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.701723 (+   127us) service_pool.cc:224] Handling call
0709 12:22:08.702044 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.702309 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:09.171596 (+469287us) log.cc:844] Serialized 14 byte log entry
0709 12:22:09.890209 (+718613us) log.cc:844] Serialized 14 byte log entry
0709 12:22:09.890511 (+   302us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:09.890585 (+    74us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:09.890663 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.890765 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.890966 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14952,"prepare.queue_time_us":49254,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":1479,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":192,"thread_start_us":1109783,"threads_started":2}
I20260709 12:22:09.900182 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:09.901079 27581 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:09.908658  4745 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:09.910866  4750 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:09.911513  4750 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:09.918949 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.922501  4959 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:09.923595  4959 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:22:09.924002  4959 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:22:09.933065 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.933772 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.942345 27583 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:09.952342  4750 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.956602 27735 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:09.960134 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.962245 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:09.981974 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.988663 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:09.990738 27734 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:09.997988  4938 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:09.998584  4750 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.013345 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.023972  4847 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:10.024784  4847 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.025352  4847 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:09.955346  4643 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:10.018105 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:10.043089 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:10.015010 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.032913 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.044153 27735 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:22:10.044459 27663 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:10.044589 27583 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:09.903949  4958 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:10.071906 27664 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.007s	sys 0.032s
W20260709 12:22:10.072316 27664 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.007s	sys 0.032s
I20260709 12:22:10.054801 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:10.073863 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:10.074993 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.018s	sys 0.017s
W20260709 12:22:10.075336 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.018s	sys 0.017s
I20260709 12:22:10.075610 27734 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:10.076269 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4111) took 1720 ms. Trace:
I20260709 12:22:10.076687 27657 rpcz_store.cc:276] 0709 12:22:08.355958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.282119 (+926161us) service_pool.cc:224] Handling call
0709 12:22:09.337153 (+ 55034us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.337361 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:09.356013 (+ 18652us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.075878 (+719865us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.075930 (+    52us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:10.075981 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.076131 (+   150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.076227 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":62837,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":12696,"thread_start_us":719457,"threads_started":1}
W20260709 12:22:10.082497 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.026s	sys 0.062s
W20260709 12:22:10.083128 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.026s	sys 0.062s
I20260709 12:22:10.086326 27663 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:10.093974 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.100672 27663 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:10.060016  4745 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:10.051540  4977 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:10.111703  4834 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:10.082486  4990 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:10.112224  4977 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:10.110705 27585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.021s	sys 0.025s
I20260709 12:22:10.114858 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4104) took 1932 ms. Trace:
W20260709 12:22:10.137362  4931 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.589s	user 0.010s	sys 0.000s
I20260709 12:22:09.925004  4959 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.137494 27662 rpcz_store.cc:276] 0709 12:22:08.182883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.155260 (+972377us) service_pool.cc:224] Handling call
0709 12:22:09.155645 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.156088 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:09.156425 (+   337us) log.cc:844] Serialized 14 byte log entry
0709 12:22:09.696334 (+539909us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.755457 (+ 59123us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.755691 (+   234us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:09.755742 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.756503 (+   761us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.114653 (+358150us) raft_consensus.cc:1685] finished
0709 12:22:10.114672 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.114803 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3763,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":103323,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":153,"thread_start_us":539499,"threads_started":1,"wal-append.queue_time_us":833497}
I20260709 12:22:10.139156  4959 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:10.115280 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:10.115609 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:10.112953 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.119127 28330 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:10.124873  4628 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.072021  4940 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.125381  4977 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:10.108377  4779 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:10.143945  4779 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:10.144405  4779 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:10.147892 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.021s	sys 0.001s
W20260709 12:22:10.148360 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.021s	sys 0.001s
W20260709 12:22:10.148653 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 789537us
I20260709 12:22:10.151221 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:10.151932 27734 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:10.119514 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:10.136442 27585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.022s	sys 0.025s
I20260709 12:22:10.120281 28336 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:22:10.157021 27657 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:10.125881 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:10.172590 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49516.5us
I20260709 12:22:10.175477 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3347) took 1605 ms. Trace:
I20260709 12:22:10.176101 27657 rpcz_store.cc:276] 0709 12:22:08.570016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.157964 (+1587948us) service_pool.cc:224] Handling call
0709 12:22:10.174759 (+ 16795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.175000 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.175048 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.175103 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:10.175161 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.175231 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.175402 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:10.154982 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.018s	sys 0.017s
W20260709 12:22:10.179207 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.018s	sys 0.018s
W20260709 12:22:10.179984 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 731252us
W20260709 12:22:10.182298 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.001s	sys 0.038s
W20260709 12:22:10.182811 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.001s	sys 0.038s
I20260709 12:22:10.185117 27583 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
W20260709 12:22:10.185679 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.016s	sys 0.023s
W20260709 12:22:10.187129 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.016s	sys 0.023s
W20260709 12:22:10.189142 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 736590us
I20260709 12:22:10.191668 27581 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:10.192001 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:10.125869  4935 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.520s	user 0.007s	sys 0.002s
I20260709 12:22:10.193974 27657 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:22:10.194790 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1670) took 1611 ms. Trace:
I20260709 12:22:10.195374 27657 rpcz_store.cc:276] 0709 12:22:08.583872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.177442 (+1593570us) service_pool.cc:224] Handling call
0709 12:22:10.193674 (+ 16232us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.194478 (+   804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.194726 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:10.134148  4783 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.582s	user 0.007s	sys 0.000s
I20260709 12:22:10.200240 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.137836  4931 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:22:10.212401 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:10.148805  4877 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.017s	sys 0.002s
W20260709 12:22:10.124524  4984 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.186s	user 0.002s	sys 0.000s
I20260709 12:22:10.133864  4725 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:10.177712  4983 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.269s	user 0.000s	sys 0.005s
I20260709 12:22:10.127173  4845 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:10.209332 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.004s	sys 0.038s
I20260709 12:22:10.150691  4628 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:10.215668 27661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s	user 0.001s	sys 0.023s
W20260709 12:22:10.216110 27582 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.008s	sys 0.017s
I20260709 12:22:10.212822 27657 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:10.217383 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:10.208030  4767 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.077s
W20260709 12:22:10.216671 27656 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.027s	sys 0.028s
W20260709 12:22:10.351643 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.352494 27656 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.027s	sys 0.028s
W20260709 12:22:10.339299 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.325315 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4249) took 1573 ms. Trace:
W20260709 12:22:10.353753 27661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s	user 0.003s	sys 0.028s
W20260709 12:22:10.268227 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86223.2us
I20260709 12:22:10.329053 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:10.356513 27582 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.009s	sys 0.022s
W20260709 12:22:10.285511 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.028s	sys 0.022s
I20260709 12:22:10.218786 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:10.291776 27584 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
W20260709 12:22:10.307068 27658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.951s	user 0.018s	sys 0.034s
I20260709 12:22:10.286463 27581 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
W20260709 12:22:10.359877 27658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.004s	user 0.018s	sys 0.035s
W20260709 12:22:10.315970 27738 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.000s	sys 0.028s
W20260709 12:22:10.360432 27738 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.000s	sys 0.030s
W20260709 12:22:10.357766 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.007s	sys 0.040s
I20260709 12:22:10.269011 27663 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:22:10.320843 27628 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.014s	sys 0.016s
W20260709 12:22:10.361552 27628 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.016s	sys 0.016s
W20260709 12:22:10.356911 27660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.018s	sys 0.025s
W20260709 12:22:10.361958 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 832546us
W20260709 12:22:10.362496 27660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.019s	sys 0.025s
W20260709 12:22:10.357455 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.030s	sys 0.022s
I20260709 12:22:10.357877 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3348) took 1758 ms. Trace:
I20260709 12:22:10.364449 27657 rpcz_store.cc:276] 0709 12:22:08.599205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.196139 (+1596934us) service_pool.cc:224] Handling call
0709 12:22:10.212400 (+ 16261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.357624 (+145224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.357824 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:10.363859 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 796371us
I20260709 12:22:10.194917  4830 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.370527 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:10.371380 27581 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:10.372648 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.373917 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4099) took 2192 ms. Trace:
I20260709 12:22:10.375212 27660 rpcz_store.cc:276] 0709 12:22:08.181641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.807933 (+626292us) service_pool.cc:224] Handling call
0709 12:22:08.808202 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.820952 (+ 12750us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:08.962454 (+141502us) log.cc:844] Serialized 14 byte log entry
0709 12:22:09.341676 (+379222us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.342271 (+   595us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.342421 (+   150us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:10.362839 (+1020418us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.373486 (+ 10647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.373596 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.373655 (+    59us) raft_consensus.cc:1685] finished
0709 12:22:10.373675 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.373841 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":242725,"prepare.queue_time_us":500817,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":96,"raft.queue_time_us":19085,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":28839,"thread_start_us":1243149,"threads_started":2,"wal-append.queue_time_us":571463}
I20260709 12:22:10.376948 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3349) took 1777 ms. Trace:
I20260709 12:22:10.208263  4847 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:10.213590  4848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.013s	sys 0.032s
I20260709 12:22:10.383312 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:10.371151 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 152481us
W20260709 12:22:10.357591  4965 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.000s	sys 0.033s
W20260709 12:22:10.215384  4877 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.018s	sys 0.002s
I20260709 12:22:10.211982  4783 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:10.380671 27657 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:22:10.379837 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:10.414950 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.420843 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4118) took 1066 ms. Trace:
I20260709 12:22:10.421362 27657 rpcz_store.cc:276] 0709 12:22:09.354685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.376308 (+1021623us) service_pool.cc:224] Handling call
0709 12:22:10.376634 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.420560 (+ 43926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.420785 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:10.368842 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:10.424151 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55397.2us
I20260709 12:22:10.212844  4789 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.176s
I20260709 12:22:10.353780 27580 rpcz_store.cc:276] 0709 12:22:08.752035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.752339 (+   304us) service_pool.cc:224] Handling call
0709 12:22:09.065875 (+313536us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:22:09.066079 (+   204us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.112610 (+ 46531us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.112639 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.113138 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.460646 (+347508us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:09.460678 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.460730 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.123978 (+663248us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:10.325020 (+201042us) raft_consensus.cc:1685] finished
0709 12:22:10.325051 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.325236 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48547,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":5764,"spinlock_wait_cycles":134069248,"thread_start_us":347234,"threads_started":1,"wal-append.queue_time_us":777420}
W20260709 12:22:10.407204  4965 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.000s	sys 0.034s
W20260709 12:22:10.356406  4055 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.008s	sys 0.035s
I20260709 12:22:10.377915  4845 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.316941  4873 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:10.291738  4761 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:10.439950 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.378142 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.382725  4931 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.012s
W20260709 12:22:10.395485  4987 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:22:10.395879 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4247) took 1692 ms. Trace:
I20260709 12:22:10.216624  4785 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.172s
I20260709 12:22:10.215349  4940 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.173s
I20260709 12:22:10.212178  4935 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.383313  4830 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:10.384616  4848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.016s	sys 0.043s
I20260709 12:22:10.370061  4796 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:10.401471 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.015s	user 0.021s	sys 0.034s
I20260709 12:22:10.423072 28333 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:10.383606 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:10.485846 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.100s	user 0.021s	sys 0.034s
I20260709 12:22:10.426424 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.485975 27662 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: [34.21-34.21]
W20260709 12:22:10.282891  4945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.024s	sys 0.017s
I20260709 12:22:10.423158 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.487035 27584 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:10.409850  4591 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.825s	user 0.012s	sys 0.000s
I20260709 12:22:10.377874 27663 rpcz_store.cc:276] 0709 12:22:08.599634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.375946 (+1776312us) service_pool.cc:224] Handling call
0709 12:22:10.376449 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.376635 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.376662 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.376705 (+    43us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:10.376738 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.376802 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.376896 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:10.491452 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.492041 27663 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:10.431712 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.423295  4958 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.433257 28332 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:22:10.425952 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:10.436146 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4259) took 1214 ms. Trace:
I20260709 12:22:10.427781  4938 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:10.369683  4944 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:10.422966  4847 raft_consensus.cc:2764] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:22:10.442045 27588 rpcz_store.cc:276] 0709 12:22:08.703843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.715183 (+ 11340us) service_pool.cc:224] Handling call
0709 12:22:09.063882 (+348699us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:22:09.064082 (+   200us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.064481 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.064509 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.064991 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.460623 (+395632us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:09.460662 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.460723 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.143868 (+683145us) spinlock_profiling.cc:243] Waited 88.9 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:10.395626 (+251758us) raft_consensus.cc:1685] finished
0709 12:22:10.395651 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.395814 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":216890880,"thread_start_us":385342,"threads_started":1,"wal-append.queue_time_us":713977}
I20260709 12:22:10.441727 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:10.430383  4055 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.969s	user 0.008s	sys 0.035s
I20260709 12:22:10.429205  4877 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:10.475492 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:10.431820  4535 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:10.441466  4783 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:10.491366  4834 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:10.486855 27657 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:10.488672 27659 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:10.490327 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:10.528370 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102475us
I20260709 12:22:10.495975 27583 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:10.526638 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.587081 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.587711 27580 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:10.588243 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.588960 27657 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:10.443931  4761 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.595043 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1669) took 2037 ms. Trace:
I20260709 12:22:10.595479 27662 rpcz_store.cc:276] 0709 12:22:08.557226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.139311 (+1582085us) service_pool.cc:224] Handling call
0709 12:22:10.140424 (+  1113us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.486508 (+346084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.594788 (+108280us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54001e3cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:10.594984 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":794029440}
I20260709 12:22:10.527597 27585 rpcz_store.cc:276] 0709 12:22:09.221620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.221823 (+   203us) service_pool.cc:224] Handling call
0709 12:22:09.222203 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.222636 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:09.339951 (+117315us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.155273 (+815322us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.155866 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.156036 (+   170us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:10.156080 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.156165 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.435916 (+279751us) raft_consensus.cc:1685] finished
0709 12:22:10.435937 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.436084 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":132717,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":19727,"raft.queue_time_us":412,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":20437,"thread_start_us":814931,"threads_started":1,"wal-append.queue_time_us":1088385}
I20260709 12:22:10.484036  4848 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:10.542689 27660 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:10.543171 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.601234 27585 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:10.601593 27581 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:10.544067 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.546339 27732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.029s	sys 0.025s
W20260709 12:22:10.602629 27732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.029s	sys 0.026s
I20260709 12:22:10.547158 27663 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:10.487147 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:10.558629 27588 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
W20260709 12:22:10.552070 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 76655.6us
I20260709 12:22:10.604974 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:10.487421  4945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.024s	sys 0.020s
W20260709 12:22:10.550129  4905 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.023s	sys 0.028s
I20260709 12:22:10.533444  4751 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:10.598148 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:10.584924  4592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.011s	sys 0.011s
I20260709 12:22:10.531244  4938 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:10.601239 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.604369 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.607945 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.604351 28329 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:10.608893 27581 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:22:10.611236 27585 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21. Ops: []
I20260709 12:22:10.493112  4591 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.554800  4877 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:10.617915  4905 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.024s	sys 0.028s
I20260709 12:22:10.654601 27660 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:22:10.655315 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3363) took 1181 ms. Trace:
I20260709 12:22:10.655706 27660 rpcz_store.cc:276] 0709 12:22:09.474206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.644686 (+1170480us) service_pool.cc:224] Handling call
0709 12:22:10.644987 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.655008 (+ 10021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.655249 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:10.628587  4592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.012s	sys 0.012s
I20260709 12:22:10.489137  4796 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.545399  4055 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:10.623754 27729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.013s	sys 0.024s
W20260709 12:22:10.665166 27729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.014s	sys 0.024s
I20260709 12:22:10.628063  4751 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.601076 27580 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:10.626207 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:10.601406  4535 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:10.627151  5001 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.648419 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.617301 27584 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
W20260709 12:22:10.659017 27731 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.007s	sys 0.023s
W20260709 12:22:10.612219  4988 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.006s	sys 0.018s
I20260709 12:22:10.638077  4725 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:10.622622 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:10.626982  4958 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:10.615660  4847 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.620946  4945 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:10.650880  4993 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
W20260709 12:22:10.594254  4895 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.007s	sys 0.017s
I20260709 12:22:10.629666  4927 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:10.629627  4938 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:10.672101 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:10.730439 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 107896us
I20260709 12:22:10.732066 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:10.637619 27657 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:10.675279 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:10.631101  4789 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:22:10.600541  4834 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.615765  4753 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:22:10.676867 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:10.698406 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.004s	sys 0.012s
W20260709 12:22:10.764101 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.004s	sys 0.015s
I20260709 12:22:10.651221  5003 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.077s
I20260709 12:22:10.725355 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3359) took 1273 ms. Trace:
I20260709 12:22:10.652904  4877 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:10.679394 27731 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.008s	sys 0.024s
I20260709 12:22:10.710073 27662 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:22:10.657924  4376 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:10.660847  5000 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:10.706844  4785 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:10.738014 27664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.019s	sys 0.013s
W20260709 12:22:10.748257 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.002s	sys 0.015s
W20260709 12:22:10.726679  4988 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.006s	sys 0.025s
W20260709 12:22:10.749408 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 379132us
W20260709 12:22:10.792196  4996 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:10.791729  4945 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.750404 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:10.793817 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4101) took 2611 ms. Trace:
I20260709 12:22:10.766422 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3507) took 1209 ms. Trace:
I20260709 12:22:10.767092 27583 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:22:10.797013 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.003s	sys 0.016s
I20260709 12:22:10.674115  4751 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:10.774375 27659 rpcz_store.cc:276] 0709 12:22:09.452266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.799929 (+347663us) service_pool.cc:224] Handling call
0709 12:22:10.725153 (+925224us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b54000e42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:10.725286 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1425523968}
I20260709 12:22:10.677755  4750 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:22:10.721741  4990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.012s	sys 0.035s
I20260709 12:22:10.778939 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3482) took 2517 ms. Trace:
I20260709 12:22:10.736603  4999 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:10.804631 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 587271us
I20260709 12:22:10.729724  4845 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:10.791909  4779 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.647s	user 0.007s	sys 0.006s
I20260709 12:22:10.811401 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:10.794520 27738 rpcz_store.cc:276] 0709 12:22:09.557011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.559669 (+  2658us) service_pool.cc:224] Handling call
0709 12:22:09.571985 (+ 12316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.572193 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:10.360846 (+788653us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.766104 (+405258us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.766155 (+    51us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:10.766204 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.766270 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.766372 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":757497,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"thread_start_us":1171238,"threads_started":2}
I20260709 12:22:10.794238 27655 rpcz_store.cc:276] 0709 12:22:08.182168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.085321 (+903153us) service_pool.cc:224] Handling call
0709 12:22:09.085800 (+   479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.186854 (+101054us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.186887 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.187628 (+   741us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.083673 (+896045us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:10.131848 (+ 48175us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.132292 (+   444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.132365 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.793586 (+661221us) raft_consensus.cc:1685] finished
0709 12:22:10.793607 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.793750 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100267,"prepare.queue_time_us":47620,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":100486,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":183,"thread_start_us":895573,"threads_started":1,"wal-append.queue_time_us":1477654}
I20260709 12:22:10.803686 27732 rpcz_store.cc:276] 0709 12:22:08.247443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.547969 (+300526us) service_pool.cc:224] Handling call
0709 12:22:09.165331 (+617362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.165586 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:09.892413 (+726827us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.615776 (+723363us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.615855 (+    79us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:10.615936 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.616032 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.764461 (+148429us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54000a46d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:10.764572 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12632,"prepare.queue_time_us":1119670,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":1611,"spinlock_wait_cycles":1378420096,"thread_start_us":1436676,"threads_started":2}
I20260709 12:22:10.737344  4927 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:10.735282  4895 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.008s	sys 0.017s
I20260709 12:22:10.801744  4592 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:22:10.812136  4990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.013s	sys 0.037s
I20260709 12:22:10.681066  4628 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.114s
W20260709 12:22:10.812789 27664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.024s	sys 0.013s
I20260709 12:22:10.737612  5002 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
W20260709 12:22:10.802557 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 414922us
I20260709 12:22:10.794468  4935 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:10.794893  4830 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.762372  4958 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.691161  5001 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.775882  4847 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:10.836241 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 161023us
I20260709 12:22:10.864266 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:10.807983 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 197304us
I20260709 12:22:10.869127 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:10.869735 27737 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:10.832129  4785 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:10.831094  4990 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:10.795428  4376 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:10.850607 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.698859  4848 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.773801  4725 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.858667 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:10.871069  5006 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.219s	user 0.001s	sys 0.001s
I20260709 12:22:10.871738 27736 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:10.811793  5000 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:10.833248  4761 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.728574  4055 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.877442 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:10.819651  4779 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.879839 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:10.884099  4977 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.740s	user 0.013s	sys 0.000s
W20260709 12:22:10.885911  5005 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:22:10.905745 27729 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:10.778699  4535 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:10.888255 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.853473  4935 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:10.891824 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:10.872803  4796 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.896301 27738 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:22:10.880635 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.881464 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 131171us
W20260709 12:22:10.913872 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.914285 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.914597 27704 connection.cc:256] RPC call timeout handler was delayed by -1.04635820600015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
I20260709 12:22:10.914723 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:10.889088  4845 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.879606  4591 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:10.898836 27737 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:10.918015 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44337.8us
I20260709 12:22:10.924700 28336 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:10.937939 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.939641 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:10.938338 27737 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:10.941710 27584 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:10.945837 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:10.947492 28362 heartbeater.cc:646] Failed to heartbeat to 127.25.202.62:43715 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.25.202.62:43715 timed out after 15.000s (SENT)
I20260709 12:22:10.952457 27659 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:10.946801 27704 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:10.954403 27655 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:10.889704  5033 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.071s
I20260709 12:22:10.900022  5025 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 12:22:10.924224  4830 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.918412  4750 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:10.959021 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.959672 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4260) took 1536 ms. Trace:
I20260709 12:22:10.973451 27582 rpcz_store.cc:276] 0709 12:22:09.423074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.423210 (+   136us) service_pool.cc:224] Handling call
0709 12:22:09.452997 (+ 29787us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.460205 (+  7208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.460237 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.460681 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.357160 (+896479us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:10.357229 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.357292 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.959455 (+602163us) raft_consensus.cc:1685] finished
0709 12:22:10.959478 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.959615 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6827,"raft.queue_time_us":6939,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":22470,"thread_start_us":796289,"threads_started":1,"wal-append.queue_time_us":1016114}
W20260709 12:22:10.977703 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69952.4us
I20260709 12:22:10.962777 27737 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:10.954486 27704 leader_election.cc:403] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
W20260709 12:22:10.963634 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105074us
I20260709 12:22:10.955965 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:10.950771  4991 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:22:10.925524  5007 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:10.970799 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1668) took 2472 ms. Trace:
W20260709 12:22:10.985673 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 112274us
I20260709 12:22:10.987119 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1665) took 2679 ms. Trace:
I20260709 12:22:10.986712  5007 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.987506 27661 rpcz_store.cc:276] 0709 12:22:08.307257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.264059 (+956802us) service_pool.cc:224] Handling call
0709 12:22:09.264540 (+   481us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:09.338493 (+ 73953us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.338523 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:09.339057 (+   534us) log.cc:844] Serialized 52 byte log entry
0709 12:22:10.147476 (+808419us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:10.355146 (+207670us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:10.355186 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.355264 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.986847 (+631583us) raft_consensus.cc:1685] finished
0709 12:22:10.986865 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.987066 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":674,"raft.queue_time_us":2916,"raft.run_cpu_time_us":8527,"raft.run_wall_time_us":796992,"spinlock_wait_cycles":13704960,"thread_start_us":1015764,"threads_started":1,"wal-append.queue_time_us":1350943}
I20260709 12:22:10.989704  5007 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:10.990584 27737 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.986652 27656 rpcz_store.cc:276] 0709 12:22:08.498673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.317876 (+819203us) service_pool.cc:224] Handling call
0709 12:22:09.318196 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.339557 (+ 21361us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:22:10.353155 (+1013598us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.749770 (+396615us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.750375 (+   605us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.750561 (+   186us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:10.750870 (+   309us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.751055 (+   185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.751149 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:10.970603 (+219454us) raft_consensus.cc:1685] finished
0709 12:22:10.970622 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.970741 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78923,"prepare.queue_time_us":228984,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":3400,"raft.queue_time_us":145775,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":27040,"spinlock_wait_cycles":13567744,"thread_start_us":1360687,"threads_started":2,"wal-append.queue_time_us":509653}
I20260709 12:22:10.985527 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1557) took 1021 ms. Trace:
I20260709 12:22:10.993053 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:10.992555 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:10.991984 27737 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:22:10.955766  5016 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:22:10.946460  4935 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:11.001042 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:10.908550  4977 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:10.993255 27731 rpcz_store.cc:276] 0709 12:22:09.963872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.964014 (+   142us) service_pool.cc:224] Handling call
0709 12:22:09.964432 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.964649 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:22:10.783731 (+819082us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.984051 (+200320us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.984467 (+   416us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.984708 (+   241us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:10.984902 (+   194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.985058 (+   156us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.985336 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4746,"prepare.queue_time_us":416282,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":3041,"raft.queue_time_us":80671,"thread_start_us":989380,"threads_started":3}
I20260709 12:22:11.015585 27656 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:10.956280  4847 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:10.958066  5033 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:11.015820 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.020795 27737 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:22:11.021756 27737 pending_rounds.cc:85] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 17
I20260709 12:22:11.022392 27737 pending_rounds.cc:107] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 27.18
I20260709 12:22:11.022784 27737 raft_consensus.cc:2889] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:10.895931  4927 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.999260  4789 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.942693  4845 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:10.985313  4592 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:11.047825 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55362us
W20260709 12:22:11.022400  5017 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
W20260709 12:22:11.050871 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55620.9us
I20260709 12:22:11.063105 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:10.957746  5031 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.089s
W20260709 12:22:11.050213  5015 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.225s	user 0.001s	sys 0.001s
W20260709 12:22:11.075511 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93286.5us
I20260709 12:22:11.078878 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:11.073524 28333 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:11.084471 27661 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:22:11.021332  5030 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:22:10.997566  4830 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:11.106431 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50339.3us
I20260709 12:22:11.043794  4834 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:11.088471 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:11.104194 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:11.115960 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:11.118919 27583 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:11.116051 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.120334 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.120726 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.115983 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:11.081838  5024 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:22:11.121801 27657 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:11.120755 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.123457 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.124035 27657 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:11.125372 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:11.124862 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.126833 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:11.082192  5013 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:11.121369 27662 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:11.128446 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.134001 27659 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:11.128480 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:10.955578  4990 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.132221 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.135177 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:11.092736  4935 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:11.143198 27655 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:11.081077  4965 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:22:11.100994  5030 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:11.047364  5033 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.109228  4847 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:11.173919 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:11.154654 27583 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:11.117571  5041 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:11.060086  4789 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.161031  5051 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:11.060304  5011 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.085s
W20260709 12:22:11.184702 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 105923us
I20260709 12:22:11.174891 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:11.136726  5013 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:11.114568  4592 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.203933  5013 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.205205  4845 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:11.200616  5022 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:22:11.163467  5030 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:22:11.216050 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 111936us
I20260709 12:22:11.106851  4750 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.222141 27661 raft_consensus.cc:606] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:11.222679 27661 raft_consensus.cc:614] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:11.181983  4376 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:11.224962 27734 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:11.228066 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:11.139559  4628 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.057s
I20260709 12:22:11.231884 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:11.234927 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:11.216110  5040 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.272s	user 0.001s	sys 0.000s
I20260709 12:22:11.187682  4958 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:11.236354 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:11.183280  5001 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:11.205303  5013 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:11.237504 28330 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:11.163923  4935 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.238387 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:11.227365  4895 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:11.260900 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:11.261897 27735 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:11.262030 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:11.263298 27734 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:22:11.263988 27656 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:11.264408 27734 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:22:11.265892 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.267356 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.275661 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:11.259912  5036 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.258s	user 0.000s	sys 0.001s
I20260709 12:22:11.260861  4895 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:11.253123  5037 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:11.273415  5051 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:11.317587 27580 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:11.268941  4725 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:11.276356  5004 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:11.330564 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:11.278170  4845 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:22:11.366353 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 320150us
W20260709 12:22:11.350481 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 215405us
I20260709 12:22:11.371444 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:11.281103  4965 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:11.371948 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:11.373231 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:11.390604 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:11.349625  4643 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.293s	user 0.013s	sys 0.002s
I20260709 12:22:11.394619 27656 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:22:11.333076  5042 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:11.395818 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.396189 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:11.331640  4977 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:11.399878 27587 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:22:11.344615  5062 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:11.344899  4895 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:11.349298  5037 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.408406  4789 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:11.409672  5041 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:11.347841  5051 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:11.413723 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:11.414467 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:11.366667  4845 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:11.394711  5044 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:11.396885  4945 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:11.359726  4767 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:22:11.427455 28330 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
W20260709 12:22:11.459955 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.398818  4965 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:11.399148  5042 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:11.416538  5062 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:11.423271  5037 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:11.422945  4628 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:11.468590  5051 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:11.472071  4845 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:11.411502  5034 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.089s
I20260709 12:22:11.476203  4789 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:11.480633  4806 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:22:11.497520 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 301939us
W20260709 12:22:11.502677 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 75848us
I20260709 12:22:11.485968  5041 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:22:11.506335 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.506603 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 115987us
I20260709 12:22:11.506860 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:11.510677 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.489216  4628 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.510764 28330 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
W20260709 12:22:11.521320 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.528661 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.528872 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.529526 27580 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:11.498914  4806 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:11.502418  5033 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:11.515941  5041 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:11.548605 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44338.1us
I20260709 12:22:11.550719 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.520200  4628 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:11.550777 27582 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
W20260709 12:22:11.555761  5039 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:22:11.555598  4376 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:11.552760 27580 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:11.556702  4376 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:11.567205 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:11.548173  5045 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.553644  5041 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:11.551975  4935 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:11.610689 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43579.4us
I20260709 12:22:11.570492 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:11.568813  4376 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:11.570912  5068 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:11.622588  5073 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 12:22:11.622712  5032 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
W20260709 12:22:11.650038 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79660.2us
I20260709 12:22:11.653972 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.653869 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:11.675495 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 114663us
W20260709 12:22:11.604833  5049 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.257s	user 0.001s	sys 0.000s
I20260709 12:22:11.656662 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:11.679612 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 308538us
I20260709 12:22:11.647724  5046 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.551222  4965 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.686142 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:11.687901 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.534626  5042 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.604153 27704 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 20. Current term is 21.
I20260709 12:22:11.598016  5061 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.105s
I20260709 12:22:11.598368  5052 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:11.647716  4935 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:11.816125  4935 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.545439  5062 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.680393  5065 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:22:11.692821 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89228.2us
I20260709 12:22:11.695015  5046 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:11.744045  5078 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:22:11.628813  5045 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.744640  5060 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:11.694941  4055 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:11.674916  5066 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.057s
W20260709 12:22:11.748055  5082 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:11.656517  5068 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.749944  4830 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:11.758665 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1675) took 1341 ms. Trace:
I20260709 12:22:11.761466  4834 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:11.770511  5079 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:22:11.857014  5045 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:11.857190 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4274) took 1413 ms. Trace:
I20260709 12:22:11.628502  4806 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.857623  5045 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:11.858171  5045 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.857640 27585 rpcz_store.cc:276] 0709 12:22:10.443662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.603258 (+159596us) service_pool.cc:224] Handling call
0709 12:22:10.611029 (+  7771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.629656 (+ 18627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.856924 (+1227268us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400204a20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34ba02 00007f0e9a347dd6 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:22:11.857136 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343969664}
I20260709 12:22:11.634222  5057 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:11.859458 27582 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:11.859778 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.860589 27656 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:22:11.778232  5077 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.145s	user 0.000s	sys 0.006s
W20260709 12:22:11.788034  4877 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.986s	user 0.007s	sys 0.005s
I20260709 12:22:11.791667  4945 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.794260  4628 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.652846  5033 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:11.817203 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 140623us
I20260709 12:22:11.688309 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:11.877405 27580 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:11.878428 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.691809  5053 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:11.879549  5053 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.879956  5053 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:22:11.834568 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 420153us
I20260709 12:22:11.883584  4761 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:11.885319  4761 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:11.839811 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:11.885783  4761 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.887638 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.887765  4779 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:11.888362 27657 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:11.817108  4999 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.890197 27661 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:22:11.890017  4999 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.823387  5061 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:11.831552  5065 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.852329 27663 rpcz_store.cc:276] 0709 12:22:10.417263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.604087 (+186824us) service_pool.cc:224] Handling call
0709 12:22:11.758394 (+1154307us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:11.758586 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2040201472}
I20260709 12:22:11.843519  5060 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:11.849872  4055 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.855412  5080 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.855636  4834 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.875087 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1563) took 1007 ms. Trace:
I20260709 12:22:11.915788 27738 rpcz_store.cc:276] 0709 12:22:10.867831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.984049 (+116218us) service_pool.cc:224] Handling call
0709 12:22:10.984440 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.984823 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.984858 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.985360 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.471166 (+485806us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:11.471243 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.471372 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.761120 (+289748us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:11.872154 (+111034us) raft_consensus.cc:1685] finished
0709 12:22:11.872184 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.875005 (+  2821us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139404,"raft.run_cpu_time_us":2581,"raft.run_wall_time_us":632638,"spinlock_wait_cycles":11280000,"thread_start_us":483965,"threads_started":1,"wal-append.queue_time_us":689660}
I20260709 12:22:11.858471  5068 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:11.713493  4376 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.897125 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:11.859651  5057 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:11.860101  5066 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 12:22:11.903728 27585 raft_consensus.cc:2368] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:11.904533  5065 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:11.906177 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.921156 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.939890 27663 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:22:11.905964 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.900020  5031 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:11.938624 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 104213us
I20260709 12:22:11.926702 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:11.839555  5052 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.940234 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:11.876482  4945 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.892452  4877 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:11.904225  5061 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.941308  5066 consensus_queue.cc:1243] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 28.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:22:11.926134  5080 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:11.950095  4751 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.140s	user 0.009s	sys 0.005s
I20260709 12:22:11.972582  4751 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.942380  4848 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:11.925372 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 235416us
W20260709 12:22:12.003611  4998 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.198s	user 0.004s	sys 0.000s
W20260709 12:22:12.408581 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.042084  5090 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:22:12.417836 27588 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:12.419049  4535 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:11.949594  5031 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.465233 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1664) took 4158 ms. Trace:
I20260709 12:22:12.464998  4535 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:12.406329 27553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.006s	sys 0.005s
W20260709 12:22:12.467567 27553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.006s	sys 0.005s
I20260709 12:22:12.465727 27658 rpcz_store.cc:276] 0709 12:22:08.306947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.220045 (+913098us) service_pool.cc:224] Handling call
0709 12:22:09.220465 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.291154 (+ 70689us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.291194 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.340532 (+ 49338us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.360383 (+1019851us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:10.360426 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.360617 (+   191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.002975 (+1642358us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:12.464452 (+461477us) raft_consensus.cc:1685] finished
0709 12:22:12.464503 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.464989 (+   486us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70142,"raft.queue_time_us":117006,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":11430,"spinlock_wait_cycles":4985600,"thread_start_us":1004493,"threads_started":1,"wal-append.queue_time_us":1358586}
W20260709 12:22:12.468211 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 575923us
W20260709 12:22:12.102023  5051 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.000s
W20260709 12:22:12.264252  5030 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.028s	user 0.012s	sys 0.002s
I20260709 12:22:12.472215  5051 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.473909  5070 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:11.976926 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3520) took 1088 ms. Trace:
I20260709 12:22:12.475755  5068 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.475435  5070 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.008932 28333 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:12.476680  5068 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.479847  5068 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.463296  5061 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.476193  5030 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.463141  4848 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.467429  4877 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.476169 27737 rpcz_store.cc:276] 0709 12:22:10.888830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.024206 (+135376us) service_pool.cc:224] Handling call
0709 12:22:11.024676 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.516948 (+492272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.517015 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.789628 (+272613us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.895112 (+105484us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:11.895162 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.895258 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.976695 (+ 81437us) raft_consensus.cc:1685] finished
0709 12:22:11.976729 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.976860 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":248721,"raft.queue_time_us":204700,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":14131,"thread_start_us":571235,"threads_started":2,"wal-append.queue_time_us":157285}
I20260709 12:22:12.484596  5080 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.486673  5061 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.487728  5030 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.490087  4785 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:12.491274  5030 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.491052  4785 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.491629  5034 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:12.492172  5030 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.488207  5061 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.496397  5034 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.479884  4535 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:12.102521  4938 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.347s	user 0.010s	sys 0.010s
W20260709 12:22:12.019462 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.465895 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 523254us
I20260709 12:22:11.935405  5057 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:12.494459  4945 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:12.214082  5013 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.954s	user 0.013s	sys 0.001s
I20260709 12:22:12.507287  4938 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.526549 27589 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 27. Ops: []
I20260709 12:22:12.352414  5065 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.032526  5045 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.005738  4055 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.512209  5102 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:12.515597  5109 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.059s	user 0.001s	sys 0.001s
I20260709 12:22:12.516624 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:12.039760  5060 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.517802 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:12.525615  4591 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.006s
I20260709 12:22:11.952867  4931 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.567s
I20260709 12:22:12.520147  5052 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:12.222012  4895 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.797s	user 0.012s	sys 0.000s
I20260709 12:22:12.538616  4895 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.542608  4895 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.523404  5080 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
W20260709 12:22:12.189371  5041 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.540s	user 0.007s	sys 0.003s
I20260709 12:22:12.546746 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:12.552757 27583 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:12.553606 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4281) took 1024 ms. Trace:
I20260709 12:22:12.553965 27583 rpcz_store.cc:276] 0709 12:22:11.529519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.542444 (+1012925us) service_pool.cc:224] Handling call
0709 12:22:12.543215 (+   771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.553258 (+ 10043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.553507 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:12.144068  5037 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.650s	user 0.009s	sys 0.003s
I20260709 12:22:12.519047 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:12.180751  4628 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.531530 28331 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:12.535982  4945 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:12.539898 28329 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:12.357767  5053 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:11.975807  4761 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.548265  5041 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.549077 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3373) took 1180 ms. Trace:
I20260709 12:22:12.562036 27656 rpcz_store.cc:276] 0709 12:22:11.368917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.395573 (+ 26656us) service_pool.cc:224] Handling call
0709 12:22:12.548673 (+1153100us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54001e50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:12.548867 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059952384}
I20260709 12:22:12.549633  5102 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:12.224893  4845 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.724s	user 0.005s	sys 0.006s
I20260709 12:22:12.558526 28332 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:12.559031 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:12.559278 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:12.571833 28337 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:12.570366  4845 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.572834  4938 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.575372  4847 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:12.573625  4945 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.577956  4845 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.579257  4945 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.574833  5065 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.583451  5053 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.574534 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:12.584540  5102 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.585201  5047 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:12.563900  5041 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.588100 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:12.591672  5003 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:12.592180  5003 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.592615  5003 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:12.594745  4845 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:12.600162 28331 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:12.601622 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:12.602706  4938 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:12.603267  4938 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:12.609660  4938 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.610391  4785 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.611706  5030 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.601480  4848 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.613427  5037 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.614102  5046 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.616065  5070 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.604030  5052 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:12.632063  5057 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.640123  5070 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:12.640612  5068 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:12.641010  5070 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.644090  5068 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.652139  5070 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.652458  5068 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.652721  5061 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.652913  5046 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:12.653292  4643 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:12.653740  5046 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.654177  4643 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:12.655664 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.659368 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 134384us
I20260709 12:22:12.665083  5057 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.666383 27734 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:12.654748  5051 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.665443 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:12.670655  5046 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.671629  5068 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:12.672513  5068 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:12.674100  5068 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.676045 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1673) took 2289 ms. Trace:
I20260709 12:22:12.676501 27660 rpcz_store.cc:276] 0709 12:22:10.386755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.656600 (+269845us) service_pool.cc:224] Handling call
0709 12:22:10.676273 (+ 19673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.808109 (+131836us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.808141 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.808181 (+    40us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:10.808232 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.675508 (+1867276us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:12.675591 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.675848 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302656896}
I20260709 12:22:12.678453  4643 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:12.692653 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.695101  5051 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.696458 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:12.696271  5051 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.698104  5037 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.705049 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1433) took 1833 ms. Trace:
I20260709 12:22:12.709417  5051 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.710208 27580 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:12.710584  5057 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:12.711913  5060 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:12.711462  5057 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.705606 28336 rpcz_store.cc:276] 0709 12:22:10.871732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.872169 (+   437us) service_pool.cc:224] Handling call
0709 12:22:10.872449 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.936664 (+ 64215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.704362 (+1767698us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:12.704808 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119694720}
I20260709 12:22:12.713634  4751 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.711944 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4280) took 1182 ms. Trace:
I20260709 12:22:12.716087  5112 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.715375 27580 rpcz_store.cc:276] 0709 12:22:11.529209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.935906 (+406697us) service_pool.cc:224] Handling call
0709 12:22:11.936257 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.710785 (+774528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.711598 (+   813us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b54000b0754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:12.711868 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1626526464}
W20260709 12:22:12.717689 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52436.9us
I20260709 12:22:12.689558  5003 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.721062 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:12.714586  5057 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.719760 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:12.718387  5112 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.722996 27580 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:12.722409  5051 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.723771  5051 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.725024  5112 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.724092  5003 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.728850  5110 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:12.729678  5110 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.736337 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:12.737589  4751 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:12.740032  5110 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:12.742501  5086 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.971s	user 0.001s	sys 0.006s
I20260709 12:22:12.747478  4628 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:12.747713  4999 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.726962 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.752305 27580 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:12.752029 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:12.757884 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1715) took 1791 ms. Trace:
I20260709 12:22:12.758167 28336 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:12.758328 27584 rpcz_store.cc:276] 0709 12:22:10.960492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.960804 (+   312us) service_pool.cc:224] Handling call
0709 12:22:10.961195 (+   391us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:10.970387 (+  9192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.970421 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:10.971020 (+   599us) log.cc:844] Serialized 52 byte log entry
0709 12:22:11.019154 (+ 48134us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:11.019195 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.019258 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.737724 (+1718466us) spinlock_profiling.cc:243] Waited 7.08 ms on lock 0x7b240013aca8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:22:12.748417 (+ 10693us) raft_consensus.cc:1685] finished
0709 12:22:12.748442 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.752043 (+  3601us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8744,"raft.queue_time_us":40107,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":5109,"spinlock_wait_cycles":16289152,"thread_start_us":47829,"threads_started":1,"wal-append.queue_time_us":784598}
I20260709 12:22:12.740633  4751 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:22:12.816938 28336 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:12.826515 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:12.828775  4935 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.823949 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4266) took 1887 ms. Trace:
I20260709 12:22:12.760700 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:12.822132 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53907.2us
I20260709 12:22:12.819979 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.748133  4628 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:12.819757  5114 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.001s	sys 0.001s
I20260709 12:22:12.842067  5037 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:12.843143  4895 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:12.843823  4895 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:12.844930  5037 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:22:12.845752 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.849282  4725 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:12.850389  4725 consensus_queue.cc:1243] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 34.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:22:12.850623  5117 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:12.834789 27580 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:12.851162  4725 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:12.822479  4751 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.851660  5117 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.852036  4751 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.852367 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.853767 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.824195  5003 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:12.854355  5117 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.833257  5051 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:12.854944  5003 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.855652 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.856267  4806 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:12.841684  4628 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:12.842994 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:12.855068 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.864701 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:12.872093  4592 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:12.872795  4592 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:12.875316  4592 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.876492 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:12.878878  4935 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:12.879662 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:12.880379  4935 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:12.881443 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:12.882637  5051 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:12.883076  5051 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:12.883672  5051 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:12.886104  5045 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:12.887540  4643 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:12.831980 27586 rpcz_store.cc:276] 0709 12:22:10.936371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.937182 (+   811us) service_pool.cc:224] Handling call
0709 12:22:10.937592 (+   410us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:10.997752 (+ 60160us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:11.122894 (+125142us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.514172 (+391278us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:11.525614 (+ 11442us) log.cc:844] Serialized 52 byte log entry
0709 12:22:11.526117 (+   503us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:11.526166 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.526263 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.741559 (+1215296us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b240013aca8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:22:12.823424 (+ 81865us) raft_consensus.cc:1685] finished
0709 12:22:12.823522 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.823768 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13141,"prepare.queue_time_us":231328,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":3469,"raft.queue_time_us":43221,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":13799,"spinlock_wait_cycles":17137920,"thread_start_us":428633,"threads_started":2,"wal-append.queue_time_us":523251}
I20260709 12:22:12.887993  4643 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.888480  4643 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:12.888792 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.889747 27586 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:12.890851 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:12.883616 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:12.844981  4895 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:12.883059  4935 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.897192 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:12.897890 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:12.899600 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:12.900506 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:12.833345  5046 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.857820  5003 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:12.892143 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:12.854394  4751 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:12.903399 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:12.906415 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:12.873903  4938 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:13.023366 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4287) took 1479 ms. Trace:
I20260709 12:22:13.114993 27656 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:12.969090 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.004896 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.851666  4725 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:13.112864 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:13.123373 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.118551  4938 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.133857 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:13.137729 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.139748 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.022284  5051 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:13.140985 27587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4263) took 2948 ms (client timeout 3270 ms). Trace:
I20260709 12:22:13.137228 27662 raft_consensus.cc:2376] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32. Re-sending same reply.
W20260709 12:22:13.142807 27587 rpcz_store.cc:269] 0709 12:22:10.190953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.191755 (+   802us) service_pool.cc:224] Handling call
0709 12:22:13.137063 (+2945308us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002b9dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.138827 (+  1764us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2775622016}
I20260709 12:22:13.141904 27658 raft_consensus.cc:2376] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27. Re-sending same reply.
I20260709 12:22:13.115258 27583 rpcz_store.cc:276] 0709 12:22:11.543116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.556696 (+1013580us) service_pool.cc:224] Handling call
0709 12:22:12.575777 (+ 19081us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.576407 (+   630us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:12.785464 (+209057us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.014868 (+229404us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:13.015720 (+   852us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:13.016929 (+  1209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.019056 (+  2127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.021501 (+  2445us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":323933,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":5890,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":402408,"threads_started":2}
I20260709 12:22:13.136217  4938 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:13.147739 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:13.148613 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4284) took 1607 ms (client timeout 1856 ms). Trace:
W20260709 12:22:13.151882 27582 rpcz_store.cc:269] 0709 12:22:11.536682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.555065 (+ 18383us) service_pool.cc:224] Handling call
0709 12:22:13.137082 (+1582017us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.143702 (+  6620us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":396342016}
I20260709 12:22:13.153030 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.154222 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.149570 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:13.156015 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.157546 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:13.158509 27580 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:13.159682 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:13.160550 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.162853 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.160107 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3525) took 1338 ms. Trace:
W20260709 12:22:13.168219  4796 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.919s	user 0.011s	sys 0.011s
W20260709 12:22:13.168649  4796 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.094s	user 0.011s	sys 0.012s
I20260709 12:22:13.168579  5034 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:13.171669 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.167048 27731 rpcz_store.cc:276] 0709 12:22:11.821856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.821971 (+   115us) service_pool.cc:224] Handling call
0709 12:22:11.822232 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.974904 (+152672us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.974943 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:11.974997 (+    54us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:11.975044 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.159666 (+1184622us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:13.159724 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.159939 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":350518656}
I20260709 12:22:13.161124 27660 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:13.178965 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:13.180536 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.175732 27584 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:13.182628 27580 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:13.157482  5051 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.144912  4643 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.165673  4628 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.179977  4935 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:13.193979 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.195356 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.198089 27582 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:13.201555 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.203419 27584 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:13.204062 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:13.202618 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.205231 27587 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:13.205183 27661 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:13.206504 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.200139  4999 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.192901  4935 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:13.199847 27630 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.671s (SENT) [suppressed 21 similar messages]
I20260709 12:22:13.183995  5046 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.209271 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.213920 27662 raft_consensus.cc:2376] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30. Re-sending same reply.
I20260709 12:22:13.215922 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.218287 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.223204 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.225070 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.221939 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.230348 27732 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:13.209391  4999 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.234978 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.235419 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:13.236891 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.238415  4753 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:13.238603 27580 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:13.243615  4753 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.243988 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.244132 27736 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:13.244661 27580 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:13.246204  4753 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:13.248150 27580 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:22:13.251119 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.252144  4999 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.252604 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.253940 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.259033 27731 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:13.237422  4643 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.215186  4725 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.267254 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.268994 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:13.274207 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.275483 27732 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:13.284703 27580 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:13.285408 27656 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
W20260709 12:22:13.286687 28292 connection.cc:256] RPC call timeout handler was delayed by -1.7462097040001936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
I20260709 12:22:13.287248  4999 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:13.286615 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48986.8us
W20260709 12:22:13.287395 28292 connection.cc:256] RPC call timeout handler was delayed by -1.5554097040003398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
I20260709 12:22:13.290268 27662 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
W20260709 12:22:13.288462 28292 connection.cc:256] RPC call timeout handler was delayed by -1.502309704000254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
I20260709 12:22:13.280992  4940 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:13.292091  4940 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.293162 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:13.294656 27583 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
W20260709 12:22:13.296530 28292 connection.cc:256] RPC call timeout handler was delayed by -1.4366097040001478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260709 12:22:13.296985 28292 connection.cc:256] RPC call timeout handler was delayed by -1.4274750950003181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260709 12:22:13.280380  4628 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.304219  4785 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.308368 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.309046  5122 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:13.309211  4796 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:13.311729 27581 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
W20260709 12:22:13.316720 28292 connection.cc:256] RPC call timeout handler was delayed by -1.3286097040004279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
I20260709 12:22:13.319394 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:13.302187  4592 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:13.320369 27655 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:13.323920 28292 connection.cc:256] RPC call timeout handler was delayed by -1.3259097040004235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.211s for a response
W20260709 12:22:13.327304 28292 connection.cc:256] RPC call timeout handler was delayed by -1.2935097040003711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2146s for a response
I20260709 12:22:13.296445  4725 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:13.328233 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:13.330678 27732 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:13.331945 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.334693 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:13.336139  4834 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:13.336782  4834 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:13.337708 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.338353 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.339309 27731 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:22:13.337427  4834 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.340562  5025 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.290099  4999 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:13.354157 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61104.3us
W20260709 12:22:13.355016 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 155302us
W20260709 12:22:13.375496 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 180742us
I20260709 12:22:13.372608 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.289529  4935 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.377867 27655 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:13.379931 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.383857 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.384087 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.384958 27658 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:13.387578 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.389283 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.391216  4999 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.391842  4834 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.378829  4796 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.392622 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:13.386389  5025 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.398165 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.399577 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.372709  4940 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.372092  4592 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:13.394538 27662 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
I20260709 12:22:13.399199  4796 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:13.382179  5048 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.415431 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:13.416313  5120 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.178s	user 0.001s	sys 0.001s
I20260709 12:22:13.417464  5025 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.401053  5030 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:13.404397  4935 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:13.424768  4592 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.431484  5046 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:13.352983  4785 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.353510  5122 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.445760 27731 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:13.446975  4834 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:22:13.447523 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60654.5us
I20260709 12:22:13.453958 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:13.422050 27661 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:13.460806 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:13.462247 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:13.404047  4938 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:13.572715 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.465613 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4271) took 2131 ms. Trace:
I20260709 12:22:13.721458 27581 rpcz_store.cc:276] 0709 12:22:11.333922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.335110 (+  1188us) service_pool.cc:224] Handling call
0709 12:22:13.311298 (+1976188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:13.340836 (+ 29538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.465291 (+124455us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b54001248d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.465526 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4543948032}
I20260709 12:22:13.630671 27737 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:22:13.625625  5030 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.723821 27737 pending_rounds.cc:85] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 15
I20260709 12:22:13.725085 27737 pending_rounds.cc:107] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:22:13.725456 27737 raft_consensus.cc:2889] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:13.725579  5030 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.689102 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.726560  4806 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:13.512764  5066 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.728219  5066 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.729101  4964 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:13.729646  4964 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.730298 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:13.730819  5066 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.723482 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:13.732540  5062 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:13.732919 27581 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [33.22-35.23]
I20260709 12:22:13.734104 28336 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:13.735632 28332 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:22:13.736107 28332 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 14
I20260709 12:22:13.736343 28332 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:22:13.736582 28332 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:13.736761 27586 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:13.731978  4964 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:13.737891  4938 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.511884 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:13.739198 27736 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:13.735692 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.747437 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4292) took 1075 ms. Trace:
I20260709 12:22:13.748255 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.748968 27661 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:13.747889 27582 rpcz_store.cc:276] 0709 12:22:12.644320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.153939 (+509619us) service_pool.cc:224] Handling call
0709 12:22:13.719386 (+565447us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b54000893d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.719579 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87038848}
W20260709 12:22:13.512434 27701 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.670s (SENT)
I20260709 12:22:13.462960  5025 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:13.760284 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:13.753803 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:13.765709 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3524) took 1944 ms. Trace:
I20260709 12:22:13.757972 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.769914 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:13.770044 27661 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:13.766185 27734 rpcz_store.cc:276] 0709 12:22:11.821510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.821685 (+   175us) service_pool.cc:224] Handling call
0709 12:22:11.822088 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.667010 (+844922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.745182 (+1078172us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:13.765628 (+ 20446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1940646784}
I20260709 12:22:13.784955 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.459165  5048 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.790896 27583 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:13.793943 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:13.794749 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.795536 27734 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:13.795471 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.794220 27584 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:13.796268 27737 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:13.797186 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.797968 27584 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:13.798689 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:13.799316 27730 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:13.801429  4785 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.803463  5122 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:13.803871 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 411382us
I20260709 12:22:13.772750  4938 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:13.804709 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:13.806193  4940 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:13.806661 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:13.972138 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165618us
I20260709 12:22:13.973269 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:13.974324  4964 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:13.974874  4964 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:13.975317  4964 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:13.977823 27736 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
W20260709 12:22:13.841674 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54505.9us
I20260709 12:22:13.979732 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:13.980870 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.981338 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.981264  5060 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:13.982024  5060 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:13.891106 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4295) took 1232 ms. Trace:
I20260709 12:22:13.919262  5095 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:13.919529 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:13.983896 28329 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:13.983793  5095 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:13.985183 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:13.986414  5095 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:13.920755 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4285) took 2380 ms (client timeout 2086 ms). Trace:
I20260709 12:22:13.920895 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:13.987592  5132 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:13.983479 27586 rpcz_store.cc:276] 0709 12:22:12.659086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.166447 (+507361us) service_pool.cc:224] Handling call
0709 12:22:13.183318 (+ 16871us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:13.737215 (+553897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.890772 (+153557us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b54000bd1d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:13.891022 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1272191232}
W20260709 12:22:13.921501 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1427) took 3806 ms (client timeout 1764 ms). Trace:
I20260709 12:22:13.922147 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1515) took 2761 ms. Trace:
I20260709 12:22:13.922274 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:13.987610 27585 rpcz_store.cc:269] 0709 12:22:11.539844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.859517 (+319673us) service_pool.cc:224] Handling call
0709 12:22:13.901703 (+2042186us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b540011ce54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.920571 (+ 18868us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99211648}
I20260709 12:22:13.990720 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.991411 27585 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:13.757383  5025 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:13.988540 28335 rpcz_store.cc:269] 0709 12:22:10.114412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.115378 (+   966us) service_pool.cc:224] Handling call
0709 12:22:13.888371 (+3772993us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.921258 (+ 32887us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4097297152}
I20260709 12:22:13.992763  5025 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:13.989140 28333 rpcz_store.cc:276] 0709 12:22:11.160231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.160418 (+   187us) service_pool.cc:224] Handling call
0709 12:22:11.160800 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.477173 (+1316373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.892208 (+1415035us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54006d6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:13.921965 (+ 29757us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1949788928}
I20260709 12:22:13.988937 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.996963 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:13.998034 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.004602  5131 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:13.999217 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.008340  5054 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:14.009752  5054 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:14.010202  5054 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:14.011652 28329 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:13.925084  5048 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.925592 27662 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:13.925976 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.928004 28330 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:22:13.928437  5122 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:13.929000 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:13.929464  5031 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.931787 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3340) took 5676 ms. Trace:
I20260709 12:22:13.932381  5101 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:22:13.932996  5124 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.774s	user 0.006s	sys 0.002s
I20260709 12:22:13.934122 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4304) took 1225 ms. Trace:
I20260709 12:22:13.935448 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:13.939791 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:13.940557  4931 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:22:13.944077 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.947662 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4278) took 2419 ms (client timeout 2311 ms). Trace:
W20260709 12:22:14.076339 27588 rpcz_store.cc:269] 0709 12:22:11.528526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.528663 (+   137us) service_pool.cc:224] Handling call
0709 12:22:13.947399 (+2418736us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.947591 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2040733952}
W20260709 12:22:13.948575  5133 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.191s	user 0.001s	sys 0.003s
I20260709 12:22:13.774076  5046 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.079145  5046 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.082545  5046 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.084316 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.085301 27656 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:14.086608 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:13.961529  4725 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.000s
I20260709 12:22:13.964329 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4255) took 3779 ms. Trace:
I20260709 12:22:14.087977 27589 rpcz_store.cc:276] 0709 12:22:10.184826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.184953 (+   127us) service_pool.cc:224] Handling call
0709 12:22:10.959742 (+774789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.527134 (+1567392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.310588 (+783454us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54001248d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:13.964097 (+653509us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001248d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.964263 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5312630528}
I20260709 12:22:14.090031 27732 raft_consensus.cc:2410] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 36 because replica has last-logged OpId of term: 35 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 22.
I20260709 12:22:13.965165 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:13.965874  5123 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.815s	user 0.001s	sys 0.005s
I20260709 12:22:13.970867  4376 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:13.807942 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 354057us
I20260709 12:22:14.102327 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:13.982546  5066 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.920023 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:14.103802 27663 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:14.103665  5066 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.106163  5066 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.989810 28331 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:14.107713  4783 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:14.108103 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:14.108415  4783 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.109681 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.111505  4767 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:14.112087  4767 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:14.112413 27481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 236) took 48747 ms (client timeout 15000 ms). Trace:
I20260709 12:22:13.992251 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.114287  4767 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:13.924321 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:14.116385 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.116636 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.117776 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.118017 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.011307  5054 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:14.119470 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52677.5us
I20260709 12:22:14.119908  5054 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:14.121130 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:14.068495 27664 rpcz_store.cc:276] 0709 12:22:08.255091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.172337 (+917246us) service_pool.cc:224] Handling call
0709 12:22:09.255110 (+ 82773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.338044 (+ 82934us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:10.000594 (+662550us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:10.083602 (+ 83008us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.833843 (+750241us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.834772 (+   929us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.835117 (+   345us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:10.835197 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.835448 (+   251us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.367381 (+531933us) raft_consensus.cc:1685] finished
0709 12:22:11.367461 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.892439 (+2524978us) spinlock_profiling.cc:243] Waited 82.3 ms on lock 0x7b5400210550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.931557 (+ 39118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100560,"prepare.queue_time_us":120832,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":83726,"raft.run_cpu_time_us":9064,"raft.run_wall_time_us":663013,"spinlock_wait_cycles":203086976,"thread_start_us":1435440,"threads_started":2,"wal-append.queue_time_us":931534,"wal-append.run_cpu_time_us":4307,"wal-append.run_wall_time_us":1436475}
I20260709 12:22:14.074787 27587 rpcz_store.cc:276] 0709 12:22:12.708994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.159445 (+450451us) service_pool.cc:224] Handling call
0709 12:22:13.933885 (+774440us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b54000be5d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:13.934047 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94719232}
I20260709 12:22:14.123481  4931 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:14.123816 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.124397 28337 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:14.124804  4931 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.126075  4931 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:14.126417  5092 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.123273 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:14.012866  5060 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:13.809480 27731 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:13.807008  4940 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:14.017805 27479 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 25 to 27, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:14.024322  5042 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:14.025249 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.025395 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.026800 28336 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:13.989187 27583 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:14.026824 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:13.981359 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.066639 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:14.151453 27661 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:14.112393  5122 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.111043 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.144050 27664 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:14.148103 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:14.155457 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.160612 27662 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:14.161834 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:14.113032 27481 rpcz_store.cc:269] 0709 12:21:25.364577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.364806 (+   229us) service_pool.cc:224] Handling call
0709 12:22:14.112116 (+48747310us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:03.679648 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:03.680353 (+   705us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:03.680415 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:22:03.680452 (+    37us) tablet.cc:662] Decoding operations
0709 12:22:03.709541 (+ 29089us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:03.709660 (+   119us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:03.709688 (+    28us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:22:03.711256 (+  1568us) tablet.cc:701] Row locks acquired
0709 12:22:03.711304 (+    48us) write_op.cc:260] PREPARE: finished
0709 12:22:03.723659 (+ 12355us) write_op.cc:270] Start()
0709 12:22:03.723877 (+   218us) write_op.cc:276] Timestamp: P: 1783599723723556 usec, L: 0
0709 12:22:03.723923 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:22:03.724835 (+   912us) log.cc:844] Serialized 7455 byte log entry
0709 12:22:04.431549 (+706714us) op_driver.cc:464] REPLICATION: finished
0709 12:22:04.672448 (+240899us) write_op.cc:301] APPLY: starting
0709 12:22:04.672556 (+   108us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:04.975837 (+303281us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:05.009668 (+ 33831us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:06.323170 (+1313502us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b1800012e20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9a0870ec 00007f0e9a0d3c0a 00007f0e9a0d34f7 00007f0e9a0d3490 00007f0e9a08af48 00007f0e9dd9f1bb 00007f0e9dde65d4 00007f0e9dc928da 00007f0e9dc953cb
0709 12:22:06.575443 (+252273us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:06.576013 (+   570us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0709 12:22:06.576061 (+    48us) write_op.cc:312] APPLY: finished
0709 12:22:06.576505 (+   444us) log.cc:844] Serialized 172 byte log entry
0709 12:22:06.687622 (+111117us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:06.703367 (+ 15745us) write_op.cc:454] Released schema lock
0709 12:22:06.759645 (+ 56278us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:06.760280 (+   635us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:06.760316 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":164999,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":233129,"apply.run_cpu_time_us":54140,"apply.run_wall_time_us":2087809,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83547,"num_ops":19,"prepare.queue_time_us":431575,"prepare.run_cpu_time_us":26621,"prepare.run_wall_time_us":220518,"raft.queue_time_us":315314,"raft.run_cpu_time_us":63161,"raft.run_wall_time_us":217179,"replication_time_us":692796,"spinlock_wait_cycles":53518080,"thread_start_us":742857,"threads_started":5,"wal-append.queue_time_us":548739,"wal-append.run_cpu_time_us":36311,"wal-append.run_wall_time_us":2452950}]]}
I20260709 12:22:14.110975  4783 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.159198 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:14.127027  5092 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.170317 27663 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:14.173238 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.175740 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3380) took 1045 ms. Trace:
I20260709 12:22:14.177157 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:14.179096 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49970.1us
I20260709 12:22:14.180368 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:14.176302 27663 rpcz_store.cc:276] 0709 12:22:13.130776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.900122 (+769346us) service_pool.cc:224] Handling call
0709 12:22:14.175679 (+275557us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:14.149890  5048 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:14.149650  4725 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.054721  5025 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:14.197454 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49758.6us
I20260709 12:22:14.201970 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.202073 27657 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:22:14.202307 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 215273us
I20260709 12:22:14.201711  4806 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:14.204541 28334 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:14.204727  4806 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.205219  4806 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:14.207517  5007 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:14.207988  5007 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.208405  5007 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:14.208773 27479 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 29, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 29 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:14.212544  4830 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:14.213060  4830 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.213424  4830 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:14.213559  5095 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:14.214262  5095 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:14.214957  5095 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:14.207988  4783 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.008s
I20260709 12:22:14.216612  4783 consensus_queue.cc:1243] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 31.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:14.217159  4783 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:14.217520  4783 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:14.218313  4783 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.220714 27731 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:14.210932 27479 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 18 to 28, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 28 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:14.221772  4785 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:14.222693  4785 consensus_queue.cc:1243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:22:14.231258 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.235430  5030 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:14.235669  4783 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:14.235962  5030 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:14.237039  5030 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.189527  5048 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:14.241326 28328 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:14.240063 27656 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:22:14.245048 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 219923us
I20260709 12:22:14.239019  5030 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:14.246083 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:14.256256 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:14.241443  4767 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:22:14.186872  4938 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
W20260709 12:22:14.260876 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56876.8us
I20260709 12:22:14.240039  4751 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.246779  5048 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:14.199913  4725 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:14.264300 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 751890us
I20260709 12:22:14.266695 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:14.268838 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.277693 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.263080  5101 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:14.280263  5101 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.280560 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.280673  5101 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:14.283939 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.222195 27479 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 29 to 32. New cstate: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:14.286036 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.287098 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.288033 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.290784 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.297139 27581 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:14.292734 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:14.299269 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.292531 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.267808  5080 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.020s
W20260709 12:22:14.310997 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59596.7us
W20260709 12:22:14.308364 27701 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.749s (SENT)
I20260709 12:22:14.309279 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.293355 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.322275 27580 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:14.308758 27588 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:14.324798 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.298823  5139 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:14.276367  5068 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
W20260709 12:22:14.330261 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 330354us
I20260709 12:22:14.280798  4751 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:14.353981 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51490.5us
I20260709 12:22:14.357338 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.359992 27730 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:14.363080 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.361102 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:14.376760 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 175614us
I20260709 12:22:14.378417 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.375792 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.341044  5139 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:14.321919  5132 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:14.388707 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.330495  4964 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.398562 27737 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 35 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:22:14.402628 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:14.408018 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 64416.2us
I20260709 12:22:14.409821 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.383869  5025 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:14.414925 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.415308  5156 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:22:14.416225 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 143972us
W20260709 12:22:14.418550 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.417084 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.422734 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3398) took 1153 ms. Trace:
I20260709 12:22:14.433573 27660 rpcz_store.cc:276] 0709 12:22:13.269692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.388609 (+118917us) service_pool.cc:224] Handling call
0709 12:22:13.394990 (+  6381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.461512 (+ 66522us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:13.461557 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.771983 (+310426us) log.cc:844] Serialized 26 byte log entry
0709 12:22:14.309180 (+537197us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:14.309253 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.309340 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.422486 (+113146us) raft_consensus.cc:1685] finished
0709 12:22:14.422509 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.422658 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":26218,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":27566,"thread_start_us":473698,"threads_started":1,"wal-append.queue_time_us":590301}
I20260709 12:22:14.385066 27479 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:14.435559  5139 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.436602  4750 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:14.436288 27479 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:14.437104  5139 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:14.425017 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:14.425271  4751 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.426810  4834 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.427879 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.383073  5125 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 24.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.032s
I20260709 12:22:14.419251  5156 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.446867  4834 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.438277 27479 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:22:14.448166  5145 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:22:14.448865  4725 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:14.449240  5066 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:22:14.449982  5122 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:14.450809 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.450336 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.456666  4931 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:14.464600  5068 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:14.485586 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.486882 27730 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 34 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:22:14.413218  5149 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:14.464269  4783 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:14.419857  5092 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.497368  4834 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:14.496778 27479 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 23 to 24, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 24 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:14.498142  4931 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.485098  5068 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:14.499626  5132 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.500384  5068 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 24.20, Last appended by leader: 20, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.465351  5025 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:14.456719  4964 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.481022  5066 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:14.511310 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61057.9us
I20260709 12:22:14.489923  5156 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.442842  5095 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:14.553870  5148 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.192s	user 0.000s	sys 0.000s
I20260709 12:22:14.490279  5149 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:14.536373  4830 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.541472  5101 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:14.560760 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 207203us
I20260709 12:22:14.546693  5150 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:22:14.547592 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 202095us
I20260709 12:22:14.562044 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.529323  5043 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.553507  5156 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.557960 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.558141 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.564707 27730 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:14.565630 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.566008 27730 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:22:14.566273 27737 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:14.507535  4725 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:14.542634 27479 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 25 to 27, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:14.568367 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.532492  5066 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:14.550806  4806 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.574054 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:14.589794 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164821us
I20260709 12:22:14.547618  5159 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:22:14.591647 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 174630us
I20260709 12:22:14.588338 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:14.594146 27583 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:14.559171  5122 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:14.596063 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.597055 27585 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:14.597723  5007 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.599571 27589 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:14.599602 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.601363  5007 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.602849 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.603911 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.608326 27737 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 28 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:22:14.608407 27732 raft_consensus.cc:2368] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31.
I20260709 12:22:14.564111  5095 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.613677 27731 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:14.569176  5030 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:14.616422 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:14.591670  4751 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:14.590956  5156 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.591159  5150 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:14.594025  5159 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:14.606127  5122 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:14.630595 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.631120 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73441.3us
W20260709 12:22:14.631455 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 270413us
I20260709 12:22:14.614264  5092 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.615339  4376 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:14.619436  5007 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:14.592003  5043 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.617995  4785 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.016s
I20260709 12:22:14.648696 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.668232 27588 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:14.669062  4785 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:14.652395  4927 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:14.680985  4927 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:14.680838  5144 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:14.681496  4927 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:14.653740 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.684288  4783 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:14.684818 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.685700  5144 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:14.685770  5165 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:14.686442  5165 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:14.687073 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.687537  5165 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.688316  4806 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:14.655747  4931 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.634742  5095 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:14.634833  5159 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:14.661211  5048 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.662606  5149 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.650128  5158 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.018s
I20260709 12:22:14.664289  5150 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.639601  5122 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:14.670699 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.708199 27736 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:14.629735  4751 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:14.654458  5151 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:22:14.611821  5161 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:22:14.647876  4376 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:14.678479  5092 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.680035 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.688221 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.658253 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:14.690624 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:14.703958 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55782.7us
I20260709 12:22:14.633306  5156 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.677635  5007 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:14.711336 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:14.696233 27479 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 29, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 29 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:14.728425 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.689896  5144 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:14.726454 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.725087 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.677563  4785 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:14.723191 27729 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:14.714210  4751 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:14.785885  5048 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.785939  4785 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.787155 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.787724  4785 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:14.787993 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:14.789088 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.789386 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:14.725528  4931 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:14.722442  5101 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:14.748503 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:14.791694  5167 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:14.791749  4931 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:14.792713 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44334.3us
I20260709 12:22:14.793475  4931 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.795610  5125 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:14.796607  4783 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:14.724123 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3399) took 1437 ms. Trace:
I20260709 12:22:14.798516 27659 rpcz_store.cc:276] 0709 12:22:13.286090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.417040 (+130950us) service_pool.cc:224] Handling call
0709 12:22:13.417401 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:13.724442 (+307041us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:14.043721 (+319279us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.242495 (+198774us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.400952 (+158457us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b1800255940. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:14.549002 (+148050us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b1800255940. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:14.569496 (+ 20494us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:14.569570 (+    74us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:14.569641 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.723763 (+154122us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:14.723822 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.723950 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20622,"prepare.queue_time_us":315252,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":184987,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":197,"spinlock_wait_cycles":726306816,"thread_start_us":614143,"threads_started":3,"wal-append.queue_time_us":290185}
I20260709 12:22:14.740015 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.722265  4376 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:14.723770 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.716996 27738 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:14.802516 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:14.736212 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87393.5us
I20260709 12:22:14.727257 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.748030 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.726029  4830 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.736569  5131 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:14.750535 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.749758 27479 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 22 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 25 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:14.751161  4834 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:14.807232 27732 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
W20260709 12:22:14.807790 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 84114.9us
I20260709 12:22:14.808425 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.752710  4964 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:14.808620 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.810088 27732 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:14.811049 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.811504  5101 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:22:14.812156 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.812295  5101 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:14.807188 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.753867  5149 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:14.724249  5092 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:14.746016 27660 raft_consensus.cc:606] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.816497 27660 raft_consensus.cc:614] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:14.746322 27733 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:14.725570  5156 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.757438  5046 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:14.758922 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.759198 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.759347  5163 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.760859 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.761765 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.826767 27658 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 31 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
W20260709 12:22:14.767710  5042 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.587s	user 0.006s	sys 0.005s
I20260709 12:22:14.768505  5043 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:14.769340  5007 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.772475 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.776129  5165 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:14.777789  5068 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:22:14.779198 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.784790 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.785475 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.705210  4767 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:14.790126 27731 raft_consensus.cc:2410] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:14.802142 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:14.797277  5103 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.009s	sys 0.002s
I20260709 12:22:14.803263 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:14.805009 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65072.1us
I20260709 12:22:14.806100  5164 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:14.806210 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.751878 27735 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:22:14.745992  5144 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:14.820758 27656 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:14.834553 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.838071 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:14.839063 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88620.2us
I20260709 12:22:14.816771  4927 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.849062  5003 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.859750 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.860275  5003 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.860647 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.861135  4927 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:14.837102  5150 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:22:14.846498  5103 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.009s	sys 0.002s
I20260709 12:22:14.862413 27580 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:14.844017 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:14.863365  5144 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:14.850351 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.844317 27729 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:14.820523  5046 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:14.855090 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:14.865814 27731 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:22:14.802398  5066 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.834929  4725 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:14.861588 27585 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
I20260709 12:22:14.868275 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.862494 27738 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:22:14.853956 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:14.869375 27663 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:14.869485 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:14.865015 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.867663  5139 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:14.873030 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:14.873354  4806 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:14.874150  5046 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.873296 27663 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:22:14.873271 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66229.8us
I20260709 12:22:14.876940  5066 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:14.877096 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.874292  4927 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:22:14.882221 28335 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:22:14.884508 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.885504 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:14.885881 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:14.883847 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:14.887498 27736 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:14.881040 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:14.891183 27584 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:14.892396 27737 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:14.889477 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:14.897130 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101892us
I20260709 12:22:14.898352 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.901068 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.901919 27735 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 18. (term mismatch)
I20260709 12:22:14.897032 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.905118 27729 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:14.903630 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:14.902431 27735 pending_rounds.cc:85] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 17
I20260709 12:22:14.923175 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:14.899044 27587 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:14.924242 27588 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:14.923714 27735 pending_rounds.cc:107] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 30.18
I20260709 12:22:14.925318 27735 raft_consensus.cc:2889] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:14.912400 27736 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:14.841964  5163 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:14.930235 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.930896 27738 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:14.931365 27656 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:22:14.925717 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60737.4us
I20260709 12:22:14.935070 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:14.905752  5054 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:14.896283  4940 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:14.941637 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:14.943681 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:14.944095 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:14.940506  5043 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:14.944566  4751 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.948451 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:15.000108 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.980243 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47315.4us
I20260709 12:22:14.980633  5169 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:14.980108  5179 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:14.974836 27659 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:14.950548  5101 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:14.940028  5168 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:15.002952 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.936873  4725 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:14.887770  4767 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:14.889056  4783 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:14.978281 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:15.065237 27660 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:22:15.069267 27588 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:15.073796 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:15.086423 27586 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: [27.20-27.20]
I20260709 12:22:15.087033 27730 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:15.088831 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:15.090039 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.090662 27729 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:15.097868 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:15.098707 27660 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:15.099045 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:15.107838 27656 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:15.050825  5179 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.116472 27583 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
I20260709 12:22:15.121515 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:15.124044 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.130436 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:15.087090  5054 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:15.101498  4725 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.838924 27479 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 11 to 13, term changed from 20 to 28, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to <none>, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 28 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:15.089751  5043 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:15.134449 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 192867us
I20260709 12:22:15.143173 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:15.144382 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.133728 27657 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:15.141607 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.125106  5048 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:22:15.150601 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 211684us
W20260709 12:22:15.153014 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94986.8us
I20260709 12:22:15.116813  5179 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:15.100320  4783 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.159973 27735 raft_consensus.cc:2410] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:22:15.053821  5169 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.067010  5168 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.164260  5163 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.165067  4783 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.118048  5092 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:22:15.157387  5048 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:15.166412 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45024.3us
I20260709 12:22:15.170265 27729 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:15.140915  5030 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:22:15.147683  5043 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:15.162536 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.146723  4725 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.220490 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:15.220893 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 109836us
I20260709 12:22:15.187556 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:15.224182 27729 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:15.161369  5159 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.179714  5163 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:15.179090  4834 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:15.205562 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.224649 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:15.206367  5176 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
W20260709 12:22:15.190955 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51269.1us
I20260709 12:22:15.235544 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:15.237464 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.237347  4751 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.247574  4767 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.248207 27733 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:15.251592 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.230192 27479 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 23 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:15.253826 27479 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:15.254321 27657 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:15.201964  4783 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:15.258016 27583 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:15.259554 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 107623us
I20260709 12:22:15.260319 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.261318 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.251809 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:15.263742 27733 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:15.259002  5190 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:15.233932  5080 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:22:15.267794  5177 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:22:15.258234  4376 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.269418 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3402) took 1381 ms. Trace:
I20260709 12:22:15.266129 27734 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 20.
I20260709 12:22:15.269194 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:15.267817  5080 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:15.270576  5068 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:15.270629  5080 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:15.271245  5068 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:15.271905  5068 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:15.269853 27655 rpcz_store.cc:276] 0709 12:22:13.888491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.181806 (+293315us) service_pool.cc:224] Handling call
0709 12:22:14.194686 (+ 12880us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:14.585916 (+391230us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:14.619594 (+ 33678us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.767834 (+148240us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b180042e320. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:14.845167 (+ 77333us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:14.852127 (+  6960us) log.cc:844] Serialized 26 byte log entry
0709 12:22:14.852274 (+   147us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:14.852309 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.852376 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:15.142919 (+290543us) spinlock_profiling.cc:243] Waited 6.85 ms on lock 0x7b180042e320. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:15.269162 (+126243us) raft_consensus.cc:1685] finished
0709 12:22:15.269183 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.269337 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":156483,"prepare.queue_time_us":25530,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":172405,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":9468,"spinlock_wait_cycles":22698752,"thread_start_us":255320,"threads_started":2,"wal-append.queue_time_us":210593}
I20260709 12:22:15.273660 27655 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
I20260709 12:22:15.273996 27655 pending_rounds.cc:85] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 19
I20260709 12:22:15.274274 27655 pending_rounds.cc:107] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:22:15.274505 27655 raft_consensus.cc:2889] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:15.272207 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:15.188246  5164 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.074s
I20260709 12:22:15.276121 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.277302 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:15.277514  5168 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.277894 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:15.278611 27729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:15.279079 27729 raft_consensus.cc:606] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.279413 27729 raft_consensus.cc:614] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:15.280740 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.269846  4376 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:15.281648 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:15.282421 27655 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
W20260709 12:22:15.283001 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.283823 27655 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: [34.21-34.21]
I20260709 12:22:15.284921  5159 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:15.285141  5007 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:15.285534  4592 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:15.285739  5007 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:15.263036 27659 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:15.286921  5169 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.236248  5187 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:15.286748  5007 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.291100  5068 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.293977 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.282524 27733 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
W20260709 12:22:15.300076 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94609.5us
I20260709 12:22:15.229023  5122 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.296326 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:15.303084 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.303277 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.312479 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.264911  4751 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:15.315671 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.283058  5167 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:22:15.320338 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 84884.3us
I20260709 12:22:15.290653  5080 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.324268 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:15.313799  5002 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:22:15.315402  5104 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:22:15.331009 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:15.331261 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.332397 28331 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:15.318922  5168 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:15.319839  5187 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:15.328512 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:15.335809 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.328362  5117 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:15.345523  5122 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.346261  5179 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.346664  4725 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.348207 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.348613 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.349272  4783 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:15.350119  4783 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:15.335444  5104 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:15.351361  5104 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:15.351758 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.354962 28336 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:15.370932 27663 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:15.387835 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.389494 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.353199  5131 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:22:15.362221  5122 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:15.358425  5159 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:15.394791 27660 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:15.396114 27657 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:15.399436 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:15.392771  5188 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:22:15.392189  5167 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:15.394865 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:15.398793  4785 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.611s	user 0.008s	sys 0.004s
I20260709 12:22:15.393852  5131 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:15.398491  5187 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:15.395262  5122 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.408180  5117 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:15.417868 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:15.383229  5168 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.024s
I20260709 12:22:15.420331 27627 leader_election.cc:400] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31.
I20260709 12:22:15.406544  5149 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:15.428632 27627 leader_election.cc:403] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:15.407500  5167 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.417096  5190 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:15.414350  5131 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:15.320921 27479 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 20 to 21, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 21 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:15.420544  5122 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:15.433481  5165 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.593s	user 0.006s	sys 0.005s
I20260709 12:22:15.435906  5167 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:15.440685 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.446522  5139 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.439318  5149 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:15.451040 27479 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 12 to 15, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 15 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:15.452342  5103 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:15.427106  5169 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.475867  5169 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.478749  5169 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.443813  5187 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:15.442209  5190 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.484043  5190 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:15.467993 27581 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:15.455485  5068 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:15.463819  5144 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.600s	user 0.006s	sys 0.006s
W20260709 12:22:15.469112 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58940.1us
I20260709 12:22:15.461009  5117 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.472658  5179 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:15.472324 27479 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 13 to 22. New cstate: current_term: 22 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:15.513655  5190 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.520746 27479 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 30 to 32, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 32 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:15.522531 27479 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 18 to 24. New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:15.523679 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.524310 27479 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 14 to 24, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 24 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:15.525843  5190 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.468346 27662 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:15.527791 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.490278  5149 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.494278  5003 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.505517  5046 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:15.507107  5092 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:15.511834 27734 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:15.512763  5068 raft_consensus.cc:2764] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election decision vote started in defunct term 30: won
I20260709 12:22:15.513281  5168 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:15.513631  5048 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.515175  5080 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:15.517210  5180 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.333s	user 0.002s	sys 0.000s
I20260709 12:22:15.468886  5139 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:15.531128 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.525900 27479 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 20 to 25. New cstate: current_term: 25 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:15.488934  5103 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:15.501824  5144 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.558488 27734 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:22:15.535620  5092 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:15.562377 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.567022 27655 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:22:15.538764  5043 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.568264 27732 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:15.554412  5165 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.015s
I20260709 12:22:15.565908  4785 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:15.598379  5144 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.605926  5122 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.610019 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.612569  5122 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.602985  5187 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31.
W20260709 12:22:15.614130 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.627758  5092 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.629500  5092 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:15.596699  5103 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.640516  5103 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:15.639756 27479 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: config changed from index 5 to 11, term changed from 15 to 17, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 17 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:15.641805 27479 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 21 to 22, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 22 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:15.646733 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 212273us
W20260709 12:22:15.651050 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:15.651611 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.652140 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:15.651676 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:15.654472 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:15.654515 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3553) took 1008 ms. Trace:
I20260709 12:22:15.663857 27655 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:15.618969  5167 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.657924  5150 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.659842 27738 rpcz_store.cc:276] 0709 12:22:14.646174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.940163 (+293989us) service_pool.cc:224] Handling call
0709 12:22:14.945805 (+  5642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.051615 (+105810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:15.051667 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.068689 (+ 17022us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:15.081023 (+ 12334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.654322 (+573299us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:15.654362 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.654464 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205756672}
I20260709 12:22:15.669683 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:15.671433 28328 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:15.676476 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.679049 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.682168  5003 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:15.682233  5104 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.678735  5042 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.005s
I20260709 12:22:15.679019  4940 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.006s
I20260709 12:22:15.688336  5068 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:22:15.665606  5198 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:22:15.676044  5150 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.695832  5150 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:15.697428  5043 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:15.688352  4751 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:15.699977 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3548) took 1060 ms. Trace:
I20260709 12:22:15.700389 27731 rpcz_store.cc:276] 0709 12:22:14.639705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.866442 (+226737us) service_pool.cc:224] Handling call
0709 12:22:14.866748 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.881293 (+ 14545us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:14.937195 (+ 55902us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.110759 (+173564us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.509207 (+398448us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b18004cd260. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:15.699523 (+190316us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b18004cd260. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:15.699652 (+   129us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.699698 (+    46us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:15.699756 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.699821 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.699924 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6598,"prepare.queue_time_us":40307,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":85,"raft.queue_time_us":104,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5808,"spinlock_wait_cycles":41838720,"thread_start_us":116187,"threads_started":1,"wal-append.queue_time_us":211867}
I20260709 12:22:15.702590 27731 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:15.703279 27583 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:15.703447  5159 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:22:15.704536  5159 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:15.704871  5159 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:15.705758  5159 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.706827  5104 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.706982  5159 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:15.707408 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.693502 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:15.709074 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 564744us
I20260709 12:22:15.725865 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:15.726194 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.714372 27732 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260709 12:22:15.709931 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.714928 27737 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:15.722186 28328 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:15.716639  5156 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:15.724303 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:15.727716 28329 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:15.731428 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:15.721295 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:15.738906 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.740866 27588 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:15.739647 27583 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:15.733400 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:15.744670 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51239.9us
I20260709 12:22:15.745296 27732 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:22:15.746754 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:15.747855 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3556) took 1036 ms. Trace:
I20260709 12:22:15.749265 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.749779 27732 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:15.750533 27589 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:15.748214 27729 rpcz_store.cc:276] 0709 12:22:14.712075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.280151 (+568076us) service_pool.cc:224] Handling call
0709 12:22:15.280401 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.280673 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:22:15.401689 (+121016us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.747544 (+345855us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.747592 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:15.747639 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.747711 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.747806 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":301423,"prepare.queue_time_us":104123,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"thread_start_us":144522,"threads_started":2,"wal-append.queue_time_us":63900}
I20260709 12:22:15.754949 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:15.755761 27584 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:15.757793 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.754372 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.728360  4751 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.728987  5156 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:15.769953 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:15.770262 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44536.3us
I20260709 12:22:15.778008 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.770848 27732 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
W20260709 12:22:15.770906 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49695.1us
I20260709 12:22:15.776252 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.780376 27729 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:15.769340 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:15.780930 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:15.750150  5003 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:15.785374 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3555) took 1110 ms. Trace:
I20260709 12:22:15.785270 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.786504  5101 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:15.790307 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.790942 27729 raft_consensus.cc:2376] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26. Re-sending same reply.
I20260709 12:22:15.791282 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:15.792366 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.793934 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.795578  5101 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.795889 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.755155  4830 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:22:15.797219  5131 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.792114  5003 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:15.799203  5003 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:15.801204  5205 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.228s	user 0.003s	sys 0.000s
I20260709 12:22:15.801833 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.802994 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:15.786760  5149 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:15.803534 27729 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:15.803721  5149 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:15.804158  5149 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:15.796010  5101 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:15.808987 27732 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:15.811429 27729 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:15.787055  5179 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:15.788856  5043 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:15.786129 27736 rpcz_store.cc:276] 0709 12:22:14.675391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.933150 (+257759us) service_pool.cc:224] Handling call
0709 12:22:14.989725 (+ 56575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.989941 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:15.048779 (+ 58838us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.785041 (+736262us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.785091 (+    50us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:15.785141 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.785215 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.785320 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":539718,"prepare.queue_time_us":58582,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":4606,"raft.queue_time_us":547799,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":151167,"threads_started":2,"wal-append.queue_time_us":174372}
I20260709 12:22:15.828595 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.829773 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.831018  4931 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:15.831548  4931 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.831933  4931 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:15.834740 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3552) took 1190 ms. Trace:
I20260709 12:22:15.835754  5156 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.837375  4940 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.837424  5156 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:15.838105  4940 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.840873  5054 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:15.841434  5054 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:15.841921  5054 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:15.842084 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3543) took 1204 ms. Trace:
I20260709 12:22:15.845495  4376 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:15.846227  4376 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:15.848800 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.835253 27737 rpcz_store.cc:276] 0709 12:22:14.644569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.912896 (+268327us) service_pool.cc:224] Handling call
0709 12:22:15.714754 (+801858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.728742 (+ 13988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.834469 (+105727us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b540017ded0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:15.834652 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1804425088}
I20260709 12:22:15.842797 27730 rpcz_store.cc:276] 0709 12:22:14.637287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.639412 (+  2125us) service_pool.cc:224] Handling call
0709 12:22:15.841866 (+1202454us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:15.841984 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554260864}
I20260709 12:22:15.814461  5144 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:15.781172  4751 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:15.820993 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:15.818253  4806 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:22:15.868005 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47160.5us
I20260709 12:22:15.757386 27479 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 22 to 25, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 25 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:15.871102 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.869329  5198 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:15.875435  5212 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:15.870848  4725 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:15.880028 28336 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:15.880160  4940 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.884809  4751 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:15.826792  5179 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.889223 28335 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:15.828069  5002 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:15.848935  5043 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.893139  4830 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:15.891609  5167 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:15.847409  4376 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:15.931187 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:15.891521  5183 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:15.891054  5164 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:15.890758  5198 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:15.937099 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 203776us
I20260709 12:22:15.923415  5179 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:15.949955 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 205425us
I20260709 12:22:15.930564  5043 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:15.955977 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:15.947291  4927 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 12:22:15.958220  4830 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:15.958292 27479 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 22 to 24, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:15.916471  5117 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:22:15.916011  5104 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
W20260709 12:22:15.979487 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 417015us
I20260709 12:22:15.953923  5198 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:15.928696  5042 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:22:15.950129  5164 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:15.980608 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:15.929162  5196 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:22:15.922851  5212 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:22:15.975163 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.968614  5068 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:22:15.983469 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96049.5us
I20260709 12:22:15.990882 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.992571 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.995007 27581 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 20. (term mismatch)
I20260709 12:22:15.935635  5002 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.004503 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3562) took 1149 ms. Trace:
W20260709 12:22:16.007992 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.996366 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:15.984042  5168 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:16.010126 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:15.995430 27581 pending_rounds.cc:85] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 19
I20260709 12:22:16.022298 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:16.023715 27581 pending_rounds.cc:107] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.20
I20260709 12:22:16.026338 27581 raft_consensus.cc:2889] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:16.022351  5219 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:16.005136 27738 rpcz_store.cc:276] 0709 12:22:14.855100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.708183 (+853083us) service_pool.cc:224] Handling call
0709 12:22:15.708511 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:15.822723 (+114212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:15.822769 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:15.823401 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 12:22:15.976410 (+153009us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:15.976463 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.976565 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.004282 (+ 27717us) raft_consensus.cc:1685] finished
0709 12:22:16.004303 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.004444 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7967,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":198597,"threads_started":2,"wal-append.queue_time_us":141408}
I20260709 12:22:16.009186  5139 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:16.030499 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56859.7us
I20260709 12:22:15.979725  4751 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:22:15.979059  5103 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:22:16.032910 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:15.999511  5163 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:16.038230 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61643.3us
I20260709 12:22:16.041404 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:16.025290  5144 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:16.025446  5168 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:16.048705 27731 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:16.024647  4785 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:16.050493 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.051795  5068 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:15.984283  5217 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.061s
I20260709 12:22:16.039692  5190 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:16.086359 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.047506  5219 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.093937 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.047806  4767 raft_consensus.cc:2764] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:22:16.023636  5222 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.097s
I20260709 12:22:16.054862  5159 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:16.076954  5168 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:16.076929  4785 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:16.104617  5046 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:16.055991  4931 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.085361  5122 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.092753  5190 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.041309  5163 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.089856  5167 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:16.137347 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97269.6us
I20260709 12:22:16.115403  4783 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:16.147308 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:16.104915  4940 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:22:16.135932  4785 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:16.140529  5003 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.017s
I20260709 12:22:16.108804  5217 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:22:16.162796 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:16.163937  5222 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:16.164371  5149 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.165309  5222 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:16.165796  5222 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:16.166678  5222 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.168411  5101 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.171854  5101 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.171773 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:16.175858  5215 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:22:16.179382  5210 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:16.180716  5216 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:22:16.181924 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3560) took 1377 ms. Trace:
I20260709 12:22:16.182308 27734 rpcz_store.cc:276] 0709 12:22:14.804199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.560316 (+756117us) service_pool.cc:224] Handling call
0709 12:22:15.567813 (+  7497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:15.768071 (+200258us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:15.768118 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:15.784288 (+ 16170us) log.cc:844] Serialized 26 byte log entry
0709 12:22:15.887398 (+103110us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:15.887456 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.887658 (+   202us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.181673 (+294015us) raft_consensus.cc:1685] finished
0709 12:22:16.181700 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.181862 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":197394,"raft.queue_time_us":187817,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":14120,"thread_start_us":89088,"threads_started":1,"wal-append.queue_time_us":164096}
I20260709 12:22:16.048668  5164 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.144518  5190 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:16.145839  5122 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.157788  5104 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:22:16.209143 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 115313us
I20260709 12:22:16.106564  5219 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:16.194979  5220 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:22:16.210522 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.156071  5080 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:22:16.196750  5149 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:16.222332 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:16.229417 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:16.218966  5218 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.231s	user 0.001s	sys 0.001s
I20260709 12:22:16.213317  5169 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:16.251228  5169 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:16.251791  5169 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:16.187206  5054 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.233009 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:16.233886 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71157.7us
I20260709 12:22:16.235563  5228 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:16.256125  5228 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:16.231588  5042 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.013s
I20260709 12:22:16.256901  5003 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:16.257087  5054 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.257081  5228 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.257788  5054 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:16.259093  5212 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:16.259475  5212 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:16.240675  5163 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:16.241238 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1567) took 1430 ms. Trace:
W20260709 12:22:16.241310  5221 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:22:16.241696  5213 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:22:16.242336  5210 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:16.151124 27479 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 22 to 28. New cstate: current_term: 28 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:16.243631  4931 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.244130  5139 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.244443  5002 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:16.234094  5167 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:16.210244  5068 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.232378  4783 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:16.242354  5122 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:16.247794  5131 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.220011  5104 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:16.248660  5101 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:22:16.263855 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.290309  5104 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.291512  5122 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:16.290807 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.294160  5201 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:16.294575  5201 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:16.295473  5201 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.296749  5201 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:16.269480 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.297118 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.297847  5190 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.298198 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.298969 27738 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:16.298902 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.275846  5210 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.299645 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.300519 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.300617 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.301294 27662 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:16.301362 27730 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:16.302428 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.302711 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.302306 27479 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 29. New cstate: current_term: 29 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:16.303298 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:16.303634 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.303928  5159 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:16.304329  5159 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:16.304561 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.304828 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.304381 27479 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:16.305182  5159 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.305907  5122 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:16.306262 27662 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:16.306543  5159 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:16.299659 27664 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:16.297889 27734 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:16.308030 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.307789  5168 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.308691 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.309901  5168 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.310695 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.311419 27655 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 19: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:16.311679 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.314731 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.315522 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.315750 27662 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:22:16.318089 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.320011 27659 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:16.320541 27663 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:16.260025  5198 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.267949 27735 rpcz_store.cc:276] 0709 12:22:14.810614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.657455 (+846841us) service_pool.cc:224] Handling call
0709 12:22:15.657773 (+   318us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:15.757684 (+ 99911us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:15.757713 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:15.767315 (+  9602us) log.cc:844] Serialized 52 byte log entry
0709 12:22:15.944852 (+177537us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:15.944891 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.944975 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.240997 (+296022us) raft_consensus.cc:1685] finished
0709 12:22:16.241019 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.241170 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119072,"raft.queue_time_us":100282,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":17635,"thread_start_us":137155,"threads_started":1,"wal-append.queue_time_us":195972}
I20260709 12:22:16.257390  5003 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:16.325497 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:16.325949 27656 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:22:16.326995 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:16.278353 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56004.9us
I20260709 12:22:16.257082  5219 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.269305  5046 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:16.272190  5213 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:22:16.276790  5092 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.647s	user 0.015s	sys 0.000s
I20260709 12:22:16.260279  5212 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.284454  5101 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:22:16.365034 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:16.306547  5122 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.327704 27656 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:16.338662 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:16.339411 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:16.339869 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:16.340804 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:16.341589  4376 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:16.350530 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.353875  5046 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.356431 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.332623 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.363715 27658 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:22:16.354804  5213 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.294616  5179 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.385704  5046 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:16.386119 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.386605  4785 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.370420 27732 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:16.387507 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.387835  5190 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.388502 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.365236  5101 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:16.385380 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 116526us
I20260709 12:22:16.398950 27657 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:16.391995 27663 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:16.400245 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.401285 27663 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:16.392372 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.401994 27735 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:16.403620 27730 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:16.366389 27660 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:22:16.398762 27656 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:16.405042 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.407878 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.408756 27732 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:16.385368  5164 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:16.366246  5150 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:16.418592 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.380311  5212 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:16.376454  4376 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.419494 27663 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:16.445326 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.446395 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.447402 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.447925 27663 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:16.448645 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.449115 27663 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:16.407856  5149 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.373759  5233 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:16.421746 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.457233 27660 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:16.306463 27479 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 19 to 21, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 21 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:16.418618 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.410710  5222 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:16.459451 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.460485 27658 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:16.461704  5101 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.463402  5233 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.464612 27479 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 18 to 24, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 24 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:16.467620 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:16.469194 27655 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:16.483498 27661 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:16.393059  5122 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.486966  5223 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:16.489193 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.492764 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.499789 27660 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
W20260709 12:22:16.514940  5043 consensus_meta.cc:287] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.001s
I20260709 12:22:16.476989  5131 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:16.477926  5132 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:16.516001  5210 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.521607  5233 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.460741  5149 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.575913 27580 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:16.500211  5101 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:16.459116  5244 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.086s
I20260709 12:22:16.536127 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:16.563602 27660 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:16.506242 27656 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:16.565232 27655 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:16.478067  5169 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:16.564065 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72554.7us
I20260709 12:22:16.584939 27657 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
W20260709 12:22:16.558476 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69442.4us
I20260709 12:22:16.625651 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.625914 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.626251 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.575907 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:16.626579 27657 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:16.627396 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:16.593499 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:16.563625  4931 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:16.601542 27661 raft_consensus.cc:606] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:16.644676 27661 raft_consensus.cc:614] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:16.602969 27655 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
W20260709 12:22:16.604781 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.594135 27479 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 18 to 25, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 25 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:16.610041 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.612392  5003 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.615904  5244 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:16.617269  5122 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:16.575707  5043 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.599170  4785 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:16.631373 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:16.633280  4783 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:16.638952 27729 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:22:16.669056 27729 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:16.639149  5046 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.641435  5048 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:16.672622  5230 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
I20260709 12:22:16.681517  5201 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:16.682595  5201 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:16.682950  5201 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:16.643596 27658 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:16.565667  5223 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.685221  5223 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:16.605883  5246 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:22:16.687564  5092 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:22:16.646668 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53262.7us
I20260709 12:22:16.652050 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:16.688944  5223 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:16.689713 28335 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:16.662535  5164 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:16.652760  5244 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:16.693086  5244 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.694417  5244 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:16.651767  5210 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:22:16.594285  5132 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.647092  5198 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:16.699215 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.657045  5217 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:22:16.643217  5149 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.661548 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:16.667064 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.647902  4834 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:16.588502  5131 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.672679 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.699534 27662 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:16.658679 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:16.714152 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.652081  5144 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:16.654899 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:16.723165 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56185.1us
I20260709 12:22:16.727371 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:16.727759 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.664247  5222 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:16.699129  5046 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.731194 27657 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:16.701165  5179 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:16.720470  5252 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:22:16.703500  4376 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.700404  5230 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.662305  4592 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
W20260709 12:22:16.738569 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83773.6us
I20260709 12:22:16.745330 27587 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:16.758306 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.746233 28331 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:16.749383  5222 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.751636 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:16.753758 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:16.761868 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.760249 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.763592 27580 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:16.734017  5103 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:16.730437  5228 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:16.762707 27736 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37.
I20260709 12:22:16.763634 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:16.767238 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:16.767987 28336 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:16.772733 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 83574.1us
I20260709 12:22:16.743685  5165 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:16.773643 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.701144  5201 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.779173 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.779862 27657 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:16.784668 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.762965  5046 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:16.762050  4376 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:16.763849  5232 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:16.789902 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.792264 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.792411 27736 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:16.796000 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:16.796370 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:16.796640 27657 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
W20260709 12:22:16.806504 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.790078 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:16.786741  4785 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:16.811863  5150 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:16.812882 27657 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:16.826682 27733 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:16.790122  5255 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:16.830235 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:16.832003 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76758.1us
I20260709 12:22:16.838981 27736 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:16.839403 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:16.788800  5242 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:16.762669  5230 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.792172  4376 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:16.842758 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.800858  5165 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:16.777443  5228 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:16.851845  4376 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:16.817279  5201 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:16.851861 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 193253us
I20260709 12:22:16.857669 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.836848  5212 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:16.859966 27736 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:16.917214 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.917923 27736 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:16.803576  4940 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260709 12:22:16.847425  5198 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20260709 12:22:16.860944  5132 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.867879  5131 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:16.860595 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.874383  5169 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:16.879632 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.949962 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.949899 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.881767 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:16.884070  5259 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:22:16.838800  5255 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:16.877382  4376 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:16.957124  5255 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:16.958493  4376 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:16.888430 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:16.889045 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:16.960412 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99881.1us
I20260709 12:22:16.960772 27584 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:16.961476 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:16.889664 27661 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:16.889622 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:16.962975 27586 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:16.877403  5002 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:22:16.892314  5003 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:16.980389  5210 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:16.981127  5210 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:16.984025 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.985821 27735 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:16.900748 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.988790 27734 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
W20260709 12:22:16.902945 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.903229  4927 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:22:16.905252 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.909204  5230 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:16.880545  5263 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.859778  5068 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 12:22:16.920228 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.009629 27581 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:22:17.012926 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51540.1us
I20260709 12:22:17.013929 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:16.926577 27738 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:22:16.936358 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:16.947769 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 105086us
I20260709 12:22:16.952373 27730 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:16.963467 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.934890  4751 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:22:16.970322  5007 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:17.027995 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:16.948462  5239 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:22:17.007426 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.016188 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:17.011715 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:17.021416  4376 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:17.031541  4376 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.031991  4376 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:17.023736 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:17.034793 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:16.847265  5223 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:22:17.045814 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:17.047381 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:17.036921 27738 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:16.995467  5179 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:16.995714  5002 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:17.065915 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.066967 27738 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:17.004307  4927 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.145510 27661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:17.153980 27661 raft_consensus.cc:606] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:17.018585  5169 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:17.027486  5007 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.039783  5239 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:17.037541  5217 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:17.176787  4376 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.093796 27735 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
W20260709 12:22:17.149490 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.043853  5132 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:17.061256  5212 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:17.060690  4751 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:22:17.158070 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.281750  4376 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:17.060812  5103 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:17.075174  5042 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.086s
I20260709 12:22:17.074669  4931 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:17.071008  5206 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:17.169557 27586 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
W20260709 12:22:17.172593 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148986us
I20260709 12:22:17.029462  5263 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.220155 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:16.982388  5210 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.225997 28336 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 18. Current term is 19. Ops: []
I20260709 12:22:17.171841  5165 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.229324  5216 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:22:17.172995  5169 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.232385  5132 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:17.275529  5230 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.297039 27730 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
W20260709 12:22:17.300262  5266 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:22:17.301872 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:17.281131  5222 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:17.290259 27737 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:17.282977  4751 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:17.308468 27738 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:17.289198  5201 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:17.281070  5211 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:17.317610  5211 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:17.296656  5048 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.617s	user 0.011s	sys 0.000s
I20260709 12:22:17.320384 27664 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 35 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:22:17.320863  5169 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.313846 27735 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:17.311311  4751 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:17.288280  5103 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:17.315963  5270 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:17.301136  5132 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:22:17.321930  5268 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
W20260709 12:22:17.324604  5269 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:17.297353  5206 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.314570  5201 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.325340  5228 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.303228  5165 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.341269 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.310951  5230 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.349318 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:17.288764  5002 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.352952  5002 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:22:17.357043 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.357409  5103 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.359503  4783 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.360069 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.360447  5263 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.360037  4783 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.314741  5222 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.361271 27657 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:17.302870  5007 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.370937  5222 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:17.372386  5222 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:17.372771  4783 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.374419  5165 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:17.376535  5244 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.682s	user 0.012s	sys 0.000s
I20260709 12:22:17.376473  5165 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.377185  5165 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.379724 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3561) took 2541 ms. Trace:
I20260709 12:22:17.379809  5165 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.380270 27731 rpcz_store.cc:276] 0709 12:22:14.838392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.703282 (+864890us) service_pool.cc:224] Handling call
0709 12:22:15.711306 (+  8024us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.086513 (+375207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.379509 (+1292996us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:17.379668 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":775408000}
I20260709 12:22:17.380848  5213 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:22:17.381675 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.382453  5007 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:17.385036 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4169) took 1077 ms. Trace:
I20260709 12:22:17.385505 27664 rpcz_store.cc:276] 0709 12:22:16.307387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.308462 (+  1075us) service_pool.cc:224] Handling call
0709 12:22:17.384786 (+1076324us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:17.384964 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2324624128}
I20260709 12:22:17.388617  4834 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:17.389180  4834 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:17.359514  5206 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.394133  5165 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:17.287823  5239 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.394707  5165 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.397607  5165 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.397857  5230 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:17.399673  5201 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.400727  5201 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:17.407716  4830 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:17.408162  4830 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.409052  4830 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.409821  5211 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:17.410035  4830 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:17.372841  5222 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.414803  5222 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:17.414958 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3426) took 1081 ms. Trace:
I20260709 12:22:17.415426 27658 rpcz_store.cc:276] 0709 12:22:16.333888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.491413 (+157525us) service_pool.cc:224] Handling call
0709 12:22:16.627248 (+135835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.683670 (+ 56422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.414771 (+731101us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400344250. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:17.414900 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":278553344}
I20260709 12:22:17.417025 27658 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:17.418191 27658 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:17.419644  5092 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:17.420203  5092 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.423141 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3416) took 1104 ms. Trace:
I20260709 12:22:17.423079  5092 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.423565 27662 rpcz_store.cc:276] 0709 12:22:16.318685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.318815 (+   130us) service_pool.cc:224] Handling call
0709 12:22:16.319066 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.724772 (+405706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.422934 (+698162us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:17.423086 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":874558848}
I20260709 12:22:17.424542  5222 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.425108 27662 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:17.426329 27662 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:17.426759  5222 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:17.429391  4783 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.429912  4783 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.390974  5255 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.431887  5048 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.432160  5080 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:17.432615  5080 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.432368  5048 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.433276  5255 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:17.433557  5080 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.434553  5048 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:17.434645  5080 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:17.435158  5048 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.435441 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.435492 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.436663 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.437345 27580 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:17.437824 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.438326 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:17.439041 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:17.439581 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.440253 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:17.436764 27586 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 19.
I20260709 12:22:17.464169 27583 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:17.432170 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:17.464133  5048 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.465322  5132 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.464574 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.483023 27589 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:17.468752 27658 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:17.463682 27662 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:22:17.463982 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.484453 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.466133 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3434) took 1007 ms. Trace:
I20260709 12:22:17.486150 27662 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:17.485184 27655 rpcz_store.cc:276] 0709 12:22:16.458311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.828123 (+369812us) service_pool.cc:224] Handling call
0709 12:22:16.853061 (+ 24938us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:16.860194 (+  7133us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.860240 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:16.953963 (+ 93723us) log.cc:844] Serialized 52 byte log entry
0709 12:22:17.281552 (+327589us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:17.281593 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.281663 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.459237 (+177574us) raft_consensus.cc:1685] finished
0709 12:22:17.459268 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.466072 (+  6804us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":36357,"thread_start_us":206825,"threads_started":1,"wal-append.queue_time_us":290154}
I20260709 12:22:17.474589 27587 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 34 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 17.
W20260709 12:22:17.478350 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46267.5us
I20260709 12:22:17.489812 27658 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:17.484580 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:17.493768 27585 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:17.467381  4783 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:17.437857  4751 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.499029 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:17.501114  4751 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.502177 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:17.503723 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.506824 27662 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:17.496023 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:17.508983 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.490818  5165 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:17.521720 27660 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
W20260709 12:22:17.558295 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65620.5us
I20260709 12:22:17.527643 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4041) took 1232 ms. Trace:
I20260709 12:22:17.536634 27581 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:17.560645 27588 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:22:17.559535 27733 rpcz_store.cc:276] 0709 12:22:16.295013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.299416 (+  4403us) service_pool.cc:224] Handling call
0709 12:22:17.527488 (+1228072us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b54000a4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:17.527587 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1210442112}
W20260709 12:22:17.548632 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.549734 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 200486us
I20260709 12:22:17.552184 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.563519 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:17.564137 27585 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:17.554657 27584 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:17.559072 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:17.566836 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.568522 27662 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:17.570438 27586 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:17.575497 27583 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:22:17.579560 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.580497 27655 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:17.579690 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:17.558174  4751 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:17.583581 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.563027  4783 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:17.585415 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:17.581611  5271 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.303s	user 0.003s	sys 0.000s
I20260709 12:22:17.585337 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:17.589666 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:17.589494 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:17.590323 27660 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:17.591857 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:17.596518 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.598477 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.599294 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.600051 28329 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:22:17.599687 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.601506 28329 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:17.601682 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:17.602082 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:17.586457  5159 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:17.585940  4783 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.587129  5165 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:17.607309 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.609957 28333 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:17.610523 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.596374  5169 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:17.597656  5255 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:17.597468  5092 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:22:17.609812 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:17.599722  5211 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.627393  5048 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.627786  5211 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.628719  5048 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:22:17.629150  5048 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:17.629421 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3437) took 1064 ms. Trace:
I20260709 12:22:17.629506  5048 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:17.604846  5159 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.624104  5092 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.630563  5159 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:17.630937  5092 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:22:17.617738  5169 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.631795  5169 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:17.588457  5235 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.022s
I20260709 12:22:17.624307 28332 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:17.635782 28328 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:17.587529  4751 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.638047  5149 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:22:17.608017  5229 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.021s
I20260709 12:22:17.608412  4783 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:17.603422  5213 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.028s
I20260709 12:22:17.599084  5222 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.026s
I20260709 12:22:17.629853 27664 rpcz_store.cc:276] 0709 12:22:16.564665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.386624 (+821959us) service_pool.cc:224] Handling call
0709 12:22:17.387058 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.411989 (+ 24931us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.412024 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.412437 (+   413us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.582618 (+170181us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:17.586296 (+  3678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.586425 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.629183 (+ 42758us) raft_consensus.cc:1685] finished
0709 12:22:17.629203 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.629331 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45552,"raft.queue_time_us":7229,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":4397,"thread_start_us":163579,"threads_started":1,"wal-append.queue_time_us":208981}
W20260709 12:22:17.668838 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.657477 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:17.669565 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.647715 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.656175 27589 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
W20260709 12:22:17.670002 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.645658 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.664144 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:17.671058 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.672030 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.673312 27655 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:17.659209 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.675715 27584 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:17.678368 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:17.619973 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:17.683023 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.622129  5255 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:17.642211  5245 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.258s	user 0.004s	sys 0.000s
I20260709 12:22:17.688870 28335 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:17.630147  5211 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:17.687954 28332 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:17.690586 28337 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:17.676206  5285 raft_consensus.cc:993] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
W20260709 12:22:17.691540 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71650us
W20260709 12:22:17.691859 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 390069us
I20260709 12:22:17.696554 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:17.641497  5239 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:17.700089  5103 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:22:17.640143  5043 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.705724 28337 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 20. (term mismatch)
I20260709 12:22:17.706111 28337 pending_rounds.cc:85] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 19
I20260709 12:22:17.706336 28337 pending_rounds.cc:107] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 27.20
I20260709 12:22:17.706501 27660 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:17.699208 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.666127  5264 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:17.696642 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:17.698302 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:17.691824 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:17.665761  5206 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:17.682260  5235 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.706554 28337 raft_consensus.cc:2889] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:17.695786  5256 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:17.706835 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:17.639238  4751 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.719607  5244 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:17.720059 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:17.720464  5054 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:17.721047  5054 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:17.721329 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:17.722044  5054 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.688555  5267 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:17.690936  5104 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:17.697767  5255 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:17.698130 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:17.715380 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.696599  5273 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.729786  5273 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.701606  4830 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20260709 12:22:17.731451  4931 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:17.732090  4931 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:17.733152  4931 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.735425  5210 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:17.739584  5273 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:17.740654  4927 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.741868  4927 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:17.713117  5211 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:17.743203 27583 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:17.726972  5223 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:22:17.723114 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:17.766791 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.704282  5144 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:17.727349  5139 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:17.688781  5280 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.047s
I20260709 12:22:17.714254  5219 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.743017  5211 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.752074 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:17.718901 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:17.756757 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:17.767894 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56349.6us
I20260709 12:22:17.714171  5183 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:17.716310  5256 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:17.782793 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:17.724658  5235 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.779419 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.780055 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:17.783557 27655 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:17.714365  5201 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:22:17.713879  5007 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.057s
W20260709 12:22:17.777411 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.727474  5104 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:17.786331 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89747us
I20260709 12:22:17.841852 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:17.762033  5068 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:17.723229  5169 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.714663  5206 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.776939  5273 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:22:17.884943 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.778302  5264 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.783380  5002 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:17.780035  5211 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:17.783166  5183 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:17.785629  5066 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.908152 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.780862  5144 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:17.773317  5228 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.071s
I20260709 12:22:17.923959  4834 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.924432  5103 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:17.776731  5213 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.129s
I20260709 12:22:17.925462  4834 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:17.746666  5267 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.926584  5228 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:17.927055  5228 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.927508  5267 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.782944  5223 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:17.928000  5228 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:17.924623 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.928578  5223 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:17.929890 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.931476 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:17.775064  5159 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.789872  5101 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:17.784380  5139 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:17.932667 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3429) took 1553 ms. Trace:
I20260709 12:22:17.824323  5276 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:17.943432 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:17.940616 27589 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:17.940163 27656 rpcz_store.cc:276] 0709 12:22:16.379114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.601049 (+221935us) service_pool.cc:224] Handling call
0709 12:22:16.601380 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.601607 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:16.720899 (+119292us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.931648 (+1210749us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.932309 (+   661us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.932362 (+    53us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:17.932402 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.932481 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.932602 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1076674,"prepare.queue_time_us":1917,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":113,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":136,"thread_start_us":252041,"threads_started":2,"wal-append.queue_time_us":427200}
I20260709 12:22:17.840237  5229 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.848672  5104 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:17.938335 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.911823  5043 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:17.955148 27660 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:17.848158  5244 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.088s
I20260709 12:22:17.931730  4830 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:17.964236  4830 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:17.963097 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.964604  5144 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.954700  5101 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.965673  4830 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.966647  4830 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:17.936208  5164 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:17.937083  5292 raft_consensus.cc:1081] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:22:17.944650  5285 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:17.784186  5219 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.967700  5101 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:17.971263 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:17.986678  5290 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:17.933903  5066 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:17.983115  5285 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.957530  5267 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:17.958626  5276 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:22:18.053637 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82500.7us
W20260709 12:22:18.096433 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.097509 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:18.035812 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:18.092689 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:18.117519 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 269680us
I20260709 12:22:17.922363  5264 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.038079  5265 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.958814  5229 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:17.979449  5273 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:17.969784  5043 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.118263 27662 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:18.118209 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 186689us
I20260709 12:22:18.117771 28336 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:18.096038  5212 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:18.146636 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.110877  5285 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:18.110587  5296 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:18.121882 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:18.119649  5003 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.159229 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:18.161363 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:18.164222 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:18.166167 28336 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:18.167057 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3435) took 1671 ms. Trace:
W20260709 12:22:18.170406 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.171456 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:18.145974  5042 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.167490 27661 rpcz_store.cc:276] 0709 12:22:16.496132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.229186 (+733054us) service_pool.cc:224] Handling call
0709 12:22:17.278971 (+ 49785us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.279262 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:22:17.955611 (+676349us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.166737 (+211126us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.166785 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:18.166832 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.166892 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.167010 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":652895,"prepare.queue_time_us":679918,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"thread_start_us":199762,"threads_started":1}
I20260709 12:22:18.176740 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:18.177929 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:18.182910 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:17.954730  5256 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.177584 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:18.191947 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.189728 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.195626 27733 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:18.199636 27662 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:18.201500 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.217648 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:18.217844 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:18.112771  5139 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.176090  5201 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:18.162303  5213 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.195276 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:18.223714 27738 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:18.181758  5144 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.162818  5273 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.181828  5003 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.226716 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.242561 27661 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:18.227268  5300 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.227434 28334 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:22:18.227821  5101 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:18.228426 27732 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:22:18.254546 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:18.184638  5002 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:22:18.231807 27736 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:18.227294 27730 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:22:18.184664  5131 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:22:18.245873 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 86723.1us
I20260709 12:22:18.145277  5265 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.247710 27734 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:18.196084  5279 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:18.219704  5256 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:18.179066  5280 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.059s
I20260709 12:22:18.190331  5042 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.264242 27660 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:22:18.296958 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:18.320742 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:18.237813  5206 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.298439 27661 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
W20260709 12:22:18.324168 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 146641us
I20260709 12:22:18.252345 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:18.363497 27660 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:18.366621 27660 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:18.236886  5201 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:18.363107  5101 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.181855  5296 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.374414 28292 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37.
I20260709 12:22:18.299451  5279 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:18.364003  5131 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:18.393638 28292 leader_election.cc:403] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
I20260709 12:22:18.246352  5213 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.393297 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:18.323344  5169 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.246186  5080 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.122s
I20260709 12:22:18.416922  4376 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:18.247866  5144 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.369093  5201 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:18.423743 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:18.392520  5269 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:18.370162  5298 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:22:18.408242  5299 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.297s	user 0.001s	sys 0.000s
W20260709 12:22:18.417793 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 165523us
I20260709 12:22:18.374925  5159 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.442528  4376 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:18.466434  4376 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.466935  4376 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:18.443466 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69196.9us
I20260709 12:22:18.259891  5223 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.166s
I20260709 12:22:18.415452  5219 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:18.464183 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 209700us
W20260709 12:22:18.444995  5288 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.349s	user 0.000s	sys 0.002s
I20260709 12:22:18.433574  5295 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:18.418584  5279 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:18.479645 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:18.481011 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87852.1us
I20260709 12:22:18.237499  5276 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 24.20, Last appended by leader: 20, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.406435  5101 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:18.465441  4834 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.003s
I20260709 12:22:18.484157 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.510608 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.511831 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:18.260102  5300 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:18.487185 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.015s	sys 0.019s
W20260709 12:22:18.516017 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.016s	sys 0.019s
W20260709 12:22:18.516393 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 674601us
I20260709 12:22:18.517131 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:18.487546 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.488129  5313 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:18.488965  5314 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37.
I20260709 12:22:18.489490  5315 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:22:18.489965  5291 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.328s	user 0.000s	sys 0.001s
I20260709 12:22:18.459461  5066 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:18.490945 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.567955 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.568291 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.458988  5131 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:18.494391  4376 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.477036  5002 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.461817  5286 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:18.465250  5169 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:18.504658 27661 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:22:18.504976 27660 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:18.518392 27738 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:22:18.497366  5092 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.866s	user 0.011s	sys 0.000s
W20260709 12:22:18.492902  5255 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.757s	user 0.004s	sys 0.007s
I20260709 12:22:18.577895 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:18.577836 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3432) took 2167 ms. Trace:
I20260709 12:22:18.579276 27730 raft_consensus.cc:2376] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22. Re-sending same reply.
I20260709 12:22:18.580446 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.580674  4376 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:18.556061  5229 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:18.564805 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69118.4us
W20260709 12:22:18.568216 27657 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.000s	sys 0.009s
I20260709 12:22:18.583313 27660 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:18.463670  5292 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:18.579281 27663 rpcz_store.cc:276] 0709 12:22:16.410527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.813153 (+402626us) service_pool.cc:224] Handling call
0709 12:22:16.813546 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.856154 (+ 42608us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.856200 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:16.891309 (+ 35109us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.620177 (+728868us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b18001efca0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:17.756366 (+136189us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:17.756418 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.756509 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.574669 (+818160us) raft_consensus.cc:1685] finished
0709 12:22:18.574689 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.577779 (+  3090us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10576,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7014,"spinlock_wait_cycles":64403712,"thread_start_us":135950,"threads_started":1,"wal-append.queue_time_us":1066287}
I20260709 12:22:18.570134 27732 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:18.583580 27657 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.000s	sys 0.010s
I20260709 12:22:18.584327 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:18.522948  5264 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:18.573452 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.586247 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.585558 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:18.574822 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 343837us
I20260709 12:22:18.576918 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.587585 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:18.561311 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.584275 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.524502  5313 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.596414 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:18.596307 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.597807 27661 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:18.597056 27730 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:18.590178 27660 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:18.601212 27660 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:18.603545 27660 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:22:18.605455 27660 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:18.590159 27736 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:18.576768  4834 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:18.582047  5311 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:22:18.609400  5308 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.324s	user 0.000s	sys 0.001s
I20260709 12:22:18.609833  5319 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.610374  5319 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.611457  5318 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:18.612004  5318 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.612504  5319 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:18.613135  4830 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.646s	user 0.009s	sys 0.000s
I20260709 12:22:18.614040  5042 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:18.614957  5304 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
I20260709 12:22:18.580081  5003 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.582361  5092 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:18.617219  4927 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.875s	user 0.008s	sys 0.004s
I20260709 12:22:18.617594  5092 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:18.617977  5048 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.988s	user 0.012s	sys 0.000s
I20260709 12:22:18.618661 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3425) took 2288 ms. Trace:
I20260709 12:22:18.575387  5002 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.619469  5326 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.620107  5326 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.620707  5002 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.619021 27659 rpcz_store.cc:276] 0709 12:22:16.329869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.330003 (+   134us) service_pool.cc:224] Handling call
0709 12:22:16.373171 (+ 43168us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.373535 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:16.568035 (+194500us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.930760 (+1362725us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b180046ff60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:22:17.930990 (+   230us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.239698 (+308708us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.242587 (+  2889us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.242792 (+   205us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:18.249019 (+  6227us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.249319 (+   300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.298975 (+ 49656us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.618455 (+319480us) raft_consensus.cc:1685] finished
0709 12:22:18.618477 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.618608 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22246,"prepare.queue_time_us":1351404,"prepare.run_cpu_time_us":181,"prepare.run_wall_time_us":49243,"raft.queue_time_us":271502,"raft.run_cpu_time_us":530,"raft.run_wall_time_us":51708,"spinlock_wait_cycles":8065280,"thread_start_us":606906,"threads_started":4,"wal-append.queue_time_us":554993,"wal-append.run_cpu_time_us":1304,"wal-append.run_wall_time_us":1058468}
I20260709 12:22:18.621214  5329 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:18.590741  5131 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.621822  5329 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:18.618371  5048 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.623598  5316 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:22:18.624217  5316 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:18.624918  5048 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:18.625149  5316 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:18.614168  5297 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
I20260709 12:22:18.499212  5295 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.632632  5003 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:18.633415  5323 raft_consensus.cc:2764] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:22:18.632740  5048 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.634503  5295 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:18.635058 27661 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:18.634567  5048 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.632598  5002 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
I20260709 12:22:18.636008  5325 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.060s
I20260709 12:22:18.637075  5048 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.580483  5315 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.579619  5169 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:18.644409  5320 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 12:22:18.645502  5301 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260709 12:22:18.646751  5144 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:18.591482  5273 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:18.629884  5317 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:18.608428  5292 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.662642 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3438) took 2069 ms. Trace:
W20260709 12:22:18.662438  5210 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.922s	user 0.011s	sys 0.001s
I20260709 12:22:18.597023  5313 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:18.582311  5255 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.614284  5296 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.662988 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.694231 27737 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:18.694739 27738 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
W20260709 12:22:18.695175 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 178148us
I20260709 12:22:18.622836  5329 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.698487 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:18.704905 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:18.705506 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:18.708247 27730 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:18.656775  5319 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:22:18.706296  5213 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:18.721746 27582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:22:18.722489 27582 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:18.722853 27582 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:18.723006  5295 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.723678 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.723580  5295 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.724278 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:18.725601  5295 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.727137  5295 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.727636 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:18.727607  5295 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.709199  5311 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.728583  5255 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.729031  5311 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.709178  5317 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.728315 27661 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:18.656908  4834 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.731632 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:18.734009 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.743175 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:18.747395 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.741879 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.768581 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:22:18.751724  4783 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.069s	user 0.011s	sys 0.001s
I20260709 12:22:18.749452 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:18.705744  5296 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:18.751467 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:18.752153 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:18.745363 27582 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.749739 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:18.856331 27583 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:18.856611 27587 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:18.750732 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:18.750355 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:18.731293 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:18.862329 27582 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.731289  5311 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.858206 27738 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:18.858546 27586 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:18.743438 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:18.749470  5300 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:18.761987  5212 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.587s	user 0.005s	sys 0.005s
I20260709 12:22:18.860517 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:18.900331  5212 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:18.900662  5300 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:18.902019 27729 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:22:18.760807 28331 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:18.807928  5104 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.835s	user 0.007s	sys 0.004s
I20260709 12:22:18.863623 27731 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:22:18.735456  5327 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.194s
I20260709 12:22:18.734494  5295 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.731935 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:18.870446 28334 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:22:18.857197 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:18.856745 27585 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:18.661856  5315 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:18.880628  5333 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:22:18.881234  5149 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:18.691475  5292 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.885980 27589 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:22:18.886085 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:18.857887 27584 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:18.946925  5286 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:18.886701 28337 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:22:18.736012  5298 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:18.858922  4783 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.896740  5265 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:18.735913  5317 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.808262  5323 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:18.899358 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.899611 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.900441  5219 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:18.910256 27661 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:18.990384 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:18.991077 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:18.871312  5296 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.917945  5319 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:18.904091  5300 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260709 12:22:18.998792 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:18.920548  5239 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.220s	user 0.013s	sys 0.000s
I20260709 12:22:18.921155  5263 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.911212  5206 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.013s
I20260709 12:22:18.875638  5101 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:18.760454  5318 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.924503  5104 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.991926  5048 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.929015 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:18.932242 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.030920 27663 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:19.032197 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.032877 27663 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:18.932904  5183 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.000s	user 0.011s	sys 0.000s
I20260709 12:22:18.933533 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.037611 27660 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:18.935199  5279 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:18.936916  5169 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:18.937422  5256 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.611s	user 0.012s	sys 0.000s
I20260709 12:22:18.939841  5230 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:19.040516  5230 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:19.041406  5230 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.042498  5230 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:18.939597  5313 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.043455  5306 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.943490  5092 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.043980  5306 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.044071  5092 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.044802  5313 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:18.941728 27656 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:19.045565 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4188) took 1280 ms. Trace:
I20260709 12:22:18.951704  5068 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.045857 27656 rpcz_store.cc:276] 0709 12:22:17.765567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.948204 (+182637us) service_pool.cc:224] Handling call
0709 12:22:18.941458 (+993254us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b5400228ea8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea01255f4 00007f0e9f30e227 00007f0e9f30dab8 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6
0709 12:22:19.045512 (+104054us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11996032}
I20260709 12:22:18.993194  5219 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:19.047448  5281 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.317s	user 0.003s	sys 0.002s
W20260709 12:22:18.971911 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52465.4us
I20260709 12:22:19.048375 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:19.048821  4834 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.049477 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:18.982379 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96919.1us
I20260709 12:22:19.050172  5104 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:18.964471  5132 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:18.983098 27732 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:19.051960 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:19.052268  5292 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.052889 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:19.053748  5315 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.046034  5092 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:19.054878  5201 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.000s
I20260709 12:22:19.055392  5201 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.055994  5002 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:19.056394  5002 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:19.056756  5002 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:19.056833  5279 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.057431  5201 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.058022  5201 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:19.058339  5201 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:18.985764  5164 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.067200 28334 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:22:18.999670 27731 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:19.067909 27589 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 20.
I20260709 12:22:19.068230 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:19.068921 27731 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:19.005499 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.006330 28333 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:18.984884  5286 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:19.070384 27661 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:18.992475  5285 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.808s	user 0.010s	sys 0.002s
I20260709 12:22:19.028614  5300 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:22:19.024962 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.075520 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.075850 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:19.024962 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:19.073173 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:19.076592 27659 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:19.076885 27659 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:18.990540  5267 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.926409  5211 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.994s	user 0.011s	sys 0.000s
I20260709 12:22:19.035750 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:18.993474  5296 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:19.039721  5305 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:19.002880  5210 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:18.943989  5311 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.046284  5068 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:18.960191  5228 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:19.047246 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:19.050782  5104 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:19.060366  5183 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.943497  5043 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.046093  5306 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.058568  5317 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.059432  5048 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.059780  5319 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:18.983842 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:19.114431 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 130684us
I20260709 12:22:19.114940 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:19.103148  5003 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 12:22:19.117281  4751 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:19.001194 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:19.117758  4751 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:19.118388 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 117328us
I20260709 12:22:19.119127 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:19.119721  5265 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:19.120373  5265 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:19.120980  5292 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:22:19.121515  5292 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:19.122491  5292 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.066277 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.123687 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:18.970074  5149 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:19.025921 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.076278 27661 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:19.138945 27660 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:19.083305 27655 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:19.003216  5273 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:18.989696  5298 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.083081  5232 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:19.071843  5092 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:18.687620 27657 rpcz_store.cc:276] 0709 12:22:16.593132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.412685 (+819553us) service_pool.cc:224] Handling call
0709 12:22:17.412926 (+   241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.413078 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:18.584338 (+1171260us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.662283 (+ 77945us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.662348 (+    65us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:18.662404 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.662474 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.662583 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":550030,"prepare.queue_time_us":548099,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2683,"raft.queue_time_us":3273,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":10572,"thread_start_us":698698,"threads_started":2}
I20260709 12:22:19.087621  5311 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.071894  5280 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.088351  5267 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.089568 27659 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.052162 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.071085  5201 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:19.098670 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:19.105624  5324 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.103s
I20260709 12:22:19.026528  5263 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.071589  5042 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:22:19.091609  5211 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.110450 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.219954 27656 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:19.220803 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:19.221257 27656 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26.
I20260709 12:22:19.120292 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:19.125708 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:19.139956 28337 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:19.138980 27738 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:19.202868  5002 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.142565 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:19.226218 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:19.079432 27663 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:19.144130 27731 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:19.147651 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1547) took 1541 ms. Trace:
I20260709 12:22:19.138824 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:19.071266  5054 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.329s	user 0.013s	sys 0.018s
I20260709 12:22:19.074373  5164 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.151949 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:19.083870  5318 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:22:19.153187 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:19.176154 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:19.139477  5244 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:19.047211  5219 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.138762  4592 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:19.148095  4783 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.118645  4751 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:18.993296  5323 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:19.267877 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.250278  5279 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:19.269004  5279 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:19.269949  5279 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.250695  5169 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:22:19.276715  5263 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.277489  5263 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.278082  5263 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.253911  4592 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:19.280251  4592 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.280467  4834 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.243662  5043 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:19.281385  4834 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.281956  5043 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:19.195717 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 99670.9us
I20260709 12:22:19.093783  5210 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.296614  5267 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.298159  5210 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:19.198514 27627 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.712s (SENT) [suppressed 9 similar messages]
I20260709 12:22:19.299500 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1433) took 1371 ms. Trace:
I20260709 12:22:19.148481  5206 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.300668  5206 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.078455  5285 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.149556  5232 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.153015  5092 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.147781  5305 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:19.201828 27734 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:19.212813 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:19.213817 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.072409  5315 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.215005 27659 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:19.215101 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:19.179858  5317 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.222476 27662 raft_consensus.cc:2368] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:22:19.141364 27736 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
W20260709 12:22:19.224442 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 104225us
I20260709 12:22:19.154042  5273 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.229753 28328 rpcz_store.cc:276] 0709 12:22:17.598320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.598428 (+   108us) service_pool.cc:224] Handling call
0709 12:22:19.123500 (+1525072us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:19.139266 (+ 15766us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56676224}
I20260709 12:22:19.244482 27663 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:22:19.245254 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116834us
I20260709 12:22:19.245683 27660 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:19.247220 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 94121.4us
I20260709 12:22:19.247308 28334 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:19.249089 27738 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:19.249194 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.181646  5280 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:19.243089  5054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.501s	user 0.014s	sys 0.018s
W20260709 12:22:19.254722  5342 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 12:22:19.255041  5343 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:22:19.211669  5149 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.260865  5286 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.261724  5132 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.262650 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3601) took 1062 ms. Trace:
I20260709 12:22:19.264870  5002 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.266600  5211 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.288753  5219 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:19.300145  5296 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.283155  5043 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.301275  5326 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.536545  5311 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.548892 27580 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:19.552417  5296 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.502633  5042 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.484011 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.556682 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:19.499228 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.481736 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.299930 28335 rpcz_store.cc:276] 0709 12:22:17.927997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.940729 (+ 12732us) service_pool.cc:224] Handling call
0709 12:22:17.941098 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.376190 (+435092us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.376238 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.378057 (+  1819us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.514020 (+135963us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:18.514073 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.285122 (+771049us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:19.285171 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.285220 (+    49us) raft_consensus.cc:1685] finished
0709 12:22:19.285232 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.299427 (+ 14195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":70559,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":40334,"spinlock_wait_cycles":558803328,"thread_start_us":266493,"threads_started":2,"wal-append.queue_time_us":239465}
I20260709 12:22:19.569756 27732 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:19.570441 27738 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:19.571033 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.443660 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:19.575023 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:19.576699 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:19.559202 27733 rpcz_store.cc:276] 0709 12:22:18.199776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.199948 (+   172us) service_pool.cc:224] Handling call
0709 12:22:18.217506 (+ 17558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.944869 (+727363us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b18004cdb00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:22:18.944997 (+   128us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:22:18.945023 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.945763 (+   740us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.078804 (+133041us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:19.078837 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.260098 (+181261us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54000a5fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:19.260172 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.262426 (+  2254us) raft_consensus.cc:1685] finished
0709 12:22:19.262445 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.262594 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8617,"raft.queue_time_us":21013,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":20325,"spinlock_wait_cycles":669400064,"thread_start_us":171628,"threads_started":2,"wal-append.queue_time_us":201541}
W20260709 12:22:19.579305 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.582623 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
W20260709 12:22:19.582309 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 383787us
I20260709 12:22:19.586642 28292 leader_election.cc:400] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26.
I20260709 12:22:19.587123 28292 leader_election.cc:403] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
W20260709 12:22:19.469019  5303 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.056s	user 0.004s	sys 0.000s
I20260709 12:22:19.249260  5244 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:19.385958  5233 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:19.560904  5232 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:19.465538  5263 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.302978  5206 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.600524 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.618623 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:19.438870  5346 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:22:19.285282  5104 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.285168  5169 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:19.622223 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.622854 27735 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:19.561586  5305 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:19.598848 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:19.552451  5183 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.479985  5323 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.561851  5092 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:19.621182 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:19.736167 27662 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:19.281383  4592 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:19.580261  5298 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.585244  4751 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:19.766039  5048 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.595985  4931 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:19.767953  4931 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.729588 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:19.580139  5217 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.606938  5306 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.770953  5217 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.771049  5306 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.773115  5217 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.773670  5217 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26.
I20260709 12:22:19.570070  5315 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.774369  5217 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.774864  5217 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.775633  5164 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.775918  5315 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.580652  5267 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:19.776592  5267 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.777146 27662 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: [30.17-30.17]
I20260709 12:22:19.777340  5217 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.784457  5285 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.611392  5265 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.174s
I20260709 12:22:19.786034  5265 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.786506  5265 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.790385 27732 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:19.795750 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.731178 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:19.731817 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:19.800282 27738 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:19.373945  5324 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.249s
I20260709 12:22:19.596840  5210 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.733397 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 134633us
W20260709 12:22:19.733633 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1546) took 2135 ms (client timeout 1902 ms). Trace:
I20260709 12:22:19.604147  5335 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.733745 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 115202us
I20260709 12:22:19.734341 28332 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:19.609748  4834 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.611821  5179 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:19.596474  5068 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.731851 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:19.822007 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90953.2us
I20260709 12:22:19.823421 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:19.824501 27730 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:19.734783  5104 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.745896  5235 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.747473  5206 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:19.583633  5132 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.749076 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.751016  5054 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.753118 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.754632 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.756170 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3607) took 1527 ms. Trace:
W20260709 12:22:19.760175  5310 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.320s	user 0.003s	sys 0.007s
I20260709 12:22:19.764770  5286 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.616138  5233 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:19.805281 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:19.956166 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4350) took 1205 ms. Trace:
I20260709 12:22:19.956619 27580 rpcz_store.cc:276] 0709 12:22:18.750341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.750509 (+   168us) service_pool.cc:224] Handling call
0709 12:22:19.956024 (+1205515us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54001301d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:19.956112 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589189376}
I20260709 12:22:19.810484 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:19.809079 28331 rpcz_store.cc:269] 0709 12:22:17.598020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.598191 (+   171us) service_pool.cc:224] Handling call
0709 12:22:19.733405 (+2135214us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54005d4654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:19.733560 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2555325184}
I20260709 12:22:19.580806  5042 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.580694  5317 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.815166 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.815508 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:19.962409 27738 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:19.962453 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:19.811103 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:19.963379 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:19.963711 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 152674us
I20260709 12:22:19.741493 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:19.964515 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3468) took 1003 ms. Trace:
I20260709 12:22:19.964892 27659 rpcz_store.cc:276] 0709 12:22:18.960819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.992185 (+ 31366us) service_pool.cc:224] Handling call
0709 12:22:19.964444 (+972259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:19.827879 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.802218  5165 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.792450  5263 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:22:19.969529 28333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1468) took 1792 ms (client timeout 1627 ms). Trace:
W20260709 12:22:19.970206 28333 rpcz_store.cc:269] 0709 12:22:18.176973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.177093 (+   120us) service_pool.cc:224] Handling call
0709 12:22:19.969354 (+1792261us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54003705d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:19.969459 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1778115968}
W20260709 12:22:19.970626 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 160219us
W20260709 12:22:19.866236 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 295478us
I20260709 12:22:19.817981  5179 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:19.822789  5201 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.003s
I20260709 12:22:19.966032 27735 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:19.905762 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1466) took 1731 ms. Trace:
I20260709 12:22:19.906378 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3464) took 1164 ms. Trace:
I20260709 12:22:19.968539 27660 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:22:19.741336  5292 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.900931 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:19.875979 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.971287 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:19.964407 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.015106 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:19.818639  5228 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.148s
W20260709 12:22:20.061152 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 89963us
I20260709 12:22:20.011442 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.964038 28331 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
W20260709 12:22:20.063614 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99284.9us
I20260709 12:22:19.973465 27737 rpcz_store.cc:276] 0709 12:22:18.228437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.233314 (+  4877us) service_pool.cc:224] Handling call
0709 12:22:18.260223 (+ 26909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.694618 (+434395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.755870 (+1061252us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:19.756102 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":997298048}
I20260709 12:22:19.980036  5233 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.981141  5347 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:19.980069 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:19.985436  5132 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.985937  5305 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:19.986421 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:19.825345  5280 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:19.992097  5306 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.993192  4834 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:19.993407  5265 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:19.997502  5324 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.002526  5267 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:19.821223  5335 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.003489  5165 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.109941 27735 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:20.004241  5263 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:19.864125  5206 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:20.006166 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.007486 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:20.016829 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.016665 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:20.119095 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.119377 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.120004 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.120627 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.121414 28335 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:20.089434  5223 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:22:20.012423 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.972771 27733 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:19.825601  5244 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.064293 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.064828 27655 rpcz_store.cc:276] 0709 12:22:18.741640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.898978 (+157338us) service_pool.cc:224] Handling call
0709 12:22:19.906169 (+1007191us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:19.906310 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211286912}
W20260709 12:22:20.132746 27656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.008s	sys 0.024s
W20260709 12:22:20.133234 27656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.008s	sys 0.024s
I20260709 12:22:20.065655 28332 rpcz_store.cc:276] 0709 12:22:18.174552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.175090 (+   538us) service_pool.cc:224] Handling call
0709 12:22:19.734170 (+1559080us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:19.809478 (+ 75308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.905525 (+ 96047us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54005d4650. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:19.905687 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3583969536}
I20260709 12:22:20.072386 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:20.074000 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:20.087286 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:19.965447  5327 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:19.895023  5164 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:19.821154  5232 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.138478 27733 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:20.090843 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
W20260709 12:22:20.139134 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.068s	sys 0.125s
I20260709 12:22:19.818236  5210 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:19.895260  5315 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:20.018936  5179 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:20.087203  4834 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.073450  5347 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.973421  5054 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.107558 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:20.120595 28331 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:20.072899  5002 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:20.065115 27734 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:20.066391 27729 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:20.156548 27737 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:20.136674  5164 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.097666  5233 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:20.133107 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 116551us
I20260709 12:22:20.136147 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.172048 28332 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:20.067255  5201 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:20.137157 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72950.3us
W20260709 12:22:20.179008 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 387792us
I20260709 12:22:20.182564 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:20.129487  5286 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:20.139796 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.139595 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.069s	sys 0.125s
W20260709 12:22:20.187840 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 601296us
I20260709 12:22:20.148914 27730 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 25 index: 21. (index mismatch)
I20260709 12:22:20.125147  5206 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:20.114419  5263 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:20.074664  5292 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:20.152868 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.017s	sys 0.017s
I20260709 12:22:20.126819  5223 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.156787 28333 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.24-35.24]
I20260709 12:22:20.136842  5327 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:20.139222  5048 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:20.129914 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.175215 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.150549  5267 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:20.231709 28335 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:20.132624  4751 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
W20260709 12:22:20.200980 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.092482  5169 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.236008  5267 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:20.250872  5267 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:20.156533  5317 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.188706 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:20.204713 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.017s	sys 0.017s
W20260709 12:22:20.256062 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67475.5us
I20260709 12:22:20.256217 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.256940 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:20.257022 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:20.141043  5092 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.210223 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.161175  4834 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:20.167716  5347 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:20.219897 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.167918  5228 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.204344 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.176013  5164 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:20.264672 28330 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:20.221858  5311 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.270901 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.231242 28336 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:20.231760 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3463) took 1490 ms. Trace:
I20260709 12:22:20.223894 28328 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:22:20.272104 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.021s	sys 0.020s
W20260709 12:22:20.272552 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.021s	sys 0.020s
I20260709 12:22:20.224164  5183 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.184990 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.172423  5002 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:20.244267  5210 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.245111  5104 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.245777  5068 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.245921  5007 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:20.261353 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:20.293955 27659 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:20.222327  5286 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.262943 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.233566 27729 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:20.188446  5326 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:20.223172 28331 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:20.264998 28335 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
W20260709 12:22:20.263291 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53171.8us
I20260709 12:22:20.266696  5317 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:20.224471  5323 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:20.225083  5223 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:20.270355 28330 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:20.271757 27656 rpcz_store.cc:276] 0709 12:22:18.741401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.221599 (+480198us) service_pool.cc:224] Handling call
0709 12:22:19.221780 (+   181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:19.221922 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:19.613081 (+391159us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.231186 (+618105us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.231404 (+   218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.231460 (+    56us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:20.231510 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.231574 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.231697 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88546,"prepare.queue_time_us":797698,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":51130,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":24868,"spinlock_wait_cycles":45248384,"thread_start_us":888026,"threads_started":2}
I20260709 12:22:20.280071 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:20.295929 27735 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:20.296754 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:20.312203 27738 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:20.275579  5132 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.022s
I20260709 12:22:20.275871  5228 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.299216  5305 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.187309  5298 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.293535 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:20.106208  5324 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.303769 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.263952  5355 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:22:20.287618  5352 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.282s	user 0.001s	sys 0.001s
W20260709 12:22:20.288326  5350 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:22:20.281527  5276 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:20.343230 27659 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:20.343484 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.293308  5007 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:20.304898  5317 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:20.345685 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65732us
I20260709 12:22:20.349570 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.288769  5183 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.377414 27585 raft_consensus.cc:2368] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:22:20.376060 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.349933 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.379012 27589 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:20.351279 27581 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:22:20.352054 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.354225 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.380918 27586 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:20.355165 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:20.356884 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.382489 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.307106  5323 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.308983  5104 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.306459  5296 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:22:20.283376  5311 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.372992 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.335107  5210 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.385974 27583 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 14. (term mismatch)
I20260709 12:22:20.397099 27583 pending_rounds.cc:85] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:22:20.397370 27583 pending_rounds.cc:107] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260709 12:22:20.363759  5324 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.340231  5306 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.390170 27581 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:22:20.345288  5355 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.359579  5280 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:20.395236 27585 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:20.397504 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.397663 27583 raft_consensus.cc:2889] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.375700 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:20.411237 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.412299 27581 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:20.402647 27586 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:22:20.411093 27589 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:20.386292  5317 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:20.389539  5323 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:20.310381  5326 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.414984 27587 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:20.296037  5327 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.388711  5305 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:20.416702  5048 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:20.434234 27663 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.010s	sys 0.009s
I20260709 12:22:20.345678  5298 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.391454  5165 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.402232  5232 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.401081  5210 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.415422  5280 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:20.474786 27663 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.011s	sys 0.009s
I20260709 12:22:20.489462 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.411070  5355 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:20.498184 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194492us
I20260709 12:22:20.404066  5357 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.080s
I20260709 12:22:20.396801  5104 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.468286  5324 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.509616 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.486363  5327 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:20.510622  5363 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.510852  5068 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.514209  5048 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.485909  5305 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:20.473948  5335 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:20.536980  5358 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:22:20.537261  5363 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.539554  5363 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:20.539922  5286 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.540042  5068 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.527153  5165 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:20.542531  5165 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.542901  5165 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:20.545717  5232 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:20.546193  5228 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.546257  5232 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.549103  5232 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.474336  5007 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.552088  5007 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:20.523103 27655 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:22:20.524127 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:20.398910  5002 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.567545 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.471686  5370 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.103s
I20260709 12:22:20.568370 27655 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:20.569176 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:20.525453 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140514us
I20260709 12:22:20.570147 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.508026  5280 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:20.526850 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.414358  5306 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.486222  5201 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.536844  5210 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:20.566447  5361 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:22:20.587193 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.587859  5132 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:20.589471  5132 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:20.589860  5132 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:20.590281  5132 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:20.571695 27580 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:22:20.576574 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1718) took 2821 ms. Trace:
I20260709 12:22:20.591955 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.592322 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.592639 27589 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:20.594039  5267 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.582870  5370 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.610823 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.611707 28337 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:20.583602  5232 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:20.585721  5372 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:20.585924  5306 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:20.566810 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.591418 27658 rpcz_store.cc:276] 0709 12:22:17.754722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.754874 (+   152us) service_pool.cc:224] Handling call
0709 12:22:17.755189 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.159946 (+404757us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:18.195308 (+ 35362us) log.cc:844] Serialized 14 byte log entry
0709 12:22:19.263988 (+1068680us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b180036d3c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:19.622365 (+358377us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.728214 (+105849us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.731402 (+  3188us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:20.187504 (+456102us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.187792 (+   288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.187867 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.576368 (+388501us) raft_consensus.cc:1685] finished
0709 12:22:20.576389 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.576514 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124446,"prepare.queue_time_us":289518,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":10539,"raft.queue_time_us":289775,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":80324,"spinlock_wait_cycles":93129856,"thread_start_us":1133563,"threads_started":3,"wal-append.queue_time_us":1831931}
I20260709 12:22:20.627720 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.596154  5298 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.570500  5276 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:20.597368 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1472) took 1031 ms. Trace:
I20260709 12:22:20.576498 27587 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:20.600708 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:20.601843 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.637183 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:20.602149 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.602466 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.511441  5054 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:20.606931 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:20.615361 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.528009  5375 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.638404 28330 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:20.597294  5311 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.628515 27658 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:20.599972  5309 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.159s	user 0.021s	sys 0.003s
I20260709 12:22:20.634548 28329 rpcz_store.cc:276] 0709 12:22:19.566447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.566595 (+   148us) service_pool.cc:224] Handling call
0709 12:22:19.567037 (+   442us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.897122 (+330085us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:22:19.897166 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.897641 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.312457 (+414816us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:20.312643 (+   186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.312706 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.597137 (+284431us) raft_consensus.cc:1685] finished
0709 12:22:20.597177 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.597308 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3366,"raft.queue_time_us":45477,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":29238,"thread_start_us":546382,"threads_started":2,"wal-append.queue_time_us":522780}
I20260709 12:22:20.648007 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.649362 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.643975 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:20.650375 27658 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:20.596864  5355 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.524885  5324 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:20.646443 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.642187 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.657892 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.653101 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.661203 27658 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:20.663021 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.655606  4830 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:20.642585  5327 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.650735  4592 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:20.674271 28332 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:20.648854  5201 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:20.668200 27552 leader_election.cc:400] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:22:20.620321  5370 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.665757  4830 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.682981 27552 leader_election.cc:403] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:22:20.626272  5232 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.680886 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.686518 27658 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:20.690442 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:20.679724  5267 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:20.679795  4592 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.692019  4592 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:20.695789  5380 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:20.701020  5165 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.701835  5092 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.702452 27655 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:20.704376  5165 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:20.705348 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 873937us
I20260709 12:22:20.708106 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1717) took 3039 ms. Trace:
I20260709 12:22:20.708505 27664 rpcz_store.cc:276] 0709 12:22:17.668332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.669501 (+  1169us) service_pool.cc:224] Handling call
0709 12:22:17.689446 (+ 19945us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.774449 (+ 85003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.774481 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.774970 (+   489us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.158197 (+383227us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:18.158979 (+   782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.159067 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.388516 (+2229449us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:20.707816 (+319300us) raft_consensus.cc:1685] finished
0709 12:22:20.707836 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.708047 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6256,"raft.queue_time_us":214873,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9157,"spinlock_wait_cycles":17536384,"thread_start_us":338977,"threads_started":2,"wal-append.queue_time_us":450109}
I20260709 12:22:20.710387 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.710640 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.711082 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:20.711709 27582 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:20.712875 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.717638 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:20.718489 27704 leader_election.cc:400] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:22:20.718838 27704 leader_election.cc:403] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:20.651734  5375 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.724095 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.686477  4830 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:20.755276 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.761618 27585 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:20.755584 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:20.679427  5373 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.085s
I20260709 12:22:20.776638  5370 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.756240  5276 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.778191  5355 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:20.778772  5382 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:22:20.780232 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:20.770602  5347 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.758991 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.757794 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.681672  5201 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:20.763191 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45648.7us
I20260709 12:22:20.763665  5043 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:20.764501  5335 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:22:20.800163  5201 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:20.801234 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:20.802080  5370 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:22:20.773991  5206 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.543s	user 0.006s	sys 0.003s
I20260709 12:22:20.771152  5386 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:20.802793  5370 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:20.803258  5370 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:20.802961  5206 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.803388  5386 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:20.774397  5223 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.783355 27584 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:20.785959 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.804729  5386 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.805423 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.805825 27730 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:22:20.782076 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.806079 27658 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:20.807192 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.805193 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.758605 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.807269 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.808857 27586 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:20.809257 27664 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:22:20.806023  5223 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:20.810170  5223 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:20.810585 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.809719 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.801996  5335 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:20.812201 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.809345 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:20.812944 27658 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:20.813905  5373 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:20.815663 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.816074 27658 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:22:20.686470  5002 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:20.719913  4751 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.662495  5305 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.720885  5384 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.678082  5327 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.817158 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.810544  5206 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:20.760120  5315 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.607s	user 0.008s	sys 0.003s
I20260709 12:22:20.752045  5363 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:22:20.835268 27658 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:20.713099  5183 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:20.721139  5228 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:20.722956  5374 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.858428 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.935827 27585 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:20.699138  5280 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:20.796111  5233 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.605s	user 0.005s	sys 0.005s
W20260709 12:22:20.938396 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 180668us
I20260709 12:22:20.970185 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:20.970851 28333 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:22:20.969362 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:20.819408  5385 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:22:20.866724 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:20.866746 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:20.985333 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:20.985783 27586 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:20.810618  5223 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:20.859786  5219 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.274s	user 0.012s	sys 0.000s
I20260709 12:22:20.910105 27588 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:20.992760 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:20.759912  5210 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.104s
I20260709 12:22:20.858263  4751 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.908212 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:20.756343  5306 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.105s
W20260709 12:22:20.875974  5263 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.640s	user 0.010s	sys 0.000s
I20260709 12:22:20.875937  5315 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:22:20.924033  5179 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.758s	user 0.011s	sys 0.000s
I20260709 12:22:21.013491  5179 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.014390 27660 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:22:21.015561 27733 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:21.015714  5179 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.016398  5179 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:20.928520  5375 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.016923 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.017225  5375 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.017725  5375 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:20.901600  5370 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.019825  5375 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.020934  5370 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:20.965826 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 211142us
I20260709 12:22:20.876155  5228 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:21.022490  5228 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:20.859565  5323 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:20.988234 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:21.018247 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.027837 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:21.022799  4830 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.029841 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:20.920020  5324 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.031708  5179 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:21.031867 27589 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:21.032146  5179 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.032570  5179 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:21.028282 27656 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:22:21.034654 27656 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:20.989657  5043 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:22:20.993556 27732 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 36 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:22:20.994904 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.036054  5043 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:21.036448  5043 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:21.000188 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.003193 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:21.036986  5043 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:21.037243 27583 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:21.037688 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.038271 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.039155 27586 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:20.864898  5384 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:21.040081  5393 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:21.040504 27586 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:20.968073  5374 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.041908  5384 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.043286  5374 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.009315  4592 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.044421 27583 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:21.045694  5164 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.766s	user 0.001s	sys 0.008s
I20260709 12:22:21.045476  5201 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.045975  4592 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.031117 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:21.046135  5164 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.047501 27582 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:21.049759  5280 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:20.990765  5223 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:21.052358  5179 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.054487  5179 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:20.996981 27658 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 19.
I20260709 12:22:21.055020  5179 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:21.000526  5132 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.056089  5179 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:21.056471  5179 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.056828  5179 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:21.056938  5232 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.057536  5132 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.058279 28328 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:21.058920  5219 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:21.059363  5219 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:21.059571  5327 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.059769  5219 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:20.996167  5210 consensus_queue.cc:1243] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 34.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:22:21.064507  5335 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:21.064659  5210 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:20.971127  5317 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.065718 27585 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:21.049778 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3469) took 2088 ms. Trace:
I20260709 12:22:21.067205  5317 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.068089 27664 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.068424  5183 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:21.012477  5263 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.071153  5263 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:21.072058 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.011222  5347 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.036381 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.073079 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:21.073915 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:21.066509 27663 rpcz_store.cc:276] 0709 12:22:18.961148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.573378 (+612230us) service_pool.cc:224] Handling call
0709 12:22:19.573815 (+   437us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:19.679529 (+105714us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:22:20.490226 (+810697us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.647163 (+156937us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:20.661768 (+ 14605us) log.cc:844] Serialized 52 byte log entry
0709 12:22:20.661930 (+   162us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:20.662215 (+   285us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.678031 (+ 15816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.678117 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.049557 (+371440us) raft_consensus.cc:1685] finished
0709 12:22:21.049578 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.049720 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66550,"prepare.queue_time_us":250105,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":4719,"raft.queue_time_us":69063,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":97707,"spinlock_wait_cycles":5083904,"thread_start_us":789707,"threads_started":2,"wal-append.queue_time_us":471108}
I20260709 12:22:21.028218  5323 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:21.071686  5007 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.519s	user 0.007s	sys 0.003s
I20260709 12:22:21.084780 27656 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
W20260709 12:22:21.085170 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 267768us
W20260709 12:22:21.087066 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.051223  5306 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:21.089666  5104 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.090754  5179 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.084039  5263 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:21.092654  5164 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.051251  4751 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.095022 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:21.087278 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:21.122527  5397 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:21.085211  5335 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:21.197247  5397 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.094684  5363 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:22:21.197680  5397 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:21.095480 27584 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:22:21.196383 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 416206us
I20260709 12:22:21.118206  4830 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.199579 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:21.200891 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.201193  5326 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.201774  5363 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:21.203358 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:21.204046  5306 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.204483  5306 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:21.122774  5201 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.132041  5002 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:22:21.138903  5223 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.208618 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.208999  5223 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.209204  5363 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:22:21.173749 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.210826 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 123630us
I20260709 12:22:21.216590 27586 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
W20260709 12:22:21.219041 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 171842us
W20260709 12:22:21.220520 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.221088 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 990726us
I20260709 12:22:21.222520 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:21.223788 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.214578 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:21.224818 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:21.226636 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.214995  5201 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:21.228246 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.230017 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:21.225069 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.231933 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:21.233911 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:21.228222  5201 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.174703  5398 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.209055  5335 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:21.209610  5363 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:21.263758  5273 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.264813  5335 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:21.265034  5273 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:21.267412 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.239459 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:21.224009  5263 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:21.229590  5326 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:21.340180  5326 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:21.341102  5326 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.272517 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.274253  5068 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.276122  5285 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:21.277227 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3466) took 2362 ms. Trace:
I20260709 12:22:21.346287  5319 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.344901 27661 rpcz_store.cc:276] 0709 12:22:18.914393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.226907 (+312514us) service_pool.cc:224] Handling call
0709 12:22:19.227251 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.007889 (+780638us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:20.079882 (+ 71993us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.611678 (+531796us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.611735 (+    57us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:20.611802 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.277011 (+665209us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b5400211954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:21.277052 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.277175 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":72845,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":14272,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":1794665856,"thread_start_us":531315,"threads_started":2,"wal-append.queue_time_us":487064}
I20260709 12:22:21.238739  5317 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:21.346814  5319 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.347592  5317 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:21.348238  5317 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:21.349637  5319 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.238900  5280 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:21.351673  5280 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:21.352556  5280 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:21.270376  4834 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.995s	user 0.008s	sys 0.004s
I20260709 12:22:21.354964  4834 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.357520  4834 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.358898 27736 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:22:21.362322 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3628) took 1068 ms. Trace:
I20260709 12:22:21.362672 27736 rpcz_store.cc:276] 0709 12:22:20.293784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.293919 (+   135us) service_pool.cc:224] Handling call
0709 12:22:20.294311 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.359493 (+1065182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.362055 (+  2562us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:21.362263 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2447495424}
I20260709 12:22:21.263118  5150 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.016s
I20260709 12:22:21.281571 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.385797 27589 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:21.268623  5201 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:21.285665  4783 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.289709  5267 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:21.291121 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.291759 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.292451 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1730) took 1711 ms. Trace:
I20260709 12:22:21.423921 27660 rpcz_store.cc:276] 0709 12:22:19.581247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.098571 (+517324us) service_pool.cc:224] Handling call
0709 12:22:20.128395 (+ 29824us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.014751 (+886356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.292215 (+277464us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:21.292389 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2036729216}
I20260709 12:22:21.292793  5219 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.293645 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3462) took 2554 ms. Trace:
I20260709 12:22:21.425045 27657 rpcz_store.cc:276] 0709 12:22:18.740136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.150011 (+409875us) service_pool.cc:224] Handling call
0709 12:22:19.227659 (+ 77648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.227995 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.228022 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.245441 (+ 17419us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.611308 (+365867us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:19.611368 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.611460 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.487696 (+876236us) raft_consensus.cc:1685] finished
0709 12:22:20.487728 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.293417 (+805689us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b54001e4bd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:21.293592 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14742,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":3899,"spinlock_wait_cycles":177859328,"thread_start_us":324677,"threads_started":1,"wal-append.queue_time_us":570953}
I20260709 12:22:21.294338  5397 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.295148 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.432320 27580 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:21.432433 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.236629  5323 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:21.296665 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1598) took 1858 ms (client timeout 2174 ms). Trace:
I20260709 12:22:21.436371 27587 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:21.438112 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.299428  5263 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:21.301702 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.440730 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.441483 27585 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:21.304791 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.306604 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.308708  5404 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:21.315882  5179 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.316210  5042 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.320881  5244 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:21.328629  4931 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.336732 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.337272 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.337319 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.338001 28336 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:22:21.339291 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:21.451272 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 112030us
I20260709 12:22:21.422313  5285 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.237607  5164 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:21.263078  5381 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:21.267439  5363 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:21.366904  5359 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:21.238018  5405 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.133s
I20260709 12:22:21.267426  5398 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.369591 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:21.374279 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.228215  5356 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:21.429927 27589 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
W20260709 12:22:21.443826 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.436460 27732 rpcz_store.cc:269] 0709 12:22:19.438313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.904718 (+466405us) service_pool.cc:224] Handling call
0709 12:22:21.296414 (+1391696us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:21.296590 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2135016320}
I20260709 12:22:21.439266 27588 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:21.423524 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.473387 27655 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:21.472605 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 226344us
I20260709 12:22:21.450057 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.475028 27661 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.449925 28335 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:21.452509 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.449997 28333 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:21.459154 27585 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:21.448431  5394 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:21.453109  5306 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.463410  5042 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:21.422628  5267 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:21.468274 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.488904 27587 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:21.472641 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.490085 27659 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:21.476889 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.477170 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.479763 27656 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:22:21.493818 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.499480 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.500295 27589 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:21.501183 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.480149 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.481565 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.484308 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.509739 28332 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:21.509812 27589 raft_consensus.cc:2376] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33. Re-sending same reply.
I20260709 12:22:21.511018 27588 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:21.465701  5164 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:21.470295  4834 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:21.485682 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.486129 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.484560 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.519671 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.520215 28336 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.487605 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.503563 28331 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:21.503597 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.504354 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.431047  5043 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.527016 27584 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:22:21.523963 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:21.479897 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 253323us
I20260709 12:22:21.507246 27580 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:21.528026 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.508535 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.430896  4783 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.485224 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.528954 27582 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:21.529033 28329 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:21.523582 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.485167 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:21.530049 28330 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:21.513849 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.517180 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:21.478950  5356 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.514181 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.479528  5394 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:21.530719 27661 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:22:21.532688 27589 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:21.533190 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.534159 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.533977 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.535061 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.520955 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.537300 27588 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:22:21.539337 27588 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:22:21.539976 27656 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.541208 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.541914 27656 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.542727 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.543640 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.544394 27580 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.544701 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.545274 27729 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:21.545192 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.546830 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.547070 27588 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.14-28.14]
I20260709 12:22:21.500877  5381 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.469259  5285 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.512390  4592 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.537822 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.557626 27583 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:21.477715  5359 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.499672  5042 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:21.550148 27655 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:21.547528 27733 raft_consensus.cc:2376] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36. Re-sending same reply.
I20260709 12:22:21.552831 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.554044 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.555066 27588 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:22:21.572364 27736 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:21.546485 27584 pending_rounds.cc:85] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 21
I20260709 12:22:21.572827 27584 pending_rounds.cc:107] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 34.22
I20260709 12:22:21.573169 27584 raft_consensus.cc:2889] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:21.573609 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.547567  5356 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:21.556195 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.586498 27659 raft_consensus.cc:2368] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:21.557621 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.587337 27735 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:21.525238  5327 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.521678  5164 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:21.549526 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.590463 27664 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.18-32.18]
I20260709 12:22:21.561537 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.561689 27729 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:21.553005 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.562737 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.591982 27661 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:21.592710 27732 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:22:21.563032 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.593505 27734 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 34 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:22:21.530318  5265 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:22:21.604624 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55186.6us
I20260709 12:22:21.563123 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.605710 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.518152  5223 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.048s
I20260709 12:22:21.519505  4834 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:21.568398 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.555902 27580 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: [32.20-32.20]
W20260709 12:22:21.548144 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 311806us
I20260709 12:22:21.574182 27655 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:21.574348 27731 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:21.531809  5305 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:21.581614 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.545126  5183 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:21.626267 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.626971 27731 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:21.628247 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.551664  5405 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 12:22:21.596511 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.597769 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.598829 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:21.599673 27583 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:21.520480  5228 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.566980 27587 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:21.563468  5324 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:22:21.623134 27730 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:21.628867 27733 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:21.610198  5265 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.633061 27655 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:21.633623 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.610698  5335 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:21.649575 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.667137 27735 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:21.589471  5409 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:21.594573 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.682494 27661 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:21.643668 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.559175  5381 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.662071 27731 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:21.568005  4592 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.670857 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.671670 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:21.672328 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.697196 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.698009 27659 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:21.699585 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.700171 27659 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:22:21.690337 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 129896us
I20260709 12:22:21.646179  5405 consensus_queue.cc:1243] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 32.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:22:21.704011 27553 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:21.710837 27553 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:22:21.703823 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.652638  5324 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.707003 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.559026  5267 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.646008  5411 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:21.705116 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.714354 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:21.728329 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:21.728636 28328 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 19.
I20260709 12:22:21.611384  5327 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.713335 27731 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:21.727125 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.734071 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:21.743659 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.744454 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:21.711175  5335 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:21.729076 27627 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.712s (SENT)
I20260709 12:22:21.665829  5265 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:21.721756  5324 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:21.641582  5305 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.763692 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.769810 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.746146 27661 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:22:21.765022 27627 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.786s (SENT)
I20260709 12:22:21.725592 27733 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:21.717092  5219 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.641793  5394 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.21, Last appended by leader: 21, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.782912 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.802116 28330 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:21.804782  5306 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.784467 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.817121 27733 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:21.726693  5411 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:21.735463  5276 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.772574 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.799088 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.792869 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.831962 27734 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:21.833853 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.834486 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
W20260709 12:22:21.814889 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 172440us
I20260709 12:22:21.736690  5416 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:22:21.832615 27735 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:21.841034 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.820863  5292 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:21.703142  5409 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.851926 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.771919  4834 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.771100  5335 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.790450  5298 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.828397 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.827543  5374 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:21.736277  5397 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.847047 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.786774  4783 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:21.797451  5359 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:21.857352 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.918215 27655 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:21.828459  5411 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:21.853582 27733 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:21.827137  5280 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.080s
I20260709 12:22:21.865072 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:21.866199 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.932687 27663 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.867066 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.730666  5317 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:21.896864 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 192982us
I20260709 12:22:21.933667 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:21.899286 27661 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:21.934883 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.932029 27735 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:21.842584  5043 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:21.822151  5398 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:21.910811 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.853385 27731 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:21.924036 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:21.927815 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.863736 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.840117  5068 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.031s
W20260709 12:22:21.929296 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 131342us
I20260709 12:22:21.929237 27655 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:21.935391 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:21.935750 27661 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:21.905503 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:21.939918 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:21.941286 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.732231  5042 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:21.918646  5427 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.949010 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:21.951929 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.924644  5285 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:21.985906 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 52328.1us
I20260709 12:22:21.994558 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:21.986114 27587 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:21.949930 27582 raft_consensus.cc:2376] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36. Re-sending same reply.
W20260709 12:22:21.992486 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.918489  5405 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:21.950145 27663 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:21.840534  5276 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:21.933785  4834 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:21.951438 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46066.1us
I20260709 12:22:21.995304 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:22.045014 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:21.999361 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:22.046360 27735 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
W20260709 12:22:22.002630 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50792.8us
I20260709 12:22:22.004357 27658 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:22.005546 27582 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:22:21.900985  5298 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.056456 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.999680 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.045208 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:22.057858 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:21.989518 27656 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:21.994889  5043 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:21.945811  5399 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.049692 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:21.985450  5357 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.071895 27658 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:22:22.000103  5374 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:22:22.061192 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61595.6us
I20260709 12:22:21.996484  5228 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.066694 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.078856 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:21.998797  4783 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:22.047964 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:22.006654  5292 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:22.090556 27661 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:22.085693 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.001618  4592 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.082608 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.045117  5315 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:22.078235 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.059391  5398 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:22.096676 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.062505  5386 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:22.094204 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.948873  5426 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.144s
I20260709 12:22:22.095031 28330 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:22.103883 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:22.104724 27587 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:22.095062 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:22.101524 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:22.106029 27735 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:22.081518  5374 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.108865 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:22.116868 27658 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: [34.21-36.22]
I20260709 12:22:21.999500  5427 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.045791  5405 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.090502  5327 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.093880  5306 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:22:22.121696 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 124222us
I20260709 12:22:22.129904 27661 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:22.129766 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:22.097738  5431 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:22.102737  5285 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:22.143543 27587 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:22:22.118273  5374 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:22.079895  5429 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:22:22.119835  5292 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:22.162477 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:22.084358  5357 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:22.174562 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.170897 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65966.5us
I20260709 12:22:22.081871  5399 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.105707  5315 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.118072  5228 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:22:22.129608  5426 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:22:22.115000  5386 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.098351  4783 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:22.403416 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.353065 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.210144 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:22.363497 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:22.378139 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:22.138835  5438 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:22.409711 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:22.203043 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:22.410594 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 20.
I20260709 12:22:22.416213 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:22.204559 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:22.409323 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 199246us
I20260709 12:22:22.416396 27580 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:22:22.148061  5285 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.203540 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:22.163473  5412 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.312s	user 0.001s	sys 0.000s
I20260709 12:22:22.161963  5398 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:22:22.364796  5326 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.005s	sys 0.003s
W20260709 12:22:22.223948  5421 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:22:22.427937 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:22.429805 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 370485us
I20260709 12:22:22.426523 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:22.165185  5244 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:22.201933  5436 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:22.161176  5381 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:22.361899  5441 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 12:22:22.432474 27582 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 34 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:22:22.417608 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:22.418310 28328 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:22.432494 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:22.487756 28328 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:22.364006  5267 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:22.144457  5434 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:22:22.435778  5326 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.007s	sys 0.004s
I20260709 12:22:22.202430  5298 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.163604  5431 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:22.212706  5335 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.209558  5317 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.419783  5444 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.417593 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:22.424260  5424 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:22.209872  5054 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.226s
I20260709 12:22:22.459159 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:22.493156 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:22.423069  5285 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:22.487470 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.172812  5327 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.302s
I20260709 12:22:22.439962  5244 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:22.524103 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.511706 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 445036us
W20260709 12:22:22.559738 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72383us
W20260709 12:22:22.561182 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.210062  5292 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.537242 28328 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.552132 27583 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:22:22.507802  5421 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:22:22.536724 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:22.498514  5442 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.080s	user 0.000s	sys 0.008s
W20260709 12:22:22.568105 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 215849us
I20260709 12:22:22.565737 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:22.566651 27580 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:22.565747 28328 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:22.512360  5434 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:22.573318 27588 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:22.571098 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.510505  5398 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:22.556353  5424 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:22.564275  5244 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:22.589916 27735 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:22.507292  5381 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.063s
I20260709 12:22:22.582863 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:22:22.586328 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3492) took 1035 ms. Trace:
W20260709 12:22:22.595893 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 169459us
I20260709 12:22:22.568203  5054 consensus_queue.cc:1243] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 30.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
W20260709 12:22:22.597188 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 178406us
I20260709 12:22:22.574371  5327 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:22:22.564597  5445 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:22.598963 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.603315 27656 rpcz_store.cc:276] 0709 12:22:21.551301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.552698 (+  1397us) service_pool.cc:224] Handling call
0709 12:22:22.586182 (+1033484us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:22.586274 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":901554560}
I20260709 12:22:22.606401 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:22.614850 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:22.600423 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.608119 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:22.615588 27735 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:22.616019 27656 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.14-28.14]
I20260709 12:22:22.617245 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:22.596174  5424 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:22.618763 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:22.620189 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:22.621011 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.612789 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:22.622524 27661 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:22.623986  5452 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:22.624495  5452 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.624832  5452 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:22.625219 27656 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:22:22.626281 27656 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:22.610886 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:22.631718 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.632916 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.633248  5276 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.633852  5335 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:22.633913  5298 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:22.634646  5335 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:22.635491  5446 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:22.635797  5298 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:22.637192 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3472) took 3511 ms. Trace:
I20260709 12:22:22.637573 27662 rpcz_store.cc:276] 0709 12:22:19.125934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.777981 (+652047us) service_pool.cc:224] Handling call
0709 12:22:19.778306 (+   325us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:19.818692 (+ 40386us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:21.271247 (+1452555us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b1800009340. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:22:21.513171 (+241924us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.949370 (+436199us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:21.994061 (+ 44691us) log.cc:844] Serialized 52 byte log entry
0709 12:22:21.994222 (+   161us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:22.006307 (+ 12085us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.044110 (+ 37803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.044208 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.636980 (+592772us) raft_consensus.cc:1685] finished
0709 12:22:22.637003 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.637137 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76794,"prepare.queue_time_us":164950,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":83,"raft.queue_time_us":140999,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":20090,"spinlock_wait_cycles":131586944,"thread_start_us":613683,"threads_started":3,"wal-append.queue_time_us":310969}
W20260709 12:22:22.639042  5324 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.854s	user 0.008s	sys 0.000s
I20260709 12:22:22.563975  5444 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.643498  5438 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:22.644086  5317 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:22.661208 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:22.663035 28328 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:22:22.657567 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:22.644804  5417 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.648s	user 0.004s	sys 0.000s
W20260709 12:22:22.645071  5448 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:22:22.678632  5438 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:22:22.645563  5439 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:22:22.679595  5438 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:22.639732  5386 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.681016  5438 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:22.682301  5415 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.870s	user 0.003s	sys 0.003s
I20260709 12:22:22.645820 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:22.645922  5263 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.194s	user 0.001s	sys 0.009s
W20260709 12:22:22.646008  5440 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.392s	user 0.001s	sys 0.000s
I20260709 12:22:22.654980 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:22.655346 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.612494  5223 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:22.641727  5434 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.642035  5398 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.635247  5335 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:22:22.639189  5363 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.151s	user 0.009s	sys 0.000s
I20260709 12:22:22.685911 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:22.686810 27734 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:22:22.653584  5326 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:22.613899  5405 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.664841 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:22.669222  5357 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:22.627249  5427 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:22.627562  5054 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:22.614717  5445 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:22.642333  5356 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.032s	user 0.009s	sys 0.000s
W20260709 12:22:22.676213  5443 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.274s	user 0.001s	sys 0.001s
I20260709 12:22:22.618359  5327 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.683871  5263 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:22.672654  5265 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.889s	user 0.011s	sys 0.000s
I20260709 12:22:22.659418  5324 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.718508  5447 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:22.667872 28336 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:22:22.719621 28328 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:22.679728  5219 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:22.656517  5436 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:22.657491  5149 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:22.687664  5223 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.773869  5007 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:22.774232  5223 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:22.774313  5007 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:22.775276  5007 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.776301  5007 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:22.777932  5324 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.624035  5431 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.769322  5265 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.692045 28329 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:22.753171  4834 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.696s	user 0.012s	sys 0.000s
I20260709 12:22:22.690155  5363 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.769785  5219 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:22.780766 28334 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:22.781353  5279 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.781800  5363 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:22.772979  5164 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.146s	user 0.007s	sys 0.007s
I20260709 12:22:22.775879  5445 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.781910  5279 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.782883  5445 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:22.782667  5164 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.747488  5002 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:22.781440  5219 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.751621  5327 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:22.718817  5326 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:22.787609  5444 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.788130  5164 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.782815  5436 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.721887  5210 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:22:22.780690  5265 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:22.783612  5447 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:22.802539  5285 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.802932  5447 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:22.803947  5323 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.359s	user 0.010s	sys 0.002s
I20260709 12:22:22.804383  5323 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:22:22.805611  5411 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.871s	user 0.008s	sys 0.001s
I20260709 12:22:22.809505  5315 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.748483  5356 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.814663  5356 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:22.812942  5043 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.730s	user 0.007s	sys 0.005s
I20260709 12:22:22.815533  5043 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:22:22.817121  5374 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.639s	user 0.006s	sys 0.007s
W20260709 12:22:22.815101  5201 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.393s	user 0.010s	sys 0.005s
I20260709 12:22:22.794104  5326 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:22.817759  5374 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.826370  5201 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.806015  5411 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.797940  5263 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.812140  5219 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:22.831185  5374 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.828804  5394 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:22.829943  5409 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.838302  5206 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:22.838798  5206 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:22.784693  5452 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.840746  5327 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.848645  5411 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:22.850131  5452 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.887938 27587 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:22.888594 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4399) took 1372 ms. Trace:
I20260709 12:22:22.891481 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:22.952658 28334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1478) took 2768 ms (client timeout 2050 ms). Trace:
I20260709 12:22:22.892925 27659 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:22.953440 28334 rpcz_store.cc:269] 0709 12:22:20.183752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.184661 (+   909us) service_pool.cc:224] Handling call
0709 12:22:22.951723 (+2767062us) spinlock_profiling.cc:243] Waited 2.5 s on lock 0x7b5400491bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:22.952037 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5743801728}
I20260709 12:22:22.894791 27661 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:22.896350 27732 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:22.744542 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:22.958274 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:22.706944 27735 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:22.952104 27584 rpcz_store.cc:276] 0709 12:22:21.515785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.545953 (+ 30168us) service_pool.cc:224] Handling call
0709 12:22:21.546270 (+   317us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:21.865344 (+319074us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.866110 (+   766us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:21.869695 (+  3585us) log.cc:844] Serialized 78 byte log entry
0709 12:22:22.101184 (+231489us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:22.101238 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.102043 (+   805us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.887785 (+785742us) raft_consensus.cc:1685] finished
0709 12:22:22.887850 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.888285 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":145493,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6989,"thread_start_us":419766,"threads_started":2,"wal-append.queue_time_us":262887}
I20260709 12:22:22.960088 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3634) took 1254 ms. Trace:
I20260709 12:22:22.960772 27735 rpcz_store.cc:276] 0709 12:22:21.705706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.647675 (+941969us) service_pool.cc:224] Handling call
0709 12:22:22.959818 (+312143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:22.943043 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:22.962026 27663 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:22.896641 27730 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:22.967741 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3641) took 1248 ms. Trace:
I20260709 12:22:22.968142 27730 rpcz_store.cc:276] 0709 12:22:21.722134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.736804 (+ 14670us) service_pool.cc:224] Handling call
0709 12:22:22.967441 (+1230637us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:22.967660 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2450514304}
I20260709 12:22:22.969762 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:22.898113 27733 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:22.786451  5002 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:22.971550 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.973376 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.973096  5002 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.973938 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:22.899946 27586 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:22.902346  5201 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.889801 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:22.975940  5235 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:22.976217  5356 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:22.976545  5235 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:22.903440 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:22.904881  5223 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:22.977186  5235 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:22.977548 28331 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:22.978040 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.904431 27662 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:22.979755  5223 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:22.909488 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:22.980922 28330 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:22.911199 28335 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:22.913228  5244 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.988727  5434 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.990036  5244 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.917126  5285 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.920291 27738 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:22:22.993934 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4087) took 1449 ms. Trace:
W20260709 12:22:22.994187 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.994357 27738 rpcz_store.cc:276] 0709 12:22:21.544110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.544235 (+   125us) service_pool.cc:224] Handling call
0709 12:22:21.544536 (+   301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:22.991819 (+1447283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.993527 (+  1708us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:22.993824 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3113642240}
W20260709 12:22:22.894977 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 261962us
I20260709 12:22:22.912834  5206 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.998711 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:22.924389  5430 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:22.932642 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3482) took 1501 ms. Trace:
W20260709 12:22:22.935732 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.936509 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3480) took 1538 ms. Trace:
I20260709 12:22:22.833354  5424 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.939816 27660 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:22.962759 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:22.718874 27736 raft_consensus.cc:2368] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:22.963621 27734 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:23.018082 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1609) took 1320 ms. Trace:
I20260709 12:22:23.018573 27734 rpcz_store.cc:276] 0709 12:22:21.697485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.957417 (+1259932us) service_pool.cc:224] Handling call
0709 12:22:23.017963 (+ 60546us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.020025 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:23.021147 27734 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:23.021682 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4095) took 1441 ms. Trace:
I20260709 12:22:23.022066 27734 rpcz_store.cc:276] 0709 12:22:21.579996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.019598 (+1439602us) service_pool.cc:224] Handling call
0709 12:22:23.021597 (+  1999us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:22.953246 27656 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:23.023775  5424 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:22.976770  5356 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.908102  4834 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.012439 27664 rpcz_store.cc:276] 0709 12:22:21.398172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.398323 (+   151us) service_pool.cc:224] Handling call
0709 12:22:21.436185 (+ 37862us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.590920 (+154735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.935784 (+1344864us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540022a454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:22.936261 (+   477us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":354193664}
I20260709 12:22:23.029167  5356 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:23.012102 27657 rpcz_store.cc:276] 0709 12:22:21.431549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.432163 (+   614us) service_pool.cc:224] Handling call
0709 12:22:22.932120 (+1499957us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400210f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:22.932358 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284916608}
I20260709 12:22:23.029595  4834 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.033279 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:23.034168 28334 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:23.036541 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:23.037346 28328 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:22:23.042228  5235 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.044792 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1735) took 1136 ms. Trace:
I20260709 12:22:23.045302 27658 rpcz_store.cc:276] 0709 12:22:21.908019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.073094 (+165075us) service_pool.cc:224] Handling call
0709 12:22:22.116599 (+ 43505us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:22.117314 (+   715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.044295 (+926981us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b54001e3cd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.044572 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98994048}
I20260709 12:22:23.046154  5235 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.049731 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:22:23.056038 28333 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
W20260709 12:22:22.984251 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1455) took 1490 ms (client timeout 1839 ms). Trace:
I20260709 12:22:22.975122 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:22.989677  5434 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:22.998601  5043 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:22:23.001288  5305 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.002223  5399 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:22.918087  5306 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:23.004679  4783 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:23.071450 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:22.993963  5327 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:23.009629  5279 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:23.072532 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74056.6us
I20260709 12:22:23.010110 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.072921 27664 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:23.011584 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.073560 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:23.074379 27655 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:23.015568  5223 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.016712 27735 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 36 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:22:23.075853 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1617) took 1174 ms. Trace:
I20260709 12:22:23.075744  5223 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.005745  5183 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:23.076475 27735 rpcz_store.cc:276] 0709 12:22:21.901883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.962182 (+1060299us) service_pool.cc:224] Handling call
0709 12:22:23.075757 (+113575us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:23.059304 28335 rpcz_store.cc:269] 0709 12:22:21.493327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.493512 (+   185us) service_pool.cc:224] Handling call
0709 12:22:22.983988 (+1490476us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54003705d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:22.984162 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3179229952}
I20260709 12:22:23.066485  5306 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:23.078665 27659 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:23.079213 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:23.080456  5327 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:22:23.080271  5306 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:23.081182 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106373us
I20260709 12:22:23.080925  5326 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.081859  5279 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:23.082659  5279 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:23.076052  5398 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:23.083504 27730 raft_consensus.cc:2376] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34. Re-sending same reply.
I20260709 12:22:23.084236  5306 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:23.081202  5327 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:23.084651  5223 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:23.085691  5219 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:23.085945  5327 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.087481 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3653) took 1265 ms. Trace:
I20260709 12:22:23.088048  5434 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:23.087888 27732 rpcz_store.cc:276] 0709 12:22:21.821650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.585746 (+764096us) service_pool.cc:224] Handling call
0709 12:22:23.087241 (+501495us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.087394 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":644191488}
I20260709 12:22:23.066920 28331 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:23.084261  5279 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.090111  4834 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:23.090713 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4359) took 2030 ms (client timeout 2177 ms). Trace:
W20260709 12:22:23.091208 27585 rpcz_store.cc:269] 0709 12:22:21.060389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.476871 (+416482us) service_pool.cc:224] Handling call
0709 12:22:23.090498 (+1613627us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.090620 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2595932416}
W20260709 12:22:23.093338 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.093425 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.098940 27664 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:23.099478 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.100378 27655 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:22:23.101454  5326 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:23.094056 27586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4368) took 2031 ms (client timeout 2388 ms). Trace:
I20260709 12:22:23.103832 28334 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
W20260709 12:22:23.103665 27586 rpcz_store.cc:269] 0709 12:22:21.063060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.485470 (+422410us) service_pool.cc:224] Handling call
0709 12:22:23.093803 (+1608333us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.093994 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3172095872}
I20260709 12:22:23.097445  5179 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:23.105222 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.105263  5179 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:23.106148 27587 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:23.107570  5319 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:23.107997  5319 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:23.108666 27581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4386) took 2396 ms (client timeout 1644 ms). Trace:
W20260709 12:22:23.109107 27581 rpcz_store.cc:269] 0709 12:22:20.712469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.712592 (+   123us) service_pool.cc:224] Handling call
0709 12:22:23.108287 (+2395695us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.108459 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2783725952}
I20260709 12:22:23.109205  5319 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.107095  5179 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.112521 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.112872  5179 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:23.114017 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.114466 27662 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:23.114730 27657 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:23.115829 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.116235 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:23.116819 27662 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:23.117659 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:23.110862 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.119143 27585 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 35 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:22:23.120608  4592 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:23.121096  4592 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:23.119834 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.122220 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.122682 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.122346  4592 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.063472  5002 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:23.115854 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.123360 27662 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:22:23.119208 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:23.126932 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1614) took 1285 ms (client timeout 1524 ms). Trace:
I20260709 12:22:23.146799 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4097) took 1241 ms. Trace:
I20260709 12:22:23.123178 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:23.148211 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3638) took 1436 ms. Trace:
I20260709 12:22:23.148741 27734 rpcz_store.cc:276] 0709 12:22:21.706766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.118865 (+1412099us) service_pool.cc:224] Handling call
0709 12:22:23.143193 (+ 24328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.147446 27730 rpcz_store.cc:276] 0709 12:22:21.905081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.969261 (+1064180us) service_pool.cc:224] Handling call
0709 12:22:23.146545 (+177284us) spinlock_profiling.cc:243] Waited 69.2 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.146732 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159239424}
I20260709 12:22:23.145529 27657 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:23.145094 27733 rpcz_store.cc:269] 0709 12:22:21.841215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.927556 (+ 86341us) service_pool.cc:224] Handling call
0709 12:22:23.124136 (+1196580us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b54002e9ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.126877 (+  2741us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2192919424}
I20260709 12:22:23.142203  5319 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:23.172530 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:23.175069 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.188782 27585 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:23.176864 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:23.152189  5306 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:23.179169 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.172361  4592 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:23.213529 28328 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 21.
W20260709 12:22:23.217248 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99774.8us
I20260709 12:22:23.214425 27734 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:23.224473 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:23.226046 27737 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:23.228335 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3643) took 1495 ms. Trace:
I20260709 12:22:23.228698 27737 rpcz_store.cc:276] 0709 12:22:21.732646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.122726 (+1390080us) service_pool.cc:224] Handling call
0709 12:22:23.228264 (+105538us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.233856  5357 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.234124 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:23.234709 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.235071 27731 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:23.234579  5357 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.235745 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3650) took 1434 ms. Trace:
I20260709 12:22:23.236315 27731 rpcz_store.cc:276] 0709 12:22:21.801558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.171777 (+1370219us) service_pool.cc:224] Handling call
0709 12:22:23.235664 (+ 63887us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.239056 27656 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:22:23.241223 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.242162 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.243172 27580 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:23.243744 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.244251 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.244735 27587 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:22:23.244988 27580 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:23.245643 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.230489  4830 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.232101  5326 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:23.231496 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:23.245886 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.280467 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:23.247689 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107750us
W20260709 12:22:23.259724 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184527us
W20260709 12:22:23.273221 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.242210 27586 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:23.282698 27585 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:23.267660 27734 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:23.247074 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.269766 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1607) took 1585 ms. Trace:
I20260709 12:22:23.289463 27662 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:23.247575 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:23.289358 27733 rpcz_store.cc:276] 0709 12:22:21.684634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.172861 (+1488227us) service_pool.cc:224] Handling call
0709 12:22:23.231036 (+ 58175us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.269341 (+ 38305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.269377 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.269442 (+    65us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:23.269490 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.269554 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.269686 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.294477 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1608) took 1609 ms. Trace:
I20260709 12:22:23.294942 27734 rpcz_store.cc:276] 0709 12:22:21.684941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.227140 (+1542199us) service_pool.cc:224] Handling call
0709 12:22:23.267391 (+ 40251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.294180 (+ 26789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.294415 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.300491 27657 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:23.246881  4830 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.305804  5405 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:22:23.306999  5405 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:23.307489  5405 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:23.308164  4830 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.308651  5405 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.310906  5356 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.311553 27584 raft_consensus.cc:2376] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33. Re-sending same reply.
I20260709 12:22:23.313501  5356 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.315131 27737 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:22:23.315989 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3632) took 1611 ms. Trace:
I20260709 12:22:23.316350 27737 rpcz_store.cc:276] 0709 12:22:21.705071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.314182 (+1609111us) service_pool.cc:224] Handling call
0709 12:22:23.314716 (+   534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.315675 (+   959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.315903 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.248406 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:23.269007 27588 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.695s	user 0.010s	sys 0.011s
I20260709 12:22:23.270604 27730 raft_consensus.cc:606] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:23.322332 27730 raft_consensus.cc:614] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:23.326767 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.327680 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:23.248265  5397 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:22:23.326248 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 145798us
I20260709 12:22:23.329411 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 54) took 1472 ms. Trace:
I20260709 12:22:23.331732 27730 rpcz_store.cc:276] 0709 12:22:21.850033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.171807 (+1321774us) service_pool.cc:224] Handling call
0709 12:22:23.322773 (+150966us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.334547 27581 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28.
W20260709 12:22:23.338593 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.332683 27580 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:23.320447  5424 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:23.324100  5405 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:23.465286 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 143603us
I20260709 12:22:23.256152  5326 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.339476 27588 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:23.331092  5397 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.486099 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:23.492051 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4407) took 1082 ms. Trace:
I20260709 12:22:23.492748 27588 rpcz_store.cc:276] 0709 12:22:22.409478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.565355 (+155877us) service_pool.cc:224] Handling call
0709 12:22:23.488680 (+923325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.495676  5424 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:23.497071 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 223270us
I20260709 12:22:23.498494  5424 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.499554  5306 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:23.500077  5306 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.500507  5306 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:23.505220 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:23.508073  5416 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:23.508754  5416 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:23.509864  5416 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.493628  5397 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:23.465293  5356 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:22:23.515554  4834 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:23.516099 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 347491us
I20260709 12:22:23.516397  4834 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.518388 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:23.516397 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 183155us
W20260709 12:22:23.526193 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.530916  5434 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:23.531615  5434 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:23.532752  5434 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:23.533774 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.536341 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 194711us
I20260709 12:22:23.538144  5306 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.519150  4834 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:23.540668 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:23.543473 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 82596.3us
I20260709 12:22:23.544555 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:23.544827  5306 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:23.545470  5356 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:23.546029  5356 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:23.546499  5356 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:23.546743 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:23.561455 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.562381 27662 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:23.588961 27580 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:23.591717 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.594720 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.555310  5424 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:23.600979 27663 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:23.602236 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1739) took 1486 ms. Trace:
I20260709 12:22:23.603466  5409 raft_consensus.cc:2764] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:22:23.603691  5411 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:22:23.604235 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.607476  5239 raft_consensus.cc:2764] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:22:23.610867 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.602666 27661 rpcz_store.cc:276] 0709 12:22:22.115452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.623206 (+507754us) service_pool.cc:224] Handling call
0709 12:22:22.658520 (+ 35314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.955567 (+297047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.590523 (+634956us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:23.602175 (+ 11652us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":540804992}
I20260709 12:22:23.619737 27658 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:22:23.616128 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:23.627194  5179 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.004s
I20260709 12:22:23.630460 27730 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:23.631542 27655 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:23.633715 27656 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 35 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:22:23.642237 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3640) took 1920 ms. Trace:
I20260709 12:22:23.642753 27730 rpcz_store.cc:276] 0709 12:22:21.714216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.489986 (+1775770us) service_pool.cc:224] Handling call
0709 12:22:23.611947 (+121961us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.634835 (+ 22888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.635019 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.648021 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:23.650137  5239 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:23.650823  5239 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:23.668840 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:23.616595  5285 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.045s
W20260709 12:22:23.674206 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 133707us
I20260709 12:22:23.676353 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:23.677193  5476 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:22:23.686729 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:23.690205 27730 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:23.703229 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1611) took 1979 ms. Trace:
I20260709 12:22:23.703761 27730 rpcz_store.cc:276] 0709 12:22:21.718221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.664177 (+1945956us) service_pool.cc:224] Handling call
0709 12:22:23.690018 (+ 25841us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.697312 (+  7294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.697610 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.711889  5394 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:22:23.728619 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:23.737334 27662 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:23.734401 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:23.749221 27583 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 17
I20260709 12:22:23.749704 27583 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 32.18
I20260709 12:22:23.750213 27583 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:23.760114 27582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4386) took 1688 ms (client timeout 1712 ms). Trace:
W20260709 12:22:23.760603 27582 rpcz_store.cc:269] 0709 12:22:22.071948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.072826 (+   878us) service_pool.cc:224] Handling call
0709 12:22:23.742157 (+1669331us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54000bd1d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.759922 (+ 17765us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":751673088}
I20260709 12:22:23.759601 27730 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:23.762951 27738 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:22:23.763671 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1612) took 2029 ms. Trace:
I20260709 12:22:23.763988 27738 rpcz_store.cc:276] 0709 12:22:21.734776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.695459 (+1960683us) service_pool.cc:224] Handling call
0709 12:22:23.731968 (+ 36509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.763425 (+ 31457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.763620 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.764272 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1613) took 2018 ms. Trace:
I20260709 12:22:23.769799  4931 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.770679  4931 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.769466 27730 rpcz_store.cc:276] 0709 12:22:21.830223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.705257 (+1875034us) service_pool.cc:224] Handling call
0709 12:22:23.742859 (+ 37602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.764086 (+ 21227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.764233 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.774087  5424 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:23.774883  5424 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.722012  5239 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.780263 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:23.783759  5285 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.784492  5285 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.791488  5179 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.792224  5179 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.794983  5179 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:23.787122  5285 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:23.808557  5481 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:22:23.810473 27734 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:22:23.810539 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 125477us
I20260709 12:22:23.747642  5279 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.056s
I20260709 12:22:23.777666  5424 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.812927 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1610) took 2105 ms. Trace:
I20260709 12:22:23.801326  5306 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.816951  5306 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.811406 27731 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.18-32.18]
I20260709 12:22:23.768936  5416 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.801527  5305 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:23.815253 27734 rpcz_store.cc:276] 0709 12:22:21.707314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.740820 (+2033506us) service_pool.cc:224] Handling call
0709 12:22:23.767591 (+ 26771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.811035 (+ 43444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.812823 (+  1788us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.814400 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:23.821693 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:23.822923 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4101) took 1235 ms. Trace:
I20260709 12:22:23.823302 27732 rpcz_store.cc:276] 0709 12:22:22.587088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.761725 (+1174637us) service_pool.cc:224] Handling call
0709 12:22:23.822845 (+ 61120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.823623 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3633) took 2118 ms. Trace:
I20260709 12:22:23.825026 27731 rpcz_store.cc:276] 0709 12:22:21.705404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.741354 (+2035950us) service_pool.cc:224] Handling call
0709 12:22:23.801821 (+ 60467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.820636 (+ 18815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.823557 (+  2921us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.818869  5356 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:23.830350 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 111011us
W20260709 12:22:23.838372 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3646) took 2071 ms (client timeout 1988 ms). Trace:
W20260709 12:22:23.838868 27736 rpcz_store.cc:269] 0709 12:22:21.767121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.645549 (+878428us) service_pool.cc:224] Handling call
0709 12:22:23.837374 (+1191825us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b54000a6ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.838150 (+   776us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80602496}
I20260709 12:22:23.848758 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4103) took 1261 ms. Trace:
I20260709 12:22:23.849365 27731 rpcz_store.cc:276] 0709 12:22:22.587702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.829062 (+1241360us) service_pool.cc:224] Handling call
0709 12:22:23.839380 (+ 10318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.848264 (+  8884us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.848319 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.848382 (+    63us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:23.848435 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.848518 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.848684 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.833758  5416 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.855399  5485 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:23.856289  5485 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:23.857647  5485 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.863467 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:23.866864 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:23.867614 27738 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:23.869341 27734 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:22:23.846283  5279 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:23.869022  5306 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:23.875983  5279 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.876400  5279 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:23.878132  5356 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:23.887920  5179 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.888530  5179 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.889950 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:23.892587 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71982.2us
W20260709 12:22:23.893405 27627 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.686s (SENT)
I20260709 12:22:23.893550 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:23.893997 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:23.894534 28334 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:23.895437 28337 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:22:23.895867 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:23.896807 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:23.896989  5434 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:23.897316 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:23.898234  5306 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:23.898890  5306 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:23.898095 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:23.899964 28328 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:23.901103 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:23.905329  5306 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.905836  5434 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:23.906941  5434 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:22:23.907316  5434 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:23.908330  5434 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.909675  5434 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:23.891027  5179 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:23.920511 27737 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:23.921489 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3636) took 2215 ms. Trace:
I20260709 12:22:23.921895 27737 rpcz_store.cc:276] 0709 12:22:21.706262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.919695 (+2213433us) service_pool.cc:224] Handling call
0709 12:22:23.920156 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.921112 (+   956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.921390 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.924163 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4232) took 2073 ms. Trace:
I20260709 12:22:23.924715 27659 rpcz_store.cc:276] 0709 12:22:21.851072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.851674 (+   602us) service_pool.cc:224] Handling call
0709 12:22:23.923868 (+2072194us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400211e54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.924084 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2391241088}
I20260709 12:22:23.928501 27730 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:22:23.872282 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3654) took 2037 ms. Trace:
I20260709 12:22:23.931823 27738 rpcz_store.cc:276] 0709 12:22:21.834897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.812112 (+1977215us) service_pool.cc:224] Handling call
0709 12:22:23.833236 (+ 21124us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.871955 (+ 38719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.872206 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.875166  5305 raft_consensus.cc:2764] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:22:23.935768 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.937539 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.938166 27658 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:23.939826 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.940457 27659 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:23.940951 27659 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:22:23.941056 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3637) took 2234 ms. Trace:
I20260709 12:22:23.941886 27733 rpcz_store.cc:276] 0709 12:22:21.706505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.939790 (+2233285us) service_pool.cc:224] Handling call
0709 12:22:23.940259 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.940642 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.940681 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.940724 (+    43us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:23.940773 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.940854 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.940987 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.943889 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:23.944561 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.945437 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.946480 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:23.946806 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:23.947525 27659 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:23.947782 27735 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:22:23.948426 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4100) took 1527 ms. Trace:
I20260709 12:22:23.949045 27735 rpcz_store.cc:276] 0709 12:22:22.420761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.946103 (+1525342us) service_pool.cc:224] Handling call
0709 12:22:23.948349 (+  2246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.954062 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1776) took 2668 ms. Trace:
I20260709 12:22:23.955847  5434 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:23.956348 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:23.956979  5434 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:23.957662  5434 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:23.957810  5285 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:23.958407  5285 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:23.958915  5434 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.959409  5285 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:23.961113  5285 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:23.964027 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.963882 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 785398us
I20260709 12:22:23.964635  5425 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:23.964782 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:23.965298  5425 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:23.965451  5357 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.965785 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:23.966598  5425 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.971603  5444 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.972450  5444 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:23.974224 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.974921 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.967927  5423 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:23.968694  5324 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:22:23.968963 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.943940 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4230) took 2105 ms. Trace:
I20260709 12:22:23.970306  5179 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:24.001020  5179 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:24.002681  5179 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:24.005434  5279 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.007036 28336 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:22:24.008934  5279 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:23.871240 27731 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
W20260709 12:22:23.978503 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.980804 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4250) took 1095 ms. Trace:
I20260709 12:22:24.044705 27663 rpcz_store.cc:276] 0709 12:22:22.885107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.965510 (+ 80403us) service_pool.cc:224] Handling call
0709 12:22:23.595731 (+630221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.614963 (+ 19232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.980580 (+365617us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54001e41d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.980728 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1445795200}
W20260709 12:22:23.983705 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 465421us
I20260709 12:22:24.048254 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:24.050133 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.052042 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 185320us
I20260709 12:22:24.053416  5318 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:24.054347  5305 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:22:23.982693 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.055547 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.056216 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.991390 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:23.980266  5324 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:24.056800 28329 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:24.057781 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.057574  5324 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:23.992345 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:23.979282  5423 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:23.976411  5374 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.059692  5374 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.059968  5423 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.061705 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:24.061823 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.063102 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.874837 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3655) took 2024 ms. Trace:
I20260709 12:22:24.065210 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:23.995249 28333 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:23.997035 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:24.065768 27734 rpcz_store.cc:276] 0709 12:22:21.850625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.829776 (+1979151us) service_pool.cc:224] Handling call
0709 12:22:23.846674 (+ 16898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.872759 (+ 26085us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.874760 (+  2001us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.954676 27589 rpcz_store.cc:276] 0709 12:22:21.285156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.536907 (+251751us) service_pool.cc:224] Handling call
0709 12:22:21.537128 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.537305 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:22:23.033469 (+1496164us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b18000d8520. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:22:23.033723 (+   254us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.952938 (+919215us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b18000d8520. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:23.953308 (+   370us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.953405 (+    97us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:22:23.953479 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.953583 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.953800 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19979,"prepare.queue_time_us":114006,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":17478,"raft.queue_time_us":738315,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"spinlock_wait_cycles":47528064,"thread_start_us":296390,"threads_started":2,"wal-append.queue_time_us":188972}
I20260709 12:22:23.996168 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:23.998294 28328 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 20. Current term is 21. Ops: [20.17-20.17]
I20260709 12:22:24.067484 28331 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28.
I20260709 12:22:23.999219 27736 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:23.957967  4931 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:24.077812 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3660) took 1134 ms. Trace:
I20260709 12:22:24.078243 27736 rpcz_store.cc:276] 0709 12:22:22.943018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.984820 (+1041802us) service_pool.cc:224] Handling call
0709 12:22:23.998982 (+ 14162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.068391 (+ 69409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.077757 (+  9366us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.079245 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3659) took 1137 ms. Trace:
W20260709 12:22:24.078647 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 252325us
I20260709 12:22:24.023447  5206 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:24.025693 27659 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:24.027572 27658 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:22:23.879041  5356 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.149s
I20260709 12:22:23.969202  5326 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:24.028849 27730 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:24.028885 27656 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:22:24.085877 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:24.032883 27662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:23.869860  5416 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:24.039759 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.055433  5305 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:24.058477 27582 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:23.975301  5444 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:23.984089 27660 rpcz_store.cc:276] 0709 12:22:21.838644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.838813 (+   169us) service_pool.cc:224] Handling call
0709 12:22:22.938890 (+1100077us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.013089 (+ 74199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.943627 (+930538us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540022a450. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:23.943839 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2469442432}
I20260709 12:22:24.075645 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:24.079916 27731 rpcz_store.cc:276] 0709 12:22:22.942062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.850623 (+908561us) service_pool.cc:224] Handling call
0709 12:22:23.851056 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.078740 (+227684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.078978 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.015894 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:24.099220 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 83384.7us
I20260709 12:22:24.089015 27738 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:23.970187  5462 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:24.092797 27580 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:24.094908 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 314715us
I20260709 12:22:24.092761 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.105994 27736 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:22:24.108050 27585 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:24.108476 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.109413 27735 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 35 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:22:24.111639 28337 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 28: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:24.108268 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:24.112272 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:24.096681  5397 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.582s	user 0.013s	sys 0.000s
I20260709 12:22:24.117362 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.117765 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.118512 27587 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:24.086992  5424 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.125761 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.117659 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1572) took 1002 ms. Trace:
I20260709 12:22:24.087231 27662 raft_consensus.cc:606] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:24.126801 28336 rpcz_store.cc:276] 0709 12:22:23.105687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.891962 (+786275us) service_pool.cc:224] Handling call
0709 12:22:23.892399 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.007495 (+115096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.098415 (+ 90920us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54005d93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:24.108350 (+  9935us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":262834944}
W20260709 12:22:24.137198 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46055.6us
I20260709 12:22:24.132203 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.141388 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:24.142222 27738 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:24.143221 27701 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:22:24.143591 27701 leader_election.cc:403] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:24.143971 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:24.144098 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44021us
I20260709 12:22:24.145283 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:24.146559 27582 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:22:24.148082 27733 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:24.148670 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:24.109652  5416 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:24.111455  5326 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.150866 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.156373 27589 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:24.155191 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:24.158676 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:24.159660 27730 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:24.159582 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111468us
I20260709 12:22:24.163223 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.165881 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:24.147699  5409 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:24.155382  5405 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:24.176501 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:24.155150  4931 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:22:24.180059 28333 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:24.151641  5416 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.191803 27734 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:24.192251 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:24.172475 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:24.132886  5397 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:24.209681 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.148914  5424 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:24.211527 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:24.240656 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.251449 27731 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:22:24.253358 27580 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:24.255533 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.186357  5405 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.258467 27586 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:24.258675  5405 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:24.177943  5285 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.077s
I20260709 12:22:24.259516 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:24.241781 27736 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:22:24.260134 27586 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:24.256942 27581 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:24.212404  5416 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:24.257850 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46427.5us
I20260709 12:22:24.277570 27733 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:24.165374  5356 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.099s
I20260709 12:22:24.313647 27586 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:24.311551 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.215212  5462 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.314366 27589 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:24.312423 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.315538 27736 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:24.313311 27729 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:24.313048 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52438us
I20260709 12:22:24.211002  5409 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:24.311456 27704 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:24.328284 27585 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:24.329492 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:24.330868 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.178299  5326 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:24.317612 27582 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:24.315276 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:24.348564 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.311846  5239 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:24.355963  5450 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.356609  5450 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.359462  5357 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:24.359753  5490 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:24.360616  5490 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:24.361994  5490 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.366902 27582 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:22:24.254540  4834 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.085s
I20260709 12:22:24.368201 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.371774 28337 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:24.372481 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:24.380060 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:24.381443 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:24.382375 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 167483us
I20260709 12:22:24.261139  5411 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.081s
I20260709 12:22:24.261452  5394 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.086s
I20260709 12:22:24.359078  5450 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:24.388322 28328 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:22:24.374315  5279 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:24.392225  5279 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:24.371207  5239 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.353466  5374 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:24.339253  5462 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.412554 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:24.416280  5429 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.007s
I20260709 12:22:24.400583  5239 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:24.338526  5318 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:22:24.443694 27738 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:24.445119 27731 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:22:24.432165 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:24.460305 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:24.462767 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 342375us
I20260709 12:22:24.463079 28336 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:24.403666  5450 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.056s
I20260709 12:22:24.470193 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:24.473145 28334 raft_consensus.cc:1813] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Handling vote request from an unknown peer ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:24.473834 28334 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
W20260709 12:22:24.481875 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 143184us
I20260709 12:22:24.409768  5430 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.055s
W20260709 12:22:24.486804 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 86286.9us
I20260709 12:22:24.490293 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:24.492602  5462 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:24.493598  5430 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:22:24.497474 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.484668  4834 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:24.503980 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:24.506273  5460 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:24.521939  4834 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.563875  4834 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:24.569630 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74542.2us
I20260709 12:22:24.513731  5397 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:24.576764  5424 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:24.577458  5424 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:24.557186  5434 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.009s	sys 0.022s
W20260709 12:22:24.579430  5434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.010s	sys 0.022s
I20260709 12:22:24.580770  5434 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:24.517443 27738 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:24.501670 27583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.030s	sys 0.047s
W20260709 12:22:24.603538 27583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.030s	sys 0.047s
W20260709 12:22:24.593088 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.517731  5430 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:24.592677 27587 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:24.614099 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:24.617162 27589 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:24.556336  5318 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:24.596448  5425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.012s	sys 0.027s
I20260709 12:22:24.484246  5409 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:24.582728  5326 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:24.629006 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:24.634346 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:24.654464 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:24.673712 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.633597  5425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.013s	sys 0.027s
W20260709 12:22:24.684686 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.607403  5485 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.014s	sys 0.029s
W20260709 12:22:24.694849  5485 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.015s	sys 0.031s
I20260709 12:22:24.703019 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:24.667989  5326 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:24.581351  5434 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.728825 27737 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:24.636464  5318 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:24.727905 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4102) took 2140 ms (client timeout 1904 ms). Trace:
W20260709 12:22:24.734681 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 477291us
W20260709 12:22:24.735780 27729 rpcz_store.cc:269] 0709 12:22:22.587454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.889322 (+1301868us) service_pool.cc:224] Handling call
0709 12:22:24.727633 (+838311us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b540039dd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:24.727815 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":913665792}
W20260709 12:22:24.738557 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 423359us
W20260709 12:22:24.736933 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 433309us
W20260709 12:22:24.740609 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.006s	sys 0.018s
W20260709 12:22:24.748126 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.006s	sys 0.018s
W20260709 12:22:24.748982 27735 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.595s	user 0.005s	sys 0.005s
I20260709 12:22:24.749383 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:24.749641 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 438248us
W20260709 12:22:24.744918 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.758929 28331 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.007s	sys 0.014s
W20260709 12:22:24.802343 28331 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.007s	sys 0.014s
I20260709 12:22:24.758467 27735 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:22:24.765952 27734 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:22:24.786966 27730 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.615s	user 0.007s	sys 0.000s
I20260709 12:22:24.788635 27729 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:24.738993  5444 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:24.804121 27730 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:24.803346 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:24.804370 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:24.757573 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.732671  5326 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.819269 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:24.816650 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76264.1us
I20260709 12:22:24.822310 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:24.824368 27735 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
W20260709 12:22:24.801492  5285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.010s	sys 0.018s
W20260709 12:22:24.826908 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.826741 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:24.751298  5405 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:24.826699 27584 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.014s	sys 0.026s
I20260709 12:22:24.725569  5430 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.754957  5397 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.066s
W20260709 12:22:24.832495 27584 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.015s	sys 0.026s
W20260709 12:22:24.827922 27627 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.720s (SENT) [suppressed 30 similar messages]
W20260709 12:22:24.828012  5285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.011s	sys 0.018s
I20260709 12:22:24.781000  5357 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.059s
I20260709 12:22:24.752895  5498 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.085s
I20260709 12:22:24.816198  5444 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:24.827531  5232 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.028s	sys 0.062s
I20260709 12:22:24.844123 27736 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:24.832212  5306 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:24.848255 27733 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.003s
I20260709 12:22:24.848721 27733 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:24.833320  5411 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:22:24.855103 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 247799us
W20260709 12:22:24.847425  5232 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.028s	sys 0.062s
W20260709 12:22:24.875691 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49051.6us
W20260709 12:22:24.875845 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 371059us
I20260709 12:22:24.877012  5179 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:24.878862  5179 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:24.879501  5179 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:24.881179 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:24.882119 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:24.878167 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:24.836946  5416 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:24.856868  5411 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.900492 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:24.903781 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.905200 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:24.915601 27738 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:24.918368 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.919690 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.920137 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.858631  5306 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:24.897042  5411 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:24.928912 27628 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28.
I20260709 12:22:24.929596 27628 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:22:24.914589  5357 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:24.914398  5505 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:24.904510  5318 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:24.926738  5499 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.958514 27580 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
W20260709 12:22:24.962144 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 158909us
I20260709 12:22:24.986526 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3661) took 2012 ms. Trace:
I20260709 12:22:24.987083 27737 rpcz_store.cc:276] 0709 12:22:22.944069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.922937 (+978868us) service_pool.cc:224] Handling call
0709 12:22:24.728627 (+805690us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.731762 (+  3135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.956566 (+224804us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b540039dd50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:24.956724 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1709145856}
I20260709 12:22:24.941910  5500 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:24.944269  5397 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:22:24.956804  5492 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:22:24.952834  5305 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:22:24.955763  5505 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:25.017302 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50463.4us
I20260709 12:22:24.983462  5318 raft_consensus.cc:2764] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:22:25.026345 27585 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
W20260709 12:22:25.026176 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 203963us
I20260709 12:22:24.912616  5444 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:25.041927 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.011s	sys 0.010s
W20260709 12:22:25.045358 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.012s	sys 0.010s
I20260709 12:22:25.047469 27589 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:25.018796  5305 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:25.023671  5505 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:24.989892  5499 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.066360 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184307us
W20260709 12:22:25.068205 28292 connection.cc:256] RPC call timeout handler was delayed by -1.9597124350002559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
W20260709 12:22:25.052843  5491 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.351s	user 0.003s	sys 0.000s
I20260709 12:22:25.041671  5397 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:25.048296  5501 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.061276  5405 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:25.131084 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 204263us
W20260709 12:22:25.138690 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71580.7us
I20260709 12:22:25.062785  5434 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:25.044852  5490 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:22:25.065212  5429 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:25.074545  5326 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:25.165357 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 479268us
W20260709 12:22:25.154852 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 254479us
W20260709 12:22:25.033540 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 128440us
W20260709 12:22:25.146139 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 268083us
W20260709 12:22:25.167868 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 181395us
I20260709 12:22:25.167408 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:25.142982  5514 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.060966  5279 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.152279  5405 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:25.241511 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:25.247699 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4418) took 1172 ms. Trace:
I20260709 12:22:25.158713  5429 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:25.255838 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 183233us
W20260709 12:22:25.257052 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85181.7us
I20260709 12:22:25.271682  5429 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.164498  5305 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.279742  5305 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:22:25.279947  5524 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:22:25.282053 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1616) took 3413 ms. Trace:
I20260709 12:22:25.283031 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:25.283376  5511 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.237s	user 0.000s	sys 0.000s
I20260709 12:22:25.283833  5540 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:25.284646  5540 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:25.284575 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.285086  5285 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:25.286294 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.290257  5531 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.291893  5526 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:25.292505  5499 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.292951  5526 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:25.293711  5499 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.172449  5326 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:22:25.294492  5326 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.294448  5499 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.296170 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.295080  5326 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:22:25.301805  5506 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.239s	user 0.000s	sys 0.000s
I20260709 12:22:25.295174  5512 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:25.302928 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1786) took 1155 ms. Trace:
I20260709 12:22:25.303375 27581 rpcz_store.cc:276] 0709 12:22:24.148025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.287697 (+139672us) service_pool.cc:224] Handling call
0709 12:22:24.288431 (+   734us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.355934 (+ 67503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.355971 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:24.356592 (+   621us) log.cc:844] Serialized 26 byte log entry
0709 12:22:24.782994 (+426402us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:24.783029 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.783090 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.302688 (+519598us) raft_consensus.cc:1685] finished
0709 12:22:25.302707 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.302851 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4400,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":426008,"threads_started":1,"wal-append.queue_time_us":634584}
W20260709 12:22:25.304708  5518 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
W20260709 12:22:25.307752 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.286796  5285 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:22:25.308238 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.309376  5533 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:25.309863  5533 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:25.302932  5512 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:25.311343  5512 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:25.310775  5533 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.312366  5512 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.313966 27552 leader_election.cc:400] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:25.314388 27552 leader_election.cc:403] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:22:25.315230 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.290921  5531 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.316617  5423 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:25.317334  5423 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:25.317811  5423 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:25.317848 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:25.318313  5531 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.319307 27662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:25.319924 27662 raft_consensus.cc:606] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:25.320385 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.143668  5501 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.321277  5516 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.321811  5516 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.322351  5279 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:25.322815  5501 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.320255  5517 raft_consensus.cc:2764] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:22:25.320266 27662 raft_consensus.cc:614] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:25.318924 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:25.331468 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.293772  5526 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:25.256749 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 89450.2us
W20260709 12:22:25.236567  5239 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.755s	user 0.015s	sys 0.000s
I20260709 12:22:25.367484 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.357410 27702 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:22:25.241247  5450 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:25.248399 27584 rpcz_store.cc:276] 0709 12:22:24.075594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.075726 (+   132us) service_pool.cc:224] Handling call
0709 12:22:24.076076 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.076402 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:22:24.846967 (+770565us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.247093 (+400126us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.247179 (+    86us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:25.247257 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.247422 (+   165us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.247611 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":724662,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":240068,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":110336,"thread_start_us":1133467,"threads_started":3,"wal-append.queue_time_us":202856}
I20260709 12:22:25.256623  5462 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:22:25.282557 27732 rpcz_store.cc:276] 0709 12:22:21.868534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.829084 (+1960550us) service_pool.cc:224] Handling call
0709 12:22:23.837374 (+  8290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.851742 (+ 14368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.851774 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:23.852340 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:22:24.335587 (+483247us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:24.873561 (+537974us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.873866 (+   305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.879006 (+  5140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.051950 (+172944us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:25.243267 (+191317us) raft_consensus.cc:1685] finished
0709 12:22:25.263779 (+ 20512us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.281949 (+ 18170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":954220,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":160245,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":23641,"spinlock_wait_cycles":6428032,"thread_start_us":965544,"threads_started":2,"wal-append.queue_time_us":781853}
I20260709 12:22:25.381235 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4424) took 1073 ms. Trace:
W20260709 12:22:25.384511 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 213764us
I20260709 12:22:25.384722 28336 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:22:25.382537 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 245873us
I20260709 12:22:25.385344 28336 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:25.389001 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.391882 27662 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:25.393857 27659 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:22:25.318823  5423 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.377381  5450 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.382544  5512 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.237854  5514 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.383224 27582 rpcz_store.cc:276] 0709 12:22:24.307311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.393657 (+ 86346us) service_pool.cc:224] Handling call
0709 12:22:24.583219 (+189562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.632769 (+ 49550us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:22:24.883244 (+250475us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.240911 (+357667us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.375788 (+134877us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.375991 (+   203us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.376049 (+    58us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:22:25.376118 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.376208 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.381143 (+  4935us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":134234,"prepare.queue_time_us":225901,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":57792,"raft.queue_time_us":289262,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"thread_start_us":468466,"threads_started":2}
I20260709 12:22:25.402552 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:25.402666 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45735.8us
W20260709 12:22:25.403891 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58233.6us
I20260709 12:22:25.402141 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:25.406747 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:25.351645  5397 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.406879 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.399966 28336 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.413520 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.416456 27659 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:22:25.399480  5450 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:25.398985  5429 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:25.415367 27663 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:25.415386 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:25.422801 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.423720  5514 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:25.424329  5528 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.424784 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:25.425395  5206 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.320s	user 0.012s	sys 0.000s
I20260709 12:22:25.424870  5528 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.425793 27734 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 17.
I20260709 12:22:25.426970 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.439630 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:25.427619 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:25.441684 27736 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:25.428260 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.429240 27730 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:25.429754 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:25.431252  5424 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.853s	user 0.008s	sys 0.003s
I20260709 12:22:25.446448  5424 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:25.431840  5544 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.447172  5430 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:25.440908 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.447403  5544 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.432145 28336 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:25.432246  5530 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:25.415058  5539 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.451251 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.452062 27661 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:25.451773  5539 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.453030 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.453598 27661 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:25.454423 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.433588  5537 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:25.460354 27588 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:22:25.436789  5496 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.612s	user 0.001s	sys 0.015s
I20260709 12:22:25.455649  5536 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.456223  5501 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:25.400787  5416 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.383666  5239 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:25.433251  4834 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.869s	user 0.010s	sys 0.002s
I20260709 12:22:25.505920  4834 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.507534  5525 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.088s
I20260709 12:22:25.509094  5525 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.509541 27735 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:25.509609  5525 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.510586  5424 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:25.511529  4834 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:25.511620  5416 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:25.458986 27732 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:25.508690 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.439152 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.512356  5416 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.513018 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.513118 27731 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
W20260709 12:22:25.513885  5527 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:25.514343  5540 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28.
I20260709 12:22:25.514638  5416 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.515170 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:25.513123  5239 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:25.519026 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4410) took 3108 ms. Trace:
I20260709 12:22:25.519418 27583 rpcz_store.cc:276] 0709 12:22:22.410296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.602433 (+192137us) service_pool.cc:224] Handling call
0709 12:22:23.748704 (+1146271us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:23.980285 (+231581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.980320 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:23.981259 (+   939us) log.cc:844] Serialized 52 byte log entry
0709 12:22:24.604013 (+622754us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:24.604068 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.616574 (+ 12506us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.945336 (+328762us) raft_consensus.cc:1685] finished
0709 12:22:24.945361 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.518806 (+573445us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000bd1d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:25.518972 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":220074,"raft.queue_time_us":220345,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"spinlock_wait_cycles":2574428416,"thread_start_us":622299,"threads_started":1,"wal-append.queue_time_us":807590}
I20260709 12:22:25.489606  5528 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.527971  5526 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:25.528401  5533 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:25.531939  5462 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:25.532465  5462 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:22:25.420332  4931 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.204s	user 0.005s	sys 0.009s
I20260709 12:22:25.535580  4931 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23.
I20260709 12:22:25.486634 28333 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:25.538322 27482 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:25.537613 28333 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:25.540086 27482 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 24, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:25.540558 28333 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.541644 28333 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:25.541612 27482 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:25.411063  5405 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.491581 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.494870  5547 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:25.543232 27730 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:25.543350  5547 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:25.543666 27482 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 27, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 27 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:25.544180  5405 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.544458  5547 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.545401 27482 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 22, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 22 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:25.546692  5405 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.547367 27482 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:25.531980 27659 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:25.490491 27662 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:25.499969 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.402351  5512 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.491082  5179 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.611s	user 0.008s	sys 0.003s
I20260709 12:22:25.503252 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:25.504030 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97377.8us
I20260709 12:22:25.505296 28335 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:22:25.510062 27737 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:25.445382 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:25.461289 27661 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:25.462073 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 143268us
I20260709 12:22:25.583945 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.486303 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.570552 28336 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 37 index: 22. (term mismatch)
I20260709 12:22:25.583897 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.586665 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:25.608639 27662 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:22:25.583729 27482 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 28, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 28 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:25.588757 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:25.497617  5516 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:25.609952 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107531us
I20260709 12:22:25.614091 28336 pending_rounds.cc:85] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 21
I20260709 12:22:25.610445  5526 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:25.611424  5513 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:25.574172  4931 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:25.612802 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:25.596678 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.614610 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:25.503553  5501 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.627386 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.628300 27663 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:22:25.503468  5505 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.600514  5405 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:22:25.621291  5285 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.012s
W20260709 12:22:25.634016 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 135047us
I20260709 12:22:25.634343 27663 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:25.634529  5507 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:25.600334  5179 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.623346  4834 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:22:25.635294 27585 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:25.642987 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:25.637647 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.669423 27580 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:25.648057 27662 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:25.666935 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:25.637866  5497 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.816s	user 0.005s	sys 0.006s
I20260709 12:22:25.652881  5525 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.622117  5513 consensus_queue.cc:1243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:22:25.656623  5500 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.000s
I20260709 12:22:25.658293  5505 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:25.659536  4931 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:25.660789 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46272.8us
I20260709 12:22:25.682621 27627 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:25.682963 27627 leader_election.cc:403] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:25.664332  5540 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:25.642737 28333 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:22:25.643188 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:25.643653 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.668694 27585 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:25.637392 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.644332 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:25.714054 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4116) took 1368 ms. Trace:
I20260709 12:22:25.637861 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.714296 27589 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:25.714483 27738 rpcz_store.cc:276] 0709 12:22:24.346177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.628450 (+282273us) service_pool.cc:224] Handling call
0709 12:22:25.713850 (+1085400us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:25.713993 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":597059456}
I20260709 12:22:25.715068 27587 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:22:25.614362 28336 pending_rounds.cc:107] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 36.22
I20260709 12:22:25.715664 28336 raft_consensus.cc:2889] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:25.664628  5450 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.631790  5397 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.645988  5326 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.675611 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:25.641611  5411 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.688s	user 0.004s	sys 0.004s
I20260709 12:22:25.645628  5306 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:25.674698 27661 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:25.731426 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.732086 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.682502 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59825.3us
I20260709 12:22:25.503669  5536 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.685310 27586 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:22:25.713487 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:25.737119 27580 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:25.738045 27630 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:25.738288 27630 leader_election.cc:403] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:22:25.720892 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.731460 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.682279  5500 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:22:25.740738 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.742205 27585 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.743891 27585 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:25.745203 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.746536 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.719890  5424 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:25.747579 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:25.723515  5528 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.724216  5525 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.723526  5540 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:25.767143 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.790413 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.850519 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.823722 27586 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:22:25.735447  5516 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:25.853469  5516 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:25.854501  5516 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.824255 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77960.3us
I20260709 12:22:25.856566 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.764241 27589 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:25.736815  5411 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.858309 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: [34.19-36.20]
I20260709 12:22:25.859402  5411 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.833307 27581 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:25.847993  5530 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.860491  5530 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.862754  5530 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:25.834851  5552 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:22:25.863770  5500 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:25.864629  5500 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:25.865129 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:25.843259 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.843961 27580 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:25.747421  5514 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.851995 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:25.874578 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:25.859611  4834 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.863384 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 137445us
I20260709 12:22:25.865285  5500 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:25.866595  5540 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:25.865064 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116585us
I20260709 12:22:25.749163 27664 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: [22.16-22.16]
W20260709 12:22:25.872189 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.875654 28333 raft_consensus.cc:2368] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32.
I20260709 12:22:25.877990 27581 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
W20260709 12:22:25.878859 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.882972 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.740325  5397 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.884788 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:25.890069 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:25.881385  5530 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:25.876008  5501 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.844192  5409 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:22:25.868939  5525 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.916515  5493 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.076s	user 0.011s	sys 0.001s
W20260709 12:22:25.916651  5554 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:25.868830  5528 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:25.878618  5514 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:25.921877 27585 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:22:25.891574  5512 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.879789  4834 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:25.928857 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.941972 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.919471  5500 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:22:25.941589 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51573.5us
I20260709 12:22:25.951069 27661 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:22:25.931192  4834 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.960016 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.960772 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:25.967242  5326 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:25.970789 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:25.971864  5551 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
W20260709 12:22:25.972395 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4414) took 1909 ms (client timeout 1710 ms). Trace:
W20260709 12:22:25.972820 27588 rpcz_store.cc:269] 0709 12:22:24.062720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.062848 (+   128us) service_pool.cc:224] Handling call
0709 12:22:25.972197 (+1909349us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540011ce54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:25.972325 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3132671744}
I20260709 12:22:25.962044 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.974316 27658 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:25.975006  5450 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:25.975250 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.975941 27658 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 21.
I20260709 12:22:25.976779 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.977939 27658 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:22:25.979132 27655 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:22:25.941123  5434 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:25.981714  5434 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:25.982169  5434 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:25.983389 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:25.984198 28333 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 38 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:22:25.984349 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.985092 27657 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:22:25.985394 28336 raft_consensus.cc:1217] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Deduplicated request from leader. Original: 23.14->[25.15-34.19]   Dedup: 29.17->[32.18-34.19]
I20260709 12:22:25.985860 28336 pending_rounds.cc:85] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 17
I20260709 12:22:25.986076 28336 pending_rounds.cc:107] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 31.18
I20260709 12:22:25.986320 28336 raft_consensus.cc:2889] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:25.987208 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:25.987208 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:25.986204 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:25.988287 27663 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:25.987880 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:25.988896 27738 raft_consensus.cc:2376] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31. Re-sending same reply.
I20260709 12:22:25.990721 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:25.991444 27663 raft_consensus.cc:606] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:25.991835 27663 raft_consensus.cc:614] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:25.994724  5397 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:25.955862  5487 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 37 current term: 37 is higher.
I20260709 12:22:25.941390  5285 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.996340 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:25.997300 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.957274  5553 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.010190 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.010931 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:26.011539 27730 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:26.011844 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.012959 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.014633 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.015354 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:26.005018  5429 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:25.999154  5536 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.010607  5285 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.027951  5507 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:26.042912 27660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:54366
I20260709 12:22:26.053005 27660 raft_consensus.cc:606] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:26.053517 27660 raft_consensus.cc:614] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:26.004745  5232 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:22:26.028947  5429 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:26.059657 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 182970us
I20260709 12:22:26.061913 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.024380  5553 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.066318 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:26.090363  5553 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:26.062763  5429 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:26.096088  5514 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:26.097333 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:26.098176 28335 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:26.098510  5528 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.060657  5525 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.101075 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.101732 27482 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.102407 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:26.104426 27482 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.106036 27482 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 28. New cstate: current_term: 28 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.106179  5285 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.110425  5179 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:26.113323  4931 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:26.115760  5430 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.668s	user 0.006s	sys 0.005s
I20260709 12:22:26.116374  5498 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:26.116816  5498 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:26.117161  5498 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:26.118727 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:26.119287 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53080.2us
W20260709 12:22:26.119700 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.122829 27730 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:22:26.123133  5434 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.124619 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.125471  5533 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:22:26.126060  5533 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:26.126806  5434 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:26.073166  5507 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:22:26.127270  5533 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:26.131301 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 892579us
I20260709 12:22:26.131664  5503 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:26.133106  5503 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:26.133589  5503 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:26.135507  5356 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.136123  5356 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.137591 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:26.138404 28333 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:22:26.139045 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.139747  5516 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:26.140406  5516 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:26.142338  5490 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.142946  5490 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.145926 27583 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:22:26.146936  5504 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.147598  5504 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.148759 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:26.141400  5516 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.149354 27587 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:26.151600  5521 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:26.152271  5521 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.153662  5536 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:26.154387  5536 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:26.155445 28329 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:26.155400  5536 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.156026 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.157020  5530 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:26.159510 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 738223us
I20260709 12:22:26.159775 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:26.161073 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53031.3us
W20260709 12:22:26.133277 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.165870 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.161422 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.174995 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:26.176296 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52813.8us
I20260709 12:22:26.178525 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.179239 27657 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:26.180640 27702 leader_election.cc:400] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32.
I20260709 12:22:26.181038 27702 leader_election.cc:403] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:22:26.182077 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.187955 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.194651 27735 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:26.195559 27731 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:26.196107 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:26.120285  5514 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:26.199381  5356 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.134608  5503 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.199676 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.199826  5528 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:26.200268 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.200938 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.202647 27655 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:26.196802  4931 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.014s
I20260709 12:22:26.202684  5525 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:26.180074  5543 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:26.180218  5531 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:26.187615  5416 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.017s
I20260709 12:22:26.204819 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:26.222447 27734 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:26.205343 27630 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28.
W20260709 12:22:26.205636 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.187762  5500 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:26.205888 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:26.207218 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.227147 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.208009 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.214965  5462 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.682s	user 0.008s	sys 0.003s
I20260709 12:22:26.212513 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.212708 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.233484 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.210716 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.215808 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.216188 27730 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 32. Ops: []
I20260709 12:22:26.216893 27587 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:26.218564 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1494) took 2129 ms. Trace:
I20260709 12:22:26.220785 27663 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:26.237651 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.238431 27663 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 21 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 16.
W20260709 12:22:26.238746 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.209059 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.206326  5179 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:26.243402 27664 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:26.244141 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.240630 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:26.248307 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:26.249364 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.249949 27657 raft_consensus.cc:2376] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35. Re-sending same reply.
I20260709 12:22:26.251948 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.246341 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.222524  5531 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.226186  5416 consensus_queue.cc:1243] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 36.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:22:26.239595 28331 rpcz_store.cc:276] 0709 12:22:24.082523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.082685 (+   162us) service_pool.cc:224] Handling call
0709 12:22:24.087497 (+  4812us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.107956 (+ 20459us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:22:24.815984 (+708028us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.227724 (+411740us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.522003 (+294279us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b18003e77c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:26.129508 (+607505us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b18003e77c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:26.186105 (+ 56597us) log.cc:844] Serialized 14 byte log entry
0709 12:22:26.211542 (+ 25437us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:26.211595 (+    53us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:22:26.211643 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.211705 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:26.211811 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":718480,"prepare.queue_time_us":922260,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":109,"raft.queue_time_us":938670,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":73670,"spinlock_wait_cycles":11428608,"thread_start_us":1055844,"threads_started":3,"wal-append.queue_time_us":312960}
I20260709 12:22:26.257357 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.245249 27737 raft_consensus.cc:2376] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35. Re-sending same reply.
I20260709 12:22:26.258603 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.264281 27657 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:26.264672 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.265622 27661 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:26.268087 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.268811 27657 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:26.216540  5279 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:22:26.231120  5525 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:26.267684 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:26.273795 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.273167 27729 raft_consensus.cc:2376] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26. Re-sending same reply.
I20260709 12:22:26.279785 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:26.267761  5568 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:22:26.280524 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:26.282510 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.287746 27660 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:26.243561  5462 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.289458 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.290104 27730 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:26.290167  5462 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.291008  5521 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.284036  5512 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.012s
I20260709 12:22:26.274435  5285 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:26.295209 27482 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.295853  5525 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.284466  5356 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:22:26.253840  5179 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:26.267094  5232 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.266875  5394 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.061s
I20260709 12:22:26.270520  5397 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:26.284734  5490 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.325692  5504 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.505862 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.501749 27734 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:22:26.298183 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.296947 27482 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 20 to 22, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 22 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.358914 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:26.534658 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.272029  5405 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.351954  5501 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:26.356595 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.260437  5531 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:26.325392 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.299633 27659 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:22:26.433085 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.553994 27659 pending_rounds.cc:85] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 15
I20260709 12:22:26.554540 27659 pending_rounds.cc:107] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:22:26.528128 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.544099  5499 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.554457 27660 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
W20260709 12:22:26.298688 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.548929 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.557230 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.557883 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.558346 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.555191 27732 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
W20260709 12:22:26.528195 27552 connection.cc:256] RPC call timeout handler was delayed by -3.2844318289999137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
I20260709 12:22:26.530294 27663 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:26.325685  5512 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.218s
I20260709 12:22:26.545761 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:26.576272 27552 connection.cc:256] RPC call timeout handler was delayed by -3.2779231919998892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24670000000000003s for a response
I20260709 12:22:26.579214 27655 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
W20260709 12:22:26.572956 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.555037 27659 raft_consensus.cc:2889] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:26.579265 27552 connection.cc:256] RPC call timeout handler was delayed by -3.2188527690000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1839s for a response
I20260709 12:22:26.580201 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:26.581516 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:26.582509 27733 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:26.582486 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:26.580798 27552 connection.cc:256] RPC call timeout handler was delayed by -3.191803230000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260709 12:22:26.562462 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:26.583870 27732 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:26.544471  5429 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.545791  5530 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:26.546527  5444 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:26.552261  5416 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:22:26.552040 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 226757us
I20260709 12:22:26.549939  5501 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:26.585011  5462 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.545765  5450 raft_consensus.cc:2764] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:22:26.587992 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.590476 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.551909 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:26.586483 27663 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:22:26.593232 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.593514 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.521286  5528 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32.
I20260709 12:22:26.597205 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.597910  5499 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:26.597839 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:26.598994 27589 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:22:26.599551 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.599723  5429 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.401765  5285 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.184s
I20260709 12:22:26.600198 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:26.668184 27584 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:26.601107 27582 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
W20260709 12:22:26.602144 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 441649us
I20260709 12:22:26.603387 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.545003  5179 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.615917 27588 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:26.547978  5232 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.617470 27580 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:26.674990  5528 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:26.654879  5544 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.655014  5539 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.617354  5462 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.682070  5462 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.682390  5394 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:26.684501  5394 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:26.682132 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.698441 27584 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
W20260709 12:22:26.700594 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.701910 27580 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:26.703794 27658 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:26.618258 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.543241  5498 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.629703  5574 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:26.639096 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.714543 27581 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:26.644178 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:26.713665 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95475.3us
I20260709 12:22:26.644457 27729 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:26.646673  4834 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.649564  5424 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:26.723937 27656 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:26.549302  5405 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.652724 27587 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:26.651664  5553 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:26.602835  5416 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.661783 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.730721 27589 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:26.666157 27655 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:22:26.661798 27583 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:22:26.670179 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66926.1us
I20260709 12:22:26.733486 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.612597 27659 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [31.18-32.19]
I20260709 12:22:26.671808 27662 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 29. Ops: []
I20260709 12:22:26.738762 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.711695 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:26.583539 27552 connection.cc:256] RPC call timeout handler was delayed by -3.1891032300000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1988s for a response
I20260709 12:22:26.723150 27586 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:26.740991 27732 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
I20260709 12:22:26.728968 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.700183  5279 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:26.708295  5462 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:26.760696 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:26.779120 27663 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:26.727147  4834 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.727835  5539 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:22:26.790117 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:26.777637 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.781692 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.790360 27581 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:26.787949 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.724383  5424 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:26.793026  5462 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.793679  5499 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:26.794605 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.786158 27661 raft_consensus.cc:1217] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Deduplicated request from leader. Original: 33.20->[34.21-36.22]   Dedup: 34.21->[36.22-36.22]
I20260709 12:22:26.795360 27655 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:26.795775 27661 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: [34.21-36.22]
I20260709 12:22:26.796624 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:26.797350 27655 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:22:26.798084 27661 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:22:26.798465 27661 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 16
I20260709 12:22:26.728021  5574 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.798717 27661 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:22:26.782812 27737 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:26.801589  5499 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:26.775831  5490 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.730136  5553 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:26.806059 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.744830  5528 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:26.799132 27661 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:26.810815 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.811887 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.744963  5416 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:26.819613 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.811967 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:26.823143 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45641.9us
I20260709 12:22:26.823056  5490 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.824559  4834 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:26.781311  5512 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:22:26.836591 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.836982 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 85198.5us
W20260709 12:22:26.838128 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.832422 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.832473 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.833875 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:26.838675 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.789541  5525 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:26.789991  5539 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:26.848057 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.854245 27580 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:26.809067 27663 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:26.855656 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.856662 27585 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:26.781873  5521 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
W20260709 12:22:26.862495 27552 connection.cc:256] RPC call timeout handler was delayed by -2.710303230000136s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.252s for a response
I20260709 12:22:26.818416  5553 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:26.860452 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78894.4us
W20260709 12:22:26.863062 27552 connection.cc:256] RPC call timeout handler was delayed by -2.3485032300000057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2922s for a response
I20260709 12:22:26.863085 27584 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 17. (term mismatch)
W20260709 12:22:26.863663 27552 connection.cc:256] RPC call timeout handler was delayed by -1.8695137549998435s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20260709 12:22:26.864187 27552 connection.cc:256] RPC call timeout handler was delayed by -1.769613754999682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20260709 12:22:26.864693 27552 connection.cc:256] RPC call timeout handler was delayed by -1.5617137549998006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1935s for a response
W20260709 12:22:26.865150 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3943137549999847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
I20260709 12:22:26.861318 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.863715 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.726692  5501 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:26.829093  5504 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:26.873893 27587 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:22:26.878463 27582 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 16. (term mismatch)
I20260709 12:22:26.878875 27582 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:22:26.840628  5487 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:26.863693 27584 pending_rounds.cc:85] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 16
I20260709 12:22:26.884306 27584 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20260709 12:22:26.884681 27584 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:26.879187 27582 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:22:26.886126 27582 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:26.860853  5539 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:22:26.861078  5540 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.021s	sys 0.017s
I20260709 12:22:26.882167 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:26.846052  5394 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:26.875411 27588 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:26.913429 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.863021  5528 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:26.914721 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.916733 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:26.916380 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.918123 27588 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 17.
I20260709 12:22:26.921343  5574 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.915990  5528 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.922725  5528 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:26.921381 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.926123 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:26.927392  5430 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:26.928081  5430 raft_consensus.cc:2764] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:22:26.862752  5239 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.051s
I20260709 12:22:26.930104 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:26.932040 27587 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:22:26.930670  5394 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:26.937891  5394 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.21, Last appended by leader: 21, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:26.941960  5503 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:26.944187  5232 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.945421  5462 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:26.857906  4834 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.054s
W20260709 12:22:26.910564  5540 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.022s	sys 0.017s
I20260709 12:22:26.899619  5591 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:26.951300 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:26.958316 27581 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:26.971517 27657 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:26.901471  5499 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:26.973692 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 115127us
I20260709 12:22:26.976300 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:26.981549 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.936218  5536 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:26.996450 27581 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
W20260709 12:22:26.998029 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81358.9us
I20260709 12:22:26.965256  5462 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:26.994822 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:27.005143 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:27.005111 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170623us
I20260709 12:22:27.008057 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:27.010768 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:26.992105  5498 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:26.917004  5424 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.013110  5485 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:27.000775  5593 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:27.027164 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:27.035929 27733 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:27.024082 27587 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: [34.19-36.20]
W20260709 12:22:27.024740 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94793.2us
I20260709 12:22:27.031373 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.036967 27730 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:26.996256  5521 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:22:27.043002 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.044029 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:27.004328  5536 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.042244 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:27.026594  5501 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:27.051862 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.057096 27730 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:27.030141  5544 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:27.023898  5405 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.054306 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:27.035492  5485 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.066018 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:27.067243 27656 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 38 because replica has last-logged OpId of term: 37 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 21.
W20260709 12:22:27.069104 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.016562  5531 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:27.075941 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.075243  5485 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:27.086104 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.087886 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:27.095089  5588 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:22:27.096858  5586 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:22:27.091120  5521 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:22:27.097561  5521 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:27.059340  5501 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.099866  5501 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:27.100214 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:22:27.101771 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.104399  5536 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.098623  5521 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.106065  5521 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:27.020172  5594 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:22:27.048671  5601 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:27.048491  5596 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:22:27.121166 27734 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: [30.22-30.22]
W20260709 12:22:27.122946 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68891.6us
I20260709 12:22:27.112934 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:27.123631 27657 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:27.141685 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:27.143630 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:27.144562 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.138284 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 136469us
W20260709 12:22:27.156889 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.157454 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.164639 27586 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:27.166800 27733 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:27.169044 27733 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:22:27.119346  5517 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:27.059110  5574 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:22:27.121809  5601 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.123659  5425 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:27.123739  5596 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.139041  5405 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:27.178479 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:27.178495 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65656.2us
I20260709 12:22:27.123365  5544 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:27.179656 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:27.120501  5462 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:22:27.191787  5462 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:27.190726 27482 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 28 to 29, leader changed from <none> to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 29 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:27.192168  5462 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:27.193317  5462 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.193475 27482 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 17 to 19, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 19 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:27.194648  5462 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:27.197263 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:27.198143 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:27.182539 27733 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:22:27.058820  5499 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.164425  5232 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:27.183884 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:27.172088  5593 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:27.171167  5536 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:27.180775  5601 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:27.190083  5596 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:22:27.218971 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.212152 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.230362 27587 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:22:27.232141 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.170910  5595 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:22:27.203215  5516 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:27.170859  5603 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 12:22:27.211006  5594 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:27.251737 27552 leader_election.cc:400] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:27.223659  5593 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:27.256692 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:27.264262 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.259377 27586 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
W20260709 12:22:27.250660  5444 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.650s	user 0.010s	sys 0.003s
I20260709 12:22:27.242590  5232 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:27.296641 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49537.1us
I20260709 12:22:27.258051  5594 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:27.239962  5536 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.266014  5593 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:27.305225 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.304855  5531 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:27.311515  5602 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:22:27.314872 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.315604 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:27.315291  5594 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.321403 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.322507 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.300518  5416 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.341718  5409 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:27.341729  5500 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:27.299240  5544 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:27.436407 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 111570us
I20260709 12:22:27.438267 27732 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:27.440789  5498 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.441445  5498 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.444432  5514 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.445091  5514 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.448165 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.450860  5514 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.449623  5498 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:27.452682 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62214.2us
I20260709 12:22:27.386445  5500 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:27.434568  5507 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.375010  5601 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.456277  5540 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:27.489315  5562 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:27.457126  5429 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:22:27.382833  5232 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.509696 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.427668  5485 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.513892 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3676) took 1235 ms. Trace:
I20260709 12:22:27.514400 28328 pending_rounds.cc:85] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 23
I20260709 12:22:27.514782 27730 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:22:27.521275 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3516) took 1331 ms (client timeout 1725 ms). Trace:
I20260709 12:22:27.541237 28328 pending_rounds.cc:107] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 36.24
I20260709 12:22:27.541945 28328 raft_consensus.cc:2889] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:27.473963  5500 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:27.541745 27737 rpcz_store.cc:276] 0709 12:22:26.278601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.298381 (+ 19780us) service_pool.cc:224] Handling call
0709 12:22:27.513705 (+1215324us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:27.513818 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538388352}
W20260709 12:22:27.541592 27658 rpcz_store.cc:269] 0709 12:22:26.190348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.190463 (+   115us) service_pool.cc:224] Handling call
0709 12:22:27.521112 (+1330649us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b540022b354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:27.521219 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":877800704}
I20260709 12:22:27.524483 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:27.514780 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:27.513834  5613 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
W20260709 12:22:27.514389  5619 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:22:27.474463  5514 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:22:27.553071 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:27.500233  5429 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:27.513137  5416 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.503048  5424 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:27.494807  5553 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.611s	user 0.008s	sys 0.000s
I20260709 12:22:27.562832 28337 raft_consensus.cc:2368] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
W20260709 12:22:27.563170 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48479.5us
I20260709 12:22:27.565317 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:27.480756  5507 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.571128  5600 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:27.571682  5504 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:27.484273  5498 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.077s
I20260709 12:22:27.560755  5429 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:27.523530  5501 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.491291  5562 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:22:27.575913 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:27.580420 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.581308 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.568545  5553 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.584877 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:27.585355 27589 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 27 index: 22. (index mismatch)
W20260709 12:22:27.593096 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.592775 27587 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:27.575033  5600 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:27.574126  5425 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.024s
W20260709 12:22:27.607443 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 121488us
I20260709 12:22:27.609391 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.609531 28333 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:22:27.610512 28333 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 15
I20260709 12:22:27.610818 28333 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:22:27.611159 28333 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:27.594269  5618 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:27.613941 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:27.619827  5397 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.620339  5397 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:27.624107  5485 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.624536  5596 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.604629  5600 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:27.627403  5490 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:27.629920  5567 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.373s	user 0.009s	sys 0.002s
I20260709 12:22:27.597622  5416 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.630618  5416 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:27.631982 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.633116  5416 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:27.635519  5514 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:22:27.636758 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67553.9us
I20260709 12:22:27.639446  5514 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.639724 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4133) took 1643 ms. Trace:
I20260709 12:22:27.640066  5516 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:27.640717  5516 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:27.640190 27738 rpcz_store.cc:276] 0709 12:22:25.996583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.996705 (+   122us) service_pool.cc:224] Handling call
0709 12:22:26.004771 (+  8066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.010997 (+  6226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.011028 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.028549 (+ 17521us) log.cc:844] Serialized 26 byte log entry
0709 12:22:26.181841 (+153292us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:26.181882 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.181945 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.541309 (+1359364us) spinlock_profiling.cc:243] Waited 9.71 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:27.639393 (+ 98084us) raft_consensus.cc:1685] finished
0709 12:22:27.639418 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.639659 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":9489,"spinlock_wait_cycles":22329728,"thread_start_us":130082,"threads_started":1,"wal-append.queue_time_us":205791}
W20260709 12:22:27.643072  5622 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:22:27.646775 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:27.639943  5514 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:27.635099  5503 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.693s	user 0.004s	sys 0.004s
I20260709 12:22:27.652968 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.656471 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:27.659548 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.665160 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:27.645151  5621 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:27.666261 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.665603 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:27.668241 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:27.670768 27663 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:27.615190  5536 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:27.672380  5540 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:27.672890  5540 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:27.673345  5540 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:27.676007 27737 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 27 index: 22. (index mismatch)
I20260709 12:22:27.677631 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.678321 27738 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:27.680485  5553 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.683384  5507 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.684320  5503 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:27.687839 27663 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:27.691663  5630 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.692402  5630 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.700641 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:27.711597 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45419.3us
I20260709 12:22:27.650494  5462 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:27.718256 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 173268us
W20260709 12:22:27.718401  5626 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:22:27.720150 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:27.720614  5514 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.716662  5490 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:27.740561  5490 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.681558  5593 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.707038  5394 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:27.712875  5521 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:27.713627  5543 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:27.712175  5601 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.665278  5279 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.078s
I20260709 12:22:27.809122  5501 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.804618  5357 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.013s
I20260709 12:22:27.809842  5628 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.859899  5574 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:27.860332  5574 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:27.860146  5628 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.861256  5574 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.862334  5628 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.805619  5507 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:22:27.810114  5618 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:27.819227  5630 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.847168 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:27.864040  5630 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:27.864243 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4144) took 1173 ms. Trace:
I20260709 12:22:27.864699 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:27.865501  5562 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:27.866032  5562 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.866408  5562 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:27.824347  5239 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.867604  5239 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.869802  5239 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.791121  5514 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:27.870515  5514 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.870911  5514 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:27.871611  5425 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:22:27.873730  5528 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.950s	user 0.013s	sys 0.000s
I20260709 12:22:27.874207  5528 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.827088 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1635) took 1062 ms. Trace:
I20260709 12:22:27.832177  5423 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:27.875569  5628 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:22:27.742432  5462 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.721956  5596 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.877828  5462 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.833757 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3680) took 1213 ms. Trace:
I20260709 12:22:27.879354 27732 rpcz_store.cc:276] 0709 12:22:26.620793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.755462 (+134669us) service_pool.cc:224] Handling call
0709 12:22:26.755768 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.439926 (+684158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.833536 (+393610us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540017e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:27.833699 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1569076096}
I20260709 12:22:27.864693 27730 rpcz_store.cc:276] 0709 12:22:26.690883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.078808 (+387925us) service_pool.cc:224] Handling call
0709 12:22:27.514453 (+435645us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.541294 (+ 26841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.864020 (+322726us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b5400317250. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:27.864177 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":983732480}
I20260709 12:22:27.880862 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.881313 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.872046  5425 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:27.824721  5593 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.834753  5179 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:27.883249  5179 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:27.882951  5425 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.836625  5490 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.814819  5394 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.841667  5405 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:22:27.255738 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:27.876260  5528 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.815093  5498 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:22:27.851334 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: [34.19-36.20]
I20260709 12:22:27.854861 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:27.905334 27735 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:27.716553  5536 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.099s
I20260709 12:22:27.855911 27738 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:22:27.856801 27737 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:22:27.857374 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.858012 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:27.884140  5179 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.918835  5179 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:27.921355  5516 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.862670  5574 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:27.860348  5625 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260709 12:22:27.650612  5429 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.946506  5429 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:27.865535  5507 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:27.842348  5397 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:27.869868  5564 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.795s	user 0.000s	sys 0.015s
I20260709 12:22:27.951820  5507 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:27.879019  5539 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.960s	user 0.011s	sys 0.000s
I20260709 12:22:27.952865  5539 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.953321  5507 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.832693  5600 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:27.957329  5600 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.875484 27734 rpcz_store.cc:276] 0709 12:22:26.764507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.101035 (+336528us) service_pool.cc:224] Handling call
0709 12:22:27.101375 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.125221 (+ 23846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.826792 (+701571us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:27.827032 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44869376}
I20260709 12:22:27.881587 27732 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:27.887450 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.011s	sys 0.003s
W20260709 12:22:27.960209 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.804s	user 0.011s	sys 0.003s
I20260709 12:22:27.960834  5539 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:27.960949 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 817361us
I20260709 12:22:27.962386 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:27.893841 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.964352  5539 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:27.966351 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:27.966727  5591 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:27.967372  5591 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:27.967846  5591 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:27.898838  5394 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:27.887857  5405 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:27.973893  5405 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.975301  5405 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:22:27.976080 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.978035  5540 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:27.821708  5533 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:27.997776  5533 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:27.998999  5533 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:28.000818 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 239748us
I20260709 12:22:28.000916  5533 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:28.002216 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:28.003204  5536 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:27.834712  5409 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:27.958884 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4131) took 1973 ms. Trace:
I20260709 12:22:28.003794  5536 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:28.004760  5536 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.004134 27736 rpcz_store.cc:276] 0709 12:22:25.985037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.985167 (+   130us) service_pool.cc:224] Handling call
0709 12:22:25.985501 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.985860 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.985893 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:25.986324 (+   431us) log.cc:844] Serialized 26 byte log entry
0709 12:22:26.054449 (+ 68125us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:26.054510 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.054606 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.869304 (+1814698us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:27.958579 (+ 89275us) raft_consensus.cc:1685] finished
0709 12:22:27.958600 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.958821 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1584,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":105008000,"thread_start_us":67703,"threads_started":1,"wal-append.queue_time_us":87509}
I20260709 12:22:27.986616  5424 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:28.007445 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 290537us
I20260709 12:22:28.007921  5424 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.009316  5503 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:22:28.009763 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.010263  5627 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.010389  5503 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.010839  5627 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:28.014045 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 467893us
W20260709 12:22:28.015676 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.016150 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.016619 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.016975 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.017153 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 413209us
I20260709 12:22:28.018414  5644 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:22:27.991271 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 269443us
I20260709 12:22:27.995889  5528 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:28.019675 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.020217  5528 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.020454 27584 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:28.020967  5528 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:28.019387 28328 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: []
W20260709 12:22:28.019941 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.022720 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:28.024405  5627 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:28.024971  5643 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:28.019219 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:28.025555 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.025853  5444 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:28.026290  5444 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:28.026486  5643 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:28.027299 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:28.027388  5444 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.028601  5643 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:28.029227  5643 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:28.029107  5444 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:28.031782 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:28.032829  5279 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:28.033197 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.033586 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:28.033893 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:22:28.030622  5643 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.034884 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.035557 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:28.037711  5462 raft_consensus.cc:2764] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election decision vote started in defunct term 26: won
I20260709 12:22:28.037762  5232 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:28.038079 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:28.038470  5232 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.037664  5394 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:28.038978  5232 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:28.039529  5394 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:28.039651 27553 leader_election.cc:400] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:28.040099 27553 leader_election.cc:403] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:28.040654  5462 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:28.040514  5394 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.041081  5462 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:28.041489  5462 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:28.047931 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:28.048524 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 183893us
I20260709 12:22:27.969923  5500 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.053022  5628 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:28.053372  5500 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:28.054757 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:28.055709  5628 raft_consensus.cc:2764] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:22:27.989360 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:28.057858 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68614.6us
I20260709 12:22:28.059022  5543 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:28.059514  5544 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:28.059949  5544 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:28.060322  5544 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:28.060477  5543 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:28.049006 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.061692 27659 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:28.062247  5543 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:28.064942  5424 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:28.066093 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:28.067860 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.071346  5423 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:28.071903  5423 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:28.072790  5423 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.082157  5425 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:28.083752 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.075150  5397 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:27.945150  5516 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:28.060952 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.094265 27736 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:28.018512 28295 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:28.099144 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 80768us
W20260709 12:22:28.104125 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64608us
I20260709 12:22:28.060809 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:28.115586 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54840.4us
I20260709 12:22:28.120563 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:28.137172 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.147882 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.150256 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.150501 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.150904 27580 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:28.151791 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:28.152105 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 104251us
I20260709 12:22:28.152956 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.152805 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:28.166852 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.230001 27662 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:28.173820 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:28.163918 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.234346 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:28.163995 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.161243 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.235752 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:28.238296 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55670.3us
I20260709 12:22:28.235229 27582 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:28.243198 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.243815 27731 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:28.246654 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.248039 27581 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:22:28.248844 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.249325 27664 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:28.235469 27589 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:22:28.252640 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 99918.2us
I20260709 12:22:28.251785 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:28.259377 27736 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:28.128047  5516 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:22:28.120772  5485 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:28.253384 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:28.262996 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:28.281531 27729 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:28.283175 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.273177 27580 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:28.120795  5498 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:28.145606  5424 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:28.274518 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.276628 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.291759 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.292016 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.292681 27662 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:28.244637  5536 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:28.247812  5627 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:28.298934 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.300324 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.112006  5232 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:28.303304 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 272889us
I20260709 12:22:28.306581 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.298583  5536 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.310374  5536 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:28.310601 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.311380 27589 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:28.276731  5429 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:28.155941  5514 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.162120  5517 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.142s
I20260709 12:22:28.315364 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.286652 27659 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:28.280095  5485 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.244385  5462 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.320726 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.321527 27586 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 19. (term mismatch)
I20260709 12:22:28.330157 27586 pending_rounds.cc:85] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 18
I20260709 12:22:28.330471 27586 pending_rounds.cc:107] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 32.19
I20260709 12:22:28.330806 27586 raft_consensus.cc:2889] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:28.322371 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.323141 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:28.325857 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.331935 27581 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:28.327081 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:28.340719 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57506.5us
I20260709 12:22:28.349731 28329 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:22:28.306140  5643 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:28.364594 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.365126 28331 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:22:28.315603  5543 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:28.366423 27663 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.19-35.19]
I20260709 12:22:28.315212  5429 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.375864 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.378047 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:28.323688  5485 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:28.384271 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 210565us
I20260709 12:22:28.399537 27581 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:28.384109 27627 leader_election.cc:400] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:22:28.397341 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:28.328541  5540 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:28.399575 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.400977 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:28.410969 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.312403  5627 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.309495  5498 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.412263 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:28.421343 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.422014 27656 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:28.412351 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.401093 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.410254 27729 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:22:28.429715 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.429800 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.431310 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.431919 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:28.432727 27581 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:28.432755 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.433987 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.434430 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:28.434803 27581 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:28.413265 27627 leader_election.cc:403] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:22:28.400682 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.366093  5643 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.395246  5429 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:28.394068  5521 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:28.449582 27660 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:28.401819  5507 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:28.452057 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:28.454499 27662 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:28.397868  5444 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:28.383488  5615 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.072s
I20260709 12:22:28.419236  5543 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.419766  5501 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:28.373378  5630 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.072s
I20260709 12:22:28.393721  5424 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:22:28.465845 27581 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:28.421921  5600 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:28.448603  5540 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:28.446122  5643 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:28.370803  5649 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.107s
I20260709 12:22:28.381254  5547 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.075s
I20260709 12:22:28.483834  5644 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:22:28.488540  5500 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:28.489033  5232 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:28.477612 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.496958 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:28.499763 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.501140 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:28.504106  5655 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.365s	user 0.000s	sys 0.001s
W20260709 12:22:28.504419  5661 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.063s	user 0.002s	sys 0.000s
W20260709 12:22:28.510622  5663 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
W20260709 12:22:28.513888 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 94943.9us
W20260709 12:22:28.514958  5574 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.569s	user 0.007s	sys 0.000s
W20260709 12:22:28.515844  5658 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.376s	user 0.002s	sys 0.000s
W20260709 12:22:28.518730  5405 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.000s
I20260709 12:22:28.522616 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:28.524632  5462 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:28.524792 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.526033 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.461901  5507 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.542002 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:28.665186 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.665963 27664 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:28.446396 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87802.1us
W20260709 12:22:28.572101 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.577353 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.588912 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92017.5us
I20260709 12:22:28.461516  5444 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:28.536588  5562 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.670s	user 0.012s	sys 0.000s
W20260709 12:22:28.594848 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 117318us
W20260709 12:22:28.538825  5591 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.001s
I20260709 12:22:28.627385 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.509307  5544 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.539377 27581 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:22:28.566126 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.670486 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:28.586882  5501 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.714217 28328 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:22:28.715935 27736 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:28.599362  5644 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:28.683652 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:28.535384  5543 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.725417 27581 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:28.725816 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:28.690902 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.635694  5232 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:28.553742  5600 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:28.704121 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.739586 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.625952  5615 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:28.719305 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:28.750573 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:28.723129 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:28.752074 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:28.665257  5649 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:28.562675  5540 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:28.730692 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60323.3us
I20260709 12:22:28.765722 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:28.766050 27736 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:22:28.727522 27659 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:28.727013 27663 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:28.731853 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.679175  5667 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:28.588287  5630 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 32.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:28.740406 27662 raft_consensus.cc:2368] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
W20260709 12:22:28.742837 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.740710 27664 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
W20260709 12:22:28.716876  5528 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.695s	user 0.010s	sys 0.002s
I20260709 12:22:28.747756 27733 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:28.755146 27730 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
W20260709 12:22:28.728086  5653 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:22:28.738641  5600 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:28.827309 27661 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:28.828519 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:28.829144 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:28.835068 27663 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:28.746819  5397 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:28.746647  5499 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:28.698740  5562 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.717383  5405 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.106s
I20260709 12:22:28.763748  5593 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:28.722270  5591 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.746894  5553 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:28.827512 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:28.863462 27582 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:22:28.750465  5615 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:28.728331  5501 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.763136  5514 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:28.748776  5232 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:28.763473  5450 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.131s
W20260709 12:22:28.837406  5536 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.000s
W20260709 12:22:28.928946 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101534us
I20260709 12:22:28.964003 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:28.964727 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
W20260709 12:22:28.924703 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 299825us
I20260709 12:22:28.967020 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:28.842130  5179 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.918s	user 0.010s	sys 0.001s
I20260709 12:22:28.769548  5423 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.133s
I20260709 12:22:28.759932  5649 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.824862  5595 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.098s
I20260709 12:22:28.823563  5667 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:28.824662  5462 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:22:29.006688 27656 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.584s	user 0.008s	sys 0.000s
I20260709 12:22:28.893249  5499 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:28.979571  5653 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:22:29.035773 27734 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:28.963842  5630 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:28.836436  5528 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:28.978086  5615 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:29.032842  5627 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:28.996644  5356 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.046856 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.059636 27656 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:29.232745 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4316) took 1082 ms. Trace:
I20260709 12:22:29.059240 27661 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:28.996410  5498 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:29.183096 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.036s	sys 0.063s
W20260709 12:22:29.234232 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.036s	sys 0.063s
W20260709 12:22:29.042440  5425 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.960s	user 0.007s	sys 0.004s
I20260709 12:22:29.034366  5637 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:29.036841  5649 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:29.038522  5683 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.244189 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 280241us
I20260709 12:22:29.233196 27656 rpcz_store.cc:276] 0709 12:22:28.150386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.419002 (+268616us) service_pool.cc:224] Handling call
0709 12:22:29.232666 (+813664us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:28.993353  5536 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.206336  5499 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:29.260556 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.003s	sys 0.007s
I20260709 12:22:29.253978 27552 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:28.860685  5444 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.254060 27583 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:22:29.260848 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:29.258627 27660 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.790s	user 0.008s	sys 0.002s
I20260709 12:22:29.286962 27660 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:22:28.899744 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:29.219727  5630 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.262036 27732 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:22:29.183350  5179 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:29.243932  5424 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
W20260709 12:22:29.261272 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.003s	sys 0.007s
I20260709 12:22:29.220587  5617 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:29.299432 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4314) took 1149 ms. Trace:
I20260709 12:22:29.333403 27662 rpcz_store.cc:276] 0709 12:22:28.149805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.837874 (+688069us) service_pool.cc:224] Handling call
0709 12:22:29.299351 (+461477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:28.979364  5644 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:29.303694 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.061s	sys 0.065s
W20260709 12:22:29.341550 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.061s	sys 0.065s
W20260709 12:22:29.308199 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.773999  5540 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:29.318110 27664 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:28.832921  5547 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:29.344437  5547 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:29.345005  5681 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:22:29.345463  5671 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.303s	user 0.000s	sys 0.000s
I20260709 12:22:29.250049  5462 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:29.345726 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4443) took 1194 ms. Trace:
I20260709 12:22:29.260258  5397 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:29.346848 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4301) took 1221 ms. Trace:
W20260709 12:22:29.347908  5682 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:22:29.315691 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4471) took 1035 ms. Trace:
I20260709 12:22:29.347112  5397 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:29.348850  5664 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.421s	user 0.000s	sys 0.001s
I20260709 12:22:29.348332 27582 rpcz_store.cc:276] 0709 12:22:28.280907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.314665 (+1033758us) service_pool.cc:224] Handling call
0709 12:22:29.315005 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.315220 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.315258 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.315357 (+    99us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:22:29.315406 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.315476 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.315620 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.350052 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4433) took 1201 ms. Trace:
W20260709 12:22:29.344561  5673 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20260709 12:22:29.350382 27588 rpcz_store.cc:276] 0709 12:22:28.148293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.148419 (+   126us) service_pool.cc:224] Handling call
0709 12:22:28.148706 (+   287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.149020 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.149046 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.149555 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.326839 (+177284us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:28.326898 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.326988 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.349877 (+1022889us) raft_consensus.cc:1685] finished
0709 12:22:29.349895 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.350001 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2391,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":176844,"threads_started":1,"wal-append.queue_time_us":562394}
I20260709 12:22:29.346125 27581 rpcz_store.cc:276] 0709 12:22:28.151150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.726174 (+575024us) service_pool.cc:224] Handling call
0709 12:22:28.736423 (+ 10249us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:28.736893 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.736938 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:28.737679 (+   741us) log.cc:844] Serialized 52 byte log entry
0709 12:22:29.262361 (+524682us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:29.262489 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.262690 (+   201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.345501 (+ 82811us) raft_consensus.cc:1685] finished
0709 12:22:29.345524 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.345670 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10895,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":14574,"thread_start_us":424983,"threads_started":1,"wal-append.queue_time_us":574742}
I20260709 12:22:29.285781  5648 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.263211  5498 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:29.305111  5630 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:29.372283 27736 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.003s	sys 0.007s
W20260709 12:22:29.332923  5669 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.493s	user 0.000s	sys 0.001s
I20260709 12:22:29.254616  5356 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.347189 27658 rpcz_store.cc:276] 0709 12:22:28.125430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.125654 (+   224us) service_pool.cc:224] Handling call
0709 12:22:28.245947 (+120293us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:28.246343 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.246363 (+    20us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:28.261490 (+ 15127us) log.cc:844] Serialized 52 byte log entry
0709 12:22:28.562880 (+301390us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:22:28.562937 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.563028 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.346634 (+783606us) raft_consensus.cc:1685] finished
0709 12:22:29.346655 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.346793 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18736,"raft.queue_time_us":9360,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":15290,"thread_start_us":249999,"threads_started":1,"wal-append.queue_time_us":641275}
I20260709 12:22:29.367007 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1819) took 1070 ms. Trace:
I20260709 12:22:29.384671 27581 rpcz_store.cc:276] 0709 12:22:28.296937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.352109 (+1055172us) service_pool.cc:224] Handling call
0709 12:22:29.366382 (+ 14273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.366622 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.366659 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.366711 (+    52us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:29.366748 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.366804 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.366936 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.385859 27662 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:22:29.386686 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4320) took 1235 ms. Trace:
I20260709 12:22:29.387091 27662 rpcz_store.cc:276] 0709 12:22:28.151573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.365242 (+1213669us) service_pool.cc:224] Handling call
0709 12:22:29.385434 (+ 20192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.386362 (+   928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.386612 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.254477  5637 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.314371  5514 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:29.378145 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.352743  5423 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:29.375248  5544 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:29.354552 27730 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28. Ops: [26.18-27.19]
I20260709 12:22:29.347628  5547 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:29.397145 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 145039us
I20260709 12:22:29.373718  5444 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:29.400642  5595 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:29.372997 27736 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.003s	sys 0.008s
W20260709 12:22:29.403641 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.999s	user 0.000s	sys 0.006s
I20260709 12:22:29.372411  5507 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.368829  5553 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:29.410483 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.327168 27659 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 19. (term mismatch)
I20260709 12:22:29.437837 27659 pending_rounds.cc:85] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 18
I20260709 12:22:29.438153 27659 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 34.19
I20260709 12:22:29.438455 27659 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.438959 27659 pending_rounds.cc:107] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 36.20
I20260709 12:22:29.439239 27659 raft_consensus.cc:2889] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.439284 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:29.395895  5450 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:29.264684  5683 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.425547 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3688) took 1048 ms. Trace:
I20260709 12:22:29.330397  5179 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:29.390714  5394 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.008s	sys 0.008s
I20260709 12:22:29.433250 28329 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
W20260709 12:22:29.435433 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.031s	user 0.000s	sys 0.007s
I20260709 12:22:29.445904 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4319) took 1294 ms. Trace:
I20260709 12:22:29.446311 27730 rpcz_store.cc:276] 0709 12:22:28.377157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.306413 (+929256us) service_pool.cc:224] Handling call
0709 12:22:29.354235 (+ 47822us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:29.425218 (+ 70983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.425451 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.413388 27662 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:29.365478  5617 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.446702 27659 rpcz_store.cc:276] 0709 12:22:28.151243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.311496 (+1160253us) service_pool.cc:224] Handling call
0709 12:22:29.311899 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.445628 (+133729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.445847 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.447000 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4307) took 1315 ms. Trace:
I20260709 12:22:29.450851 27659 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:29.431108  5593 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:29.435782 27585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.003s	sys 0.016s
W20260709 12:22:29.453320 27585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.004s	sys 0.017s
I20260709 12:22:29.454489 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3543) took 1107 ms. Trace:
I20260709 12:22:29.454825 27659 rpcz_store.cc:276] 0709 12:22:28.346806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.450349 (+1103543us) service_pool.cc:224] Handling call
0709 12:22:29.450654 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.454288 (+  3634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.454448 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:29.455879  5660 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.698s	user 0.001s	sys 0.001s
W20260709 12:22:29.440968  5668 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.531s	user 0.000s	sys 0.001s
W20260709 12:22:29.457036 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.009s	sys 0.007s
I20260709 12:22:29.353180  5667 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:29.460562 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.010s	sys 0.008s
W20260709 12:22:29.463137 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.374848  5648 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.435212  5562 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.465631 28328 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:22:29.442158  5544 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.468362 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4159) took 1227 ms. Trace:
I20260709 12:22:29.472215 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:29.440632  5232 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.475538  5232 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:22:29.475853  5678 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:22:29.476562  5699 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:29.477129 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4158) took 1236 ms. Trace:
I20260709 12:22:29.425918  5514 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.472456 27738 rpcz_store.cc:276] 0709 12:22:28.241270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.241536 (+   266us) service_pool.cc:224] Handling call
0709 12:22:28.241829 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.260507 (+ 18678us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:28.260540 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.277410 (+ 16870us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.457998 (+180588us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:28.458059 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.458198 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.468159 (+1009961us) raft_consensus.cc:1685] finished
0709 12:22:29.468179 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.468307 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25392,"raft.queue_time_us":21562,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":15541,"thread_start_us":142460,"threads_started":1,"wal-append.queue_time_us":383411}
I20260709 12:22:29.475718  5690 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.472925 28331 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:22:29.464630  5533 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.024s	sys 0.013s
I20260709 12:22:29.488024  5677 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:29.488217 27730 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:29.488544  5677 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:29.488950 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3690) took 1070 ms. Trace:
I20260709 12:22:29.489020  5677 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:29.489339 27730 rpcz_store.cc:276] 0709 12:22:28.419058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.474140 (+1055082us) service_pool.cc:224] Handling call
0709 12:22:29.488035 (+ 13895us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.488714 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.488896 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.449369  5425 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:22:29.455873  5593 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.469722  5501 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.477499 27737 rpcz_store.cc:276] 0709 12:22:28.240974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.241074 (+   100us) service_pool.cc:224] Handling call
0709 12:22:28.252708 (+ 11634us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.279723 (+ 27015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.279756 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.280210 (+   454us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.491142 (+210932us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:28.719166 (+228024us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:28.719464 (+   298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.719820 (+   356us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.476859 (+757039us) raft_consensus.cc:1685] finished
0709 12:22:29.476879 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.477071 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18749,"raft.queue_time_us":27633,"raft.run_cpu_time_us":2759,"raft.run_wall_time_us":208477,"spinlock_wait_cycles":33975040,"thread_start_us":380212,"threads_started":1,"wal-append.queue_time_us":740868}
I20260709 12:22:29.457136  5591 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:29.501032  5485 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.085s	user 0.008s	sys 0.002s
I20260709 12:22:29.501381 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4431) took 1432 ms. Trace:
I20260709 12:22:29.501483  5485 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.501838 27584 rpcz_store.cc:276] 0709 12:22:28.069307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.069772 (+   465us) service_pool.cc:224] Handling call
0709 12:22:28.070120 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.121066 (+ 50946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.121095 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.121558 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.418490 (+296932us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:28.844417 (+425927us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.845146 (+   729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.845258 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.501173 (+655915us) raft_consensus.cc:1685] finished
0709 12:22:29.501201 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.501326 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35772,"prepare.queue_time_us":922359,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":95223,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":34750,"thread_start_us":526619,"threads_started":2,"wal-append.queue_time_us":473073}
I20260709 12:22:29.504010 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:29.505024  5648 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.505308 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:29.506108 27584 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:29.508270  5637 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
W20260709 12:22:29.508632  5516 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.230s	user 0.010s	sys 0.000s
I20260709 12:22:29.509045  5516 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:29.452826  5450 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.509224  5279 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:29.509637  5279 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:29.511670 27738 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:29.426818  5423 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.512285 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3692) took 1038 ms. Trace:
I20260709 12:22:29.521567 27738 rpcz_store.cc:276] 0709 12:22:28.474150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.511200 (+1037050us) service_pool.cc:224] Handling call
0709 12:22:29.511492 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.512047 (+   555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.512233 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.512169 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:29.508315 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4309) took 1374 ms. Trace:
I20260709 12:22:29.514972 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:29.525871 27584 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:29.534416 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:29.533263 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:29.546576 27580 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:29.483357  5667 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:29.587759 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:29.510439  5279 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.523568 27655 rpcz_store.cc:276] 0709 12:22:28.133553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.134036 (+   483us) service_pool.cc:224] Handling call
0709 12:22:28.243005 (+108969us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.334162 (+ 91157us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:28.737667 (+403505us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.326279 (+588612us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.326334 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:29.326380 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.508113 (+181733us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:29.508154 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.508259 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46123,"prepare.queue_time_us":137340,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":99283,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":7127,"spinlock_wait_cycles":209085184,"thread_start_us":915635,"threads_started":2,"wal-append.queue_time_us":733065}
I20260709 12:22:29.511049  5450 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:29.487282  5533 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.024s	sys 0.013s
I20260709 12:22:29.589638 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:29.595417 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4475) took 1283 ms. Trace:
W20260709 12:22:29.529744  5521 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.073s	user 0.010s	sys 0.001s
I20260709 12:22:29.644160 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:29.595748 27729 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.587s	user 0.009s	sys 0.000s
I20260709 12:22:29.595764 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:29.603210 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:29.446285  5595 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.528278  5405 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:29.591810  5672 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.454s	user 0.001s	sys 0.000s
I20260709 12:22:29.659075 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:29.663090 27584 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:29.665685  5356 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:29.666260  5450 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:29.667698  5520 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.668255  5520 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:29.670077 27661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4317) took 1519 ms (client timeout 2023 ms). Trace:
W20260709 12:22:29.670469 27661 rpcz_store.cc:269] 0709 12:22:28.150661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.361894 (+211233us) service_pool.cc:224] Handling call
0709 12:22:29.669903 (+1308009us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:29.670019 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":756531456}
I20260709 12:22:29.671916 27661 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:29.673043 27664 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:29.674978 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3549) took 1156 ms. Trace:
I20260709 12:22:29.676651  5405 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.678869 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4462) took 1573 ms. Trace:
I20260709 12:22:29.601977  5637 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:29.680191  5536 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.679263 27587 rpcz_store.cc:276] 0709 12:22:28.105994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.145953 (+ 39959us) service_pool.cc:224] Handling call
0709 12:22:28.245192 (+ 99239us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.275262 (+ 30070us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.275308 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.296610 (+ 21302us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.553366 (+256756us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:28.553432 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.553530 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.678629 (+1125099us) raft_consensus.cc:1685] finished
0709 12:22:29.678650 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.678809 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14855,"raft.queue_time_us":29729,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":14939,"thread_start_us":217195,"threads_started":1,"wal-append.queue_time_us":523293}
I20260709 12:22:29.679824  5637 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.527098  5500 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.096s
I20260709 12:22:29.681439 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:29.682041 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:29.682876  5637 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:29.589845 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4323) took 1437 ms (client timeout 1807 ms). Trace:
I20260709 12:22:29.619220 27581 rpcz_store.cc:276] 0709 12:22:28.311995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.385987 (+1073992us) service_pool.cc:224] Handling call
0709 12:22:29.389362 (+  3375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.389593 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:29.431828 (+ 42235us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.595079 (+163251us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.595130 (+    51us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:29.595186 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.595244 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.595362 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7389,"prepare.queue_time_us":37678,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":3874,"raft.queue_time_us":23567,"thread_start_us":148236,"threads_started":1}
W20260709 12:22:29.617004 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.483455  5544 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:29.699307 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.000s	sys 0.017s
I20260709 12:22:29.590226  5498 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:29.643661  5707 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.654327 27552 leader_election.cc:400] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:29.592947  5528 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:29.713088 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58953.9us
I20260709 12:22:29.660173  5686 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:29.623855  5507 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.684099  5688 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:22:29.684579 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:29.685307  5708 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 12:22:29.678248  5673 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.199s	user 0.003s	sys 0.000s
W20260709 12:22:29.685585 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.098s	user 0.027s	sys 0.040s
W20260709 12:22:29.723678 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.156s	user 0.027s	sys 0.040s
I20260709 12:22:29.605435 27730 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:22:29.671921 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1487) took 1138 ms. Trace:
I20260709 12:22:29.686946 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4477) took 1315 ms. Trace:
I20260709 12:22:29.692008 27661 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:29.690271 27664 rpcz_store.cc:276] 0709 12:22:28.518605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.494082 (+975477us) service_pool.cc:224] Handling call
0709 12:22:29.672877 (+178795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.673424 (+   547us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.674784 (+  1360us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400210a50. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:29.674919 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":410499840}
I20260709 12:22:29.626336  5279 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:29.689802 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:29.698510 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:29.482227  5630 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.700845 27655 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
W20260709 12:22:29.702841 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.000s	sys 0.017s
W20260709 12:22:29.696611 27660 rpcz_store.cc:269] 0709 12:22:28.152446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.322912 (+170466us) service_pool.cc:224] Handling call
0709 12:22:29.589688 (+1266776us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54001e5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:29.589789 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268668672}
W20260709 12:22:29.704382 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.010s	sys 0.034s
W20260709 12:22:29.735108 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.011s	sys 0.034s
I20260709 12:22:29.735251 27737 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:29.736043 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 769058us
I20260709 12:22:29.511284  5690 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.651891 27729 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:29.649648  5644 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:29.717581 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:29.740849 27581 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:29.624491  5501 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.084s
I20260709 12:22:29.741214 27664 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:29.670469  5520 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.726523 27584 rpcz_store.cc:276] 0709 12:22:28.371448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.659089 (+1287641us) service_pool.cc:224] Handling call
0709 12:22:29.659483 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.686567 (+ 27084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.686880 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.725095 28329 rpcz_store.cc:276] 0709 12:22:28.533678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.533874 (+   196us) service_pool.cc:224] Handling call
0709 12:22:28.534223 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.448068 (+913845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.671685 (+223617us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400962654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:29.671840 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066970368}
I20260709 12:22:29.703886  5498 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:29.743752 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:29.707811  5356 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:29.748889 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:29.685926  5521 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:29.720052  5536 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.752200 27584 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:29.749368 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:29.714160 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:29.788347 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74305us
I20260709 12:22:29.715746  5695 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:29.601198  5705 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.150s
W20260709 12:22:29.759362 27583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4438) took 1609 ms (client timeout 2040 ms). Trace:
I20260709 12:22:29.726445  5591 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.751750 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4465) took 1634 ms. Trace:
I20260709 12:22:29.789474 27581 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:22:29.650879  5553 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.800472  5517 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:29.801873  5703 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.003s
I20260709 12:22:29.805439  5703 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:29.805853  5703 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:29.810025  5648 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:29.810817  5648 consensus_queue.cc:1243] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:22:29.721102  5707 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.813438  5707 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.814121  5707 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:29.815018  5462 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:29.815418  5462 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:29.816461 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4161) took 1538 ms. Trace:
I20260709 12:22:29.816186  5462 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.817332  5462 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:29.816835 27731 rpcz_store.cc:276] 0709 12:22:28.278475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.282935 (+  4460us) service_pool.cc:224] Handling call
0709 12:22:28.283252 (+   317us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:28.283793 (+   541us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.283831 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:28.314999 (+ 31168us) log.cc:844] Serialized 52 byte log entry
0709 12:22:28.600564 (+285565us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:28.661939 (+ 61375us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.683518 (+ 21579us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.713373 (+ 29855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.713485 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.489004 (+775519us) spinlock_profiling.cc:243] Waited 5.52 ms on lock 0x7b18004bfc40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b557 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:29.677979 (+188975us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b24002c5aa8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:22:29.816238 (+138259us) raft_consensus.cc:1685] finished
0709 12:22:29.816260 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.816406 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32761,"prepare.queue_time_us":61784,"prepare.run_wall_time_us":43,"raft.queue_time_us":87617,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":48194,"spinlock_wait_cycles":118723584,"thread_start_us":222043,"threads_started":1,"wal-append.queue_time_us":560805}
I20260709 12:22:29.791738 27585 rpcz_store.cc:276] 0709 12:22:28.117209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.117345 (+   136us) service_pool.cc:224] Handling call
0709 12:22:28.237266 (+119921us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.282925 (+ 45659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.282955 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.298313 (+ 15358us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.636074 (+337761us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b18004291c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:29.453966 (+817892us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:29.742590 (+288624us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.742887 (+   297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.742955 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.742991 (+    36us) raft_consensus.cc:1685] finished
0709 12:22:29.751556 (+  8565us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.751688 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62801,"prepare.queue_time_us":17586,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":56316,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":21938,"spinlock_wait_cycles":28165888,"thread_start_us":1073266,"threads_started":2,"wal-append.queue_time_us":1174560}
I20260709 12:22:29.820971  5690 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.793988  5423 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.749511  5503 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:29.822885  5503 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.823393  5503 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:29.826609  5429 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.370s	user 0.008s	sys 0.001s
I20260709 12:22:29.794620  5521 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:29.755457  5644 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.827556  5521 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.828081  5521 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:29.828492  5485 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.829027  5644 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:29.796720 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:29.829310  5516 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:29.829733  5516 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:29.830029 28332 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 18.
I20260709 12:22:29.830096  5516 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:29.831228 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:29.791461 27583 rpcz_store.cc:269] 0709 12:22:28.149877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.150260 (+   383us) service_pool.cc:224] Handling call
0709 12:22:29.759147 (+1608887us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000889d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:29.759290 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2523871104}
I20260709 12:22:29.833173 27583 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:29.833698 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4484) took 1340 ms. Trace:
I20260709 12:22:29.834004 27583 rpcz_store.cc:276] 0709 12:22:28.492945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.832591 (+1339646us) service_pool.cc:224] Handling call
0709 12:22:29.832892 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.833489 (+   597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.833649 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.729295  5507 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.802907  5517 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.837092  5507 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.835232 27583 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:22:29.838105 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4485) took 1344 ms. Trace:
I20260709 12:22:29.838430 27583 rpcz_store.cc:276] 0709 12:22:28.493261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.834684 (+1341423us) service_pool.cc:224] Handling call
0709 12:22:29.834951 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.837866 (+  2915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.838045 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.811246  5648 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:22:29.839247  5648 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:29.839740 27583 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 17. (term mismatch)
I20260709 12:22:29.840093 27583 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 16
I20260709 12:22:29.840292 27583 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 30.17
I20260709 12:22:29.840648 27583 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.840314  5648 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.841619 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4486) took 1154 ms. Trace:
I20260709 12:22:29.842334 27583 rpcz_store.cc:276] 0709 12:22:28.686817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.839132 (+1152315us) service_pool.cc:224] Handling call
0709 12:22:29.839433 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.841405 (+  1972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.841570 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.842694  5648 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:29.843839 27583 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:22:29.850420 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:29.792232 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:29.852577 27730 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:29.852669 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4487) took 1099 ms. Trace:
I20260709 12:22:29.857112 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:29.857187 27583 rpcz_store.cc:276] 0709 12:22:28.753348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.843270 (+1089922us) service_pool.cc:224] Handling call
0709 12:22:29.843556 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.852437 (+  8881us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.852620 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:29.756287  5600 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.873s	user 0.006s	sys 0.005s
I20260709 12:22:29.858649 27587 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:22:29.856406  5686 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:29.864908 27582 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.010s	sys 0.029s
W20260709 12:22:29.866370 27582 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.010s	sys 0.029s
I20260709 12:22:29.866704 27552 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30.
I20260709 12:22:29.837574  5517 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:29.880631 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.884941 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:29.886308 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:29.888494 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.861435  5423 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.760040  5498 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:29.907271 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:29.864312  5507 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:29.867369  5462 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.919747 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 176075us
I20260709 12:22:29.921537 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:29.827196  5429 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.909752 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1781) took 1393 ms. Trace:
W20260709 12:22:29.915699 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 166424us
W20260709 12:22:29.919126 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 165068us
W20260709 12:22:29.959118 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.960690 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:29.961227 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54032.5us
I20260709 12:22:29.961467 27661 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:29.954219 27659 rpcz_store.cc:276] 0709 12:22:28.516254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.491401 (+975147us) service_pool.cc:224] Handling call
0709 12:22:29.491684 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.491892 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:29.492163 (+   271us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.909377 (+417214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.909433 (+    56us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:29.909486 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.909562 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.909691 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21811,"prepare.queue_time_us":559,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"thread_start_us":395028,"threads_started":1}
I20260709 12:22:29.791395  5356 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.806809  5703 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:29.758548  5637 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:29.973377 27584 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:22:30.000092 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4479) took 1538 ms. Trace:
I20260709 12:22:30.000476 27584 rpcz_store.cc:276] 0709 12:22:28.461343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.751703 (+1290360us) service_pool.cc:224] Handling call
0709 12:22:29.752067 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.999800 (+247733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.000036 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.001464 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:29.974051 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.002743 27738 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:29.858234  5536 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.866254  5690 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.069s
I20260709 12:22:29.861984  5591 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.908811  5517 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:29.937579  5700 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:29.920495  5699 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:29.868151  5600 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.012905 27730 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:29.970650  5179 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.922353  5514 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:29.973412  5682 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:29.937834  5617 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.859222  5695 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.975639  5232 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:30.025933 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.013520 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:30.045926 27584 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:30.033700 27661 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:29.977289  5683 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:29.874609  5686 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:29.953241  5423 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.055734 28328 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:30.041889 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1820) took 1180 ms. Trace:
I20260709 12:22:29.963881  5462 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:30.054409 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.043605  5711 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:22:29.963570  5507 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.063381 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.073623 27730 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:30.058151 27583 rpcz_store.cc:276] 0709 12:22:28.861814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.858197 (+996383us) service_pool.cc:224] Handling call
0709 12:22:29.858485 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.858683 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:29.861135 (+  2452us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.041473 (+180338us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.041533 (+    60us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:30.041585 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.041703 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.041834 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5808,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":5571,"thread_start_us":177881,"threads_started":1}
I20260709 12:22:30.065744 27738 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:30.035980  5637 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:30.075232 27583 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:30.077359 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1821) took 1215 ms. Trace:
I20260709 12:22:30.043846  5514 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:30.077944 27583 rpcz_store.cc:276] 0709 12:22:28.862166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.074703 (+1212537us) service_pool.cc:224] Handling call
0709 12:22:30.075027 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.076662 (+  1635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.077268 (+   606us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:30.060329  5677 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.571s	user 0.011s	sys 0.000s
I20260709 12:22:30.038519  5425 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:22:30.060700  5595 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.101080  5507 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.074087  5539 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:22:30.103981  5499 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.801s	user 0.007s	sys 0.004s
I20260709 12:22:30.095906  5514 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.078235  5544 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.104900  5650 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:30.134590 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.127453 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:30.105407  5536 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.210467 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.096175  5677 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.108961 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4313) took 1959 ms. Trace:
I20260709 12:22:30.112457  5713 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.192s
I20260709 12:22:30.114370  5600 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.114171 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.116039 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.277319 27731 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:30.265669  5593 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:22:29.450510 27657 rpcz_store.cc:276] 0709 12:22:28.131144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.131625 (+   481us) service_pool.cc:224] Handling call
0709 12:22:28.131965 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.132200 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:22:28.403638 (+271438us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.446573 (+1042935us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.446648 (+    75us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:29.446708 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.446799 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.446935 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":170534,"prepare.queue_time_us":165903,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":1343,"thread_start_us":1213800,"threads_started":2}
I20260709 12:22:30.280975 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.281646 27657 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 21.
I20260709 12:22:30.282678 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:30.267710 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140301us
I20260709 12:22:30.283401 27657 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:30.136955  5685 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.549s	user 0.000s	sys 0.000s
I20260709 12:22:30.139523 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:30.313488 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 174041us
I20260709 12:22:30.314328 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:30.156569 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.164180 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:30.238857 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 269332us
I20260709 12:22:30.315210 27664 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:30.315656 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:30.153334 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.316416 27660 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:30.118417 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4478) took 1697 ms. Trace:
I20260709 12:22:30.318554 27587 rpcz_store.cc:276] 0709 12:22:28.420844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.682645 (+1261801us) service_pool.cc:224] Handling call
0709 12:22:29.682959 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.862278 (+179319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.118127 (+255849us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.118345 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381556992}
I20260709 12:22:30.133420 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1817) took 1842 ms. Trace:
I20260709 12:22:30.234701 27584 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:30.251955 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4483) took 1759 ms. Trace:
I20260709 12:22:30.267503 27656 rpcz_store.cc:276] 0709 12:22:28.149498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.267767 (+1118269us) service_pool.cc:224] Handling call
0709 12:22:29.313167 (+ 45400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.313442 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:29.371033 (+ 57591us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.465013 (+ 93980us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.465079 (+    66us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:29.465134 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.465218 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.108774 (+643556us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b5400210550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:30.108899 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71437,"prepare.queue_time_us":46060,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":2298,"raft.queue_time_us":1024,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":13283,"spinlock_wait_cycles":103375744,"thread_start_us":93261,"threads_started":1,"wal-append.queue_time_us":343694}
I20260709 12:22:30.319473 27582 rpcz_store.cc:276] 0709 12:22:28.290849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.349672 (+1058823us) service_pool.cc:224] Handling call
0709 12:22:29.350003 (+   331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:29.350354 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.350385 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:29.350937 (+   552us) log.cc:844] Serialized 52 byte log entry
0709 12:22:29.866735 (+515798us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:29.866763 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.866811 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.133161 (+266350us) raft_consensus.cc:1685] finished
0709 12:22:30.133191 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.133347 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1191,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":10218,"thread_start_us":515430,"threads_started":1,"wal-append.queue_time_us":740920}
I20260709 12:22:30.161870  5601 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.129678  5695 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.274662 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:30.116384  5540 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.744s	user 0.010s	sys 0.000s
W20260709 12:22:30.146312  5662 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.420s	user 0.000s	sys 0.007s
W20260709 12:22:30.204965  5643 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.569s	user 0.004s	sys 0.008s
W20260709 12:22:30.122397  5444 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.685s	user 0.006s	sys 0.002s
W20260709 12:22:30.158496  5615 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.897s	user 0.007s	sys 0.002s
I20260709 12:22:30.155300 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:30.092149  5649 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.792s	user 0.001s	sys 0.009s
I20260709 12:22:30.287895 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:30.322152 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 158057us
I20260709 12:22:30.329492  5713 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:30.333624  5600 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:30.333885  5562 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.323546 27660 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:22:30.320184 27585 rpcz_store.cc:276] 0709 12:22:28.492629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.820642 (+1328013us) service_pool.cc:224] Handling call
0709 12:22:29.821046 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.821303 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:29.821697 (+   394us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.251560 (+429863us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.251621 (+    61us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:30.251668 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.251747 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.251882 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1123,"prepare.queue_time_us":696,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":95215,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":7463,"thread_start_us":427426,"threads_started":2,"wal-append.queue_time_us":345254}
I20260709 12:22:30.161949  5630 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.168797  5517 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.390875 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.392125 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.392928 27585 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 35 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:22:30.393633 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:30.130957  5462 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.117499  5405 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.152973  5686 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.170260  5429 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.332912 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.403723 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:30.334460 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.128019  5423 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.267727  5539 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:30.339663 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:30.337917  5601 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.412179 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4436) took 2263 ms. Trace:
I20260709 12:22:30.412786 27738 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:30.284881  5667 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:30.413546  5667 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:30.414693  5601 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.265618  5650 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:30.342423  5697 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.066s
I20260709 12:22:30.412547 27586 rpcz_store.cc:276] 0709 12:22:28.149288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.532260 (+382972us) service_pool.cc:224] Handling call
0709 12:22:28.532681 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.571843 (+ 39162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.571875 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.572371 (+   496us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.037966 (+465595us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:29.038020 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.038088 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.136169 (+1098081us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:30.411857 (+275688us) raft_consensus.cc:1685] finished
0709 12:22:30.411883 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.412121 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38663,"raft.queue_time_us":39275,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":9854,"spinlock_wait_cycles":7139200,"thread_start_us":375094,"threads_started":1,"wal-append.queue_time_us":854929}
I20260709 12:22:30.416359 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.415858  5650 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.417084 27586 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:30.263182  5179 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.075s
I20260709 12:22:30.412498 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.116950  5516 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.335175  5528 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:30.329715  5628 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.374656 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.329986  5536 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:30.330250  5593 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:30.360113  5540 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.235055  5617 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.106s
I20260709 12:22:30.376822 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.446583 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:30.377601 27655 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 18.
I20260709 12:22:30.417198  5650 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:30.166903  5614 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.171s
I20260709 12:22:30.380980  5726 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.383512 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.343245  5647 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.265247  5499 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.380757  5649 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.355795  5507 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.376118  5503 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.000s
I20260709 12:22:30.407537 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.414461  5667 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.209511  5450 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.123s
I20260709 12:22:30.419086 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.320941  5521 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.375385  5356 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.381982  5444 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.409168  5731 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.297s	user 0.002s	sys 0.000s
W20260709 12:22:30.444793 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 482591us
I20260709 12:22:30.481869 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:30.452705  5593 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:30.322186 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:30.448750 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.391028  5517 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.373489  5643 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:30.449368 28335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.000s	sys 0.027s
I20260709 12:22:30.450357 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.422781  5686 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.451782 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:30.416827  5539 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.128552  5637 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.459962 27658 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:30.402767  5562 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.381742  5615 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.462858 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:30.446971  5528 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:30.471609 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.387995  5713 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.479867 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.447957 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.451341  5628 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.455046  5737 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:30.458693  5726 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:30.628051 28335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.011s	sys 0.048s
I20260709 12:22:30.503669 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:30.628924 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 306793us
W20260709 12:22:30.629285 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.629833 27586 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:30.501617 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:30.489912 28336 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.20-28.20]
I20260709 12:22:30.551307 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:30.457733  5536 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:30.489447 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.631989 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:30.628304 27730 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:30.462275  5232 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:30.629921 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
W20260709 12:22:30.637301 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 184817us
W20260709 12:22:30.468125  5705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.016s	sys 0.037s
W20260709 12:22:30.483345  5648 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.619s	user 0.010s	sys 0.000s
W20260709 12:22:30.520138  5498 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.552s	user 0.010s	sys 0.000s
I20260709 12:22:30.655903 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:30.680819 27655 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:22:30.465792  5647 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.501401  5601 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.475625  5503 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.474936  5507 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.632669  5614 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:30.653895  5739 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:30.668164 27589 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: [30.17-30.17]
I20260709 12:22:30.694221  5739 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:30.509764  5521 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.694654  5739 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:30.652477  5705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.016s	sys 0.037s
I20260709 12:22:30.665889 27584 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:30.675200 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.704586 27659 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:22:30.678015 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 176498us
W20260709 12:22:30.684964 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.665305 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:30.511565  5514 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.705763 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:30.708141 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.679548  5650 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.724162 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:30.723698 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.656489  5517 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.656597  5649 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.691670  5540 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.501958  5735 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:22:30.651461  5539 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:30.704182  5736 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.716557 27657 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:22:30.721690 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:30.500685  5677 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.649492  5742 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:30.493185  5444 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.725577 27586 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:30.502247  5423 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.201s
I20260709 12:22:30.702481  5648 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.692096  5498 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.780786  5648 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.781415 27664 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:22:30.782760  5498 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.783257  5648 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.773473  5736 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.762537  5507 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.787695  5736 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.788626 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.789363 27582 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:30.790555 27582 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
W20260709 12:22:30.791014 27580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1812) took 2518 ms (client timeout 1913 ms). Trace:
I20260709 12:22:30.791316 27583 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
W20260709 12:22:30.791514 27580 rpcz_store.cc:269] 0709 12:22:28.272595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.300113 (+ 27518us) service_pool.cc:224] Handling call
0709 12:22:30.790826 (+2490713us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:30.790947 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2779928448}
I20260709 12:22:30.792738 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1822) took 1416 ms. Trace:
I20260709 12:22:30.793179 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1824) took 1385 ms. Trace:
W20260709 12:22:30.795313  5651 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 2.053s	user 0.004s	sys 0.007s
I20260709 12:22:30.796469 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:30.797780  5716 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:30.797899 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.798283  5716 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:30.798677 27655 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:30.790563 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.799625 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:30.800592 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.801242 27660 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:30.801841 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1595) took 2924 ms. Trace:
I20260709 12:22:30.803627 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:30.802285 28333 rpcz_store.cc:276] 0709 12:22:27.877744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.877897 (+   153us) service_pool.cc:224] Handling call
0709 12:22:27.878236 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.878626 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:27.878947 (+   321us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.283395 (+404448us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.298805 (+ 15410us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.298946 (+   141us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:28.298984 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.299050 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.667608 (+2368558us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:30.801530 (+133922us) raft_consensus.cc:1685] finished
0709 12:22:30.801644 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.801781 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14257,"prepare.queue_time_us":662,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":9944,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":3959,"spinlock_wait_cycles":29829376,"thread_start_us":395461,"threads_started":1,"wal-append.queue_time_us":688038}
I20260709 12:22:30.804890 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:30.793090 27583 rpcz_store.cc:276] 0709 12:22:29.376323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.095501 (+719178us) service_pool.cc:224] Handling call
0709 12:22:30.790949 (+695448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.791747 (+   798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.792542 (+   795us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b5400188dd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:30.792673 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1598450560}
I20260709 12:22:30.806360 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.798081 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1823) took 1390 ms. Trace:
I20260709 12:22:30.806813  5647 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.654835  5450 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260709 12:22:30.663879  5617 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.064s
I20260709 12:22:30.701663  5516 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.736642 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.639249  5686 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.639112  5737 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.762529  5726 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.763094  5593 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.726670  5547 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:22:30.729508  5697 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:22:30.730106  5405 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:22:30.731957  5650 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.765765 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1601) took 1079 ms. Trace:
I20260709 12:22:30.723678  5232 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:30.726397  5614 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:30.681308 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.726119  5536 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.802593 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:30.793639 27580 rpcz_store.cc:276] 0709 12:22:29.408105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.792403 (+1384298us) service_pool.cc:224] Handling call
0709 12:22:30.792676 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.792859 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.792887 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.792929 (+    42us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:22:30.792970 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.793025 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.793130 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.807184 27661 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:30.808079 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1644) took 2722 ms. Trace:
I20260709 12:22:30.809916 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4439) took 2659 ms. Trace:
I20260709 12:22:30.813287 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:30.788681 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1490) took 1046 ms. Trace:
W20260709 12:22:30.729089 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.848575 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:30.854656 27655 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:30.852085 28334 rpcz_store.cc:276] 0709 12:22:29.686289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.686443 (+   154us) service_pool.cc:224] Handling call
0709 12:22:29.686725 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.687083 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.687108 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.694328 (+  7220us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.148298 (+453970us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b18003fb680. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:30.631923 (+483625us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:30.632002 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.632063 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.765383 (+133320us) raft_consensus.cc:1685] finished
0709 12:22:30.765407 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.765679 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9750,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":10919,"spinlock_wait_cycles":6951168,"thread_start_us":483464,"threads_started":1,"wal-append.queue_time_us":1011104}
I20260709 12:22:30.851169 27589 rpcz_store.cc:276] 0709 12:22:28.150101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.597389 (+447288us) service_pool.cc:224] Handling call
0709 12:22:28.661743 (+ 64354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.694567 (+2032824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.809627 (+115060us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.809840 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4613720320}
I20260709 12:22:30.814846 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.850463 27732 rpcz_store.cc:276] 0709 12:22:28.085401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.085945 (+   544us) service_pool.cc:224] Handling call
0709 12:22:28.086366 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.297587 (+1211221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.807741 (+1510154us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54002e9fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.807982 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2703072128}
I20260709 12:22:30.703820  5601 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:30.860195 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47637.6us
I20260709 12:22:30.853705 27660 pending_rounds.cc:85] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 16
I20260709 12:22:30.857738 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.862219 27660 pending_rounds.cc:107] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 20.17
I20260709 12:22:30.862628 27660 raft_consensus.cc:2889] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:30.854138 28337 rpcz_store.cc:276] 0709 12:22:29.742352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.742524 (+   172us) service_pool.cc:224] Handling call
0709 12:22:29.785686 (+ 43162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.063680 (+277994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.063710 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.063751 (+    41us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:30.063782 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.788443 (+724661us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400520dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.788506 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.788626 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":638841600}
I20260709 12:22:30.863277 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.815418 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3541) took 2470 ms. Trace:
I20260709 12:22:30.858055 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:30.866138 27662 rpcz_store.cc:276] 0709 12:22:28.344751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.395234 (+1050483us) service_pool.cc:224] Handling call
0709 12:22:29.395602 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.449692 (+ 54090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.815150 (+1365458us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.815357 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40392576}
I20260709 12:22:30.856998 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:30.807273 27582 rpcz_store.cc:276] 0709 12:22:29.407698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.790060 (+1382362us) service_pool.cc:224] Handling call
0709 12:22:30.790382 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.797819 (+  7437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.798026 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.868854 27580 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:22:30.869585 27732 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:30.869993 27580 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:30.858464 27731 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
W20260709 12:22:30.872466 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 150862us
I20260709 12:22:30.777804  5735 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:30.777024  5499 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.874977 27580 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:22:30.870004 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3695) took 1051 ms. Trace:
I20260709 12:22:30.733862  5544 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.124s
I20260709 12:22:30.814797  5503 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.880187 27732 rpcz_store.cc:276] 0709 12:22:29.818933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.862977 (+1044044us) service_pool.cc:224] Handling call
0709 12:22:30.869951 (+  6974us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.875995 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.809128  5725 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:30.877446 27655 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:22:30.873606  5614 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:30.816121  5643 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.817188  5628 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.893507 27659 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:30.883993 27730 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:30.889986 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:30.875911  5405 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:30.881781 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.892069  5517 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:22:30.880633  5232 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:30.893765  5429 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:30.875602  5734 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:30.881790 27731 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:22:30.901907 27663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.012s	sys 0.033s
W20260709 12:22:30.913334 27663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.012s	sys 0.033s
I20260709 12:22:30.909694 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1493) took 1149 ms. Trace:
I20260709 12:22:30.914757 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:30.874390  5697 consensus_queue.cc:1243] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 32.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:22:30.904771  5714 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.869s	user 0.003s	sys 0.000s
I20260709 12:22:30.903888  5735 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.907855  5232 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.913934 28330 rpcz_store.cc:276] 0709 12:22:29.760346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.760619 (+   273us) service_pool.cc:224] Handling call
0709 12:22:29.815688 (+ 55069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.850105 (+ 34417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.850140 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.850179 (+    39us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:29.850215 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.909465 (+1059250us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.909509 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.909628 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78597888}
I20260709 12:22:30.908542  5429 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.902750  5405 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.908849  5734 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:30.927209  5734 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:30.928020 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3555) took 1110 ms. Trace:
I20260709 12:22:30.928416 27657 rpcz_store.cc:276] 0709 12:22:29.817208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.631513 (+814305us) service_pool.cc:224] Handling call
0709 12:22:30.674201 (+ 42688us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.738430 (+ 64229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.927771 (+189341us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:30.927962 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96027008}
I20260709 12:22:30.868624  5536 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.929898 27657 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.19-35.19]
I20260709 12:22:30.931200  5536 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.931491  5517 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:30.916275 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:30.815164  5528 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:30.904430 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1652) took 1255 ms. Trace:
I20260709 12:22:30.931250 27657 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:22:30.936125 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.941270 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1647) took 2589 ms. Trace:
I20260709 12:22:30.941632 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.940359 27732 rpcz_store.cc:276] 0709 12:22:29.648763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.881554 (+1232791us) service_pool.cc:224] Handling call
0709 12:22:30.904363 (+ 22809us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.941812 27736 rpcz_store.cc:276] 0709 12:22:28.351502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.768301 (+416799us) service_pool.cc:224] Handling call
0709 12:22:28.768821 (+   520us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:29.448515 (+679694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.448562 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:29.449175 (+   613us) log.cc:844] Serialized 52 byte log entry
0709 12:22:29.723509 (+274334us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:29.731351 (+  7842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.731426 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.904161 (+1172735us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:30.932361 (+ 28200us) raft_consensus.cc:1685] finished
0709 12:22:30.932390 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.941181 (+  8791us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64501,"raft.queue_time_us":263771,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":1868,"spinlock_wait_cycles":7316736,"thread_start_us":857909,"threads_started":2,"wal-append.queue_time_us":482104}
I20260709 12:22:30.944480 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.945209 27736 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:30.944770 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.947772 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.934166 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:30.952764 27733 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:30.950587 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:30.957114 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.795s (SENT) [suppressed 5 similar messages]
I20260709 12:22:30.960564 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.961452 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:30.959615  5615 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:30.963195 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.963491 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:30.963791 27732 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:30.963917  5737 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.964608 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.965202 27731 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:30.965554 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:30.965961 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:30.889397  5462 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:30.966346  5739 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:30.901784  5725 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:30.968427 27734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3691) took 2502 ms (client timeout 1795 ms). Trace:
W20260709 12:22:30.983861 27734 rpcz_store.cc:269] 0709 12:22:28.450508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.500869 (+ 50361us) service_pool.cc:224] Handling call
0709 12:22:30.952194 (+2451325us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54002e90d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:30.952392 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023228672}
I20260709 12:22:30.967262 27731 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:30.936250  5643 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:30.987823 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4171) took 1458 ms. Trace:
I20260709 12:22:30.988368 27731 rpcz_store.cc:276] 0709 12:22:29.529198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.966813 (+1437615us) service_pool.cc:224] Handling call
0709 12:22:30.967075 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.987505 (+ 20430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.987762 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.989979 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1648) took 1389 ms. Trace:
I20260709 12:22:30.990371 27731 rpcz_store.cc:276] 0709 12:22:29.600866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.989220 (+1388354us) service_pool.cc:224] Handling call
0709 12:22:30.989510 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.989679 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.989705 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.989746 (+    41us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:30.989784 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.989836 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.989931 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.991369 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:30.992964 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1655) took 1138 ms. Trace:
I20260709 12:22:30.994541 27736 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:30.997581 27732 rpcz_store.cc:276] 0709 12:22:29.854981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.962953 (+1107972us) service_pool.cc:224] Handling call
0709 12:22:30.992893 (+ 29940us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.998786 27657 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:31.000138 27732 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: [30.22-30.22]
I20260709 12:22:31.008538 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1651) took 1365 ms. Trace:
I20260709 12:22:31.010121 27732 rpcz_store.cc:276] 0709 12:22:29.643595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.999488 (+1355893us) service_pool.cc:224] Handling call
0709 12:22:30.999854 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.008084 (+  8230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.008461 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.012647 27732 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:30.993273 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1649) took 1372 ms. Trace:
I20260709 12:22:30.947827  5562 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:30.951742  5687 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:31.027424 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:30.960259  5677 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:31.028388 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3693) took 1196 ms. Trace:
I20260709 12:22:31.029099 27734 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:31.041105 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4175) took 1441 ms. Trace:
I20260709 12:22:31.040230 27732 rpcz_store.cc:276] 0709 12:22:29.818301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.012039 (+1193738us) service_pool.cc:224] Handling call
0709 12:22:31.012416 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.014392 (+  1976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.014624 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.996049  5628 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:31.041555 27734 rpcz_store.cc:276] 0709 12:22:29.599830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.984712 (+1384882us) service_pool.cc:224] Handling call
0709 12:22:31.028887 (+ 44175us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.040775 (+ 11888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.041035 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.042470 27732 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:22:31.042186  5628 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.043048 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3696) took 1223 ms. Trace:
I20260709 12:22:30.967051  5462 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.043395 27732 rpcz_store.cc:276] 0709 12:22:29.819203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.041851 (+1222648us) service_pool.cc:224] Handling call
0709 12:22:31.042146 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.042830 (+   684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.043003 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.044425  5628 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:30.967713  5516 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:31.045717  5462 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.025940  5713 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.054924  5725 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.056358  5725 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:31.033528 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 295798us
I20260709 12:22:31.025449 27731 rpcz_store.cc:276] 0709 12:22:29.621363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.992506 (+1371143us) service_pool.cc:224] Handling call
0709 12:22:30.992778 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.992947 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.992974 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.993016 (+    42us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:30.993056 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.993111 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.993223 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.024405  5536 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:31.079653 27734 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:22:31.068388 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3554) took 1251 ms. Trace:
I20260709 12:22:31.047253  5677 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.094244 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.095448 27659 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 35 because replica has last-logged OpId of term: 33 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:22:31.096755 27659 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:31.083094  5739 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.078223  5444 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.049015  5687 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.093621  5677 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:31.131979 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1492) took 1380 ms. Trace:
I20260709 12:22:31.132539 28329 rpcz_store.cc:276] 0709 12:22:29.751439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.752099 (+   660us) service_pool.cc:224] Handling call
0709 12:22:29.752348 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.024181 (+271833us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.024228 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.024285 (+    57us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:30.024331 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.131626 (+1107295us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b54006bc6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.131692 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.131899 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624575872}
I20260709 12:22:31.134686 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:31.136173 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:31.138089 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.138551 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1518) took 1559 ms. Trace:
W20260709 12:22:31.140453 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.102025  5650 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.139078 28328 rpcz_store.cc:276] 0709 12:22:29.579662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.579807 (+   145us) service_pool.cc:224] Handling call
0709 12:22:29.580088 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.056155 (+476067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.138335 (+1082180us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b54005a3554. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.138497 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1093434752}
I20260709 12:22:31.051780  5614 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.137840 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1818) took 2846 ms. Trace:
I20260709 12:22:31.090905  5536 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:31.182247 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46191.8us
I20260709 12:22:31.183167 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:31.184469 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:31.186714 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:31.110389  5601 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.190987 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:31.188616 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.130203  5686 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.193454 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:31.185550 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 228452us
I20260709 12:22:31.168088  5517 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:31.178622  5544 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:31.208163 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.208945 27737 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:31.210090 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:31.196682  5547 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:31.189422  5553 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:31.213742 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:31.194614  5450 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:31.215687 27737 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:31.194684  5615 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:31.189571  5516 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:31.226814 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.246246 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.221772  5667 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.256340 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:31.237669  5547 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:31.229228  5650 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.289333 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:31.310021 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1650) took 1687 ms. Trace:
I20260709 12:22:31.310194  5543 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:31.293308 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.236389  5516 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:31.310511 27736 rpcz_store.cc:276] 0709 12:22:29.622144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.996814 (+1374670us) service_pool.cc:224] Handling call
0709 12:22:30.997115 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.997339 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:31.049845 (+ 52506us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.309658 (+259813us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.309718 (+    60us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:31.309766 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.309834 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.309962 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11697,"prepare.queue_time_us":55645,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":33486,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":247503,"threads_started":2}
W20260709 12:22:31.253746  5748 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:22:31.315727 27732 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:31.251503  5750 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:31.223138  5753 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.073s
W20260709 12:22:31.328694 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 135820us
I20260709 12:22:31.337133 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.338675 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4177) took 1363 ms. Trace:
I20260709 12:22:31.339527 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.340818 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:31.342123 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:31.291199  5643 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:31.347012 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:31.344878 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.339339 27731 rpcz_store.cc:276] 0709 12:22:29.975436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.070270 (+1094834us) service_pool.cc:224] Handling call
0709 12:22:31.071128 (+   858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.071724 (+   596us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:31.081505 (+  9781us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.338114 (+256609us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.338212 (+    98us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:31.338292 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.338419 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.338593 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11525,"prepare.queue_time_us":9120,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"thread_start_us":148100,"threads_started":1}
I20260709 12:22:31.348995 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.314553  5547 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:31.351969 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:31.353668 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 140002us
I20260709 12:22:31.356302 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:31.325897  5750 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.360213 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:31.362341 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.287720  5667 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:31.352291  5749 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 12:22:31.345841  5462 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:31.357115  5601 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.016s
I20260709 12:22:31.375957  5601 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:22:31.376416  5670 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 2.141s	user 0.001s	sys 0.003s
I20260709 12:22:31.378125 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4160) took 3136 ms. Trace:
I20260709 12:22:31.378628 27735 rpcz_store.cc:276] 0709 12:22:28.241694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.241798 (+   104us) service_pool.cc:224] Handling call
0709 12:22:28.242062 (+   264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.280767 (+ 38705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.280804 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.294456 (+ 13652us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.593824 (+299368us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:28.593882 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.593979 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.376117 (+2782138us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b24002c5aa8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:22:31.377715 (+  1598us) raft_consensus.cc:1685] finished
0709 12:22:31.377744 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.378041 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24702,"raft.queue_time_us":26701,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9543,"spinlock_wait_cycles":70743936,"thread_start_us":272929,"threads_started":1,"wal-append.queue_time_us":657613}
I20260709 12:22:31.380620 27735 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:31.383002 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4182) took 1327 ms. Trace:
I20260709 12:22:31.350908  5737 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:31.376341  5601 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:31.358397  5547 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:31.376457  5759 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:31.374737  5462 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:31.394129  5462 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:31.326119  5650 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.397194  5650 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.356899  5630 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:31.359660  5423 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:31.398474  5423 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:31.399652  5423 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.401494 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4178) took 1393 ms. Trace:
I20260709 12:22:31.363473  5750 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:31.392174  5601 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.401975 27736 rpcz_store.cc:276] 0709 12:22:30.007894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.375189 (+1367295us) service_pool.cc:224] Handling call
0709 12:22:31.400912 (+ 25723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.401120 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.401153 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.401201 (+    48us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:31.401248 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.401307 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.401436 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.402650  5601 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:31.390276  5553 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.403867  5553 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.406349  5553 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.407601 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3558) took 1539 ms. Trace:
I20260709 12:22:31.407769  5713 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:31.408407  5713 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.408067 27660 rpcz_store.cc:276] 0709 12:22:29.867903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.852784 (+984881us) service_pool.cc:224] Handling call
0709 12:22:30.853464 (+   680us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.874639 (+ 21175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.874682 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.875140 (+   458us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.135760 (+260620us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:31.188253 (+ 52493us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.194663 (+  6410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.194739 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.407388 (+212649us) raft_consensus.cc:1685] finished
0709 12:22:31.407409 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.407536 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13059,"prepare.queue_time_us":52246,"prepare.run_wall_time_us":46,"raft.queue_time_us":6516,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3743,"spinlock_wait_cycles":10682624,"thread_start_us":239302,"threads_started":1,"wal-append.queue_time_us":363462}
I20260709 12:22:31.410454 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:31.411013  5713 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.371980  5450 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:22:31.411580 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.371424  5643 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:22:31.412562  5643 consensus_queue.cc:1243] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 30.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:22:31.413224  5630 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:31.413625  5630 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:31.416997  5667 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.369448  5593 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:31.418052  5593 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.418476  5593 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:31.383757 27735 rpcz_store.cc:276] 0709 12:22:30.055811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.380164 (+1324353us) service_pool.cc:224] Handling call
0709 12:22:31.380464 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.380958 (+   494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.382954 (+  1996us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:31.423996  5765 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 12:22:31.428061  5756 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:22:31.429405 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4489) took 1963 ms. Trace:
I20260709 12:22:31.429795 27587 rpcz_store.cc:276] 0709 12:22:29.466062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.794061 (+1327999us) service_pool.cc:224] Handling call
0709 12:22:30.794440 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.868186 (+ 73746us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.868222 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.868693 (+   471us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.074523 (+205830us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:31.074582 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.074676 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.429164 (+354488us) raft_consensus.cc:1685] finished
0709 12:22:31.429188 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.429333 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73530,"raft.queue_time_us":64974,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6956,"thread_start_us":181664,"threads_started":1,"wal-append.queue_time_us":332325}
I20260709 12:22:31.433799 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4503) took 1152 ms. Trace:
I20260709 12:22:31.434300 27582 rpcz_store.cc:276] 0709 12:22:30.281651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.876143 (+594492us) service_pool.cc:224] Handling call
0709 12:22:30.927897 (+ 51754us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.928240 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.928269 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.928724 (+   455us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.104503 (+175779us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:31.104546 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.104617 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.407192 (+302575us) raft_consensus.cc:1685] finished
0709 12:22:31.407216 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.433711 (+ 26495us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1065,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":76,"thread_start_us":175409,"threads_started":1,"wal-append.queue_time_us":301385}
I20260709 12:22:31.437705 27657 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:31.445487 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.447960  5543 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.448554  5543 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.451566  5713 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:31.452154  5713 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.452178  5547 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:31.453387  5760 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:31.455961  5547 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.454998  5713 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:31.462474 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 105522us
I20260709 12:22:31.451046  5543 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.476537 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:31.464461 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.389851  5562 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:31.442672  5752 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
W20260709 12:22:31.473585 27730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1656) took 1608 ms (client timeout 1857 ms). Trace:
W20260709 12:22:31.480883 27730 rpcz_store.cc:269] 0709 12:22:29.864698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.681062 (+816364us) service_pool.cc:224] Handling call
0709 12:22:31.473354 (+792292us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:31.473502 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74701184}
I20260709 12:22:31.474962 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:31.482136 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:31.479830 27660 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 18.
I20260709 12:22:31.483846 27735 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:31.485802 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:31.506619 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:31.532647 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.458534  5667 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:31.563618 27660 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:31.476039  5536 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.564567 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.483215  5737 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:31.568346 27660 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
W20260709 12:22:31.576267  5679 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 2.371s	user 0.000s	sys 0.004s
I20260709 12:22:31.581326  5694 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:31.581872  5694 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:31.563481  5667 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:31.593076  5767 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 12:22:31.598476 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 123544us
I20260709 12:22:31.483952  5736 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.086s
I20260709 12:22:31.610813 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:31.628993 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3699) took 1809 ms (client timeout 1818 ms). Trace:
W20260709 12:22:31.629549 27738 rpcz_store.cc:269] 0709 12:22:29.820070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.287571 (+467501us) service_pool.cc:224] Handling call
0709 12:22:31.628786 (+1341215us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:31.628923 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79377664}
I20260709 12:22:31.630240 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4501) took 1355 ms. Trace:
I20260709 12:22:31.630669 27580 rpcz_store.cc:276] 0709 12:22:30.275257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.870700 (+595443us) service_pool.cc:224] Handling call
0709 12:22:30.871000 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.879744 (+  8744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.629965 (+750221us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b54001fead4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.630169 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8221440}
I20260709 12:22:31.632978 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4322) took 3480 ms. Trace:
I20260709 12:22:31.633392 27658 rpcz_store.cc:276] 0709 12:22:28.152175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.383851 (+1231676us) service_pool.cc:224] Handling call
0709 12:22:29.411304 (+ 27453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.630582 (+1219278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.632764 (+1002182us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001e4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.632925 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2411031808}
I20260709 12:22:31.618600 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4506) took 1195 ms. Trace:
I20260709 12:22:31.621295  5648 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:22:31.623039 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3698) took 1803 ms. Trace:
I20260709 12:22:31.640746 27734 rpcz_store.cc:276] 0709 12:22:29.819798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.042682 (+1222884us) service_pool.cc:224] Handling call
0709 12:22:31.043083 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.080118 (+ 37035us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.622683 (+542565us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b540017e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.622961 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83374976}
I20260709 12:22:31.635532 27586 rpcz_store.cc:276] 0709 12:22:30.423536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.966288 (+542752us) service_pool.cc:224] Handling call
0709 12:22:30.966564 (+   276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.045580 (+ 79016us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.045623 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.046496 (+   873us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.188552 (+142056us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:31.188600 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.618032 (+429432us) spinlock_profiling.cc:243] Waited 78.6 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.618099 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.618188 (+    89us) raft_consensus.cc:1685] finished
0709 12:22:31.618209 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.618513 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":872,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":180705664,"thread_start_us":141631,"threads_started":1,"wal-append.queue_time_us":194440}
I20260709 12:22:31.624078  5179 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:31.625181 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1788) took 1733 ms. Trace:
I20260709 12:22:31.652304 27655 rpcz_store.cc:276] 0709 12:22:29.892030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.856190 (+964160us) service_pool.cc:224] Handling call
0709 12:22:30.856451 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.900002 (+ 43551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.624885 (+724883us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.625108 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47607680}
W20260709 12:22:31.627434 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.485118  5562 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.569480  5737 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.656124  5737 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:31.658524 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.661922 27660 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
W20260709 12:22:31.670666 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.672735 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:31.674558 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.640496  5648 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:31.652232  5179 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:31.680776 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.685624 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.688685 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:31.690735 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 145160us
I20260709 12:22:31.669723  5547 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.694114 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.695438 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.697567 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.676327  5753 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.676244  5684 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:31.700510 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:31.703589 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:31.705461 27735 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: [29.14-29.14]
I20260709 12:22:31.708107 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:31.708833 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 106943us
I20260709 12:22:31.709357 27735 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:31.693362  5591 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.707896  5536 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:31.709055  5543 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:31.702616  5779 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:22:31.727903 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:31.729890  5721 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.623s	user 0.000s	sys 0.004s
I20260709 12:22:31.733112 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.733314  5736 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:31.721197  5726 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.018s
I20260709 12:22:31.721498  5784 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:31.740773 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3700) took 1889 ms. Trace:
I20260709 12:22:31.738792  5593 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.914728 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4185) took 1630 ms. Trace:
I20260709 12:22:31.915158 27736 rpcz_store.cc:276] 0709 12:22:30.283974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.403103 (+1119129us) service_pool.cc:224] Handling call
0709 12:22:31.439790 (+ 36687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.459702 (+ 19912us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.459745 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.460304 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.638401 (+178097us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:31.638444 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.638511 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.914532 (+276021us) raft_consensus.cc:1685] finished
0709 12:22:31.914553 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.914674 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22878,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":6401,"thread_start_us":177732,"threads_started":1,"wal-append.queue_time_us":264249}
W20260709 12:22:31.743795  5768 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:22:31.927541  5593 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:31.748951  5781 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:22:31.928732 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:31.702330  5601 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:22:31.746841  5648 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.930444  5601 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:31.930873  5601 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:31.931175  5648 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:31.931947 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4507) took 1439 ms. Trace:
I20260709 12:22:31.932346 27587 rpcz_store.cc:276] 0709 12:22:30.492455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.431595 (+939140us) service_pool.cc:224] Handling call
0709 12:22:31.432016 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.432416 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.432453 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.433026 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.563062 (+130036us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:31.563147 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.563249 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.931738 (+368489us) raft_consensus.cc:1685] finished
0709 12:22:31.931761 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.931891 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9110,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":129545,"threads_started":1,"wal-append.queue_time_us":202129}
I20260709 12:22:31.933638  5739 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.743155  5462 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.934060 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:31.934908 27587 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:31.936379  5462 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.943903 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:31.945032 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:31.948391  5516 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.001s
I20260709 12:22:31.948963  5791 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.949033  5516 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.949723  5791 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.951756  5601 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.952574  5179 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.953572  5179 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:31.954916  5533 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:31.700309  5547 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.955300  5533 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:31.956413  5533 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:31.957244 27584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4451) took 1577 ms (client timeout 1831 ms). Trace:
I20260709 12:22:31.705076  5684 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.957583  5533 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:31.958012 27582 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:22:31.959192  5684 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.727553  5536 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:31.959532  5791 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.959885  5536 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:31.960139 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4508) took 1409 ms. Trace:
I20260709 12:22:31.960673 27582 rpcz_store.cc:276] 0709 12:22:30.550702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.439221 (+888519us) service_pool.cc:224] Handling call
0709 12:22:31.957702 (+518481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.958541 (+   839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.959890 (+  1349us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54002ba7d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:31.960062 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1189799680}
I20260709 12:22:31.962123 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:31.770092  5677 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.624s	user 0.006s	sys 0.006s
I20260709 12:22:31.963567  5677 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.934134  5739 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.965302 27580 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:22:31.965624  5677 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.967957  5739 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:31.970834  5429 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:31.972657 27729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3682) took 3628 ms (client timeout 2073 ms). Trace:
W20260709 12:22:31.973213 27729 rpcz_store.cc:269] 0709 12:22:28.344489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.434170 (+ 89681us) service_pool.cc:224] Handling call
0709 12:22:31.972427 (+3538257us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54000a55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:31.972559 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297842944}
I20260709 12:22:31.977643 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:31.180369 27588 rpcz_store.cc:276] 0709 12:22:28.295758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.351866 (+1056108us) service_pool.cc:224] Handling call
0709 12:22:29.375493 (+ 23627us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.598248 (+222755us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:29.643251 (+ 45003us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.058651 (+415400us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.058707 (+    56us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:30.058761 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.137458 (+1078697us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.137522 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.137761 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16982,"prepare.queue_time_us":45844,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":24974,"raft.queue_time_us":23451,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":15115,"spinlock_wait_cycles":512940672,"thread_start_us":392465,"threads_started":2,"wal-append.queue_time_us":520337,"wal-append.run_cpu_time_us":979,"wal-append.run_wall_time_us":1001242}
I20260709 12:22:31.978293 27655 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 22.
I20260709 12:22:31.979555 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:31.980439  5593 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:22:31.980091 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:31.980823  5593 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.981194  5593 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:31.723464  5667 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:31.704638  5771 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:22:31.711062  5713 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:22:31.747689  5543 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.986018  5543 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:31.988988  5694 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:31.909584 27732 rpcz_store.cc:276] 0709 12:22:29.851714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.044110 (+1192396us) service_pool.cc:224] Handling call
0709 12:22:31.044389 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.319444 (+275055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.740503 (+421059us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54001d24d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.740710 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623342336}
I20260709 12:22:31.723925  5750 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:31.913657 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.917270 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 207784us
I20260709 12:22:31.924230  5784 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.749120  5779 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:31.952996  5601 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:31.990329  5694 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:31.861450 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4458) took 1415 ms. Trace:
I20260709 12:22:31.705188  5753 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:31.994776 28336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1519) took 2400 ms. Trace:
I20260709 12:22:31.997191 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4310) took 3862 ms. Trace:
W20260709 12:22:31.957617 27584 rpcz_store.cc:269] 0709 12:22:30.379903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.403495 (+ 23592us) service_pool.cc:224] Handling call
0709 12:22:31.957069 (+1553574us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54002ba7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:31.957178 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580408704}
I20260709 12:22:31.978986 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:32.038542 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.036286 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:32.043421 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.919382  5725 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.863s	user 0.018s	sys 0.008s
I20260709 12:22:32.048098 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.042030 27588 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:32.004104 27589 rpcz_store.cc:276] 0709 12:22:30.445854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.214904 (+769050us) service_pool.cc:224] Handling call
0709 12:22:31.227843 (+ 12939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.228056 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:31.388082 (+160026us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.663336 (+275254us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.853183 (+189847us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.853250 (+    67us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:31.853305 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.853403 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.861359 (+  7956us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":164408,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":1618,"raft.queue_time_us":184811,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":8572,"spinlock_wait_cycles":2122112,"thread_start_us":599192,"threads_started":4,"wal-append.queue_time_us":296359}
I20260709 12:22:32.050259 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.042220 28336 rpcz_store.cc:276] 0709 12:22:29.594141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.594290 (+   149us) service_pool.cc:224] Handling call
0709 12:22:29.594684 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.630957 (+1036273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.994435 (+1363478us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:31.994700 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2058167552}
I20260709 12:22:32.052138 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.042559 27663 rpcz_store.cc:276] 0709 12:22:28.134362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.997314 (+862952us) service_pool.cc:224] Handling call
0709 12:22:28.997714 (+   400us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:30.169057 (+1171343us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b1800110500. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a340bfb 00007f0e9a34b64b
0709 12:22:30.169239 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.169270 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:30.169868 (+   598us) log.cc:844] Serialized 52 byte log entry
0709 12:22:30.913919 (+744051us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:30.948906 (+ 34987us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.949178 (+   272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.949290 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.985004 (+1035714us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b1800110500. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:31.996916 (+ 11912us) raft_consensus.cc:1685] finished
0709 12:22:31.996947 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.997118 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43844,"prepare.queue_time_us":53081,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":373527,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":19150,"spinlock_wait_cycles":45432576,"thread_start_us":1079070,"threads_started":2,"wal-append.queue_time_us":862149}
I20260709 12:22:32.055621 27663 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:22:32.058152 27663 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:22:31.999780  5734 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.072s	user 0.021s	sys 0.003s
I20260709 12:22:32.049063 27582 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:32.061216 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.063588 27738 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:32.066872 27588 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:32.068498 27582 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:22:32.001209  5680 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.074033 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.076335 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:31.947028  5737 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.078348  5753 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.079900 27735 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:32.079955 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.083027  5667 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:32.083834 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4481) took 3591 ms. Trace:
I20260709 12:22:32.084014 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.088789 28336 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:22:32.089385 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.111528 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.089897 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.091154 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:31.957607  5547 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:32.090811 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:32.004412  5779 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:31.919694  5736 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.098848 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.099290 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.099718 27584 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 35 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
W20260709 12:22:32.101007 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 394951us
I20260709 12:22:32.125532 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.091261  5725 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:32.110137 27581 rpcz_store.cc:276] 0709 12:22:28.491973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.784788 (+1292815us) service_pool.cc:224] Handling call
0709 12:22:29.785163 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.808227 (+ 23064us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.083593 (+2275366us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:32.083779 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9377024}
I20260709 12:22:32.128415 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.128937 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.129315 28337 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:22:32.069926  5734 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.119470 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.119832 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.126364 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.137506 28330 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:22:32.126608 28328 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 31 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 22.
I20260709 12:22:32.118571 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.138442 27730 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:32.106303 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.110546  5726 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.148645  5726 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.149583  5703 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:32.116278  5680 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.141294  5707 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:32.141490  5737 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.144310  5516 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:32.145244 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4164) took 3784 ms. Trace:
I20260709 12:22:32.146538  5700 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:32.153138 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.090876  5536 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.159978 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.181141 27735 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:32.172127 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.182289 27733 rpcz_store.cc:276] 0709 12:22:28.361278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.303912 (+942634us) service_pool.cc:224] Handling call
0709 12:22:30.952521 (+1648609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.953304 (+   783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.145052 (+1191748us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54002e90d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:32.145187 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3673382272}
I20260709 12:22:32.184514 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.188802 28335 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:32.188879 27587 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:32.196957 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.197373 28329 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 18.
I20260709 12:22:32.115574  5533 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 12:22:32.199417 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.140602  5562 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:32.188982  5700 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.130286  5650 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:22:32.125068  5667 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.187901  5429 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:32.238250 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.240507 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:32.242151 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.247022 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.247841 28328 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:32.134389  5725 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.197940 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.251047 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.200758  5750 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.252975 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:32.254526 27584 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:32.229100  5726 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.260138 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:32.264564 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66703.8us
I20260709 12:22:32.235249  5700 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:32.235734  5681 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:32.268436 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 119373us
I20260709 12:22:32.270313 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.271216 28335 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:32.267138  5528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.887s	user 0.016s	sys 0.015s
I20260709 12:22:32.286940 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:32.289337 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 287065us
I20260709 12:22:32.290288 28328 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 29 because replica has last-logged OpId of term: 28 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
W20260709 12:22:32.291649 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.294488 27735 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:22:32.292452 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:32.295708 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 197400us
I20260709 12:22:32.239576  5787 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.049s
W20260709 12:22:32.276646  5528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.906s	user 0.017s	sys 0.015s
W20260709 12:22:32.297575 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 185555us
I20260709 12:22:32.272688  5429 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:32.272643  5562 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.302299 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.305243 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:32.307780 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:32.309114 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 225792us
I20260709 12:22:32.310923 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.311542 28328 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:32.310505 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.315313 28335 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:32.315821 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:32.316286 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.292806  5735 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.316825 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3553) took 2500 ms. Trace:
I20260709 12:22:32.332674 27664 rpcz_store.cc:276] 0709 12:22:29.816635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.317908 (+501273us) service_pool.cc:224] Handling call
0709 12:22:30.318159 (+   251us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.781860 (+463701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.316563 (+1534703us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b54000e56d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:32.316756 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1064837888}
I20260709 12:22:32.333858 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.314640 27736 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 34 because replica has last-logged OpId of term: 33 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:22:32.302026  5232 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.321755 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.319532 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.321363 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.322961 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.338272 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:32.325850 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.327471 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.279336  5681 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.303259  5784 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.291165  5593 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.329496 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.331998 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.338912 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:32.343129 27663 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:32.344696 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.346698 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.347373 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.371680  5536 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.372200 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.372650  5536 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:32.375895 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:32.376762 27582 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:32.377228  5736 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:32.336760  5803 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:32.379828  5803 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:32.380257  5803 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.380656  5803 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:32.381655 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.326215  5179 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.029s
I20260709 12:22:32.383219  5179 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:32.383610  5179 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:32.384486  5179 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.387199  5681 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:32.338209  5735 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.389863  5543 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.338382  5689 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:22:32.392189  5543 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.340838  5739 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.052s
I20260709 12:22:32.393779 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.395031 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.396595 27735 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:32.390475  5735 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:32.398139 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:32.403798 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.404422 27734 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:32.406037  5643 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:32.406438  5643 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:32.347581 27663 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:32.311728  5562 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.415805 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.346143 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:32.371788  5802 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:32.429280 27729 raft_consensus.cc:2368] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:22:32.408846 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.430183 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:32.349377 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.431465 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:32.418093 27734 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:32.413647 27582 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 29 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:22:32.433804 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.434821 28328 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:32.438304 27663 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:32.439622 28295 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:32.501096 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61548.4us
I20260709 12:22:32.407532  5179 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:32.430138  5500 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:22:32.511737 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165698us
I20260709 12:22:32.514930 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.432657  5785 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.513091 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:32.526648 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:32.527837 27663 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:32.513685 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.371680  5805 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:22:32.451372  5648 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.001s
W20260709 12:22:32.513185 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.519778 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 181554us
W20260709 12:22:32.528623 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.515646 27587 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:22:32.530635 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 233185us
W20260709 12:22:32.501746  5794 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.185s	user 0.001s	sys 0.001s
I20260709 12:22:32.532464 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:32.354331  5232 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.537726 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4342) took 1318 ms. Trace:
I20260709 12:22:32.537921 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.554858 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.541724 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.542621 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.515225  5500 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:32.540150 27736 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:32.552480 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:32.564093 27588 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:32.524487  5650 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:32.407248  5643 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:32.563463 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 225387us
I20260709 12:22:32.576579 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.559718 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.577867 27587 raft_consensus.cc:2410] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:22:32.561703 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:32.572867 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.546120  5684 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:32.582639 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.582979 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.583913 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.584594 28332 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:32.531464  5787 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 19, Time since last communication: 0.037s
I20260709 12:22:32.555925  5725 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:32.549772  5810 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:32.578024  5753 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.590898  5643 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:32.600056  5750 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:32.594401 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:32.560683  5425 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:32.592818 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:32.585453  5684 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:32.601470  5553 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:32.602288  5680 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:32.603775  5796 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 12:22:32.570324  5650 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.683516  5516 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:22:32.604763  5806 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:22:32.683267  5726 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:22:32.611827  5798 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:22:32.616222 27588 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:32.591652 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:31.081794 27656 rpcz_store.cc:276] 0709 12:22:29.816938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.320740 (+503802us) service_pool.cc:224] Handling call
0709 12:22:30.330567 (+  9827us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.342122 (+ 11555us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.342174 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.342715 (+   541us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.632212 (+289497us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:31.067724 (+435512us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.068047 (+   323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.068124 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.068163 (+    39us) raft_consensus.cc:1685] finished
0709 12:22:31.068177 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.068334 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31747,"prepare.queue_time_us":126721,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":11369,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":30874,"thread_start_us":575359,"threads_started":2,"wal-append.queue_time_us":403140}
I20260709 12:22:32.628547  5562 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:32.630565 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.662849  5803 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.635782 27585 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 30 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
W20260709 12:22:32.636765 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.553483 27658 rpcz_store.cc:276] 0709 12:22:31.219686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.634307 (+414621us) service_pool.cc:224] Handling call
0709 12:22:31.663527 (+ 29220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.663796 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:22:31.679699 (+ 15903us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.087240 (+407541us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b18003cd900. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:32.537162 (+449922us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.537341 (+   179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.537390 (+    49us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:22:32.537439 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.537524 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.537668 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15385,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":396,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":13965,"spinlock_wait_cycles":4236160,"thread_start_us":446333,"threads_started":1}
I20260709 12:22:32.642822  5601 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:32.650275 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4509) took 2089 ms. Trace:
I20260709 12:22:32.652797  5686 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:32.656386  5687 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:32.515481  5716 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.533463  5785 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.661567 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3574) took 1446 ms. Trace:
I20260709 12:22:32.701184  5753 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.663777 27655 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:22:32.676472 27581 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
W20260709 12:22:32.682531 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 170769us
I20260709 12:22:32.588009 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:32.775301 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 187445us
W20260709 12:22:32.776216 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.777148 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.570101  5735 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:32.608084  5799 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.086s	user 0.003s	sys 0.000s
W20260709 12:22:32.714722 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 122014us
I20260709 12:22:32.716861 27588 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:32.568015  5802 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:32.720427 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.709033  5425 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.727813 27580 rpcz_store.cc:276] 0709 12:22:30.560777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.631632 (+1070855us) service_pool.cc:224] Handling call
0709 12:22:31.632113 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.965677 (+333564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.649904 (+684227us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54000889d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:32.650196 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765688704}
I20260709 12:22:32.792650 27702 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:32.753436  5553 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.712782  5516 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.712620  5650 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:22:32.769783 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 178184us
I20260709 12:22:32.717931  5547 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:32.771611 27657 rpcz_store.cc:276] 0709 12:22:31.215560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.372475 (+156915us) service_pool.cc:224] Handling call
0709 12:22:32.661393 (+1288918us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b5400291154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:32.661503 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58140032}
I20260709 12:22:32.585253  5797 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.084s
I20260709 12:22:32.778105 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.724531  5781 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:32.782898 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4515) took 1044 ms. Trace:
I20260709 12:22:32.766777  5775 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.787900 27656 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:32.632037  5500 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.788213 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:32.816982 27658 raft_consensus.cc:2410] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:22:32.736327  5686 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:32.765347  5601 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.794442 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.719400  5700 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.822297 27733 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: [32.17-33.18]
I20260709 12:22:32.823050 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:32.823982 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:32.824222 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.797344 27627 leader_election.cc:400] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:22:32.773710  5593 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:32.772996  5812 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:32.804263 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.829536 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:32.805251 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.806906 27736 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:32.808751 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.801281  5539 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:32.785490  5429 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:32.813711 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.813905 27581 rpcz_store.cc:276] 0709 12:22:31.738123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.675854 (+937731us) service_pool.cc:224] Handling call
0709 12:22:32.676237 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.782552 (+106315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.782828 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:32.818233  5781 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.819509 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.793411  5516 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:32.820251 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.821672 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.824999 27627 leader_election.cc:403] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:32.783645  5735 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.826941 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.828318 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.831068 27734 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:32.865150 27729 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:32.781694  5716 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:32.832120 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:32.847950 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:32.850165 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4353) took 1129 ms. Trace:
I20260709 12:22:32.867874 27656 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:32.853291 27731 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:32.815418  5703 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:22:32.857177 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 43595us
I20260709 12:22:32.906507 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.907508  5539 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:32.908124 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:32.860904 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.864445 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67251us
I20260709 12:22:32.771799  5684 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:32.914546 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:32.798635  5553 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.826776  5802 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:22:32.904539 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76357us
I20260709 12:22:32.927811 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4345) took 1611 ms. Trace:
I20260709 12:22:32.848722  5681 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:32.920778 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:32.829972  5687 consensus_queue.cc:1175] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 31 status { last_received { term: 31 index: 19 } last_committed_idx: 19 last_received_current_leader { term: 31 index: 19 } }
I20260709 12:22:32.928359 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.902839 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4347) took 1541 ms. Trace:
I20260709 12:22:32.851441  5750 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.928197 27662 rpcz_store.cc:276] 0709 12:22:31.316965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.748997 (+432032us) service_pool.cc:224] Handling call
0709 12:22:31.749333 (+   336us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:31.773049 (+ 23716us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.908474 (+135425us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:32.002870 (+ 94396us) log.cc:844] Serialized 52 byte log entry
0709 12:22:32.261091 (+258221us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:32.283602 (+ 22511us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.289027 (+  5425us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.289100 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.927576 (+638476us) raft_consensus.cc:1685] finished
0709 12:22:32.927600 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.927754 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61097,"prepare.queue_time_us":22605,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":5294,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":69759,"thread_start_us":189446,"threads_started":1,"wal-append.queue_time_us":274314}
I20260709 12:22:32.942104 27656 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.19-35.19]
I20260709 12:22:32.867103  5500 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:32.858012  5429 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:32.864732  5812 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:32.941676 27659 rpcz_store.cc:276] 0709 12:22:31.361479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.929459 (+567980us) service_pool.cc:224] Handling call
0709 12:22:31.929775 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.991959 (+ 62184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.991990 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.992467 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.298533 (+306066us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:32.298570 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.298635 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.607477 (+308842us) spinlock_profiling.cc:243] Waited 4.64 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:32.853255 (+245778us) raft_consensus.cc:1685] finished
0709 12:22:32.853283 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.902728 (+ 49445us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61806,"raft.queue_time_us":61951,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":10681088,"thread_start_us":305687,"threads_started":1,"wal-append.queue_time_us":373097}
I20260709 12:22:32.903559  5736 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.904623  5462 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:32.954773 28336 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:32.913502  5726 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:32.925240  5791 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:32.952311 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.931291  5802 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.925930  5739 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:32.956569 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:32.958010 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.959980 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:32.926645 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.979598 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.963546 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.825574  5775 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.916069  5814 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:32.933622  5681 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.963169 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.960422  5684 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:32.981563 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:32.984383 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 152379us
I20260709 12:22:32.928733  5788 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:22:32.953280  5429 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:22:32.961387  5779 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.827s	user 0.008s	sys 0.006s
I20260709 12:22:32.929350  5809 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.043s
I20260709 12:22:32.989179 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.973848 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:32.957442  5725 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:32.992354 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:32.947800  5405 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:32.978284  5813 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:33.003444 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:32.992422  5681 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:33.004438 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:32.978961  5802 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:32.985915  5543 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:33.003494  5232 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.014106 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.015681 28332 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:22:32.953038  5750 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.004395 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:32.932636  5686 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:32.990276  5810 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:32.982880  5703 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:22:33.007885  5713 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:32.976518  5736 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.012137 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:33.062855 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50801.9us
I20260709 12:22:33.063987 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.065419 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:33.032073 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 256202us
I20260709 12:22:33.001955  5803 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.047518 27733 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:33.053354 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.055068 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:33.019512  5814 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:33.019577  5788 consensus_queue.cc:1243] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 30.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:22:33.056957  5648 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:33.019972  5784 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:33.005947  5812 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.044441  5813 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:33.006052  5779 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.032143  5785 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.085336 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:33.090026  5753 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.104506 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:33.106115 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:33.058472  5713 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:33.069454  5543 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:33.083145  5405 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:33.129202 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4471) took 1159 ms. Trace:
I20260709 12:22:33.090508  5803 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:33.089076  5813 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:33.098497  5788 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:22:33.137630 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72307.8us
W20260709 12:22:33.139156 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 135821us
I20260709 12:22:33.127830  5713 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:33.063033  5562 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
I20260709 12:22:33.107648  5601 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.108376  5814 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:33.138883  5648 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.116441  5232 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:33.144105 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.144547 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:33.138347 27588 rpcz_store.cc:276] 0709 12:22:31.970169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.786598 (+816429us) service_pool.cc:224] Handling call
0709 12:22:32.786870 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.787128 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:32.963689 (+176561us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.128843 (+165154us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.128898 (+    55us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:33.128948 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.129012 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.129146 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28320,"prepare.queue_time_us":203836,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":15526,"thread_start_us":303662,"threads_started":2}
I20260709 12:22:33.128145  5700 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.128657  5820 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:33.131352  5737 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:33.129259  5425 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.131273  5785 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:33.133404  5781 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:33.160826  5816 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 12:22:33.139221  5735 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.143186  5813 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:33.128659  5824 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.037s
I20260709 12:22:33.187930  5824 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:33.188410  5824 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:33.189512  5824 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.154116  5788 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:33.193142  5788 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.161497  5553 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:33.169734  5828 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.001s	sys 0.001s
W20260709 12:22:33.179879  5819 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
W20260709 12:22:33.181118  5823 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:33.183641  5753 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.141077  5405 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.175470  5803 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:32.867913 27658 rpcz_store.cc:276] 0709 12:22:31.720839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.817674 (+1096835us) service_pool.cc:224] Handling call
0709 12:22:32.849259 (+ 31585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.849713 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.849755 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.849814 (+    59us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:32.849867 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.849944 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.850095 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":10025}
I20260709 12:22:33.157781  5814 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.153075  5838 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:33.141831 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:33.164224  5737 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:33.226310  5826 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20260709 12:22:33.227023  5736 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:33.226361  5839 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:33.235282  5753 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:33.250496  5837 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:33.251679 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4358) took 1260 ms. Trace:
I20260709 12:22:33.252038  5753 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:33.252576  5829 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:22:33.160575  5810 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.252040 27656 rpcz_store.cc:276] 0709 12:22:31.991677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.943348 (+951671us) service_pool.cc:224] Handling call
0709 12:22:32.973585 (+ 30237us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.974001 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.974041 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.974583 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.069130 (+ 94547us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:33.069182 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.069266 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.251474 (+182208us) raft_consensus.cc:1685] finished
0709 12:22:33.251494 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.251618 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":7378,"thread_start_us":94193,"threads_started":1,"wal-append.queue_time_us":152810}
W20260709 12:22:33.223222 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 218876us
I20260709 12:22:33.257844  5734 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:33.259271  5755 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.971s	user 0.004s	sys 0.017s
I20260709 12:22:33.259166 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.260538 27733 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:33.260792  5667 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.241463  5814 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:33.261925  5781 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:33.262297 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4510) took 1525 ms. Trace:
I20260709 12:22:33.262735  5781 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.262693 27586 rpcz_store.cc:276] 0709 12:22:31.736853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.737001 (+   148us) service_pool.cc:224] Handling call
0709 12:22:31.737368 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.334547 (+597179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.334582 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.335010 (+   428us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.707956 (+372946us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:32.708013 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.187732 (+479719us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b540006f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:33.187782 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.262000 (+ 74218us) raft_consensus.cc:1685] finished
0709 12:22:33.262025 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.262240 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12330,"raft.queue_time_us":679,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":13129,"spinlock_wait_cycles":1372581248,"thread_start_us":276545,"threads_started":1,"wal-append.queue_time_us":613983}
I20260709 12:22:33.266110  5781 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.237371  5803 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:33.267695  5803 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:33.269294  5617 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:33.269933  5617 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:33.271759  5737 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.272990  5737 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:33.274825  5686 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:33.235966 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68748.2us
I20260709 12:22:33.175627  5785 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:33.295985 27656 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:22:33.164176  5648 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.200459  5232 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:33.245609  5405 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.246585  5630 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 29.14, Last appended by leader: 14, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.247852 27658 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:22:33.260923 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4490) took 3794 ms. Trace:
I20260709 12:22:33.396840 27583 rpcz_store.cc:276] 0709 12:22:29.466324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.806269 (+1339945us) service_pool.cc:224] Handling call
0709 12:22:30.806603 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.880812 (+ 74209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.880846 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.881283 (+   437us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.129743 (+248460us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:31.129794 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.129873 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.258730 (+2128857us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:33.260615 (+  1885us) raft_consensus.cc:1685] finished
0709 12:22:33.260639 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.260862 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74338,"raft.queue_time_us":78816,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":2728,"spinlock_wait_cycles":25574656,"thread_start_us":247388,"threads_started":1,"wal-append.queue_time_us":341741}
W20260709 12:22:33.276755  5770 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.577s	user 0.004s	sys 0.000s
I20260709 12:22:33.419106 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.419826 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
W20260709 12:22:33.277660  5836 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:22:33.279992 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4517) took 1541 ms. Trace:
I20260709 12:22:33.438927 27589 rpcz_store.cc:276] 0709 12:22:31.738600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.677902 (+939302us) service_pool.cc:224] Handling call
0709 12:22:32.678323 (+   421us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:32.678828 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.678875 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:32.682758 (+  3883us) log.cc:844] Serialized 52 byte log entry
0709 12:22:32.943725 (+260967us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:32.943764 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.943833 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.279749 (+335916us) raft_consensus.cc:1685] finished
0709 12:22:33.279771 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.279932 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":879,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":128,"thread_start_us":222932,"threads_started":1,"wal-append.queue_time_us":321433}
I20260709 12:22:33.191176  5824 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:33.285485  5650 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.285724  5838 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:33.287146  5802 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.463670  5814 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.465611  5802 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.168422  5841 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:33.287968  5779 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.466440  5841 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.288306  5543 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:33.288400  5840 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.290865  5643 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.000s
I20260709 12:22:33.466895  5814 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.294415 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.467378  5840 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.467845 27582 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 19. (term mismatch)
I20260709 12:22:33.298674 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4461) took 1356 ms. Trace:
I20260709 12:22:33.468221 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.468240 27582 pending_rounds.cc:85] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 18
I20260709 12:22:33.469483  5840 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.469524 27582 pending_rounds.cc:107] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 35.19
I20260709 12:22:33.470181 27582 raft_consensus.cc:2889] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:33.468381 27587 rpcz_store.cc:276] 0709 12:22:31.942240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.708087 (+765847us) service_pool.cc:224] Handling call
0709 12:22:32.708445 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.708699 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:33.158103 (+449404us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.298319 (+140216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.298372 (+    53us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:33.298423 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.298508 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.298616 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":239776,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":177,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":99424,"thread_start_us":568177,"threads_started":2}
I20260709 12:22:33.300024 27731 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: []
I20260709 12:22:33.472337 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.473050 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:33.474444 27731 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:33.290966  5725 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:33.301680 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:33.306766 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.335525 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.306044 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.431425 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.442062 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:33.442633 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.437831 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.443104 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.481582 27729 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:33.483242  5838 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.483947  5838 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.486616  5838 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.493031  5648 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:33.494197 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.495733  5785 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.499702  5232 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:33.506366 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.512554 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.276854  5750 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.304286  5462 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:33.453779  5405 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.460680  5429 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:33.292390  5834 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 12:22:33.271157  5617 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.466917  5841 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:33.535202  5716 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.621s	user 0.008s	sys 0.004s
W20260709 12:22:33.543588  5770 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:22:33.600276 27581 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 24. Current term is 25. Ops: []
W20260709 12:22:33.476557 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.603430 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1831) took 1470 ms. Trace:
I20260709 12:22:33.604009 27581 rpcz_store.cc:276] 0709 12:22:32.133269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.854565 (+721296us) service_pool.cc:224] Handling call
0709 12:22:32.854852 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.601654 (+746802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.602750 (+  1096us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:33.603251 (+   501us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1712284928}
I20260709 12:22:33.457123 27658 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
W20260709 12:22:33.624261 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 462793us
I20260709 12:22:33.626924 27658 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
W20260709 12:22:33.629709 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4450) took 3262 ms (client timeout 1737 ms). Trace:
I20260709 12:22:33.450171 27656 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:22:33.632930 27580 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:22:33.468955 27733 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:33.633442 27580 pending_rounds.cc:85] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 17
I20260709 12:22:33.633641 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4340) took 2414 ms. Trace:
I20260709 12:22:33.633980 27580 pending_rounds.cc:107] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:22:33.297251 27736 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
W20260709 12:22:33.635124 28292 connection.cc:256] RPC call timeout handler was delayed by -1.124205295000138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
W20260709 12:22:33.635727 28292 connection.cc:256] RPC call timeout handler was delayed by -1.0123232260002624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20260709 12:22:33.636023 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4357) took 1656 ms. Trace:
I20260709 12:22:33.636529 27662 rpcz_store.cc:276] 0709 12:22:31.979555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.942615 (+963060us) service_pool.cc:224] Handling call
0709 12:22:32.942948 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.002740 (+ 59792us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:33.046606 (+ 43866us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.129875 (+ 83269us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.129930 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:33.129978 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.635393 (+505415us) spinlock_profiling.cc:243] Waited 59.1 ms on lock 0x7b5400229a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:33.635489 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.635783 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4162,"prepare.queue_time_us":29028,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":6313,"raft.queue_time_us":18989,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5053,"spinlock_wait_cycles":136015616,"thread_start_us":82892,"threads_started":1,"wal-append.queue_time_us":164714}
I20260709 12:22:33.639710 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.643020 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4518) took 1904 ms. Trace:
I20260709 12:22:33.643556 27582 rpcz_store.cc:276] 0709 12:22:31.738842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.680446 (+941604us) service_pool.cc:224] Handling call
0709 12:22:32.680805 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.470632 (+789827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.642400 (+171768us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54001883d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:33.642795 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1809187072}
W20260709 12:22:33.646690 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.011s	sys 0.019s
W20260709 12:22:33.647212 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.011s	sys 0.020s
I20260709 12:22:33.647244 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:33.648020 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 541926us
I20260709 12:22:33.650128 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.634495 27580 raft_consensus.cc:2889] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:33.630508 27585 rpcz_store.cc:269] 0709 12:22:30.366940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.393421 (+ 26481us) service_pool.cc:224] Handling call
0709 12:22:33.629064 (+3235643us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b5400087ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:33.629371 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5154235776}
I20260709 12:22:33.516609 27731 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:22:33.522986 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:33.779409 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:33.780457 27585 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:33.655220 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.539479  5405 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.783047  5680 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:33.783612  5680 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:33.784773 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.785645  5405 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:33.534888  5630 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:33.786741 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1620) took 1232 ms. Trace:
I20260709 12:22:33.787348 28335 rpcz_store.cc:276] 0709 12:22:32.554441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.554598 (+   157us) service_pool.cc:224] Handling call
0709 12:22:33.647279 (+1092681us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400a1dbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:33.786453 (+139174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2501805440}
I20260709 12:22:33.788458 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4479) took 1694 ms. Trace:
I20260709 12:22:33.788941 27580 rpcz_store.cc:276] 0709 12:22:32.094506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.792742 (+698236us) service_pool.cc:224] Handling call
0709 12:22:33.631688 (+838946us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.776226 (+144538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.788229 (+ 12003us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b5400087ad0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:33.788372 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1928317184}
I20260709 12:22:33.790405 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:33.793900  5775 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.680406  5809 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:33.700929 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.798184 27587 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:33.798517 27581 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:33.634214 27660 rpcz_store.cc:276] 0709 12:22:31.219068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.601288 (+382220us) service_pool.cc:224] Handling call
0709 12:22:31.601616 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.662420 (+ 60804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.633062 (+1970642us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:33.633414 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133798272}
I20260709 12:22:33.746359  5528 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:33.800051  5528 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:33.801784  5528 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:33.747820 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 94987.1us
I20260709 12:22:33.749516 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.747630  5429 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.817062  5810 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:22:33.751451  5179 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.237s	user 0.020s	sys 0.002s
I20260709 12:22:33.752166 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.752629 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4348) took 2362 ms. Trace:
I20260709 12:22:33.819087 27655 rpcz_store.cc:276] 0709 12:22:31.390084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.981613 (+591529us) service_pool.cc:224] Handling call
0709 12:22:32.663441 (+681828us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.774746 (+111305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.750612 (+975866us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b5400291150. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:33.752417 (+  1805us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1567307648}
I20260709 12:22:33.751942 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.820988 27589 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:33.753639  5650 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:33.753907 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.756556 27588 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:22:33.823915  5232 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.762539  5684 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.758s	user 0.011s	sys 0.015s
I20260709 12:22:33.824716  5232 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.826155 27586 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:33.825963  5593 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.746727 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.762894  5517 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:33.764376  5617 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:33.835007 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4194) took 2150 ms (client timeout 2230 ms). Trace:
W20260709 12:22:33.835537 27738 rpcz_store.cc:269] 0709 12:22:31.684980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.697305 (+ 12325us) service_pool.cc:224] Handling call
0709 12:22:33.654641 (+1957336us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:33.834764 (+180123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830066944}
I20260709 12:22:33.836174  5232 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.837471 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:33.767539 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.633167 27732 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:22:33.839982 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.840438 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4202) took 1763 ms. Trace:
I20260709 12:22:33.840723 27589 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:33.840941 27732 rpcz_store.cc:276] 0709 12:22:32.077216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.079420 (+  2204us) service_pool.cc:224] Handling call
0709 12:22:33.631777 (+1552357us) spinlock_profiling.cc:243] Waited 8.64 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:22:33.632379 (+   602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.840071 (+207692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.840335 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19863808}
I20260709 12:22:33.842494  5232 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:33.843887 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:33.843575  5232 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.844760 27588 raft_consensus.cc:2376] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33. Re-sending same reply.
I20260709 12:22:33.768456 27664 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:33.747144  5444 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.847476 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.848577 28330 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:33.848075  5444 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:33.850294  5803 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.582s	user 0.013s	sys 0.005s
I20260709 12:22:33.852074 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1527) took 1753 ms. Trace:
I20260709 12:22:33.852494 28329 rpcz_store.cc:276] 0709 12:22:32.098375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.098529 (+   154us) service_pool.cc:224] Handling call
0709 12:22:33.662766 (+1564237us) spinlock_profiling.cc:243] Waited 71.5 ms on lock 0x7b54006d6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:33.851868 (+189102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164515072}
I20260709 12:22:33.854617 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:33.835278  5516 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.921s	user 0.015s	sys 0.005s
W20260709 12:22:33.857378 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.856979  5516 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.859177  5429 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.859536  5516 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.860229  5462 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.859869  5429 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.860948  5462 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.863153 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:33.805634 27580 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:33.827018  5593 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.569460  5841 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.839464  5517 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:33.850955  5803 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.877519  5533 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.878399  5533 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.880234  5841 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.883502 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.883654 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.851161  5444 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.867112  5232 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.890236 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:33.891256 28330 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:33.862993  5429 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.893256 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.900689 27582 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:33.898985 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:33.903502 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 85779.3us
I20260709 12:22:33.911562 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:33.915146 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.916519  5650 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:33.917722 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:33.918992  5650 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:33.913259 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:33.893254  5835 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.018s
I20260709 12:22:33.903868  5841 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.934237 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:33.941045  5689 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:33.941812 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.942103  5689 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.936136  5179 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.005s
I20260709 12:22:33.909597  5462 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:33.947275  5700 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.949009  5785 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:22:33.950052 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 307358us
I20260709 12:22:33.950721 28330 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:33.990947  5824 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.531s	user 0.019s	sys 0.000s
I20260709 12:22:33.991190 27660 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: [33.18-35.19]
W20260709 12:22:33.992219  5681 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.959s	user 0.015s	sys 0.017s
I20260709 12:22:33.992794  5681 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:33.952164  5805 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.995904  5805 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.951308 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:33.958244  5637 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:33.974210 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:33.998553 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47443.6us
W20260709 12:22:33.982300 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.000s	sys 0.009s
W20260709 12:22:33.982543  5739 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.999s	user 0.013s	sys 0.013s
I20260709 12:22:33.983126  5841 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:33.983582 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:33.984195 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50511.5us
I20260709 12:22:33.986439  5689 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:33.987270 27630 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.008s	sys 0.000s
I20260709 12:22:33.997782 28333 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:33.999951 27655 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
W20260709 12:22:34.007601 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.000s	sys 0.009s
W20260709 12:22:34.008275 27630 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.008s	sys 0.001s
I20260709 12:22:34.011650  5179 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:34.015421  5700 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.015933  5533 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.018049  5700 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:34.020793 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 130899us
W20260709 12:22:34.022166 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.023552  5686 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.728s	user 0.010s	sys 0.012s
I20260709 12:22:34.034478 27582 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:34.037423 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:34.047734 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.041095 27584 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
W20260709 12:22:34.031558 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 731487us
W20260709 12:22:34.089608 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65585.6us
W20260709 12:22:34.090176 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46886.8us
I20260709 12:22:34.048264 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:34.019070 27479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 242) took 44491 ms (client timeout 15000 ms). Trace:
W20260709 12:22:34.040189 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3585) took 1716 ms (client timeout 1744 ms). Trace:
W20260709 12:22:34.039381 27661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.007s	sys 0.000s
W20260709 12:22:34.045082  5737 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.772s	user 0.012s	sys 0.011s
I20260709 12:22:34.034825 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:34.096036 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69739.8us
I20260709 12:22:34.097445 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:34.093448 27664 rpcz_store.cc:269] 0709 12:22:32.323290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.347038 (+ 23748us) service_pool.cc:224] Handling call
0709 12:22:34.039934 (+1692896us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:34.040112 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2899664768}
W20260709 12:22:34.098980 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 88947.5us
I20260709 12:22:34.099252 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.100337 27664 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:34.101627 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.102445 27664 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:34.103771 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.104205 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.047740  5787 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:34.105625 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.105937  5787 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.106549 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:34.107425 27661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.009s	sys 0.001s
I20260709 12:22:34.107754 27663 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:22:34.107946 27585 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:34.108219 27663 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 17
I20260709 12:22:34.108562 27663 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:22:34.006424  5637 consensus_queue.cc:1243] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 23.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:22:34.108870 27663 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:34.109146  5737 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:34.109315  5835 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:34.110450 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.111593  5750 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:34.111810  5539 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:22:34.112309  5813 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.928s	user 0.010s	sys 0.016s
I20260709 12:22:34.112779  5813 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.113425  5539 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:22:34.113744 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.113933  5539 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:34.046418  5841 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:34.114126  5713 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.956s	user 0.013s	sys 0.014s
I20260709 12:22:34.046783  5528 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:34.135780 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.006444  5681 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:34.040401  5726 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.063s	user 0.015s	sys 0.013s
I20260709 12:22:34.144469 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.145272 27660 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:34.147668 27656 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:34.143548 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:34.115890  5855 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.257s	user 0.004s	sys 0.001s
I20260709 12:22:34.148955 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:34.152731 27585 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:34.007264  5739 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:22:34.154579 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56561.2us
I20260709 12:22:34.134477 27662 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:34.115582  5803 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.115242  5813 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.157442 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:34.166602 28335 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:22:34.169565  5787 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.164953 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:34.170472 27663 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:34.171662 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:34.172250 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.037310  5680 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.173055  5811 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:34.173767  5686 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:34.174145  5680 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:34.173905  5811 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.175187  5841 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:34.176263  5811 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:34.171428  5405 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:22:34.178035  5811 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:34.178756  5811 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.180284  5648 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.180383 27664 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:34.181097  5811 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.181016  5648 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.145885  5713 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.183475  5841 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:34.183808 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.184666 27737 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:34.185436  5713 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:34.185570 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.185837 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:34.172979 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:34.027220  5593 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:34.093920  5517 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.193163  5689 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:34.193781 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:34.196820 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:34.202099 28330 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:34.165567  5824 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:34.204921 27732 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:34.166059  5726 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24.
I20260709 12:22:34.116293  5539 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.199877 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:34.217700 27659 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:34.217517 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:34.217875 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.218783 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:34.219069 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.190691  5750 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.029s
I20260709 12:22:34.221151 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.222174 27736 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:34.228118 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:34.092486 27479 rpcz_store.cc:269] 0709 12:21:49.527700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.528487 (+   787us) service_pool.cc:224] Handling call
0709 12:22:34.018816 (+44490329us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:17.319792 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:17.320510 (+   718us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:17.320564 (+    54us) write_op.cc:435] Acquired schema lock
0709 12:22:17.320585 (+    21us) tablet.cc:662] Decoding operations
0709 12:22:17.364918 (+ 44333us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:17.365015 (+    97us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:17.365042 (+    27us) tablet.cc:685] Acquiring locks for 31 operations
0709 12:22:17.410781 (+ 45739us) tablet.cc:701] Row locks acquired
0709 12:22:17.410836 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:22:17.411032 (+   196us) write_op.cc:270] Start()
0709 12:22:17.411187 (+   155us) write_op.cc:276] Timestamp: P: 1783599737410967 usec, L: 0
0709 12:22:17.411209 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:22:17.413564 (+  2355us) log.cc:844] Serialized 12110 byte log entry
0709 12:22:17.952231 (+538667us) op_driver.cc:464] REPLICATION: finished
0709 12:22:18.295247 (+343016us) write_op.cc:301] APPLY: starting
0709 12:22:18.324249 (+ 29002us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:19.461045 (+1136796us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b1800426e20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9a0870ec 00007f0e9a0d3c0a 00007f0e9a0d34f7 00007f0e9a0d3490 00007f0e9a08af48 00007f0e9dd9f1bb 00007f0e9dd9f532 00007f0e9dde6b93 00007f0e9dc93de3
0709 12:22:19.990516 (+529471us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:19.990566 (+    50us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:21.522708 (+1532142us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:21.748215 (+225507us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=62,key_file_lookups=62,delta_file_lookups=0,mrs_lookups=0
0709 12:22:21.769490 (+ 21275us) write_op.cc:312] APPLY: finished
0709 12:22:22.081611 (+312121us) log.cc:844] Serialized 268 byte log entry
0709 12:22:22.541337 (+459726us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:22.541723 (+   386us) write_op.cc:454] Released schema lock
0709 12:22:22.541807 (+    84us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:22.542300 (+   493us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:22.542329 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":318958,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":236447,"apply.run_cpu_time_us":82187,"apply.run_wall_time_us":4304006,"cfile_cache_hit":126,"cfile_cache_hit_bytes":133707,"num_ops":31,"prepare.queue_time_us":442163,"prepare.run_cpu_time_us":25800,"prepare.run_wall_time_us":287089,"raft.queue_time_us":184586,"raft.run_cpu_time_us":43853,"raft.run_wall_time_us":255792,"replication_time_us":494940,"spinlock_wait_cycles":8279424,"thread_start_us":1099831,"threads_started":5,"wal-append.queue_time_us":764346,"wal-append.run_cpu_time_us":13871,"wal-append.run_wall_time_us":2178555}]]}
I20260709 12:22:34.217762 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:34.195825  5689 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:34.236325  5689 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.237915  5689 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:34.239889  5617 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:22:34.240199 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46621.1us
I20260709 12:22:34.218787  5824 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.241240 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:34.249730  5405 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:34.251106  5785 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:22:34.252867 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 215753us
I20260709 12:22:34.229897 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.206137  5713 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:34.259256 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:34.211983 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.261101  5617 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:34.262775  5824 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.264578  5737 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.267033 27663 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
W20260709 12:22:34.268162 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71682.5us
I20260709 12:22:34.271153 27585 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:34.264396  5517 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:34.269449 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 57554.4us
I20260709 12:22:34.259940  5713 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.265739  5617 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:34.291797 27552 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.012s	sys 0.011s
W20260709 12:22:34.300261 27552 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.012s	sys 0.012s
I20260709 12:22:34.294368 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:34.295110 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.306300  5785 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:34.273679  5739 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:34.270725  5787 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:34.308632 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:34.300662 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 697416us
I20260709 12:22:34.306852  5785 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.292822  5593 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:34.309778 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.301944  5713 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:34.303877  5617 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:34.318650 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.327456 27660 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:34.307302  5824 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.315697  5739 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.313016  5539 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:34.317468  5785 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:34.316577  5700 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:34.336918 27655 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:34.321012  5593 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.343441  5824 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.344277  5593 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:34.338078 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:34.315526  5787 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:34.345271  5787 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:34.346118  5824 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.337590 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.299059  5750 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:22:34.349382 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:34.347096 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74625.6us
I20260709 12:22:34.328755  5726 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:34.352306 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:34.354050 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51369.7us
W20260709 12:22:34.354517 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124742us
I20260709 12:22:34.354842 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.310621  5737 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.355300  5803 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:34.355832 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:34.359783 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:34.337196  5739 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:34.338867  5686 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:34.370976 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:34.341588  5700 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.369557 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:34.393807 27580 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:34.398607 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1797) took 1848 ms. Trace:
W20260709 12:22:34.412964 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 200453us
I20260709 12:22:34.413255 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:34.355166  5405 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:34.391395  5803 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.393142  5686 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.416962 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:34.415382 27582 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:34.402561 27659 rpcz_store.cc:276] 0709 12:22:32.544709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.983134 (+438425us) service_pool.cc:224] Handling call
0709 12:22:33.001602 (+ 18468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.218307 (+1216705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.392488 (+174181us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:34.392744 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2724323072}
W20260709 12:22:34.414521 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 140547us
I20260709 12:22:34.409621  5813 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.403427  5700 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:22:34.427385 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 129505us
I20260709 12:22:34.431468 27587 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:34.431809 28333 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:34.433817 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:34.438012 27581 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:22:34.439270 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:34.440279 27581 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:34.441411 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:34.442237 27581 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:34.442824 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.444550 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.445542 27731 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:34.445663 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.446408 27738 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:34.446935 27737 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:34.448429 27732 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:22:34.448606 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.434240 28333 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:34.409395  5750 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:34.450162 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.410115  5528 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:34.441442  5824 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:22:34.459165  5533 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:34.459194 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.460680 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4344) took 3205 ms. Trace:
I20260709 12:22:34.461140 28330 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 21. Current term is 22. Ops: []
I20260709 12:22:34.461129  5405 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:34.467461 27585 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:34.483608 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:34.454829 28295 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:34.432143  5827 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:34.475893  5750 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.450598 28329 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:22:34.478015 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.448441 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4514) took 2710 ms. Trace:
I20260709 12:22:34.449651  5811 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:34.491214 28333 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.424228  5803 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:34.426744  5686 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:34.405620  5179 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:22:34.479923 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55892.8us
I20260709 12:22:34.482406 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:34.494966 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.455742 27583 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:34.495512 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:34.498140 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:34.494244 27584 rpcz_store.cc:276] 0709 12:22:31.737890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.647473 (+909583us) service_pool.cc:224] Handling call
0709 12:22:32.647817 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.116094 (+1468277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.423420 (+307326us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:34.448346 (+ 24926us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3202270080}
I20260709 12:22:34.481599  5528 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:34.508461 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53760.1us
I20260709 12:22:34.492940  5533 consensus_queue.cc:1243] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 33.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:22:34.432922  5813 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.479194 27661 rpcz_store.cc:276] 0709 12:22:31.255614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.702814 (+447200us) service_pool.cc:224] Handling call
0709 12:22:31.718260 (+ 15446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.718697 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.718733 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.724598 (+  5865us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.295540 (+1570942us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b18004a8d20. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:34.108029 (+812489us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:34.108087 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.108177 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.460297 (+352120us) raft_consensus.cc:1685] finished
0709 12:22:34.460320 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.460587 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":3567,"spinlock_wait_cycles":15868800,"thread_start_us":812155,"threads_started":1,"wal-append.queue_time_us":2580724}
I20260709 12:22:34.503026  5516 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.021s
I20260709 12:22:34.505324  5179 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:34.524158  5680 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:34.495666  5405 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.542737  5405 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.545179  5405 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:34.545892  5681 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:34.546324  5681 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:34.546689  5681 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:34.530807 27580 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:34.532091 28333 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:34.534247 27587 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
W20260709 12:22:34.537408 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.539778 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51132us
I20260709 12:22:34.540925  5179 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.552856  5444 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:34.555421 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71356.7us
I20260709 12:22:34.514653  5528 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:34.568441 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 166558us
I20260709 12:22:34.563280 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:34.573601  5726 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:34.574105  5726 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:34.574524  5726 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:34.570353  5444 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.581523  5750 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:34.584000  5737 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.587697  5686 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.590045  5686 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.590317  5681 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:34.590994 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.591786  5179 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:34.591833 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.595844  5805 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.596863  5737 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:34.599468  5775 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:34.600461  5516 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:22:34.600787  5516 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:34.601739  5516 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.568747  5680 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:22:34.604298  5680 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:34.604651  5680 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:34.605538  5680 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.566880  5779 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.041s
I20260709 12:22:34.606197  5785 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.607476 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:34.616513 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.617975 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.618207 27630 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.618971  5716 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:22:34.619673  5716 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:34.623677  5827 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:34.625339 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.627635  5805 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:34.628060  5805 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:34.628448  5805 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:34.636195  5630 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:22:34.636593  5630 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:34.637461  5630 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.638782  5630 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:34.609203  5813 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.639927  5824 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:22:34.644023  5863 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 12:22:34.607091  5680 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:34.610859 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.610526 28333 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260709 12:22:34.652922 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:34.650552 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:34.654711 28333 pending_rounds.cc:85] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 19
I20260709 12:22:34.662979 28333 pending_rounds.cc:107] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260709 12:22:34.634847  5593 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:34.663646 28333 raft_consensus.cc:2889] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:34.639943  5813 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:22:34.665642 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54909.4us
I20260709 12:22:34.553171  5700 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.553010  5617 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.670454  5686 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:22:34.672962 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 147057us
I20260709 12:22:34.690639 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:34.694800 27588 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.043s	sys 0.082s
W20260709 12:22:34.698290 27588 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.043s	sys 0.082s
I20260709 12:22:34.674212  5737 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:22:34.828577 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 153290us
W20260709 12:22:34.856940 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 206425us
I20260709 12:22:34.857987 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:34.858878 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 163968us
I20260709 12:22:34.692842  5533 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:22:34.860944 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164768us
W20260709 12:22:34.838407 27656 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.013s	sys 0.036s
W20260709 12:22:34.863615 27656 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.014s	sys 0.036s
I20260709 12:22:34.696396  5813 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:34.830785  5737 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.829715  5405 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:34.876535  5835 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:34.906253 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:34.888839  5737 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:34.856298  5750 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:22:34.909174  5835 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:34.940842 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 284577us
W20260709 12:22:34.945255 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.946775  5444 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:34.946529  5788 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:22:34.947504  5444 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:34.926764  5517 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.936980  5835 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:34.970502 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.034s	sys 0.046s
W20260709 12:22:34.971061 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.034s	sys 0.046s
I20260709 12:22:34.960183  5811 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:35.000069 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.002722 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.968169  5750 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:35.006954  5750 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:35.007221 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.008198 27731 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 35 because replica has last-logged OpId of term: 33 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 16.
I20260709 12:22:34.981082 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:34.986584  5861 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.586s	user 0.001s	sys 0.012s
I20260709 12:22:34.989184 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.985319  5841 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:34.991560  5593 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:34.991397 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.995445  5785 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:34.996235 27733 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.085s	sys 0.105s
W20260709 12:22:35.020768 27733 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.085s	sys 0.105s
I20260709 12:22:34.996134 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1509) took 1123 ms. Trace:
I20260709 12:22:35.021327 28329 rpcz_store.cc:276] 0709 12:22:33.872382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.875375 (+  2993us) service_pool.cc:224] Handling call
0709 12:22:34.995863 (+1120488us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:34.996047 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1274779008}
I20260709 12:22:34.998324  5726 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.003593  5617 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:35.003885  5866 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
W20260709 12:22:34.972041  5862 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
I20260709 12:22:35.003913 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.961827  5681 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.038143 27738 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:35.038379  5681 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.039047  5681 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.038937  5811 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:34.990223 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:34.999486  5648 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.031925 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.033006  5593 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.033142  5843 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:35.035077  5726 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:35.051677 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 77418.1us
I20260709 12:22:34.978487  5517 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:35.072849 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 448417us
I20260709 12:22:35.066165  5516 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:34.895854 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:35.087469 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124915us
I20260709 12:22:35.021534  5700 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.029505  5788 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:35.008369  5750 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.025648  5841 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:35.083113  5843 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:35.146188 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.155051 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.164861 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:35.168251  5750 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:35.176798  5787 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.831s	user 0.006s	sys 0.004s
I20260709 12:22:35.177034  5700 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.177587  5787 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.177939  5700 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:35.143703  5803 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.616s	user 0.011s	sys 0.001s
I20260709 12:22:35.041575  5681 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.167080 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:35.197624 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 70358.1us
W20260709 12:22:35.176604  5689 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.938s	user 0.013s	sys 0.002s
I20260709 12:22:35.012254  5813 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:35.184196 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 305640us
I20260709 12:22:35.093695  5593 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:35.206097 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.158074  5843 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:22:35.206076  5805 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.001s
W20260709 12:22:35.216897 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 324810us
I20260709 12:22:35.216193 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
W20260709 12:22:35.209654 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.136369  5528 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:35.220670 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 336706us
W20260709 12:22:35.225889 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 367990us
I20260709 12:22:35.221077 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1798) took 2675 ms. Trace:
W20260709 12:22:35.217818  5630 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.579s	user 0.006s	sys 0.009s
I20260709 12:22:35.239022 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:35.226269 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4370) took 1228 ms. Trace:
W20260709 12:22:35.224509  5739 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.805s	user 0.008s	sys 0.004s
I20260709 12:22:35.220350  5648 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:35.232396  5539 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.892s	user 0.006s	sys 0.006s
I20260709 12:22:35.214824  5876 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:35.223387  5775 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:35.239936 27656 rpcz_store.cc:276] 0709 12:22:33.997509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.164128 (+166619us) service_pool.cc:224] Handling call
0709 12:22:34.200902 (+ 36774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.226948 (+ 26046us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:34.872890 (+645942us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.225914 (+353024us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:35.225975 (+    61us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:35.226026 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.226085 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.226215 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40328,"prepare.queue_time_us":108881,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":25767,"thread_start_us":859831,"threads_started":2}
I20260709 12:22:35.205129  5803 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:35.263586 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.227989 27657 rpcz_store.cc:276] 0709 12:22:32.545597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.193616 (+648019us) service_pool.cc:224] Handling call
0709 12:22:33.194005 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.194385 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.194419 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.194884 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.028080 (+833196us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:34.028167 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.028330 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.951401 (+923071us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:35.215076 (+263675us) raft_consensus.cc:1685] finished
0709 12:22:35.215097 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.220989 (+  5892us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":39247,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":20188,"spinlock_wait_cycles":13231872,"thread_start_us":812953,"threads_started":1,"wal-append.queue_time_us":1125489}
I20260709 12:22:35.241197  5681 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:35.263680  5716 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.644s	user 0.012s	sys 0.001s
I20260709 12:22:35.222905  5805 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.230625  5533 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:35.263474  5775 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:35.296907 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57971us
W20260709 12:22:35.298540  5179 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.706s	user 0.008s	sys 0.002s
I20260709 12:22:35.256672  5739 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.309543  5739 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.295500  5630 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.016s
I20260709 12:22:35.316224  5700 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.318567  5593 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.264467  5648 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.319687  5517 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.321206  5648 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.321429  5787 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.322253  5875 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:35.322993  5875 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:35.323921  5713 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.983s	user 0.013s	sys 0.000s
I20260709 12:22:35.292816  5716 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:35.325448 27736 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 33 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
W20260709 12:22:35.329618 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 162693us
I20260709 12:22:35.330139 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1845) took 1296 ms. Trace:
I20260709 12:22:35.333905 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3591) took 1298 ms. Trace:
I20260709 12:22:35.339223 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:35.334585 27662 rpcz_store.cc:276] 0709 12:22:34.035128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.166459 (+131331us) service_pool.cc:224] Handling call
0709 12:22:34.190905 (+ 24446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.211261 (+ 20356us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:34.271699 (+ 60438us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.961525 (+689826us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b1800356e00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:35.244938 (+283413us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.333401 (+ 88463us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:35.333484 (+    83us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:35.333555 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.333641 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.333848 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30263,"prepare.queue_time_us":108651,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":102,"raft.queue_time_us":58609,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":7882,"spinlock_wait_cycles":3062656,"thread_start_us":337453,"threads_started":2}
I20260709 12:22:35.321472  5775 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:35.340584 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:35.344727 28333 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
W20260709 12:22:35.342036 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 125657us
I20260709 12:22:35.347652  5739 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:35.348433  5739 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.349793 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:35.258860  5841 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.331517  5883 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:22:35.357748  5883 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:35.358445  5883 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.360029 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.360677  5700 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:35.361393  5700 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.362799  5593 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 12:22:35.321429  5528 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:35.364327  5528 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:35.364732  5528 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.365128  5528 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:35.365694  5716 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:35.366302  5716 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:35.366431  5405 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:35.370666  5875 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.371920  5824 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:35.372466  5824 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:35.372048  5533 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.373453  5824 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.374660  5824 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:35.376319  5179 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:35.376739  5179 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:35.364961 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.381817  5835 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.349978  5637 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:35.363530  5593 consensus_queue.cc:1243] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 29.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:22:35.384956  5179 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.395077  5700 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.396436 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:35.397293 28333 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:35.402562 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.410995 27730 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:35.414671 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.417582 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.419220 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.422134 27738 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:35.420112 27580 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 35 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
W20260709 12:22:35.422406 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72699.6us
I20260709 12:22:35.330946 27588 rpcz_store.cc:276] 0709 12:22:34.034142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.085958 (+ 51816us) service_pool.cc:224] Handling call
0709 12:22:34.112729 (+ 26771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.113066 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:22:34.825258 (+712192us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.223126 (+397868us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.329704 (+106578us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:35.329763 (+    59us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:35.329822 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.329912 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.330064 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28614,"prepare.queue_time_us":873242,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":87,"raft.queue_time_us":115315,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":20691,"thread_start_us":1000406,"threads_started":3}
I20260709 12:22:35.420068 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:35.426694 28329 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:35.427452 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.428067 27581 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:35.420742 27732 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 16
I20260709 12:22:35.430109 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.390661  5637 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:35.431175 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.431450 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:35.430078 27732 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 33.17
I20260709 12:22:35.437350 27732 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:35.433202 27584 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:35.435117  5179 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:35.438043 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75530.8us
I20260709 12:22:35.430742 27734 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:35.441856  5734 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:22:35.442379  5894 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:35.443027  5894 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.443833  5843 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.446123  5894 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.446245  5843 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:35.334987  5713 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.411556  5630 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:35.451663 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:35.430431  5686 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.734s	user 0.009s	sys 0.004s
I20260709 12:22:35.468314 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.471439 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.429816  5811 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.453183 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.429935  5700 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:35.441222  5805 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.499439 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.503672 27733 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:35.414476  5737 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.505101 27658 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:35.506053 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.502542 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:35.507220 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:35.510461 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.511092 27581 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:22:35.502967 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:35.512699 27580 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:35.514321 27660 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 35 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:22:35.475602  5686 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:35.452461  5894 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:35.522848 27588 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:35.520175 27583 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:22:35.520989 28333 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:22:35.519718 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:35.528149 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:35.529073 27737 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:22:35.529313 27658 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:35.532439 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 80875.3us
I20260709 12:22:35.536214 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.506273  5841 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:22:35.550981 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.550966 27584 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:22:35.540550 27664 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:22:35.553391 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.07835e+06us
I20260709 12:22:35.554960 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.555429  5885 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.555586 27588 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:35.556028  5885 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.558471  5885 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:35.565100  5892 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:35.565666  5892 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:35.566963  5889 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
W20260709 12:22:35.567435 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.568593  5881 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
W20260709 12:22:35.569819  5888 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:22:35.539436  5516 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:22:35.571575 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51998.5us
I20260709 12:22:35.572021 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.500808  5630 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.572702 27580 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:35.575380 28329 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:35.577383 27582 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:35.554040 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.585763 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:35.586608 27660 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:35.526638  5788 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:35.554804  5841 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:35.587998  5803 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.594084  5841 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:35.596565  5713 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.597350  5637 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.598909  5637 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:35.600389  5528 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.601562  5883 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.601719 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.588778 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.505699 27481 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 26 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 27 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.532186  5405 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.588742 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.604239 27582 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:35.604380 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.591449 27588 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:35.610869 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.611894 27588 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:35.615559 27584 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 20. (term mismatch)
I20260709 12:22:35.616035 27584 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 19
I20260709 12:22:35.616299 27584 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:22:35.616613 27584 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:35.617712 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.618535 27580 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:35.537623  5835 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:35.605808 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.622623 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.615020 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.538172  5811 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:35.626914 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.624060 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.566633  5892 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.639432 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.641215 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.650729 27585 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:35.641897 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:35.651887 27661 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:35.643507  5897 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:22:35.657891 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55609.3us
I20260709 12:22:35.658772 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:35.645982 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.646044 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:35.660300 28333 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:35.642935 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.655416  5739 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.655869 27582 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:35.656718 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.614197  5680 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:35.668522 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.665297 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.667349 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:35.669009 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.669440 27588 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:35.669971 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:35.670506 27730 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:35.632387  5885 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:35.672286 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:35.673296 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 119599us
I20260709 12:22:35.678650 27735 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:22:35.680986 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.681612 27588 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:22:35.639436  5788 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:35.687767 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.688860 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.658360  5897 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.687789 27731 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:35.748461 27582 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:35.531630  5716 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.672557  5805 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:35.621874  5894 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.756105 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:35.758831 27585 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:35.674612  5680 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.664907  5528 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:35.645146  5617 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.103s
I20260709 12:22:35.767817 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.680212  5885 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:35.664160  5539 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:35.763922 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.772643 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.772840 27582 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 22.
I20260709 12:22:35.683802  5750 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:35.768407 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.679706  5713 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:35.633045  5405 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:35.754110 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61225.2us
I20260709 12:22:35.764434  5700 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:35.771311  5680 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:35.775061 27733 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:22:35.688651  5788 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.784857 27588 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:35.774528 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:35.748765  5779 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:35.801932  5617 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 35.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:35.804452 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.760648  5897 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:35.764605  5805 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.839293  5805 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:35.841979  5905 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:22:35.818596  5533 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:35.844270  5533 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:35.844726  5533 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:35.824142 27661 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:35.770265 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.850788 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:35.851255 27580 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:35.763170  5739 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:35.809967 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:35.855130 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45245.2us
W20260709 12:22:35.766747 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.774417  5885 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:35.857438 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.859500 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:35.859812 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.859997 27658 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:35.807616 27730 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:35.813546 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.822256 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:35.822815 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59507.7us
I20260709 12:22:35.826562 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:35.828589 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.828814  5873 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:22:35.829427  5787 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:35.829941 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.830135 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.836604  5894 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.809468 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:35.819972  5899 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:22:35.812562  5750 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:35.870929 27729 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:35.884537 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.885870 27730 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:35.802130  5539 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.822968  5788 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:35.869612 27588 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:35.869769 27582 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:35.871076 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.869985 27660 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
W20260709 12:22:35.875473 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.889104 27586 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:35.858019  5877 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:35.883687  5739 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.889247  5750 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:35.901360  5630 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.935724  5893 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.936597  5516 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.936311  5893 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.937196  5516 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.938647  5893 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.938798  5894 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.939513  5894 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.940945  5739 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.902741  5405 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:35.941605  5894 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.903410 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:35.942313  5896 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.276s	user 0.004s	sys 0.000s
I20260709 12:22:35.942487 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:35.943308 27589 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:35.944343 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.944829 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:35.945499 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.946556 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.948275 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:35.948982 28333 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 23.
I20260709 12:22:35.906852 27731 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:35.911603 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.950071 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:35.922560  5785 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:22:35.890022  5681 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.002s
I20260709 12:22:35.924182 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.924906 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:35.854815  5779 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.963353  5405 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:35.964143  5878 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.964262  5405 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:35.965185  5779 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:35.965888  5405 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.929389  5873 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:35.973400 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48587.7us
W20260709 12:22:35.974035  5903 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.344s	user 0.003s	sys 0.000s
W20260709 12:22:35.975026 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.971189 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:35.931722 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74039us
I20260709 12:22:35.933400  5539 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.981225 27664 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:22:35.975879 27733 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:35.945556 27737 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:35.950318 27582 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:35.914765 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:35.912747  5811 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:35.896812 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:35.950446 27580 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:35.951393 27588 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:36.003075 27588 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:35.818221  5700 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:35.869092  5737 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.957595 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.946974  5897 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:35.967087 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 116380us
I20260709 12:22:35.967770  5891 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:22:35.971678 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.744s (SENT) [suppressed 17 similar messages]
I20260709 12:22:35.971772  5517 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.890290  5787 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.894264  5835 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:22:35.972486 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.979810  5884 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:35.974112  5873 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:35.957403 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:35.980561 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:36.054864 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98949.8us
I20260709 12:22:35.950350 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.055599 27731 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:36.007290 27733 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:36.010316 27580 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:22:36.056017 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:36.013092 27584 raft_consensus.cc:2393] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:36.010241 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.015674  5179 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:22:36.018431 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44090.2us
I20260709 12:22:36.021672 27583 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
W20260709 12:22:36.025702 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 269650us
I20260709 12:22:36.027551 27582 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:35.958617  5516 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:35.981813  5681 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:35.907124  5841 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.050824 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:35.981235  5686 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.070s
I20260709 12:22:36.073107 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:36.066357 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:36.072549 27737 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:36.074594 28330 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:22:36.075809 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118310us
I20260709 12:22:36.074461 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:36.085289 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.017465  5883 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.053s
W20260709 12:22:36.092804 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 197078us
W20260709 12:22:36.091382 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.964792  5878 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.071894  5811 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:36.096094 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.064280  5630 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:36.101832 27580 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:36.094511 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:36.074282  5713 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:36.099689 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:36.094209 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:36.105216  5726 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:36.105993  5726 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:36.106536  5726 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:36.105285  5916 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:22:36.107038 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.073028  5876 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:36.109737 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.080847  5891 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:36.116276 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.116163 28329 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:36.117408 27580 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:36.110374  5813 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:36.113646 27584 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:36.054004  5517 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.125133 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:36.126988 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48353.8us
I20260709 12:22:36.131193 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.108348  5883 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 33.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:36.108729  5713 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.137293 28335 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 22: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:36.140472 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66518.5us
I20260709 12:22:36.149060 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:36.149559  5179 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:36.154333 27584 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:36.154016  5179 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.123910  5813 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.093042  5901 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:22:36.127386  5811 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:36.118125  5630 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.174469 27656 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:36.134589  5916 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:36.151517  5910 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.160543 27661 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:36.151062  5713 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:36.147222  5716 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.177610 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.158808 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.156900 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:36.189826 28329 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:36.114207  5876 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.177062 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.192984 27659 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:36.168767  5813 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:36.163527  5894 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:36.195670 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.198587 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:36.217584 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:36.194566  5824 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.819s	user 0.017s	sys 0.003s
I20260709 12:22:36.227113 27656 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:36.227269 27661 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: [33.24-33.24]
W20260709 12:22:36.192929  5775 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.821s	user 0.012s	sys 0.000s
I20260709 12:22:36.234938 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.235863 27661 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:36.236438 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.238195 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.247825 27658 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:36.240904 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.248688 27659 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:22:36.196251  5686 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:36.201535  5923 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.231477  5884 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:22:36.264856 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:36.262892 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 130315us
W20260709 12:22:36.252053  5637 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.647s	user 0.011s	sys 0.000s
I20260709 12:22:36.281448 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:36.265353 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.287927 27658 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:36.193426  5716 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:36.293535 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 285982us
I20260709 12:22:36.296746 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.298494  5912 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.259781  5700 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.195514  5910 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.267791  5897 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:36.247903  5775 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.266259  5884 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:36.305120 27661 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:22:36.292495  5879 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:36.191061  5891 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.313812 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:36.311436  5912 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.320035  5517 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:36.321760  5912 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:36.322714  5917 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:22:36.323241 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:36.324015 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.324658 27588 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:36.325744 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.326292 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:36.327926 27589 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:22:36.330446 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:36.331969 27588 raft_consensus.cc:1217] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Deduplicated request from leader. Original: 24.12->[26.13-30.15]   Dedup: 26.13->[28.14-30.15]
I20260709 12:22:36.332415 27588 pending_rounds.cc:85] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:22:36.332659 27588 pending_rounds.cc:107] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 29.14
I20260709 12:22:36.332892 27588 raft_consensus.cc:2889] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:36.337634  5805 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:36.339860  5918 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:22:36.341440  5910 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:36.344542  5680 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.008s
I20260709 12:22:36.344696  5775 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.345021  5680 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.307554  5916 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.312423  5884 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:36.312804  5787 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.308713  5841 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:36.314896  5879 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.359140  5927 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:22:36.359907  5919 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:22:36.361388  5716 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.268745  5923 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.361763  5885 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.363395 28335 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:22:36.363955 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.376798 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.389513 27580 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:36.379117  5932 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:36.356190  5876 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.325060  5878 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:36.382853 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 411178us
W20260709 12:22:36.383579 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60428.7us
I20260709 12:22:36.294138  5811 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.348204  5680 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.390920 27663 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:36.392413 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.407495 27580 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:22:36.377511  5879 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:36.380010 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 369005us
I20260709 12:22:36.396688 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:36.392275  5931 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:22:36.403025 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.384903  5894 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:36.388624  5927 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:36.393875  5775 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:36.416548 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.416456 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.403268  5932 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:36.417567  5875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.003s	sys 0.009s
I20260709 12:22:36.428222 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:36.427601 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.438194 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.423399 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.448768 27580 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:36.439407 27738 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:36.413172  5630 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.447722 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:36.450704 27737 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:36.449345 27731 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:22:36.451776 27657 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 33 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:22:36.413957  5805 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:36.441581  5875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.003s	sys 0.010s
I20260709 12:22:36.453644 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.430748  5927 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:36.423009  5893 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:36.460536 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.419075  5750 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.464075 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.467867 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.468291 27659 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22.
I20260709 12:22:36.469108 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:36.461736  5928 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:22:36.473788 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.474061 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:36.440575  5813 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.472856 27655 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:22:36.441886  5775 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.472011 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.502013 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:36.503516 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:36.506266  5405 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:22:36.506068  5894 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.507083  5879 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:36.509483  5935 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:22:36.509632 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.510607 27583 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:36.511070 27583 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:36.511656 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.510802  5689 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:36.511890 27583 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.513031 27583 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:36.528582  5788 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.003s
I20260709 12:22:36.529193  5788 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.531723 27585 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:36.513655 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.514664 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.515235 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.534360 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
W20260709 12:22:36.516023  5938 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:22:36.516201 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.517313  5885 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:36.518895  5936 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:22:36.519654  5929 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
W20260709 12:22:36.522725  5533 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.677s	user 0.007s	sys 0.002s
W20260709 12:22:36.524577  5934 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:22:36.525053  5686 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:36.504747 27738 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:36.538822 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:36.541669 27583 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 20. (term mismatch)
I20260709 12:22:36.542748 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.550421 27583 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 19
I20260709 12:22:36.550707 27583 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 35.20
W20260709 12:22:36.545481 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75628.2us
I20260709 12:22:36.551600 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:36.554217 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.546378 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:36.548159 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.526966  5923 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:36.550928 27583 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:36.555352 27661 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:36.556214 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.557221 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.557548 27580 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:22:36.509511  5879 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:36.557622 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.579941 27583 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:22:36.556844 27734 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:36.558544 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.558486 27658 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:36.502939  5726 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.564191  5892 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:36.567775 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.568141  5539 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.571352 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.594026 28328 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:36.572173 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:36.572078  5884 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.597864 28336 raft_consensus.cc:2410] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 37 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 18.
I20260709 12:22:36.591831 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:36.572885  5750 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:36.574726  5775 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.599682 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.602025 28335 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:36.599472  5885 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.009s
I20260709 12:22:36.605824  5788 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.570593  5713 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.606541  5885 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:36.606937  5885 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:36.607885  5885 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.608573  5713 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.609261  5885 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:36.612722  5915 raft_consensus.cc:2764] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:22:36.613938  5921 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:22:36.614655  5884 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.623095  5539 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.504974  5893 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:36.579092  5405 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:22:36.683737 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.689307 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93316.4us
W20260709 12:22:36.695870 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.696359 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.703852 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3761) took 1015 ms. Trace:
I20260709 12:22:36.702927 27657 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:36.679672  5726 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.730465 27730 rpcz_store.cc:276] 0709 12:22:35.688576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.954219 (+265643us) service_pool.cc:224] Handling call
0709 12:22:36.008235 (+ 54016us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:36.019204 (+ 10969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.019260 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:36.052367 (+ 33107us) log.cc:844] Serialized 52 byte log entry
0709 12:22:36.291887 (+239520us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:36.291937 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.292047 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:36.703541 (+411494us) raft_consensus.cc:1685] finished
0709 12:22:36.703573 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.703775 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":331,"raft.queue_time_us":133394,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7217,"thread_start_us":182894,"threads_started":1,"wal-append.queue_time_us":255149}
I20260709 12:22:36.550052  5533 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:36.681545  5936 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:22:36.747725 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.749696 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:36.754647  5949 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:22:36.761487 27588 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:36.761649 27657 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:22:36.762751 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:36.762882 28328 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:36.690050  5405 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:36.767688 27660 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:36.764707 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:36.836810 27588 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:22:36.789605 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.833745 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.834728 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.835608 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.839157 27656 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:36.839927 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.841212 27659 raft_consensus.cc:2368] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
W20260709 12:22:36.842419 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 288250us
I20260709 12:22:36.840822 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.843511 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:36.558604  5932 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:36.856050 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90638.9us
I20260709 12:22:36.846676 27663 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:36.851421 28328 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:36.840386  5405 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:36.872278 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:36.878660 27588 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:36.879551 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:36.881469 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.863291  5897 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:36.888746 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:36.886333  5921 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:36.890318 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:36.886595  5689 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.006s
I20260709 12:22:36.894732  5533 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.899657 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:36.900159 27588 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:22:36.900863 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.870739  5932 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:36.898526 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.904129  5916 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:36.905391 27663 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:36.874091  5878 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.035s
I20260709 12:22:36.884562  5775 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:22:36.903479  5921 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:36.910760  5813 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.918222 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:36.919731 27588 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:36.920571 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:36.923960  5893 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:36.925206  5878 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:36.925573  5878 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:36.926409  5878 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:36.906029  5879 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.011s
W20260709 12:22:36.927515 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.930613 27588 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:36.957051 27481 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 27 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:36.961546 27481 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 29, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 29 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:36.935094  5921 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:22:36.971685 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4395) took 1373 ms. Trace:
I20260709 12:22:36.972173 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:36.972930 27660 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:36.972359 27662 rpcz_store.cc:276] 0709 12:22:35.598692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.822466 (+223774us) service_pool.cc:224] Handling call
0709 12:22:35.826029 (+  3563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.047971 (+221942us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.048019 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.048085 (+    66us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:36.048134 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.971301 (+923167us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:36.971389 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.971625 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":509790080}
I20260709 12:22:36.946465  5835 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:36.947638  5539 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:36.973968 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:36.966441  5878 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:36.956931  5912 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:36.991068 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:36.980448 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:36.940719  5897 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:22:36.995175 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.000859 27659 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 22. Current term is 23. Ops: []
I20260709 12:22:36.936012  5750 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:22:37.012965 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.017117 28335 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:37.021872 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.039297 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:37.023350 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.024154 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.024734  5630 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:22:37.033165  5927 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.566s	user 0.005s	sys 0.007s
I20260709 12:22:37.000135  5879 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:37.052309  5788 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:37.034904  5787 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:37.041514 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.047885  5891 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:37.007993  5912 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.048188 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:36.995290  5835 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:37.049353  5904 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.418s	user 0.004s	sys 0.014s
I20260709 12:22:37.050153 28332 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
W20260709 12:22:37.059713 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 175168us
I20260709 12:22:37.021552  5921 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:22:37.055053 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.015461  5775 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:37.050673 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1521) took 1031 ms. Trace:
I20260709 12:22:37.066043 28329 rpcz_store.cc:276] 0709 12:22:36.019391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.019801 (+   410us) service_pool.cc:224] Handling call
0709 12:22:37.050496 (+1030695us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:37.050603 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97060224}
I20260709 12:22:37.066957 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:37.067545 27662 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:37.069649 27587 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:22:37.068162 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:37.069408  5877 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:37.071157 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:37.071903 27659 raft_consensus.cc:2376] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 22. Re-sending same reply.
I20260709 12:22:37.053676  5897 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:37.053022  5787 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:37.071043 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:37.077055 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.089258 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.085484 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:37.092525 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.059280  5788 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:37.100725 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:37.102262 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.055951  5927 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.066073  5891 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:37.075969  5897 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:37.091135  5630 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:22:37.094256  5923 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:37.094274  5775 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:37.094403  5879 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:37.133219 27662 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:37.140344 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.106087  5956 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:22:37.146378 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69394.5us
I20260709 12:22:37.147013 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:37.137085  5892 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.003s
I20260709 12:22:37.148854 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.129264  5923 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.137705  5877 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:37.142073  5775 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:37.367203  5923 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:37.183338 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.187551  5891 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.367290  5517 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:37.368376 27658 raft_consensus.cc:2376] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30. Re-sending same reply.
W20260709 12:22:37.366724 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 217991us
I20260709 12:22:37.372965 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.192633  5927 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:37.192672  5932 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:22:37.374222  5932 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:22:37.374573  5932 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:37.374756 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.375499  5892 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:37.375414  5932 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.376214  5726 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:37.376281  5892 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:37.376736  5932 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:37.377032  5726 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.377571  5892 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:37.378005  5892 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:37.368070  5775 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.197839  5920 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.148396  5689 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:22:37.382849  5689 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:37.383715  5726 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.383894  5689 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.270035 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4246) took 1868 ms. Trace:
W20260709 12:22:37.278613 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 345104us
I20260709 12:22:37.208256  5912 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:37.157605  5897 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.291402 27582 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
W20260709 12:22:37.301316 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 170456us
I20260709 12:22:37.152928  5956 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:37.387301 27732 rpcz_store.cc:276] 0709 12:22:35.401889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.402043 (+   154us) service_pool.cc:224] Handling call
0709 12:22:35.420448 (+ 18405us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:35.466026 (+ 45578us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.466076 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:35.466764 (+   688us) log.cc:844] Serialized 52 byte log entry
0709 12:22:35.630765 (+164001us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:35.630806 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.630882 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.048733 (+1417851us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:37.155346 (+106613us) raft_consensus.cc:1685] finished
0709 12:22:37.155380 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.269931 (+114551us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17670,"raft.queue_time_us":2113,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":22899,"spinlock_wait_cycles":13669632,"thread_start_us":163528,"threads_started":1,"wal-append.queue_time_us":163992}
I20260709 12:22:37.389622 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.389534  5956 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.390287 27732 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:37.313320 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:37.311363 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:37.315559 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:37.391664 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:37.391708 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.300505  5835 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:37.392213 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.313459 27731 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:37.392940 27729 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:37.393332  5879 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:37.394119 27729 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:37.338405 27583 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:37.395259 27552 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:22:37.345341  5886 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:37.348070  5405 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:37.350956  5953 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.336s	user 0.018s	sys 0.021s
I20260709 12:22:37.352059 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:37.353065 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.395460 27729 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:37.405113 27738 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:37.392959 27733 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:22:37.406699 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4525) took 1094 ms. Trace:
I20260709 12:22:37.407074 27584 rpcz_store.cc:276] 0709 12:22:36.311887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.576972 (+265085us) service_pool.cc:224] Handling call
0709 12:22:36.577336 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:36.577694 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.577725 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:36.578331 (+   606us) log.cc:844] Serialized 52 byte log entry
0709 12:22:36.900743 (+322412us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:36.900791 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.900869 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.350357 (+449488us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:37.361206 (+ 10849us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:37.406523 (+ 45317us) raft_consensus.cc:1685] finished
0709 12:22:37.406542 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.406647 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8634,"raft.run_cpu_time_us":10435,"raft.run_wall_time_us":770457,"spinlock_wait_cycles":17258112,"thread_start_us":322001,"threads_started":1,"wal-append.queue_time_us":423785}
I20260709 12:22:37.407363  5405 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.388680 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.410071 27552 leader_election.cc:403] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:37.411055 27552 leader_election.cc:400] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:37.411401 27552 leader_election.cc:403] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:37.411551  5912 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:37.387616  5726 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.411952  5879 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31.
I20260709 12:22:37.412424  5912 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:37.412817  5879 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:22:37.413555  5835 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:37.414026  5835 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:37.414799 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:37.415007  5835 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:37.355803 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 255115us
I20260709 12:22:37.396493 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4510) took 1796 ms. Trace:
I20260709 12:22:37.416268 27589 rpcz_store.cc:276] 0709 12:22:35.599623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.945255 (+345632us) service_pool.cc:224] Handling call
0709 12:22:37.396310 (+1451055us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:37.396437 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":870175488}
I20260709 12:22:37.418154 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.420347  5835 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:37.423913 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:37.347584 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4389) took 1877 ms (client timeout 1810 ms). Trace:
I20260709 12:22:37.357713 27580 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:37.412510  5726 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.425096  5915 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:22:37.432135 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.410571  5637 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:37.358357 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4550) took 1524 ms. Trace:
I20260709 12:22:37.434013  5637 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:37.434217 27585 rpcz_store.cc:276] 0709 12:22:35.833700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.870365 (+ 36665us) service_pool.cc:224] Handling call
0709 12:22:37.357386 (+1487021us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54000b1154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:37.358240 (+   854us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1335470976}
I20260709 12:22:37.360715 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.364754 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:37.437187 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72530.7us
I20260709 12:22:37.437361  5726 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:37.437988 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.438179  5726 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:37.438926  5726 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:37.382655  5920 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.440127 27731 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
I20260709 12:22:37.440491 27731 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 15
I20260709 12:22:37.388494  5897 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:37.443432 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:37.390394 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.444166  5813 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:37.444525  5813 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.344573 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 7274fd8a2115405c9b78c61214bff736 with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:51028
I20260709 12:22:37.444881  5813 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:37.445152 27736 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:37.447641  5927 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.448133  5927 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.450630  5927 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.451349 27729 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 19. (term mismatch)
I20260709 12:22:37.451962 27729 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:22:37.452280 27729 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 36.19
I20260709 12:22:37.452924 27729 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:37.415547  5886 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:37.452423  5927 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.453766  5813 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.454432  5927 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.454769  5886 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.455997  5886 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:37.456241  5887 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:37.456609  5927 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.457758  5887 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:37.354050 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.460877 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:37.462553 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.378913  5892 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.348098 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:37.345566  5787 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.435864  5912 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.463752 27656 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:37.463886  5912 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.464757 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:37.465238 27656 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:22:37.465979 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:37.466430  5912 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.466657 27656 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:37.366766 27662 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:22:37.440719 27731 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 29.16
W20260709 12:22:37.452705 27664 rpcz_store.cc:269] 0709 12:22:35.469666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.470708 (+  1042us) service_pool.cc:224] Handling call
0709 12:22:37.347382 (+1876674us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:37.347507 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71422208}
I20260709 12:22:37.467801 27731 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:37.453606  5897 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:37.468161  5516 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:37.468617  5516 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:37.468425  5897 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.469030  5516 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:37.470633  5897 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:37.462864  5914 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:22:37.471931  5787 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:37.474431 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:37.356439 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4387) took 1887 ms. Trace:
I20260709 12:22:37.475138 27738 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:22:37.475199 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 60482.1us
I20260709 12:22:37.475478 27655 rpcz_store.cc:276] 0709 12:22:35.468990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.470602 (+  1612us) service_pool.cc:224] Handling call
0709 12:22:35.508708 (+ 38106us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.478924 (+970216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.356075 (+877151us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b5400291154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:37.356367 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2216630528}
I20260709 12:22:37.477341  5914 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:37.477064  5887 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:37.477955  5914 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:37.477236 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.479135 27738 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:37.478928  5914 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.480841 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.425365 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:37.483386 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.476318 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.485664  5879 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:37.486692 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:37.488158  5788 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:37.488818  5788 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.489315  5788 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:37.495234  5516 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.483610 27735 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:37.498646  5516 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.502544  5788 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.506940  5879 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:37.542461 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:37.467310  5813 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:37.478636  5887 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:37.489167 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:37.545011 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 476892us
W20260709 12:22:37.527818 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.546451 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.548282 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.548838 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:37.507545  5884 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:37.566885 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.567713 27583 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:37.552175 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:37.555131 27732 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:37.569823 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:37.553064 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:37.571372 28329 raft_consensus.cc:2410] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:22:37.553879 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:37.572125 28335 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:37.547669 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:37.562222 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73174us
I20260709 12:22:37.576577 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.562739  5775 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.562860  5516 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:37.563393  5630 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.564672 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.565091 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.565763 27584 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: [32.20-34.21]
I20260709 12:22:37.580479 27585 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:37.574810 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:37.565598 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:37.581811 27580 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:37.539265 27481 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:37.583837 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.580535 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
W20260709 12:22:37.585289 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 140711us
I20260709 12:22:37.593320 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.600132 27729 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: [28.16-30.17]
W20260709 12:22:37.603266 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.600186 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:37.496904  5637 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.611007 27737 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
W20260709 12:22:37.611534 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.612692 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.614905 27731 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:37.616235 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.617594 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.599937 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:37.620122 27583 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:22:37.567191  5750 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:37.628489 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.628422 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:37.629724 27585 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:37.587631  5912 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:22:37.579407  5897 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:37.630785 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.631798 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:37.628010 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.633939 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:37.666277 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.676440 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:37.601703  5893 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:37.677439 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:37.602636  5958 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:37.586191  5879 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:37.544670  5915 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.066s
I20260709 12:22:37.688448 27581 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:22:37.626915  5921 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:22:37.702306 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 278458us
I20260709 12:22:37.626200  5516 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:37.696998 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 69098.5us
W20260709 12:22:37.712100 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.677150  5637 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:37.713619  5921 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.721771  5921 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:37.715461 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:37.685585  5912 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:37.595356  5775 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.618358  5923 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.712083  5967 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:37.715464  5958 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:37.602319  5630 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:37.723789 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 241368us
I20260709 12:22:37.724048 27588 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
W20260709 12:22:37.724257 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.750249  5985 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:37.750917  5630 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:37.769225  5923 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:37.770761  5979 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:22:37.771591 27736 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:22:37.729945  5912 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:22:37.740785 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.716794  5533 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:37.744441 27585 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:37.788535 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:37.751190  5788 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:37.723620  5887 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.710091  5726 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:37.693506  5750 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.761073  5775 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:37.764249  5986 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:22:37.785518 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.714241 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:37.708859 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
W20260709 12:22:37.832877 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 118770us
W20260709 12:22:37.797150 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 246578us
I20260709 12:22:37.731109  5965 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.739274  5897 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:37.826056 27481 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 27, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:37.847474 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:37.742274  5539 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.091s
I20260709 12:22:37.846110  5961 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.898736  5968 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.898610  5958 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.875151 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.899856  5689 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.819574  5877 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:37.831851  5803 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:37.821125  5835 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.083s
I20260709 12:22:37.849583  5897 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.788681  5915 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:37.909160 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 296506us
I20260709 12:22:37.915692  5894 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:37.943533 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68465.4us
I20260709 12:22:37.944813 27580 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:37.921433  5974 tablet_replica.cc:333] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:22:37.829602  5967 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.079s
I20260709 12:22:37.958946 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.875200  5965 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:37.977576 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 226401us
I20260709 12:22:37.976411 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.954706  5785 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:37.982638 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:37.907212  5988 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.066s
I20260709 12:22:37.951902  5968 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:38.010205  5981 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:22:37.957192  5897 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:38.014640 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:37.972599  5927 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:38.017405 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.020684 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:38.021281 27582 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:37.951508  5689 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.988474  5893 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:37.982995  5983 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:22:37.988325  5785 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.986567  5787 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:38.023360 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.031451 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.045992 27580 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:37.982668  5912 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:37.953428  5877 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.015182  5987 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:37.978430  5915 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:38.043349 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.022483  5630 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:37.919488  5961 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.016038  5990 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:38.040968 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.044895  5785 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:38.053895 27735 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:22:38.055536 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:38.071753 27738 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:38.074069 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.072583 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.048851  5986 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:38.052811 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4405) took 1015 ms. Trace:
W20260709 12:22:38.050825 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 203470us
I20260709 12:22:38.070334 27586 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:38.080897 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.078804 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:38.044731  5996 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:22:38.109462 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:38.048861  5787 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:38.172876 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63509.7us
W20260709 12:22:38.191633 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.078070 27657 rpcz_store.cc:276] 0709 12:22:37.037671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.037811 (+   140us) service_pool.cc:224] Handling call
0709 12:22:37.051634 (+ 13823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.051860 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:37.482786 (+430926us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.909781 (+426995us) log.cc:844] Serialized 14 byte log entry
0709 12:22:38.052485 (+142704us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:38.052535 (+    50us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:38.052586 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.052651 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.052761 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23997,"prepare.queue_time_us":255471,"prepare.run_cpu_time_us":7505,"prepare.run_wall_time_us":23489,"raft.queue_time_us":185102,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":26648,"spinlock_wait_cycles":16919936,"thread_start_us":949933,"threads_started":3,"wal-append.queue_time_us":479719}
I20260709 12:22:38.085668  6004 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:38.089257 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:38.095750  5689 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.085381  5750 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:38.080001 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.311038 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.311158 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:38.311836 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:38.065265  5997 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:22:38.086119 27582 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:38.100420  5968 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.072609  5989 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:38.101718  5921 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:38.102299 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 125958us
W20260709 12:22:38.103812  5977 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
W20260709 12:22:38.105763  5994 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:38.106943  5920 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.257855  5528 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:38.086122 27730 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:38.086758 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.307461  5835 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:38.059811  6009 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:38.258472 28330 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:22:38.261632 27731 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:38.283031 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 107866us
I20260709 12:22:38.305365 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:38.306586 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.078049  5775 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.086931 27734 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:38.314168 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:38.114593  5975 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.216s	user 0.000s	sys 0.004s
I20260709 12:22:38.107709  5961 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.326362 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.030962  5927 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.345032  5912 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.346129  5927 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.347471 27733 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:38.348047 27736 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:38.349442 27733 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:22:38.313235  6004 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.351601  5956 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:38.352005  5956 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:38.352802  6004 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.352845  5915 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.310511  5967 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:22:38.354797 27735 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:38.338569 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:38.356997 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:38.358332 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:38.358963 28330 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:38.360653 28333 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:38.328329 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.332152 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 245474us
I20260709 12:22:38.120596 27738 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:38.315516  5517 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.338215 28334 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
W20260709 12:22:38.324496  5993 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.295s	user 0.001s	sys 0.000s
I20260709 12:22:38.341176 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:38.333282  5997 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:38.374801 27628 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:22:38.378640 27628 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:22:38.343581 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.379698 27730 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:38.379923 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:38.366652 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.380930 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:38.381213 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:38.366812 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:38.342799  5985 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:38.325870  5989 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.340842  5835 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:38.383200 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.331017  5528 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:38.381886 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.384618 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:38.387530 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:38.386425 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.389968  6003 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:38.363994  5775 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.382792  5996 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:38.395068  6018 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.395637  6011 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:38.383304  5726 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:22:38.399564  5986 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:22:38.400018 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.401237  5990 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:38.374229  6014 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:38.382925  5884 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.802s	user 0.012s	sys 0.000s
I20260709 12:22:38.447700 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:38.403625  6010 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.060s	user 0.000s	sys 0.002s
I20260709 12:22:38.377301  5517 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.404297 27734 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:38.375141  5927 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:38.352866  5956 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.408169 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.410022 28330 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:38.410158  5988 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:22:38.410643 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.411175  5835 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:38.411518  5968 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:22:38.411940  5921 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.415937 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:38.410056  5897 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.374001  6007 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:38.418987 27736 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:22:38.414340 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.421366 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:38.424853 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.427701 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.431754  5630 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:38.432076 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:38.432335 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.433323 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:38.472949 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45362.3us
I20260709 12:22:38.434588 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:38.473297 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52056.1us
I20260709 12:22:38.434906 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.429306  5997 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 12:22:38.436328  5879 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:38.437089  6021 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:38.477401 27659 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 25: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:38.429323  6018 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.440553  5996 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:38.440661  5989 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.403079  5877 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.424016  5984 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:38.441586 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.442212 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.484257 27661 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:38.383837  5985 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.424199  6009 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:38.458654 27656 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:38.459189 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.486052 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.461831 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.429764  5923 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:38.489125 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:38.472987  5966 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.489758 27655 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:38.466609 27581 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:38.460968  5884 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:38.492439 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.467219  5915 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.016s
I20260709 12:22:38.469954 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:38.471294 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57131us
I20260709 12:22:38.471803 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:38.459762  6011 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:38.476285 27663 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:38.478168 28332 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:38.479696 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:38.423646  5893 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:38.509680 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.482990 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:38.417069 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.008s	sys 0.009s
W20260709 12:22:38.521392 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.008s	sys 0.009s
W20260709 12:22:38.521680 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 614966us
I20260709 12:22:38.482153  6003 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:22:38.494184  5996 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.477831  6001 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:38.497417 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.530403 27581 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
W20260709 12:22:38.498586  6017 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:22:38.490542  6009 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.490404  5985 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:38.502288 27656 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:22:38.502887 27658 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:38.511375 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:38.516438  5775 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.545308  5897 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.545588  5877 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:38.546010  5877 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:38.546401  5877 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:38.549538  6011 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.517019 27663 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:22:38.517903  5630 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.481626  6007 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.524355  5921 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:38.477643  6005 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:38.526335 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:38.572023 27655 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:22:38.526825 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43945.8us
I20260709 12:22:38.572769 27664 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:22:38.569684 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:38.528919 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:38.530613 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.467123  6014 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.531498 27657 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: [33.24-33.24]
I20260709 12:22:38.516904  5990 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:38.531454 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.532258 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.535985  5713 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:38.536924 27661 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:22:38.540433  6001 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.542236  5887 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.004s
I20260709 12:22:38.542855  6009 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:38.519935  5893 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:38.562678 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.554739  5921 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:38.662671 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.660998 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 129613us
W20260709 12:22:38.589169  6006 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.526s	user 0.001s	sys 0.000s
I20260709 12:22:38.591560  5983 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:22:38.592866  5980 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.622s	user 0.002s	sys 0.000s
I20260709 12:22:38.583005 27586 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:38.667467 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.640642 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.641065 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.668409 27656 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:38.579643 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.519778  5966 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.531363  5997 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.051s
W20260709 12:22:38.594651  5785 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.003s
I20260709 12:22:38.580013  5986 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.714335  6018 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.714063  5986 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:38.669622 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90033.8us
I20260709 12:22:38.642460  5897 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.701310 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.701858 27661 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:38.661689  5893 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:38.750085 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:38.750936 27584 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:38.582310  5990 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:38.578163  6007 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.595630  5713 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:38.705896 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.776791 27656 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 38: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:38.776805  5713 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.778856  5713 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:38.781271 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.580334  5985 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.707015 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.783066 27583 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
W20260709 12:22:38.787287 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4274) took 1411 ms (client timeout 1828 ms). Trace:
I20260709 12:22:38.708465  5517 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.710294  6019 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:38.682253  6011 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:22:38.682448  5593 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:38.712139  6001 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.802704  5593 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.642174  5927 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.490633  5528 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.718058 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:38.595090  5968 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.702548 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.742029 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.743033  5959 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:38.814538  5968 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.745414  6003 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.669468  5921 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:38.761329 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.813585 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71658.3us
I20260709 12:22:38.747997  6014 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:38.761612 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 240572us
I20260709 12:22:38.676288  5914 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:38.657351  6028 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:38.767278 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:38.668547  5983 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:38.730005  5894 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.753s	user 0.014s	sys 0.000s
I20260709 12:22:38.661497  5775 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.100s
W20260709 12:22:38.792414 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70853.9us
W20260709 12:22:38.787765 27737 rpcz_store.cc:269] 0709 12:22:37.352809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.391430 (+ 38621us) service_pool.cc:224] Handling call
0709 12:22:38.763592 (+1372162us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54001d47d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:38.763987 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343700736}
W20260709 12:22:38.808940 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.807896  5593 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:38.810840 27586 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:38.813566 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111087us
I20260709 12:22:38.844950 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:38.845464 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:38.809618  6029 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:22:38.785869  5986 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.812685  6019 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.819582 28335 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:22:38.760659 27481 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 28 to 31. New cstate: current_term: 31 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:38.831446 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.831662 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.859727 27655 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:38.832907 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.819151  5968 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.764441  5897 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:38.818507  5999 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.762s	user 0.000s	sys 0.004s
I20260709 12:22:38.841816 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.862881 27656 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 22.
I20260709 12:22:38.816915  5539 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:22:38.769013  5990 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:38.802505  5689 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:22:38.813115  6001 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:38.821408  5984 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:38.816742  5959 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.846773 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.826967  5528 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:38.847573 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.846179 27657 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:38.845655 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:38.848500 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:38.860656 27661 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 33 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:22:38.736135  5785 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.816478  5927 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.872160  5886 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:38.913462  5886 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:38.914240  5927 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.843487  5894 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:38.903543  5959 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:38.915994  5785 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:38.905493 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58006.2us
I20260709 12:22:38.915268  5927 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:38.920878 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:38.921248  5927 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:38.873199  6025 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.441s	user 0.000s	sys 0.001s
I20260709 12:22:38.879673  5983 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.924831 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.925803  5983 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:38.866735 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1681) took 1277 ms. Trace:
I20260709 12:22:38.878773  5539 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:38.928474  5539 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.924266  5927 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.825140  6003 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.932449  6009 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.933058  6003 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.852188  5637 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.866098  5630 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:38.868494 28329 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:22:38.933991  5637 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.934623 28329 raft_consensus.cc:606] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:38.937752  5637 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.880170  5893 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.942335  5893 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.881721 27663 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:22:38.919775  5689 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:38.944460 27585 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:22:38.946091 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:38.946699 27585 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:38.895293  5877 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:38.950098 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.887436  6001 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.951077  5927 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.951710  5927 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.953047  6001 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.954026  5927 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:38.906289 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60700us
I20260709 12:22:38.954555  5927 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.863651 27584 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:38.890080  6007 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.955060  5927 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.956235 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:38.957091  5713 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:38.892225  5985 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:38.958315  5985 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:38.888162 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.925028 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.958771  5927 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.959813 27661 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:38.926534 27732 rpcz_store.cc:276] 0709 12:22:37.588985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.613196 (+ 24211us) service_pool.cc:224] Handling call
0709 12:22:37.613487 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.875109 (+261622us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.875140 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.875182 (+    42us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:37.875227 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.866486 (+991259us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54002e81d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:38.866532 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.866669 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":600496384}
I20260709 12:22:38.844419  6028 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:38.868795  6043 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:22:38.901046  5897 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.962246  5897 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:38.869109 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4568) took 1276 ms. Trace:
W20260709 12:22:38.906555  6022 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.473s	user 0.000s	sys 0.008s
I20260709 12:22:38.959132 27588 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:38.911517 28334 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
W20260709 12:22:38.915505 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.909755  5803 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:38.963792 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.964176  5803 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:38.964653 27588 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 32 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:22:38.965220  5803 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.965834 27732 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:22:38.966986  6004 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.914352  5886 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.903882 27659 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:22:38.967484  6004 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:38.968597 27580 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:22:38.886190 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.902438  6019 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:38.961750  6028 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.939301 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:38.992964 27660 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 26
W20260709 12:22:38.970211 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.971302 27586 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:38.972645 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 141090us
I20260709 12:22:38.975466 28335 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:22:38.977130 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:38.994743 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:38.958837  5985 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:38.956235 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:38.982410 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:38.997829 27732 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:38.988880 27729 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:38.990597 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:38.980445 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:38.981678 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:38.962827 27587 rpcz_store.cc:276] 0709 12:22:37.592547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.642325 (+ 49778us) service_pool.cc:224] Handling call
0709 12:22:37.691852 (+ 49527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.744694 (+ 52842us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:37.956809 (+212115us) log.cc:844] Serialized 14 byte log entry
0709 12:22:38.075296 (+118487us) log.cc:844] Serialized 14 byte log entry
0709 12:22:38.080197 (+  4901us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:38.080250 (+    53us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:38.080300 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.436646 (+356346us) spinlock_profiling.cc:243] Waited 9.94 ms on lock 0x7b18004d0b00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:38.868848 (+432202us) spinlock_profiling.cc:243] Waited 52.6 ms on lock 0x7b54001ff9d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:38.868907 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.869051 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54895,"prepare.queue_time_us":311883,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":2114,"raft.queue_time_us":274598,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":153300,"spinlock_wait_cycles":180556672,"thread_start_us":307789,"threads_started":2,"wal-append.queue_time_us":99749}
I20260709 12:22:38.993746 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:38.994758 27659 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:38.939296 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.995232 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:39.001592 28332 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:39.002866 27737 raft_consensus.cc:2410] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 33 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:22:38.999938 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:39.001868 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:39.002321 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:39.000327 27661 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:39.011276 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:39.002499 27584 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:39.012609 27732 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:39.018680 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:39.019560 27737 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:39.014317 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:39.021303 27586 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:39.022342 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:38.908551  5787 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:38.993216  5788 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.869663  5968 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:38.934928  6009 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:39.027957 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:39.026289 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:38.962731  5897 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:38.997612  5967 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.038038 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:39.045799 27587 raft_consensus.cc:2410] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 31 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:22:39.003968  6019 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:38.999243  6004 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:39.071069  5967 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.025193  6003 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:39.075407  5959 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.075825  6003 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.078399  5959 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:39.078800  6003 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:39.052901 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:39.026844  5893 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:39.064636  5877 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:39.065526  5986 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.066399 27587 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:22:39.079258  5959 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:39.044589  5788 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.068889  5593 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:39.070669  5835 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.000s
I20260709 12:22:39.031409  5630 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:39.024502  5637 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:39.044188  5787 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.080905 27586 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:22:39.081588 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:38.997911  6043 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:22:39.113574 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:38.993063  5912 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.051s
I20260709 12:22:39.015300  5984 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.069s
I20260709 12:22:39.089396 27659 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
W20260709 12:22:39.090159 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79010.1us
I20260709 12:22:39.032187  5968 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:39.082816  6005 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:39.137043 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:39.085790  5785 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:39.030198  6030 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:22:39.043911  5813 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:22:39.080476  5932 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:22:39.086019  5914 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:22:39.092854  5967 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:39.098537  5637 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.151837 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:39.110976  5959 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:39.137118  5968 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:39.147432  6005 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.098801  5630 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:39.134974  5884 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:22:39.095990  5986 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.156692  5775 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:39.151021  6003 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:39.199482 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:39.222520 27731 raft_consensus.cc:606] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:39.154661 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 136679us
I20260709 12:22:39.159647  5637 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:39.104528  5835 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:39.220158 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.131038  5787 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.223692 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:39.150264  5893 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.050s
I20260709 12:22:39.170620  6014 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:39.194141  5630 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:39.218000  6005 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:39.155929  5990 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.290081 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:39.291009 27662 raft_consensus.cc:2376] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39. Re-sending same reply.
W20260709 12:22:39.275580 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51979.5us
I20260709 12:22:39.292817 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:39.293598 27662 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:39.295558 27656 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:22:39.297176 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:39.297525 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:39.298264 28335 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:39.283676 27585 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:22:39.280727  5984 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.098956  5985 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.226852  5884 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:39.306651 27732 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:39.307377 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:39.307971 27735 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:22:39.314558 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.315428 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.316319 27661 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:39.316426 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:39.315560 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:39.275314  6007 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:39.310112 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 158336us
I20260709 12:22:39.320588 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.320618 27585 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:39.276564  5887 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.324426 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:39.648437 27731 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:39.324478 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:39.308734 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 162542us
W20260709 12:22:39.327459  6020 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:22:39.351367  5593 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:39.334600  6046 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:22:39.352219  5884 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.291810  6053 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:39.292176  6043 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:39.329406  6007 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.420002  6039 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:39.423399 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:39.424392 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:39.425877 28336 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:39.431959  5892 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:39.679553 27659 raft_consensus.cc:1217] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Deduplicated request from leader. Original: 33.19->[34.20-34.20]   Dedup: 34.20->[]
I20260709 12:22:39.682067 27659 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:22:39.427125 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:39.436014 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.462296  5966 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:39.394968 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:39.686641 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:39.687212 28333 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:39.472932  6015 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.129s	user 0.003s	sys 0.002s
I20260709 12:22:39.476758 27662 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:39.479066  5835 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:39.478916  5775 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:22:39.479453  5995 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.423s	user 0.013s	sys 0.000s
I20260709 12:22:39.479856 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.717674 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.480188  5986 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:39.315533  5984 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.484359 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.672611  6053 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:39.487457 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.486550  6052 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.196s	user 0.007s	sys 0.004s
I20260709 12:22:39.583034 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:39.380365 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.687968  6050 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.279s	user 0.002s	sys 0.003s
I20260709 12:22:39.316634  5959 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.646240 27658 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:39.671578 27580 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:39.665768 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:39.680938  6043 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.681362  5956 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:22:39.684329 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.673806 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:39.689047 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 294177us
I20260709 12:22:39.784801 27585 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:39.785523 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:39.661693  5689 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.717s	user 0.004s	sys 0.029s
I20260709 12:22:39.717593 28332 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:39.479977  5788 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:39.673002  6010 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.170s	user 0.009s	sys 0.002s
I20260709 12:22:39.727828 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.804584 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.805073 27662 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:39.729784 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:39.806119 27659 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:39.805980 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:39.682917  5921 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.003s
I20260709 12:22:39.491901 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.745051 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:39.662568  5887 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.733229  6014 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:39.781772 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:39.783830 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44287.1us
I20260709 12:22:39.818665 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:39.676079  6039 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.796278 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.797206 28333 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:39.797109 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:39.675712  5990 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:39.741323  6028 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:39.807747 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.807214 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.807734 27656 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:39.805299 27663 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:39.848271 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:39.848924 27587 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:39.824422 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:39.805501  5689 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:39.683398  5968 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.861984 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:39.739897  6053 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:39.678174  6005 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.855358 27660 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:39.855217  6014 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:39.874838  6014 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.876489 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1641) took 1449 ms. Trace:
I20260709 12:22:39.876683  5887 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:39.877235  6056 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.284s	user 0.000s	sys 0.005s
I20260709 12:22:39.855381  6021 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:39.854317 28332 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:22:39.851559 27659 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:39.867739 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:39.867959  6012 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:39.868925  6008 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.812s	user 0.006s	sys 0.007s
W20260709 12:22:39.869846  6069 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:39.847554  6028 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:39.870767 27661 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:39.881002 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:39.863502  5987 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:39.913625 27587 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 35 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:22:39.824059  6019 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:22:39.876919 28334 rpcz_store.cc:276] 0709 12:22:38.427437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.427645 (+   208us) service_pool.cc:224] Handling call
0709 12:22:39.876293 (+1448648us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b5400962654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:39.876431 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1063172352}
W20260709 12:22:39.849793  6052 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:22:39.805094  5988 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.060s
W20260709 12:22:39.906836 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109821us
I20260709 12:22:39.920150 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:39.823982  5921 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:39.922734 28337 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35.
I20260709 12:22:39.921316 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:39.878722  5787 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:39.940366 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1836) took 1016 ms. Trace:
I20260709 12:22:39.941078 27657 rpcz_store.cc:276] 0709 12:22:38.924003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.925588 (+  1585us) service_pool.cc:224] Handling call
0709 12:22:38.925921 (+   333us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:38.926289 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.926319 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:38.926898 (+   579us) log.cc:844] Serialized 52 byte log entry
0709 12:22:39.164438 (+237540us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:39.164481 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.164557 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:39.940059 (+775502us) raft_consensus.cc:1685] finished
0709 12:22:39.940088 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.940271 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1670,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"thread_start_us":237160,"threads_started":1,"wal-append.queue_time_us":297709}
I20260709 12:22:39.942373 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:39.929781  6045 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.345s	user 0.000s	sys 0.005s
I20260709 12:22:39.946924 27730 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:22:39.949224 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:39.960302 27582 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:22:39.961199 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4563) took 1100 ms. Trace:
I20260709 12:22:39.961665 27582 rpcz_store.cc:276] 0709 12:22:38.860784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.959266 (+1098482us) service_pool.cc:224] Handling call
0709 12:22:39.959859 (+   593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.960876 (+  1017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.961128 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:39.938107  6021 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:39.963711 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4558) took 2364 ms. Trace:
I20260709 12:22:39.918953  6019 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.951339  5974 raft_consensus.cc:2243] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:39.930147  5775 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:39.971006 27589 rpcz_store.cc:276] 0709 12:22:37.589647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.589765 (+   118us) service_pool.cc:224] Handling call
0709 12:22:37.617111 (+ 27346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.677025 (+ 59914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.677059 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:37.697350 (+ 20291us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.904307 (+206957us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:38.037034 (+132727us) log.cc:844] Serialized 14 byte log entry
0709 12:22:38.418096 (+381062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.418194 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:39.478541 (+1060347us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:39.945929 (+467388us) raft_consensus.cc:1685] finished
0709 12:22:39.954285 (+  8356us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.954447 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15257,"prepare.queue_time_us":143597,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":8741,"raft.queue_time_us":102525,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":27787,"spinlock_wait_cycles":43155712,"thread_start_us":672544,"threads_started":3,"wal-append.queue_time_us":341177}
W20260709 12:22:39.968163  5630 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.647s	user 0.002s	sys 0.011s
I20260709 12:22:39.979760  6028 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:39.969971  5637 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.704s	user 0.008s	sys 0.004s
I20260709 12:22:39.954553  5990 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.019177  5984 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.014268  5775 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:40.030941  6049 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.091s	user 0.002s	sys 0.001s
I20260709 12:22:40.031405  6028 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:40.031880  5775 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:40.037643 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4551) took 1658 ms. Trace:
I20260709 12:22:40.038024 27583 rpcz_store.cc:276] 0709 12:22:38.379043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.787549 (+408506us) service_pool.cc:224] Handling call
0709 12:22:38.808520 (+ 20971us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:38.810510 (+  1990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.810562 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:38.824480 (+ 13918us) log.cc:844] Serialized 52 byte log entry
0709 12:22:39.102080 (+277600us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:39.146738 (+ 44658us) log.cc:844] Serialized 14 byte log entry
0709 12:22:39.147005 (+   267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.147108 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.037379 (+890271us) raft_consensus.cc:1685] finished
0709 12:22:40.037400 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.037566 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4275,"prepare.queue_time_us":33904,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":4045,"raft.queue_time_us":2264,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":18306,"thread_start_us":240994,"threads_started":1,"wal-append.queue_time_us":351820}
I20260709 12:22:40.022306  5968 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:39.980127  6019 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:40.049826 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1536) took 1392 ms. Trace:
I20260709 12:22:40.025878 27659 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:40.050299 28330 rpcz_store.cc:276] 0709 12:22:38.657474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.657665 (+   191us) service_pool.cc:224] Handling call
0709 12:22:38.657985 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.658333 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.658361 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.658860 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:22:38.918094 (+259234us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b1800413500. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:39.094174 (+176080us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:39.094244 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.094334 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.037817 (+943483us) raft_consensus.cc:1685] finished
0709 12:22:40.037836 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.049719 (+ 11883us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":45607,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":27183,"spinlock_wait_cycles":10382208,"thread_start_us":175855,"threads_started":1,"wal-append.queue_time_us":1181298}
I20260709 12:22:40.052270  6007 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.055452  5990 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:39.923570  6012 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.041087 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4578) took 1464 ms. Trace:
I20260709 12:22:40.057901 27584 rpcz_store.cc:276] 0709 12:22:38.577133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.048276 (+471143us) service_pool.cc:224] Handling call
0709 12:22:39.048559 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.063564 (+ 15005us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:39.063965 (+   401us) log.cc:844] Serialized 14 byte log entry
0709 12:22:39.284635 (+220670us) log.cc:844] Serialized 14 byte log entry
0709 12:22:39.284827 (+   192us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.289325 (+  4498us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.289476 (+   151us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:39.289522 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.289610 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.040873 (+751263us) raft_consensus.cc:1685] finished
0709 12:22:40.040895 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.041030 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4428,"prepare.queue_time_us":141050,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":17169,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":27927,"thread_start_us":215099,"threads_started":2,"wal-append.queue_time_us":264786}
I20260709 12:22:40.060324  5921 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.060570  6043 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:40.061466 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.07375e+06us
I20260709 12:22:40.038565  5630 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.055312  5637 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:40.019137  6065 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:22:40.130566 27587 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 24. Current term is 25. Ops: []
W20260709 12:22:40.064888 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 246306us
I20260709 12:22:40.164746 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.165689 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1875) took 1284 ms. Trace:
I20260709 12:22:40.166128 27587 rpcz_store.cc:276] 0709 12:22:38.881118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.962774 (+1081656us) service_pool.cc:224] Handling call
0709 12:22:40.127351 (+164577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.165397 (+ 38046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.165616 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.166729 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.170992 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.094475  5886 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:40.155846  6018 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.148727  6084 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.174767  6040 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.369s	user 0.004s	sys 0.000s
I20260709 12:22:40.169910  5984 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:40.097180  5897 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.172597  6060 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:40.154057  6021 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:40.156042  5892 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:40.187896  6047 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.896s	user 0.006s	sys 0.003s
I20260709 12:22:40.188145  5892 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.189189 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1643) took 1464 ms. Trace:
I20260709 12:22:40.189788 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4550) took 1881 ms. Trace:
I20260709 12:22:40.176241  5988 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:40.181759  5803 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:40.145699  6039 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.183130  5990 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.151966  5991 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.195777  5991 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.196154  5637 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.184787  6018 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.190642  6084 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.194271  5803 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:40.189514 28328 rpcz_store.cc:276] 0709 12:22:38.724899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.725040 (+   141us) service_pool.cc:224] Handling call
0709 12:22:38.766756 (+ 41716us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.768754 (+  1998us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:38.768803 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.789697 (+ 20894us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.084727 (+295030us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:39.084787 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.084887 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.187075 (+1102188us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:40.188906 (+  1831us) raft_consensus.cc:1685] finished
0709 12:22:40.188928 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.189130 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1505,"raft.queue_time_us":39913,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":34168704,"thread_start_us":263672,"threads_started":1,"wal-append.queue_time_us":359835}
I20260709 12:22:40.190197 27581 rpcz_store.cc:276] 0709 12:22:38.308102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.543358 (+235256us) service_pool.cc:224] Handling call
0709 12:22:38.543735 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.710948 (+167213us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:38.711304 (+   356us) log.cc:844] Serialized 14 byte log entry
0709 12:22:38.788075 (+ 76771us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.799418 (+ 11343us) log.cc:844] Serialized 26 byte log entry
0709 12:22:38.799590 (+   172us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:38.799635 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.799717 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.173997 (+1374280us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:40.189509 (+ 15512us) raft_consensus.cc:1685] finished
0709 12:22:40.189598 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.189732 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53328,"prepare.queue_time_us":1718,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":151873,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":44843,"spinlock_wait_cycles":9235712,"thread_start_us":189361,"threads_started":2,"wal-append.queue_time_us":93585}
I20260709 12:22:40.201869 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4554) took 1788 ms. Trace:
W20260709 12:22:40.203414 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.202251 27586 rpcz_store.cc:276] 0709 12:22:38.413178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.041463 (+628285us) service_pool.cc:224] Handling call
0709 12:22:39.041970 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.105016 (+ 63046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.201619 (+1096603us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b54000b1154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:40.201815 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88765696}
I20260709 12:22:40.116662  5775 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.190341  5892 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.212980  6084 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:40.213258  6042 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.408s	user 0.000s	sys 0.004s
I20260709 12:22:40.213541  6012 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:40.215164  6079 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.156s	user 0.000s	sys 0.004s
I20260709 12:22:40.206009  5990 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.206065  6021 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.220429 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.225975 27583 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:40.227933 27584 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:22:40.208422  5803 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.229730  5803 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:40.230284 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.223384  6021 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.231863  6060 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.232406  6060 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.233194  5813 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.233378  6021 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.233609  5965 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.233764  5813 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.234433  5894 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:40.234634  6060 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.235982  5813 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.236886  5813 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:40.222672  5990 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.238057  5997 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:40.238715 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.240892 27581 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:40.242339 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.249156  5689 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.249423  5787 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.249773  5689 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.250097  5787 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.252302  5877 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.252554  5787 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:40.252816  5877 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.221562  6084 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:40.254793 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:40.253366  6060 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.255921  5637 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:22:40.255240 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 193008us
I20260709 12:22:40.257382  5630 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.239238  5990 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.262930 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.259671  5984 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.264922 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:40.246850  5965 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:40.266993 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.265110  6084 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.274557 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.276013 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.277344  6084 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.268112 27582 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:22:40.269162 28328 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:22:40.280016 28328 raft_consensus.cc:606] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:40.269652  6004 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.002s
I20260709 12:22:40.282110  6019 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.272351  6043 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:40.283703  6043 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:40.262553  5991 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.284749  6019 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.280385 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.285764  5689 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.285813 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:40.286594  5689 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:40.281690 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:40.286980  5689 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:40.288679  5894 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:40.288239  5689 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.289458  5894 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:40.291733  5637 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:40.286991 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:40.292299  5637 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:40.293159  5637 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.296345 27586 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:22:40.296402 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:40.297230 27582 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:40.298102  6007 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:40.298646  6007 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:40.298888 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1570) took 1001 ms. Trace:
I20260709 12:22:40.300014  5787 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:22:40.300724  5787 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.300922  5968 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:40.299384 28329 rpcz_store.cc:276] 0709 12:22:39.297003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.308712 (+ 11709us) service_pool.cc:224] Handling call
0709 12:22:39.378005 (+ 69293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.397487 (+ 19482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.397524 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.398202 (+   678us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.823538 (+425336us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:22:39.823602 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.830216 (+  6614us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.298641 (+468425us) raft_consensus.cc:1685] finished
0709 12:22:40.298680 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.298825 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18990,"raft.queue_time_us":108863,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":77,"thread_start_us":424805,"threads_started":1,"wal-append.queue_time_us":820970}
I20260709 12:22:40.301295  5787 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:22:40.303016  6066 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.243s	user 0.001s	sys 0.005s
I20260709 12:22:40.303089  5775 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.265990  6060 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.305724  6060 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.297176  5984 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.309700 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:40.368026 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58409.4us
W20260709 12:22:40.300597  5956 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.000s
I20260709 12:22:40.260191  6089 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:22:40.301328  5968 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.299902  5877 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.396286 28331 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:40.407187  5991 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.404695  6053 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.413264 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:40.390508  6060 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:40.407847  5991 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.428821  5991 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.400108  5968 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:40.307993  5803 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.094s
I20260709 12:22:40.450861 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.451601 27582 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:40.452780 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.455650 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.457114 27582 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:40.461649 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:40.450040  6060 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.462461 28328 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:40.462730  5984 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.461616 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.464370 27729 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:40.462837  6060 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:40.468053 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:40.470433 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.299594  6007 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.472795 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:40.474926 28328 raft_consensus.cc:2410] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 35 because replica has last-logged OpId of term: 33 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 23.
W20260709 12:22:40.474858 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 193317us
I20260709 12:22:40.473368 28329 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:40.477757 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.478497 27582 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:40.479911 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.480985 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.482894  6083 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:40.483407  5984 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:40.483510  6019 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:40.483982  5984 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:40.484411  6075 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.505s	user 0.004s	sys 0.002s
I20260709 12:22:40.484625  5984 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:40.484735 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:40.483822 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:40.486750  6083 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.487389  6083 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:40.486857 27582 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:40.491834 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.492528 27659 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:40.493076 27628 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:22:40.493523 27628 leader_election.cc:403] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:40.494274 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:40.496081 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.497795 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.502200 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.484145  6019 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.503687 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.505272 27661 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:40.504922 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.506057 27664 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:40.511857  5987 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:40.504364  5986 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:40.513129  6019 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.513437  5986 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.514040  5986 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:40.516423  6060 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.493629  5894 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:40.527084 27582 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:40.525107 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:40.533214  6095 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:40.533747  6095 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.534152  6095 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:40.504927  5637 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:40.535851  5637 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:40.536266  5637 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:40.505010  5988 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26.
I20260709 12:22:40.507762  5956 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:40.526350  6060 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:40.557132 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59509.2us
I20260709 12:22:40.568397 27582 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:22:40.570375 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:40.573280 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.578297 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:40.531914  5894 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.586351 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:40.586544  5956 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:40.561702  6060 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:40.563225  5988 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.587064 27582 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:40.588649  6060 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:40.622473  6095 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.624698  5894 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.611578  6053 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.630313  5787 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.632128  5984 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.633126  6019 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:40.635812  6060 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.637341  6005 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.638975  6018 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.640421  6083 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.599848  5988 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.642714  6102 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:22:40.643465  6097 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:40.644130 27481 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 26, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 26 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:40.650329  5637 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.656996  5966 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.688196 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 117887us
I20260709 12:22:40.704517 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:40.689208 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.706043 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.708799 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.710475 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.711987 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.713460 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:40.702497 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0321305370002847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260709 12:22:40.720029 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0096305370002483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
W20260709 12:22:40.720574 27701 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.810s (SENT)
I20260709 12:22:40.692318 27582 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:22:40.695721 27658 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31.
I20260709 12:22:40.723260 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.741930  5886 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.744071 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.746484 27661 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:40.746577  5886 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.747478 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.747678 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.748764 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.748767 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.749571 27661 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:40.749974 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.739564 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.750168  5886 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.750625 27659 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:40.745143 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:40.751605 27582 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:22:40.724488  5966 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.753608  5966 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.687747  6060 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:40.612759  5956 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.683722  6019 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.763479 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.762368 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.766583 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.767694 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.776407 27661 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:22:40.777273 27657 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:40.778533 27661 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:40.779127 27661 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:40.779915 27657 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:40.777995 27659 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:40.788935 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:40.789124 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.793744 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.798753 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:40.794294 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:40.800709 27657 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:40.785720 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.801476 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:40.780045 27661 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.802358 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:40.772337  5974 raft_consensus.cc:2272] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:40.769297  6003 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.795897 27704 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
I20260709 12:22:40.803131 27661 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:40.804180 27664 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 23. Ops: []
W20260709 12:22:40.804798 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.780087  5921 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.799780 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:40.765376  6089 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:40.807456 27704 leader_election.cc:403] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:40.807204 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:40.778952  5966 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.794615  5897 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.793725  5923 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.805365 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84032.1us
I20260709 12:22:40.812395 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:40.782176  5986 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.774494  5968 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.808043  5991 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:40.808193  6005 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:40.804219  5959 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.811769  5877 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:40.806393  5835 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.877763 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.006191 28330 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:22:40.877785 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:40.925053  5988 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:40.869683  5966 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:40.928211  6108 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:22:40.875449  6043 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:40.875741  5897 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.925750  5986 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.935194 27656 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:40.926935 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:40.865810  6103 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.102s
I20260709 12:22:40.866415  5894 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.926237  6095 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.829414  6105 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:22:40.866138  6102 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.927686  5787 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.893474  5984 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:41.050035  5593 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.054471  6043 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:41.096091  6043 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.021981  5835 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.101259  5835 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:41.102370  6019 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.103873  6019 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:41.104281  6019 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.104632  6019 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:41.107475  6110 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:41.108114  6110 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.110420  6110 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:41.111790 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4553) took 2711 ms. Trace:
I20260709 12:22:41.112164 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.112952 28333 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:41.112187 27580 rpcz_store.cc:276] 0709 12:22:38.400454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.969318 (+568864us) service_pool.cc:224] Handling call
0709 12:22:38.969607 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.743350 (+773743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.111502 (+1368152us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:41.111707 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1613747968}
I20260709 12:22:41.115175 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.115751 27580 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:41.071177  5894 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.084602  5593 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.073945  6102 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.077059  6105 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:40.994201 27661 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 19. (term mismatch)
I20260709 12:22:40.994603 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:41.139959 28328 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:40.998459  6050 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:22:41.000298 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4306) took 1258 ms. Trace:
I20260709 12:22:41.000859  5968 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.821009  5921 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.007429 27736 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:22:40.812657  6003 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:40.888942  5991 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.882853  6005 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.877282  5923 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:41.024608  6080 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.884s	user 0.011s	sys 0.000s
W20260709 12:22:41.027455  6070 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:22:41.032049  6083 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:40.938993  5959 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:40.861254  6089 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:40.996146  6111 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:22:41.043313 27657 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:41.155242  5991 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:41.048681 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3780) took 2056 ms. Trace:
I20260709 12:22:41.158120  6005 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:41.050310 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 250588us
W20260709 12:22:41.053592 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.051621 28334 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:22:41.142210 27661 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 18
I20260709 12:22:41.221855 27661 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 35.19
I20260709 12:22:41.142571 27580 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:41.141130 27729 rpcz_store.cc:276] 0709 12:22:39.741427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.839978 (+ 98551us) service_pool.cc:224] Handling call
0709 12:22:39.840419 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.464949 (+624530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.000015 (+535066us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:41.000241 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434297728}
I20260709 12:22:41.222183 27661 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.223143 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.171435 27737 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:41.045219 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 249458us
I20260709 12:22:41.224009 28329 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:41.213670 27734 rpcz_store.cc:276] 0709 12:22:38.994526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.994835 (+   309us) service_pool.cc:224] Handling call
0709 12:22:39.023905 (+ 29070us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:39.193214 (+169309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.193263 (+    49us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:39.224808 (+ 31545us) log.cc:844] Serialized 78 byte log entry
0709 12:22:39.395789 (+170981us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:39.849339 (+453550us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.775373 (+926034us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.806385 (+ 31012us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.038586 (+232201us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b18003e0e60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:41.038804 (+   218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.048329 (+  9525us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.048410 (+    81us) raft_consensus.cc:1685] finished
0709 12:22:41.048429 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.048612 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1204637,"prepare.queue_time_us":754632,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":120,"raft.queue_time_us":1164230,"raft.run_cpu_time_us":632,"raft.run_wall_time_us":15890,"spinlock_wait_cycles":14404864,"thread_start_us":599376,"threads_started":4,"wal-append.queue_time_us":492975}
I20260709 12:22:41.229132 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.187707  5897 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:41.188555  6089 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.192102  5966 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:41.200170 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1571) took 1875 ms. Trace:
I20260709 12:22:41.201093  6030 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:22:41.202380 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.194502  6019 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.212587  5959 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:40.927105  5637 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:41.216758 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.229210 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.225274 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:41.213937  6101 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:41.154914  6110 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
I20260709 12:22:41.250101  6110 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:22:41.250592  6110 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:41.250727 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:41.251394  5689 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:41.252117 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:41.251674  6110 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.253005  6110 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:22:41.256786  6054 raft_consensus.cc:670] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20260709 12:22:41.233086 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:41.186347 28328 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:41.224713  6065 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:22:41.224884  6113 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:41.234572 28335 rpcz_store.cc:276] 0709 12:22:39.325186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.325390 (+   204us) service_pool.cc:224] Handling call
0709 12:22:39.325719 (+   329us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:39.326145 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.326184 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:39.391229 (+ 65045us) log.cc:844] Serialized 52 byte log entry
0709 12:22:39.944256 (+553027us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:39.944314 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.944411 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.232852 (+288441us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b18003944e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9e21c58f 00007f0e9725b571 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd
0709 12:22:41.023977 (+791125us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:41.199866 (+175889us) raft_consensus.cc:1685] finished
0709 12:22:41.199898 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.200093 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54675,"raft.queue_time_us":8873,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":14694,"spinlock_wait_cycles":48220800,"thread_start_us":268914,"threads_started":1,"wal-append.queue_time_us":575430}
I20260709 12:22:41.263497 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.226272 27736 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:41.238086 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.236166 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:41.242129 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:41.248126 28336 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:41.264901 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:41.264932 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.265760 28328 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:41.267408 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:41.237353  6030 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:41.269675 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:41.279290 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.280090 27580 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:41.274514  6060 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:41.301442 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.274164  6053 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31.
W20260709 12:22:41.322184 27732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.011s	sys 0.020s
I20260709 12:22:41.304165 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:41.279786  6083 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:41.263399  5894 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:41.279521  6089 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:41.322561 27732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.011s	sys 0.020s
I20260709 12:22:41.338318 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:41.322988 27659 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:41.280452  5877 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:41.336252 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.341842 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:41.312150  6060 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.348479 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:41.311400  6121 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:41.259737  6105 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.356753 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:41.357908 27734 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:41.357993 27656 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:41.354885 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:41.352459 28331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1540) took 1930 ms (client timeout 2171 ms). Trace:
I20260709 12:22:41.359695 27661 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:41.361546 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.362381 27736 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:41.364173 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.364989 27734 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:22:41.359647 28331 rpcz_store.cc:269] 0709 12:22:39.422354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.426432 (+  4078us) service_pool.cc:224] Handling call
0709 12:22:41.352204 (+1925772us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b5400153454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:41.352377 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1639289600}
I20260709 12:22:41.344028  6083 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:41.371647 27659 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:41.373258 27657 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:22:41.380718 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.388484 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.389362 27734 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:41.351102  6060 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:41.352035  5921 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:41.338974  6089 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.398716 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:41.405252 27736 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:41.400698 27663 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:41.388573  5593 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:41.396441 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.349988  6126 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:22:41.393982  6102 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.411962 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.364909  6121 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.409438 27580 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:41.418212  6081 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.366s	user 0.003s	sys 0.008s
I20260709 12:22:41.419956 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1874) took 2578 ms. Trace:
I20260709 12:22:41.420397 27588 rpcz_store.cc:276] 0709 12:22:38.841274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.394039 (+552765us) service_pool.cc:224] Handling call
0709 12:22:39.394451 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.496332 (+101881us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.496382 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.537823 (+ 41441us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.908104 (+370281us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:39.908154 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.908240 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.366867 (+1458627us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:41.419596 (+ 52729us) raft_consensus.cc:1685] finished
0709 12:22:41.419622 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.419889 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101254,"raft.queue_time_us":101403,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":8127,"spinlock_wait_cycles":26901248,"thread_start_us":225056,"threads_started":1,"wal-append.queue_time_us":310472}
I20260709 12:22:41.422734 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:41.424000  6132 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:22:41.426748  6083 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.351864  6030 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.431193 27661 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:22:41.415445 27734 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 38 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
W20260709 12:22:41.407985 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 181839us
I20260709 12:22:41.434850 27588 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:41.508538  6123 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.509198  6123 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.497431  5689 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:41.522569  6121 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.522740  5689 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.523609  6137 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:41.524201  6137 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.525270  5689 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.526335  6141 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.526929  6141 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.528467  6089 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.529316  6141 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.531461 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4571) took 1302 ms. Trace:
I20260709 12:22:41.531965 27583 rpcz_store.cc:276] 0709 12:22:40.228618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.239960 (+ 11342us) service_pool.cc:224] Handling call
0709 12:22:40.241780 (+  1820us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.278753 (+ 36973us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:40.278787 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.278833 (+    46us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:40.278872 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.531166 (+1252294us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:41.531213 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.531387 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84457728}
I20260709 12:22:41.533718 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.534622 27583 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:41.529945  6141 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:41.526953 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.540128 27734 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 23: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:41.541147 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.542147 27583 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31.
I20260709 12:22:41.589422 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:41.410403  5877 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.694794 28295 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31.
I20260709 12:22:41.695277 28295 leader_election.cc:403] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:22:41.656723 28337 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
W20260709 12:22:41.615509 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.653173 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:41.441231  6019 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:41.722761 27734 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
W20260709 12:22:41.724443 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4416) took 2417 ms (client timeout 1980 ms). Trace:
W20260709 12:22:41.728165 27660 rpcz_store.cc:269] 0709 12:22:39.283050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.283228 (+   178us) service_pool.cc:224] Handling call
0709 12:22:41.691469 (+2408241us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:41.699958 (+  8489us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241216896}
I20260709 12:22:41.729826 27588 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:41.730670 28332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1568) took 2412 ms (client timeout 2859 ms). Trace:
I20260709 12:22:41.732627 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.733425 27580 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:41.731207 28332 rpcz_store.cc:269] 0709 12:22:39.296318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.297249 (+   931us) service_pool.cc:224] Handling call
0709 12:22:41.708941 (+2411692us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54003ed0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:41.709096 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":964269824}
W20260709 12:22:41.632757  6131 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.217s	user 0.000s	sys 0.000s
I20260709 12:22:41.523262  6121 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.737525 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4583) took 1206 ms. Trace:
I20260709 12:22:41.596987  6122 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.735409  6095 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31.
I20260709 12:22:41.700099 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:41.702134 27736 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:41.738529  6019 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:41.739738  6064 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 36 current term: 36 is higher.
I20260709 12:22:41.752735 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1846) took 1261 ms. Trace:
I20260709 12:22:41.739643 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:41.753206 27658 rpcz_store.cc:276] 0709 12:22:40.491545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.491655 (+   110us) service_pool.cc:224] Handling call
0709 12:22:41.752548 (+1260893us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:41.752678 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438840704}
W20260709 12:22:41.755398 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55386.4us
I20260709 12:22:41.742082 27585 rpcz_store.cc:276] 0709 12:22:40.531222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.698692 (+1167470us) service_pool.cc:224] Handling call
0709 12:22:41.736885 (+ 38193us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.737101 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.737138 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.737195 (+    57us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:41.737263 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.737333 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.737457 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:41.704668 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4579) took 1396 ms. Trace:
I20260709 12:22:41.642630  6005 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.757776 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4584) took 1225 ms. Trace:
I20260709 12:22:41.762027 27585 rpcz_store.cc:276] 0709 12:22:40.531930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.756545 (+1224615us) service_pool.cc:224] Handling call
0709 12:22:41.756827 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.756992 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.757018 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.757060 (+    42us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:41.757097 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.757153 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.757718 (+   565us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:41.745924 27734 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:22:41.757624 27581 rpcz_store.cc:276] 0709 12:22:40.307999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.929882 (+621883us) service_pool.cc:224] Handling call
0709 12:22:40.930192 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.930402 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:22:41.222811 (+292409us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.704213 (+481402us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.704284 (+    71us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:41.704343 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.704432 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.704599 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":407366,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":705140,"threads_started":2,"wal-append.queue_time_us":524344}
I20260709 12:22:41.764642 27734 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:41.765738 27585 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:41.750684  6137 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.755054 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4566) took 1543 ms. Trace:
I20260709 12:22:41.771360 27585 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:22:41.754446  5637 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:41.772505 27585 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:22:41.772944 27585 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:22:41.773377 27585 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.755620  6123 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:41.773216 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 337978us
I20260709 12:22:41.744330  6121 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.776376 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:41.774765 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.745688  6122 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.775874 27587 rpcz_store.cc:276] 0709 12:22:40.211161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.211276 (+   115us) service_pool.cc:224] Handling call
0709 12:22:40.217511 (+  6235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.217706 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:22:40.217997 (+   291us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.937705 (+719708us) spinlock_profiling.cc:243] Waited 950 us on lock 0x7b18003867a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:41.402453 (+464748us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.754683 (+352230us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.754737 (+    54us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:41.754806 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.754886 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.754998 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":204623,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":83,"raft.queue_time_us":223047,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":3556,"spinlock_wait_cycles":2183936,"thread_start_us":809702,"threads_started":3,"wal-append.queue_time_us":1100131}
I20260709 12:22:41.783255 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:41.770660  6095 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 12:22:41.756037  6019 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.788269 27588 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:22:41.794180 28292 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31.
I20260709 12:22:41.788728 27587 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:41.802675 27585 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:41.780305  6145 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:41.779971  5637 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:41.805744  6135 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:22:41.805266 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:41.817790 27658 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 18. (term mismatch)
I20260709 12:22:41.818869 27662 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:41.819057 27658 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 17
I20260709 12:22:41.819681 27658 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:22:41.807005 27580 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:41.819959 27658 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.799688  6123 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:22:41.817373  6143 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:41.801783  6146 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.788106  5689 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.042s
I20260709 12:22:41.801869  6137 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:41.802389  6126 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.838980 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1881) took 1628 ms. Trace:
I20260709 12:22:41.855971  6146 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.821286  6135 consensus_queue.cc:1243] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 23.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:22:41.860846  6154 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:41.864748 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:41.857926 27586 rpcz_store.cc:276] 0709 12:22:40.210537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.210665 (+   128us) service_pool.cc:224] Handling call
0709 12:22:40.210954 (+   289us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.296862 (+ 85908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.838678 (+1541816us) spinlock_profiling.cc:243] Waited 85.1 ms on lock 0x7b54002b98d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:41.838917 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195839360}
I20260709 12:22:41.848587 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:41.869371 27481 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 15 to 18, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 18 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:41.816469  6145 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:41.869093  6071 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.828s	user 0.011s	sys 0.011s
I20260709 12:22:41.863977  6154 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:41.817059  5637 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.887008  5775 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:41.887899  6146 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.868687  6148 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:41.890362  6142 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:22:41.890149  6148 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.891474  6145 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:41.885942  5991 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.659s	user 0.011s	sys 0.000s
I20260709 12:22:41.863289  6126 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.861491  6137 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:41.872339  6123 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:41.874187  5689 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.897404  6003 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:41.897524  6162 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.006s
I20260709 12:22:41.898628 27553 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35.
I20260709 12:22:41.911638 27553 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:41.889768  6146 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:41.892130  6154 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:41.887041  6122 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.918613 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:41.876816  6156 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.038s
I20260709 12:22:41.925516 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:41.929836 27553 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:41.930794 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:41.930176  6159 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:41.930400  6150 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:22:41.945447 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:41.901352  5991 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:41.947875 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99430.2us
I20260709 12:22:41.949476 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:41.951561 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.952148 27580 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:41.922062  6145 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35.
I20260709 12:22:41.952986 27585 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:22:41.951613 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:41.956471 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.957209 27587 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:41.929684  6146 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:41.965570 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.610s (SENT) [suppressed 13 similar messages]
I20260709 12:22:41.915292  5689 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:41.981899  5689 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.959506 28332 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:41.983188 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:41.975982 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:41.946034  6121 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.947829  6153 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:41.984205 27589 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:41.944106  6147 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:41.955976  6149 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:41.986464 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4299) took 2661 ms. Trace:
I20260709 12:22:41.989631 27580 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:41.933900  6155 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.060s
I20260709 12:22:41.947606  6105 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:41.963542  6156 consensus_queue.cc:1243] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 34.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:22:41.948465  6157 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:22:41.959620  6060 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.559s	user 0.007s	sys 0.005s
I20260709 12:22:41.989169  6019 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.930743  6005 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.995159 27735 rpcz_store.cc:276] 0709 12:22:39.318283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.318398 (+   115us) service_pool.cc:224] Handling call
0709 12:22:39.318682 (+   284us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:39.319231 (+   549us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:39.319258 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:39.351828 (+ 32570us) log.cc:844] Serialized 52 byte log entry
0709 12:22:39.736150 (+384322us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:39.745559 (+  9409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.745679 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.301841 (+556162us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:41.775194 (+1473353us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:41.979510 (+204316us) raft_consensus.cc:1685] finished
0709 12:22:41.979531 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.979824 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":395004,"raft.queue_time_us":97,"raft.run_cpu_time_us":9811,"raft.run_wall_time_us":978064,"spinlock_wait_cycles":151654400,"thread_start_us":383249,"threads_started":1,"wal-append.queue_time_us":601844}
W20260709 12:22:42.004400  6076 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.945s	user 0.004s	sys 0.011s
I20260709 12:22:41.994935  6149 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:42.020792  6152 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
W20260709 12:22:42.021207 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.026666 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4580) took 1717 ms. Trace:
I20260709 12:22:42.027143 27583 rpcz_store.cc:276] 0709 12:22:40.308842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.595295 (+1286453us) service_pool.cc:224] Handling call
0709 12:22:41.732319 (+137024us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.803095 (+ 70776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.803126 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.831546 (+ 28420us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.950442 (+118896us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:41.950490 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.950560 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.026428 (+ 75868us) raft_consensus.cc:1685] finished
0709 12:22:42.026450 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.026596 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17418,"raft.queue_time_us":89568,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":8172,"thread_start_us":155870,"threads_started":2,"wal-append.queue_time_us":152403}
I20260709 12:22:42.008877  6157 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.034634  6137 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.016716  6149 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:41.989800  6123 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.049158  5991 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:41.994935  6121 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:22:42.051980 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:42.053745 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:42.009117  6105 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.011659  6060 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.029804  6148 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:42.079423 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 113890us
I20260709 12:22:42.035120  6157 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:42.081123 28295 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:42.078470  6121 consensus_queue.cc:1243] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 35.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:22:42.102240 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:42.104374 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:42.106549 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.107065 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:42.107425 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.105062 28333 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:42.108455 27581 pending_rounds.cc:85] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 15
I20260709 12:22:42.108785 27581 pending_rounds.cc:107] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:22:42.109002 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.109046 27581 raft_consensus.cc:2889] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:42.110210 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.110507 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.111081 27583 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 31 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:22:42.037194  6146 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.095077  6148 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.111079 27587 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 23.
I20260709 12:22:42.112079 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:42.122529 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:42.125243 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:42.125741 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.126636 27584 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:22:42.099745  6005 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:42.125195 27589 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:42.092190  6134 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:22:42.138724 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.116882  5803 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:42.141129  5803 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.144032  5803 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.109750  6150 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:42.131668  6110 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.151445  6005 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:42.152339  5884 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:42.154225  6060 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.097450  6095 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:22:42.128216  6099 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.156576  6105 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.135784  6104 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:22:42.165884  6144 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.394s	user 0.001s	sys 0.000s
I20260709 12:22:42.165429  6173 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:42.164415 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:42.170316 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.146265 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:42.132630  5991 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:42.132452  6148 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.097947  6154 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.136631  6137 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:42.183800  5884 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:42.190277 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44105.9us
I20260709 12:22:42.191582 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:42.200961  6007 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:22:42.205152 27481 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:42.218456  6092 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:42.218976  6092 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:42.187459  6011 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:42.239586  6011 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:42.240271  6089 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:42.240710  6089 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:42.240463  6011 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.241629  6011 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:42.242183 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3782) took 3245 ms. Trace:
I20260709 12:22:42.221441 27580 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:22:42.215000  6007 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:42.242604 27738 rpcz_store.cc:276] 0709 12:22:38.996426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.996670 (+   244us) service_pool.cc:224] Handling call
0709 12:22:38.996942 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.997155 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:39.192750 (+195595us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.650617 (+1457867us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b1800377920. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:41.150631 (+500014us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.241807 (+1091176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:42.241862 (+    55us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:42.241925 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.241985 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.242126 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1388739,"prepare.queue_time_us":712353,"prepare.run_cpu_time_us":11110,"prepare.run_wall_time_us":109752,"raft.queue_time_us":1163359,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":22612,"spinlock_wait_cycles":34445184,"thread_start_us":925878,"threads_started":3,"wal-append.queue_time_us":550645}
I20260709 12:22:42.244278  6007 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.245350 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4304) took 2504 ms. Trace:
I20260709 12:22:42.245518  6007 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:22:42.221798  6148 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.207036  5991 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:42.190150  6104 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:42.225188  6149 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.225683 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3784) took 1414 ms. Trace:
I20260709 12:22:42.186693  6099 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.188467  6173 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:42.226899 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:42.282464  6149 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.228858  6154 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.229243  6005 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.284502  6099 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.232376 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4578) took 1925 ms. Trace:
I20260709 12:22:42.287521 27582 rpcz_store.cc:276] 0709 12:22:40.307211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.808029 (+500818us) service_pool.cc:224] Handling call
0709 12:22:40.861402 (+ 53373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.293577 (+432175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.293630 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.301393 (+  7763us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.692377 (+390984us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:41.692430 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.692521 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.165293 (+472772us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:42.231952 (+ 66659us) raft_consensus.cc:1685] finished
0709 12:22:42.231979 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.232310 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":372585,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":2846,"spinlock_wait_cycles":8123648,"thread_start_us":775858,"threads_started":2,"wal-append.queue_time_us":427442}
I20260709 12:22:42.213274  6105 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:42.290382  6105 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:42.234220  6126 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.177682  6110 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.235137  6147 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:42.236191 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:42.236912 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.36279e+06us
I20260709 12:22:42.237449  5920 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:22:42.238291 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.238767  6177 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.247877 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4303) took 2507 ms. Trace:
W20260709 12:22:42.248402 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56946.3us
I20260709 12:22:42.249408 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:42.307807 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58457.7us
I20260709 12:22:42.245736 27731 rpcz_store.cc:276] 0709 12:22:39.740825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.741304 (+   479us) service_pool.cc:224] Handling call
0709 12:22:39.741602 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.658493 (+916891us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.658542 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:40.659000 (+   458us) log.cc:844] Serialized 26 byte log entry
0709 12:22:40.810000 (+151000us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:41.209843 (+399843us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b18003ac9c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:22:41.210046 (+   203us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.020105 (+810059us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:42.245036 (+224931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.245113 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.245154 (+    41us) raft_consensus.cc:1685] finished
0709 12:22:42.245165 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.245299 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1951129,"prepare.queue_time_us":118485,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":916718,"raft.run_cpu_time_us":11827,"raft.run_wall_time_us":1342179,"spinlock_wait_cycles":17567360,"thread_start_us":369549,"threads_started":2,"wal-append.queue_time_us":370612,"wal-append.run_cpu_time_us":2873,"wal-append.run_wall_time_us":1152574}
I20260709 12:22:42.273801 27580 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:42.309756 27580 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:42.291805 27737 rpcz_store.cc:276] 0709 12:22:40.811820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.812059 (+   239us) service_pool.cc:224] Handling call
0709 12:22:42.225410 (+1413351us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:42.225604 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":685393792}
I20260709 12:22:42.311388 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:42.309773 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:42.279556 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 1.05426e+06us
I20260709 12:22:42.280720  6157 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.292073  6178 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.009s
W20260709 12:22:42.276427  6179 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:22:42.314544 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:42.310649 27580 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.271529  6137 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:42.319079 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.319380 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.320130 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.324385 27580 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:42.321724 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.327105 27735 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:42.322506 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.335207 27580 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:22:42.324708 27729 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:42.336468 27731 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:42.280098  6054 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:42.338963  6154 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:42.301867  6105 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.339941 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.408195 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.408490 27737 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:42.305392 27733 rpcz_store.cc:276] 0709 12:22:39.740481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.740632 (+   151us) service_pool.cc:224] Handling call
0709 12:22:39.740984 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.657373 (+916389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.657424 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:40.658023 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 12:22:40.818313 (+160290us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:41.152998 (+334685us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.335025 (+182027us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:42.247518 (+912493us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.247596 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.247634 (+    38us) raft_consensus.cc:1685] finished
0709 12:22:42.247712 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.247824 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2885604,"prepare.queue_time_us":110417,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":1961529,"raft.run_cpu_time_us":1509,"raft.run_wall_time_us":659829,"spinlock_wait_cycles":45290368,"thread_start_us":494232,"threads_started":2,"wal-append.queue_time_us":378573,"wal-append.run_cpu_time_us":3227,"wal-append.run_wall_time_us":1245424}
I20260709 12:22:42.409677 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.341069 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.411012 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.341459 28332 raft_consensus.cc:1217] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Deduplicated request from leader. Original: 32.21->[34.22-34.22]   Dedup: 34.22->[]
I20260709 12:22:42.411898 28332 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:22:42.294839  6173 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:42.301858  5884 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.298861  6147 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:42.351461 28335 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:42.350854 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:42.278177  6095 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.058s
I20260709 12:22:42.313122  6097 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.316113  6148 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:42.315850  5893 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:42.313257  6124 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:42.364130 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63618.8us
W20260709 12:22:42.364370 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51845.8us
I20260709 12:22:42.359127 28333 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:42.373020 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.350811  5886 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.454526 28331 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:42.439935 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.456624 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.458103 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:42.404218 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.241631  6089 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.360998  6048 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.315358  6177 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.295194  5991 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:42.413391 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.414184 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.484149 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.352695  6099 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.050s
I20260709 12:22:42.415282 27734 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 27: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:42.400606  5974 ts_tablet_manager.cc:1916] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:22:42.429821 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 79279.3us
I20260709 12:22:42.494124 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.442188 27735 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:42.428115  6178 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.450628 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3800) took 1520 ms. Trace:
I20260709 12:22:42.410840  6004 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.053s
I20260709 12:22:42.413117  6054 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:42.414299 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.484829 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:42.505892 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.462843 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4297) took 3145 ms. Trace:
I20260709 12:22:42.487088 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.487633 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.489661 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:42.442978 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:42.441419 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:42.507697 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 64833.1us
W20260709 12:22:42.497994 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 190373us
I20260709 12:22:42.498652  6089 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:42.496135 27736 rpcz_store.cc:276] 0709 12:22:40.929977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.751300 (+821323us) service_pool.cc:224] Handling call
0709 12:22:41.756314 (+  5014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.303179 (+546865us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.303219 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.303675 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.418312 (+114637us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:42.418365 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.418450 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.450356 (+ 31906us) raft_consensus.cc:1685] finished
0709 12:22:42.450386 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.450549 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":580441,"raft.queue_time_us":555832,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":20279,"thread_start_us":67319,"threads_started":1,"wal-append.queue_time_us":121881}
I20260709 12:22:42.499917 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.361475  6092 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:42.511598 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70269.1us
I20260709 12:22:42.511757 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:42.512297 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:42.437287  6148 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.509047 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:42.377105  6104 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.509616 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.462185  5593 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:42.480201  6061 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:42.563910 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.564633 27729 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:42.519598 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.438621  6154 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:22:42.566867 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53739.4us
I20260709 12:22:42.567880 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3783) took 1760 ms. Trace:
I20260709 12:22:42.462299  5920 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 12:22:42.574641 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:42.576177 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67218.3us
I20260709 12:22:42.571575 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:42.552974 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.583398 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:42.487108  5991 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:42.493237  5959 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:42.578547 27730 rpcz_store.cc:276] 0709 12:22:40.807362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.807505 (+   143us) service_pool.cc:224] Handling call
0709 12:22:40.807879 (+   374us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:41.038933 (+231054us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:41.052819 (+ 13886us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.207863 (+155044us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:22:42.182863 (+975000us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b18003f5aa0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:42.313424 (+130561us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:42.340196 (+ 26772us) log.cc:844] Serialized 52 byte log entry
0709 12:22:42.340429 (+   233us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:42.340479 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.340570 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.567510 (+226940us) raft_consensus.cc:1685] finished
0709 12:22:42.567538 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.567805 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1914388,"prepare.queue_time_us":13663,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":851844,"raft.run_cpu_time_us":2887,"raft.run_wall_time_us":165670,"spinlock_wait_cycles":49274624,"thread_start_us":344847,"threads_started":1,"wal-append.queue_time_us":442560}
I20260709 12:22:42.512960 27732 rpcz_store.cc:276] 0709 12:22:39.317605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.317790 (+   185us) service_pool.cc:224] Handling call
0709 12:22:39.318075 (+   285us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.318463 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:39.685175 (+366712us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.214518 (+529343us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:41.338146 (+1123628us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.303058 (+964912us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.303533 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.303696 (+   163us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:42.303745 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.303854 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.462563 (+158709us) raft_consensus.cc:1685] finished
0709 12:22:42.462597 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.462769 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1522344,"prepare.queue_time_us":616292,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":4439,"raft.queue_time_us":1410176,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":55405,"spinlock_wait_cycles":73799040,"thread_start_us":1390059,"threads_started":4,"wal-append.queue_time_us":698533,"wal-append.run_cpu_time_us":6592,"wal-append.run_wall_time_us":1178227}
I20260709 12:22:42.570946 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.544729  6148 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:42.589340 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.589430 27731 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:42.465092  6124 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.452483  6097 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.506912  5893 raft_consensus.cc:2764] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:22:42.483222  5886 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.515247  5897 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:42.587383 27733 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:42.515089  6155 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:42.573555  5835 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:42.573511  5967 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:42.592387 27731 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:22:42.638304 27733 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:42.638865 27731 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:22:42.585062  6092 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:42.511096  6126 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.114s
I20260709 12:22:42.620548 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.654168 27730 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
W20260709 12:22:42.621925 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.566701  5921 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:22:42.590971 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 276695us
I20260709 12:22:42.660244 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:42.625866 27732 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
W20260709 12:22:42.626991 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 43691.4us
I20260709 12:22:42.515916  6039 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:42.591137  6110 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.559923  5405 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:42.503156  6178 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.627292  5528 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:42.637590  6121 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:42.643221  5967 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.637614  5835 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.643352  6155 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.629052  6096 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:22:42.563453  6147 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.635674  5959 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:42.582572  6083 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:42.586858  6141 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:42.590771 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.651810 27733 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: [34.18-35.19]
I20260709 12:22:42.689043  6039 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:42.572726 27734 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:42.672029 27735 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:42.498029  6048 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.698907  5967 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:42.684007 27481 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 28 to 31. New cstate: current_term: 31 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:42.688103 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:42.694743 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.658957  5921 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.701756  6178 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.713565 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.651598 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.715296 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.743221 27734 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:42.697728  5405 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:42.704696  5835 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:42.698108  6121 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:42.738924 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 229348us
I20260709 12:22:42.743212 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
W20260709 12:22:42.678985 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 165198us
I20260709 12:22:42.745443 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1652) took 1655 ms. Trace:
I20260709 12:22:42.709534  6096 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.714937  6141 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.747013 27481 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 24, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 24 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:22:42.750353 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164255us
I20260709 12:22:42.654402  6054 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.756868 27737 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:42.716073  6083 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:42.753448 28330 rpcz_store.cc:276] 0709 12:22:41.090262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.090443 (+   181us) service_pool.cc:224] Handling call
0709 12:22:41.090727 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.090938 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:41.093164 (+  2226us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.216381 (+1123217us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b18004563a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a35b5e5 00007f0e9a35b568
0709 12:22:42.216631 (+   250us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.510510 (+293879us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.510840 (+   330us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.686127 (+175287us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.705750 (+ 19623us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:42.705819 (+    69us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:42.742933 (+ 37114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.745198 (+  2265us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.745370 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1127571,"prepare.queue_time_us":1198760,"prepare.run_cpu_time_us":200,"prepare.run_wall_time_us":11845,"raft.queue_time_us":177739,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":39381,"spinlock_wait_cycles":150689792,"thread_start_us":432942,"threads_started":2,"wal-append.queue_time_us":368479}
I20260709 12:22:42.762557 27731 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:22:42.765846 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.747174  5921 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:42.772221 27481 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 25. New cstate: current_term: 25 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:42.775714 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.712008  6146 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.050s
I20260709 12:22:42.747473  6005 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:42.754611  6121 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:42.769960 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.758805  6096 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:42.791894 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.758483  5886 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.759653  6141 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:42.821911 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53458.5us
I20260709 12:22:42.823057 27731 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:22:42.824359 27737 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:42.771736  6191 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.827890 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:42.829881 27734 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:22:42.777470  6060 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:42.839643 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.795004  6177 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.844975 27481 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 30. New cstate: current_term: 30 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:42.851075 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 131) took 48405 ms (client timeout 15000 ms). Trace:
I20260709 12:22:42.825802  6193 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:42.854384 27481 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:22:42.899715 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 71924.1us
I20260709 12:22:42.905582 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.905431 27481 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:42.896135 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.896380 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:22:42.907370 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.908556 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.896606 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.909473 27732 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:22:42.909682 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.896970 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.910394 27731 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:42.906766 27481 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 25 to 27, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 27 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:42.894287 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.911875 27735 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:42.912216 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:42.912808 27732 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:42.913017 27481 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 28, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 28 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:42.908561 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:42.914626 27733 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:42.856336 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:42.918596 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62378.8us
I20260709 12:22:42.921106 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.900071  6003 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:42.896696  5886 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:22:42.923787 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:42.826843  6005 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:42.851536 27482 rpcz_store.cc:269] 0709 12:21:54.307919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.311540 (+  3621us) service_pool.cc:224] Handling call
0709 12:22:42.708493 (+48396953us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:28.141161 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:28.141848 (+   687us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:28.141899 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:22:28.141931 (+    32us) tablet.cc:662] Decoding operations
0709 12:22:28.143249 (+  1318us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:28.143358 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:28.143384 (+    26us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:22:28.144496 (+  1112us) tablet.cc:701] Row locks acquired
0709 12:22:28.144539 (+    43us) write_op.cc:260] PREPARE: finished
0709 12:22:28.144702 (+   163us) write_op.cc:270] Start()
0709 12:22:28.144866 (+   164us) write_op.cc:276] Timestamp: P: 1783599748144630 usec, L: 0
0709 12:22:28.144902 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 12:22:28.347913 (+203011us) log.cc:844] Serialized 5598 byte log entry
0709 12:22:31.489772 (+3141859us) spinlock_profiling.cc:243] Waited 5 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:31.727648 (+237876us) op_driver.cc:464] REPLICATION: finished
0709 12:22:32.316276 (+588628us) write_op.cc:301] APPLY: starting
0709 12:22:32.325302 (+  9026us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:32.394646 (+ 69344us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34532a 00007f0e9a2caf14 00007f0e9a2caec3 00007f0e9a2cae38 00007f0e9a2cadf3 00007f0e9a2ca0dc 00007f0e9a2f3e16 00007f0e9a2dbcbb 00007f0e9a2c639d 00007f0e9a2c9d75
0709 12:22:33.182384 (+787738us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:33.182445 (+    61us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:33.765098 (+582653us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:33.765776 (+   678us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:22:33.765898 (+   122us) write_op.cc:312] APPLY: finished
0709 12:22:33.766777 (+   879us) log.cc:844] Serialized 129 byte log entry
0709 12:22:34.262311 (+495534us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:34.342350 (+ 80039us) write_op.cc:454] Released schema lock
0709 12:22:34.359945 (+ 17595us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:34.503443 (+143498us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:34.505316 (+  1873us) write_op.cc:454] Released schema lock
0709 12:22:34.643200 (+137884us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
Metrics: {"mutex_wait_us":3359069,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":537484,"apply.run_cpu_time_us":62167,"apply.run_wall_time_us":2131725,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62647,"num_ops":14,"prepare.queue_time_us":203763,"prepare.run_cpu_time_us":63193,"prepare.run_wall_time_us":628121,"raft.queue_time_us":119758,"raft.run_cpu_time_us":27461,"raft.run_wall_time_us":697405,"replication_time_us":3567152,"spinlock_wait_cycles":41564544,"thread_start_us":1256425,"threads_started":5,"wal-append.queue_time_us":1119188,"wal-append.run_cpu_time_us":16665,"wal-append.run_wall_time_us":5037379}]]}
I20260709 12:22:42.928685 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.838267  6191 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:42.936051 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.936916 27580 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:42.937801 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.938386 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:42.940613 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.941171 27581 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:42.932199 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:42.923077  5927 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:42.923512  6199 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:42.922286  6003 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:42.944454 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.944255 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:42.945853 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.842998  6060 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.946949  5593 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:42.955470  6124 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:42.980427 27732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:42.954833  5967 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.957849 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.988497 27582 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:42.959656 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.965106 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:42.966524 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:42.967021  6007 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.721s	user 0.007s	sys 0.004s
I20260709 12:22:42.927884  6193 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:42.969682 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:42.931746  6197 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:42.942703  5920 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:42.991886 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.012351 27737 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
W20260709 12:22:42.993825  6118 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.884s	user 0.006s	sys 0.014s
I20260709 12:22:42.999022  5923 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:43.014120 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 229) took 15918 ms (client timeout 15000 ms). Trace:
W20260709 12:22:43.014647 27482 rpcz_store.cc:269] 0709 12:22:27.095780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.013871 (+15918091us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:43.014029 (+   158us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:42.984635  5921 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.020902  5921 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:42.855123  6083 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.005522 28336 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:22:43.004174  6048 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.979987  6003 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:42.993762  6134 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.989609  5967 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:42.848476  5405 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.994479  6199 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:42.842514  6039 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:42.835431  5959 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.079823 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:43.081784 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:43.010221  6148 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.085770 27737 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:22:43.087204 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:43.035449  6155 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.102360 27733 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:22:43.101796 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:43.074126  5528 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:43.084240  6199 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.104465  5528 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.108085 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:43.075278  6083 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:43.052474  5991 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.111137 28336 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:43.084189  6007 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:43.115603 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:43.114811 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 224435us
I20260709 12:22:43.125720  5991 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.067731  6137 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:22:43.128976 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.130462 27737 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:22:43.135675 27737 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:43.131840 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.094322  6195 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:43.141119 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:43.141772 28334 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:43.144923 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.145599 27734 raft_consensus.cc:2376] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32. Re-sending same reply.
I20260709 12:22:43.146617 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.139016 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.112293  5528 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:43.148819 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.154026 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:43.156018  6060 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:43.118983  6007 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:43.156595 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1575) took 1052 ms. Trace:
I20260709 12:22:43.156978  6007 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:43.160722  5835 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:43.161659  6206 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 12:22:43.162032 27730 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:43.066865  6134 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.111838  6199 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:43.114674  6095 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:43.109467  6194 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:43.136656 27737 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.109879  6196 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:43.141477  6195 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.163626 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:43.174510 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.178717 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:43.254112 28333 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:43.256350 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:43.180299 28330 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
W20260709 12:22:43.183151 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.184958 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.262468 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.190464 28334 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
W20260709 12:22:43.263507 28337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1546) took 2031 ms (client timeout 2025 ms). Trace:
I20260709 12:22:43.210026 27737 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:43.157008 28335 rpcz_store.cc:276] 0709 12:22:42.103703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.104169 (+   466us) service_pool.cc:224] Handling call
0709 12:22:42.104477 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.421361 (+316884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.156301 (+734940us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:43.156507 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":567600640}
I20260709 12:22:43.243682 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.187258 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:43.247459 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:43.211297 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.273293 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.274266 27733 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:43.275826 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.276669 27733 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:43.278000 28336 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:43.279610 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.280378 27580 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:22:43.263980 28337 rpcz_store.cc:269] 0709 12:22:41.232473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.232671 (+   198us) service_pool.cc:224] Handling call
0709 12:22:43.263290 (+2030619us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:43.263435 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309610880}
I20260709 12:22:43.280373 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.283993 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:22:43.283979 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.285966 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.286895 27734 raft_consensus.cc:2368] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29.
I20260709 12:22:43.288583 27733 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:43.280999 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:43.290004 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 47592.1us
I20260709 12:22:43.293143 27580 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:43.297816 27581 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:43.172289  6181 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.516s	user 0.000s	sys 0.001s
W20260709 12:22:43.177013  6215 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
W20260709 12:22:43.165377  6092 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.000s
I20260709 12:22:43.299439 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:43.299681 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.304955 27580 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:43.231276  6030 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:43.186915  5775 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 35 current term: 35 is higher.
I20260709 12:22:43.374766 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.375634 27733 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:43.376856 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.378361 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.239432  6195 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:43.181456  6157 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.166651  6148 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:43.176204  6110 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:43.308487 28333 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
W20260709 12:22:43.366962 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51205.2us
I20260709 12:22:43.367175 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:43.420310 28337 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:43.239104  6191 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.243979  5920 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:43.374380 28332 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:43.384317 27663 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:22:43.385021 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:43.297715  6001 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.391777 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.442675 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.443413 28333 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:22:43.397615 27580 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
W20260709 12:22:43.353680  6121 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.527s	user 0.002s	sys 0.008s
I20260709 12:22:43.298874  6189 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.222657  6134 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.163271  5835 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:43.408406 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.414220 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:43.387012  6126 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:43.448894 27733 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:43.418207 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:43.364595 27582 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:43.414861 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:43.182855  5528 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.443394 27588 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:43.443394 27581 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:43.462670 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:43.464206 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 341339us
I20260709 12:22:43.477324 27657 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:22:43.497732 28337 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:43.502461 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:43.472400 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.503931 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.505105 27656 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:22:43.507818 27656 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:22:43.514796 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.515754 27656 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:43.426096  6124 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.448426  6089 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.900s	user 0.006s	sys 0.006s
I20260709 12:22:43.535742 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:43.530506 27657 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:43.536422 27663 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:43.526945 27737 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:22:43.539604 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.449797  6001 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:43.544768  6218 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:22:43.544515  5920 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.545459  6089 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:43.545998  6089 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:43.546172  6001 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:43.465274  6126 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:43.547039  6126 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:43.466663  6121 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.549577  6121 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.553335 27656 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
W20260709 12:22:43.525437  6141 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.696s	user 0.007s	sys 0.003s
W20260709 12:22:43.529006  6173 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.100s	user 0.013s	sys 0.000s
I20260709 12:22:43.417969  6203 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.147s
I20260709 12:22:43.527565  6134 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:43.527559  6157 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.443826  6110 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.591895 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.599395 27582 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
W20260709 12:22:43.552628  6176 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 12:22:43.553088  6117 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.445s	user 0.008s	sys 0.011s
I20260709 12:22:43.593601 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:43.606160 28337 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25. Re-sending same reply.
W20260709 12:22:43.604617 27587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.002s	sys 0.010s
I20260709 12:22:43.552299  6140 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.466171  6189 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.589342  6043 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.538010  6191 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.067s
I20260709 12:22:43.537860  6092 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:22:43.608815  6157 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.650346  6097 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:43.589712  6173 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.652326  6157 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:43.653399  6173 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.610509 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:43.637046 27587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.003s	sys 0.010s
W20260709 12:22:43.605021  6104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.009s	sys 0.005s
I20260709 12:22:43.642311 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.670367 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.642364 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:43.671257 27659 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:43.671950 27657 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:43.547004  6089 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.644168  5528 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:43.644989 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.645108  6110 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.543092  6124 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.552819  5914 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:22:43.667872 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.670951 27664 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:43.684254 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:43.672551 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.685127 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:43.677688 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.678081 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.686676 27657 raft_consensus.cc:2410] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:22:43.589839  6141 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:43.670614  6104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.011s	sys 0.006s
I20260709 12:22:43.666983  6195 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:43.692925 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.043s	sys 0.086s
W20260709 12:22:43.705142 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.043s	sys 0.086s
W20260709 12:22:43.707612 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 737946us
I20260709 12:22:43.708905 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:43.713444 27655 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:43.713474 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:43.717594 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 273041us
I20260709 12:22:43.719002 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:43.707389  6126 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.722931 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:22:43.723933 27658 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:43.603540  6134 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.638453  6140 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:43.666963  6043 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:43.723572  6211 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.807s	user 0.004s	sys 0.000s
W20260709 12:22:43.748096  6054 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.002s	sys 0.018s
W20260709 12:22:43.757663  6061 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.058s	sys 0.126s
I20260709 12:22:43.789147  6126 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:43.837791  6061 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.059s	sys 0.131s
W20260709 12:22:43.831930  6054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.004s	sys 0.021s
W20260709 12:22:43.860124  6197 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.075s	sys 0.107s
W20260709 12:22:43.885625  6197 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.076s	sys 0.107s
I20260709 12:22:43.888085  6197 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:43.888249  6232 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:22:43.889343  6197 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:43.896683  6096 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.065s	user 0.008s	sys 0.004s
I20260709 12:22:43.897187  6096 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:43.898726  6231 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:22:43.900573  6003 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.845s	user 0.010s	sys 0.000s
W20260709 12:22:43.901003  6225 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:22:43.901470  6195 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.904040  6124 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:43.922852  6150 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.280s	user 0.014s	sys 0.033s
I20260709 12:22:43.932271  6189 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.940537  6003 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:43.967187  6003 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:43.959527  6150 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.317s	user 0.014s	sys 0.033s
I20260709 12:22:43.970192  6141 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:43.978572  6030 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.598s	user 0.011s	sys 0.001s
W20260709 12:22:43.981997  6224 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:22:43.983129  6194 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.983702  6194 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:43.985517  6233 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:22:43.975672  6134 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:43.992331  6096 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:43.976586  6043 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:43.996986  6083 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.882s	user 0.004s	sys 0.007s
I20260709 12:22:44.001713  5974 ts_tablet_manager.cc:1929] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 15.10
I20260709 12:22:44.002663  5974 log.cc:1199] T 7274fd8a2115405c9b78c61214bff736 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/7274fd8a2115405c9b78c61214bff736
I20260709 12:22:44.004244  5974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51028 (request call id 0) took 6660 ms. Trace:
I20260709 12:22:44.004732  5974 rpcz_store.cc:276] 0709 12:22:37.343919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.344247 (+   328us) service_pool.cc:224] Handling call
0709 12:22:37.903651 (+559404us) ts_tablet_manager.cc:1151] Deleting tablet 7274fd8a2115405c9b78c61214bff736
0709 12:22:37.909462 (+  5811us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:22:43.231068 (+5321606us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:44.001498 (+770430us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:44.004138 (+  2640us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2341317,"tablet-delete.queue_time_us":387921,"thread_start_us":189101,"threads_started":1}
W20260709 12:22:44.095890  6180 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.441s	user 0.004s	sys 0.000s
I20260709 12:22:44.098559  6194 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:44.117238 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.173s	user 0.009s	sys 0.014s
W20260709 12:22:44.117838 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.174s	user 0.009s	sys 0.014s
W20260709 12:22:44.120826 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.18862e+06us
I20260709 12:22:44.122731 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:44.132879  6147 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:44.218207 27663 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.681s	user 0.008s	sys 0.007s
I20260709 12:22:44.218750 27663 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:22:44.251597  6187 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.607s	user 0.006s	sys 0.000s
I20260709 12:22:44.276615 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.278009 27663 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:44.279084 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:44.286887 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.288514 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.289988 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.290906 27657 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:44.293654 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:44.295508 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.297075 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:44.297883 27657 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:44.289983 27663 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:44.302106 27663 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [35.19-36.20]
I20260709 12:22:44.306197 27663 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:22:44.319036 27657 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
W20260709 12:22:44.331043 28331 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.009s	sys 0.026s
W20260709 12:22:44.331589 28331 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.011s	sys 0.026s
I20260709 12:22:44.334815 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.349004  6001 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:44.349596  6001 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:44.351635  6001 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:44.354223  6001 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:44.368041 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1903) took 1084 ms. Trace:
I20260709 12:22:44.368870 27584 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:44.368712 27585 rpcz_store.cc:276] 0709 12:22:43.278500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.278629 (+   129us) service_pool.cc:224] Handling call
0709 12:22:43.279021 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.405603 (+126582us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.405654 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.417197 (+ 11543us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.655765 (+238568us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:43.655805 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.362352 (+706547us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b540011c954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.362422 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.362508 (+    86us) raft_consensus.cc:1685] finished
0709 12:22:44.362522 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.362902 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":47089,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6416,"spinlock_wait_cycles":230110464,"thread_start_us":238228,"threads_started":1,"wal-append.queue_time_us":479494}
I20260709 12:22:44.379923 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4310) took 1812 ms. Trace:
I20260709 12:22:44.380743 27736 rpcz_store.cc:276] 0709 12:22:42.566954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.574222 (+  7268us) service_pool.cc:224] Handling call
0709 12:22:44.379241 (+1805019us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b54002ea4d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:44.379535 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141223296}
I20260709 12:22:44.391371 28329 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:44.393023 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1550) took 3134 ms. Trace:
I20260709 12:22:44.393380 28329 rpcz_store.cc:276] 0709 12:22:41.258947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.259209 (+   262us) service_pool.cc:224] Handling call
0709 12:22:41.278768 (+ 19559us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:44.391885 (+3113117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.392668 (+   783us) spinlock_profiling.cc:243] Waited 3.11 s on lock 0x7b54005a1254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.392937 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7156087040}
W20260709 12:22:44.419189  6216 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.358s	user 0.003s	sys 0.000s
W20260709 12:22:44.426285 26413 debug-util.cc:398] Leaking SignalData structure 0x7b08003f3000 after lost signal to thread 27702
W20260709 12:22:44.427016 26413 kernel_stack_watchdog.cc:198] Thread 27702 stuck at /apache/dev/git/kudu/src/kudu/rpc/outbound_call.cc:307 for 153ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260709 12:22:44.437523 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.148s	user 0.020s	sys 0.003s
I20260709 12:22:44.437738 27589 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
W20260709 12:22:44.440117 27588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.054s	sys 0.032s
W20260709 12:22:44.438148 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.148s	user 0.020s	sys 0.003s
W20260709 12:22:44.440614 27588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.054s	sys 0.032s
W20260709 12:22:44.440958 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 1.15994e+06us
I20260709 12:22:44.442175 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:44.443681 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4606) took 1476 ms. Trace:
I20260709 12:22:44.444193 27587 rpcz_store.cc:276] 0709 12:22:42.967575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.967727 (+   152us) service_pool.cc:224] Handling call
0709 12:22:42.968109 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.973676 (+  5567us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:22:42.973724 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.992190 (+ 18466us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.657118 (+664928us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:43.657177 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.657273 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.443085 (+785812us) raft_consensus.cc:1685] finished
0709 12:22:44.443140 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.443431 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5061,"raft.queue_time_us":5995,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":14299,"thread_start_us":629332,"threads_started":1,"wal-append.queue_time_us":747203}
W20260709 12:22:44.448313 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 728777us
I20260709 12:22:44.449616 27589 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:22:44.451181 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.451243 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.455264 27581 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:22:44.459309 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.460948 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4604) took 1496 ms. Trace:
I20260709 12:22:44.461526 27584 rpcz_store.cc:276] 0709 12:22:42.964642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.964800 (+   158us) service_pool.cc:224] Handling call
0709 12:22:44.454497 (+1489697us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540006fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:44.460845 (+  6348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3168734592}
I20260709 12:22:44.459280 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:44.466683 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.047s	user 0.014s	sys 0.010s
I20260709 12:22:44.466861 27584 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:44.444605 27552 leader_election.cc:400] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29.
W20260709 12:22:44.468114 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.049s	user 0.015s	sys 0.010s
I20260709 12:22:44.469657 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:44.470640 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:44.470832 27655 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:44.471547 27581 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:44.475662  5803 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:44.478818  5803 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.483242  5914 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:44.486311  5914 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.487730 27661 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.105s	user 0.012s	sys 0.010s
W20260709 12:22:44.488240 27661 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.105s	user 0.012s	sys 0.010s
I20260709 12:22:44.489164  5803 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:44.486972  5914 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.494562  6001 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:44.497295 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:44.497881 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:44.500422 27732 raft_consensus.cc:606] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:44.503494 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4321) took 1809 ms. Trace:
I20260709 12:22:44.503918 27730 rpcz_store.cc:276] 0709 12:22:42.694329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.694605 (+   276us) service_pool.cc:224] Handling call
0709 12:22:42.694900 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.162428 (+467528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.502830 (+1340402us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54000a46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.503243 (+   413us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1073827584}
I20260709 12:22:44.474808  6259 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:44.519524 27587 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:44.511313  6259 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.527163 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:44.529806  5914 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.532740  6018 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:44.533242  6018 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:44.533686  6018 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:44.554382 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4607) took 1019 ms. Trace:
I20260709 12:22:44.571128  6147 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.576480  6147 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.563511  5803 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.587285  6005 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:44.589936 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.589003  6147 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.530193  6259 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:44.591475 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75932.5us
W20260709 12:22:44.598071 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.564592 27588 rpcz_store.cc:276] 0709 12:22:43.554670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.555447 (+   777us) service_pool.cc:224] Handling call
0709 12:22:43.573513 (+ 18066us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.573756 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:43.596570 (+ 22814us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.550624 (+954054us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.553577 (+  2953us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:44.553689 (+   112us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:44.553815 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.553982 (+   167us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.554297 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69968,"prepare.queue_time_us":122165,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":12485,"raft.queue_time_us":32501,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":281,"thread_start_us":869741,"threads_started":2,"wal-append.queue_time_us":353620}
I20260709 12:22:44.600986  5884 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:44.591267  6018 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:44.602558  5884 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.603786 27662 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 22. Current term is 24. Ops: []
I20260709 12:22:44.606750 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1864) took 1289 ms. Trace:
I20260709 12:22:44.608414  6018 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.610370  5884 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.607172 27662 rpcz_store.cc:276] 0709 12:22:43.317352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.317473 (+   121us) service_pool.cc:224] Handling call
0709 12:22:43.317843 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:44.604422 (+1286579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.606361 (+  1939us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400041354. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.606650 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2955957120}
W20260709 12:22:44.610960 27658 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.886s	user 0.013s	sys 0.004s
I20260709 12:22:44.619868 27658 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:44.624142 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:44.627871 27552 connection.cc:256] RPC call timeout handler was delayed by -1.1280769369996051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
W20260709 12:22:44.628535 27552 connection.cc:256] RPC call timeout handler was delayed by -1.0938769370000045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1734s for a response
I20260709 12:22:44.632733 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1866) took 1314 ms. Trace:
I20260709 12:22:44.633234 27661 rpcz_store.cc:276] 0709 12:22:43.318119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.319218 (+  1099us) service_pool.cc:224] Handling call
0709 12:22:43.319687 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.488890 (+1169203us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:44.488953 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.489914 (+   961us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.594496 (+104582us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:44.594524 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.597507 (+  2983us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.632465 (+ 34958us) raft_consensus.cc:1685] finished
0709 12:22:44.632485 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.632669 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":95434,"raft.queue_time_us":123722,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":20462,"thread_start_us":1208482,"threads_started":2,"wal-append.queue_time_us":110458}
I20260709 12:22:44.636895 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:44.638329 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.645327 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.647624 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.648545 28329 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:44.650267 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:44.651010 28329 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:22:44.653620 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:22:44.655508 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.656517 27585 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 23.
I20260709 12:22:44.660714 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.661087 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.662593  6140 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.663630  6140 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:44.623695  6018 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.665552  6267 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.667244  6267 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.670109  6267 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.670284  6259 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:44.664532  6140 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.673085  6018 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:44.662483 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:44.680786 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.686058  6140 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.686534 27585 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:44.686619  6140 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.682971 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.687181  6140 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.687870 27582 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:44.687211  6259 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.688647 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 56) took 1719 ms. Trace:
I20260709 12:22:44.689447 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:44.690215 27582 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:44.690650 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
I20260709 12:22:44.689697  6140 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.689085 27732 rpcz_store.cc:276] 0709 12:22:42.968915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.980252 (+ 11337us) service_pool.cc:224] Handling call
0709 12:22:44.688359 (+1708107us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54001d15d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:44.688512 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3493270400}
W20260709 12:22:44.693653 28332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1662) took 1515 ms (client timeout 1951 ms). Trace:
W20260709 12:22:44.694272 28332 rpcz_store.cc:269] 0709 12:22:43.178236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.241243 (+ 63007us) service_pool.cc:224] Handling call
0709 12:22:44.693255 (+1452012us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54002fab54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:44.693436 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269822848}
I20260709 12:22:44.697018 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1580) took 1562 ms. Trace:
I20260709 12:22:44.697432 28331 rpcz_store.cc:276] 0709 12:22:43.134761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.134883 (+   122us) service_pool.cc:224] Handling call
0709 12:22:43.155341 (+ 20458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.241937 (+ 86596us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:22:43.247231 (+  5294us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.653456 (+406225us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b18003382c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:44.336441 (+682985us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.341147 (+  4706us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:44.341227 (+    80us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:44.341352 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.696713 (+355361us) spinlock_profiling.cc:243] Waited 86.3 ms on lock 0x7b5400617454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.696773 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.696939 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":4939,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":97,"raft.queue_time_us":262,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":344,"spinlock_wait_cycles":201716480,"thread_start_us":681916,"threads_started":2,"wal-append.queue_time_us":731486}
I20260709 12:22:44.699607  6267 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.700350  6267 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.702437 27553 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:44.706008  5775 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:44.706188  6268 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:44.707000  6268 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:44.709404  6268 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.709936  6268 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:44.712641  6269 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:44.713117  6269 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:44.714200 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.719871 28334 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:44.719920 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.720808 28336 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:44.721139 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.722057 28335 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:44.722180 28331 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:44.707227  5775 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:44.725092  6018 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.725291  5775 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:44.726675  5775 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.727615 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.725337 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:44.711174  6268 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.019s
I20260709 12:22:44.714365  6269 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.734862  6269 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:44.723939 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:44.703022  6267 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:44.744891 27580 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.055s	user 0.000s	sys 0.020s
W20260709 12:22:44.745417 27580 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.001s	sys 0.021s
I20260709 12:22:44.749969 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:44.750818 27582 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 24.
W20260709 12:22:44.753094 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90920.1us
I20260709 12:22:44.753263 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4610) took 1114 ms. Trace:
W20260709 12:22:44.754372  6148 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.182s	user 0.010s	sys 0.009s
W20260709 12:22:44.754761  6148 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.216s	user 0.011s	sys 0.010s
I20260709 12:22:44.754058 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:44.756325 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1693) took 2415 ms. Trace:
I20260709 12:22:44.757592  6148 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:44.757823  6089 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:44.758347  6148 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.758494  6089 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:44.759001 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.759064  6089 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:44.759582  6148 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:44.756976 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:44.768879  6271 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:44.769572  6271 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:44.769968  6271 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:22:44.778436 27586 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.062s	user 0.024s	sys 0.029s
W20260709 12:22:44.778872 27586 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.069s	user 0.024s	sys 0.029s
W20260709 12:22:44.783793 27660 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.364s	user 0.005s	sys 0.028s
W20260709 12:22:44.787725 27660 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.368s	user 0.005s	sys 0.028s
I20260709 12:22:44.788189  6267 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:44.768352 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:44.789104 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.789671  6267 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:44.757558 27738 rpcz_store.cc:276] 0709 12:22:42.341187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.351152 (+  9965us) service_pool.cc:224] Handling call
0709 12:22:42.351515 (+   363us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:42.654560 (+303045us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.654589 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:42.683897 (+ 29308us) log.cc:844] Serialized 52 byte log entry
0709 12:22:42.751093 (+ 67196us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:42.935962 (+184869us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.936597 (+   635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.160002 (+223405us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x7b1800370060. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db
0709 12:22:43.161120 (+  1118us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:44.755686 (+1594566us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.755774 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.755833 (+    59us) raft_consensus.cc:1685] finished
0709 12:22:44.755844 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.756077 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":181850,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":14507,"raft.queue_time_us":33899,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":8898,"spinlock_wait_cycles":717663744,"thread_start_us":200203,"threads_started":2,"wal-append.queue_time_us":127962,"wal-append.run_cpu_time_us":6028,"wal-append.run_wall_time_us":1360870}
W20260709 12:22:44.804864 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.807716 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1901) took 1530 ms. Trace:
I20260709 12:22:44.807631  6007 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.808486 27586 rpcz_store.cc:276] 0709 12:22:43.277792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.277957 (+   165us) service_pool.cc:224] Handling call
0709 12:22:43.278276 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.278500 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:43.362116 (+ 83616us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.705382 (+1343266us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b18002f89c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:44.779244 (+ 73862us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.798077 (+ 18833us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:44.801703 (+  3626us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:44.804333 (+  2630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.806748 (+  2415us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.807653 (+   905us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17812,"prepare.queue_time_us":198845,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":83,"raft.queue_time_us":46635,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":14731,"spinlock_wait_cycles":38385024,"thread_start_us":1392190,"threads_started":3,"wal-append.queue_time_us":66265,"wal-append.run_cpu_time_us":3259,"wal-append.run_wall_time_us":1318820}
I20260709 12:22:44.816383  6005 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:44.801402  6147 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.016s
I20260709 12:22:44.823650  6005 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.826447 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1863) took 1509 ms. Trace:
I20260709 12:22:44.827898  6147 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:44.828423  6147 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:44.826880 27658 rpcz_store.cc:276] 0709 12:22:43.316964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.317107 (+   143us) service_pool.cc:224] Handling call
0709 12:22:44.824168 (+1507061us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b540005fd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:44.826326 (+  2158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":681362304}
I20260709 12:22:44.829703  6147 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 28.23, Last appended by leader: 23, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.831131  6147 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:44.818243  6007 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.833864  6007 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:44.834415  6005 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:44.825162 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.841578 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4339) took 1793 ms. Trace:
I20260709 12:22:44.842012 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.841903 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.843689 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.845238 27582 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:22:44.846819 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:44.847757 27586 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 32 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:22:44.753750 27580 rpcz_store.cc:276] 0709 12:22:43.638827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.665959 (+ 27132us) service_pool.cc:224] Handling call
0709 12:22:43.666329 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.666564 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:43.688996 (+ 22432us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.752726 (+1063730us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:44.752843 (+   117us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:44.752924 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.753032 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.753193 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3505,"prepare.queue_time_us":43969,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":10780,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"thread_start_us":1056476,"threads_started":1,"wal-append.queue_time_us":288386}
I20260709 12:22:44.849363  6157 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:44.842051 27731 rpcz_store.cc:276] 0709 12:22:43.048401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.048562 (+   161us) service_pool.cc:224] Handling call
0709 12:22:43.048852 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.049058 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:43.074303 (+ 25245us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.129283 (+ 54980us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.219750 (+ 90467us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b1800472420. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:44.346030 (+1126280us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b1800472420. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:44.346066 (+    36us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b1800472420. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:44.839474 (+493408us) spinlock_profiling.cc:243] Waited 6.56 ms on lock 0x7b1800472420. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:44.840053 (+   579us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.840458 (+   405us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.840760 (+   302us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.840913 (+   153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:44.840977 (+    64us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:44.841056 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.841198 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.841390 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18153,"prepare.queue_time_us":36049,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":89,"raft.queue_time_us":6727,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":12059,"spinlock_wait_cycles":125734528,"thread_start_us":54215,"threads_started":1,"wal-append.queue_time_us":65357}
I20260709 12:22:44.852712  6007 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:22:44.861599  6157 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:22:44.864243 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96583.2us
I20260709 12:22:44.865212 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:44.865746  6007 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:22:44.866318  6016 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:44.867865 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:44.868820  6016 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:44.869304  6016 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:44.869426 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:44.869861  6016 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:44.876884  6196 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:44.879933  6271 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.881438  6156 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:44.882014  6156 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.882535  6156 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:44.882603  6133 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:44.884250 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:44.881377 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:44.885004 28328 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:44.887648  6196 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:44.888289  6196 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:44.889616 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4337) took 1957 ms. Trace:
I20260709 12:22:44.889454  6196 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.889988 27735 rpcz_store.cc:276] 0709 12:22:42.932530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.932742 (+   212us) service_pool.cc:224] Handling call
0709 12:22:42.971884 (+ 39142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.037254 (+ 65370us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:22:43.058388 (+ 21134us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.885869 (+1827481us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:44.885961 (+    92us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:44.886046 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.889171 (+  3125us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b54002e90d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:44.889215 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.889429 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1353792,"prepare.queue_time_us":23478,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":4101,"spinlock_wait_cycles":149621120,"thread_start_us":473164,"threads_started":1,"wal-append.queue_time_us":579384}
I20260709 12:22:44.891232  6172 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:22:44.892607 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 418986us
I20260709 12:22:44.893335  6011 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:22:44.893770 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 295967us
I20260709 12:22:44.894277  6242 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:22:44.894953 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 258326us
I20260709 12:22:44.895697  6191 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:44.876873 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:44.919618 28328 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:44.902308 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:44.927878 28292 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.575s (SENT)
I20260709 12:22:44.930281 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:44.930655 28292 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 1.612s (SENT)
W20260709 12:22:44.931090 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 1.21748e+06us
W20260709 12:22:44.935997 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 813573us
I20260709 12:22:44.936987 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:44.937790  6089 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.939495  6141 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:44.940145  6141 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.944983  6089 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.847462 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 7274fd8a2115405c9b78c61214bff736 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:22:44.954954 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:44.955634 28329 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:44.913810  6271 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.955129 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:44.967074 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1862) took 1650 ms. Trace:
I20260709 12:22:44.964345  6190 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:44.971889  6270 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:44.972147 27471 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124880us
I20260709 12:22:44.973127  5528 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:44.973645  5528 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.974079  5528 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:44.975500  6272 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:22:44.975910 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:44.976388  6272 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:44.977190  5835 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:44.977456  6272 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.979549  5803 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:22:44.979954  6190 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:44.980145  5803 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:44.981349  6133 raft_consensus.cc:2764] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:22:44.943025  6148 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.981702  5803 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:44.983696 27583 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:44.983819  6148 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:44.970202  6156 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:44.987468 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88330.2us
I20260709 12:22:44.991655 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:44.978034  5835 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:44.995723 27586 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 30 because replica has last-logged OpId of term: 28 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
I20260709 12:22:44.970975 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:44.979177 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:45.000517 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:45.001195 28328 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:44.970443 27660 rpcz_store.cc:276] 0709 12:22:43.316288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.316511 (+   223us) service_pool.cc:224] Handling call
0709 12:22:43.383179 (+ 66668us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:44.788346 (+1405167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.788419 (+    73us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:44.789537 (+  1118us) log.cc:844] Serialized 52 byte log entry
0709 12:22:44.817454 (+ 27917us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:44.817490 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.819134 (+  1644us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.948828 (+129694us) raft_consensus.cc:1685] finished
0709 12:22:44.948859 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.967014 (+ 18155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35715,"raft.queue_time_us":80076,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":38514,"thread_start_us":1396053,"threads_started":2,"wal-append.queue_time_us":123927}
I20260709 12:22:45.015126  6156 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.001145  6133 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.015s
I20260709 12:22:45.020076  6141 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:45.020256 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 143635us
W20260709 12:22:45.021034 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74308.2us
I20260709 12:22:45.021211  6011 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:45.021847  6011 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.021899 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:45.022392  6011 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:45.022943  5884 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:45.023154 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:45.023372  5884 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:45.024425  5884 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:45.029592 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.031929  6011 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.032323  6148 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.033361 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 103278us
I20260709 12:22:45.033044  6148 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.034183 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.035025  6242 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:45.033066 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.037832 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:45.021873 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.039610 28328 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:45.024411 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:45.043025 28292 leader_election.cc:336] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.104s (SENT)
I20260709 12:22:45.041332 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:45.036720 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 1.90219e+06us
I20260709 12:22:45.050107 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:45.056348 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 77339.6us
I20260709 12:22:45.071702 28329 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:45.045295  6270 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:45.077236 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:45.096094 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.058203  6148 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:45.089102  6242 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:22:45.089768 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67975.6us
I20260709 12:22:45.097103  5914 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:45.097028  6197 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:45.097765 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:45.098738  5914 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:45.096306 28328 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:45.099211  5914 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:45.099686  5914 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:45.099912  6133 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.100234 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:45.100499  6133 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.101451  6148 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:45.098937 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.073158 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:45.102118 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64232.9us
I20260709 12:22:45.103868 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:45.114813 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44689.9us
I20260709 12:22:45.115654  6148 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:45.117568  6011 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.055518  6141 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:22:45.122260 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.122730 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:45.072638  6190 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:45.077605  6172 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:45.077569  6089 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:45.097043  6242 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:45.144150 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 85364.2us
W20260709 12:22:45.152808 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.154311 28328 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
W20260709 12:22:45.157800 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78690.9us
I20260709 12:22:45.114566  6191 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:22:45.122008  6156 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:45.113848  6270 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:45.137823  6271 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:22:45.184913 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85905.4us
I20260709 12:22:45.139695  6141 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:45.203032  6141 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:45.214934 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:45.141026  6190 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.156754  6242 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:45.204910 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.207201  6133 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:45.207798  6289 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
W20260709 12:22:45.208580 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 109761us
I20260709 12:22:45.137339  6001 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:45.164366  6011 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:22:45.217211  6190 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:45.185837  6156 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:45.199681  6271 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.224066 28331 raft_consensus.cc:2368] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33.
I20260709 12:22:45.142570  6016 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.233939 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.160778  6089 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.249266 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.250048  6011 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.220974  6148 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.230826  6156 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:45.258888 27481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 132) took 28877 ms (client timeout 15000 ms). Trace:
I20260709 12:22:45.259014 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.259994 28328 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:45.246130 27480 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 24, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 24 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:45.262058 27731 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:22:45.262705 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.263835  5914 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.264763 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.265465  5967 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.266269 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 319209us
I20260709 12:22:45.270951  5405 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.271379  6191 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:45.267107 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.233004  6271 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:45.282050 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3819) took 2036 ms. Trace:
I20260709 12:22:45.282506  6189 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:45.284744  6240 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:45.284869  6188 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:22:45.292611 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 147567us
I20260709 12:22:45.225100  6140 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:45.313465 27736 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:45.292443  6269 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:22:45.307287 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.319253 27737 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:45.319298 27729 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:45.309548 27733 rpcz_store.cc:276] 0709 12:22:43.245582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.464981 (+219399us) service_pool.cc:224] Handling call
0709 12:22:43.465424 (+   443us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:44.846297 (+1380873us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:44.965246 (+118949us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.271604 (+306358us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:45.272243 (+   639us) log.cc:844] Serialized 52 byte log entry
0709 12:22:45.272419 (+   176us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:45.272620 (+   201us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.272807 (+   187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.272911 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.281640 (+  8729us) raft_consensus.cc:1685] finished
0709 12:22:45.281658 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.281846 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1502831,"prepare.queue_time_us":11301,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":1629375,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":237,"thread_start_us":208252,"threads_started":2,"wal-append.queue_time_us":107765}
I20260709 12:22:45.323593 27738 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:22:45.327704 28292 leader_election.cc:336] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.220s (SENT)
I20260709 12:22:45.327836  5927 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:45.328238 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.328437  5927 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.330694  6184 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:45.312438  6001 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.316579  5405 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.330766  5927 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:45.321594  6272 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:22:45.321185  5967 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:45.335528 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 86384.3us
W20260709 12:22:45.339692 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 114619us
W20260709 12:22:45.343577 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44283.1us
I20260709 12:22:45.345091  6189 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.008s
I20260709 12:22:45.318800  6269 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:45.332758  6270 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.332544  6188 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:45.360414 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.360934  6188 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.361816  6188 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.362442  6240 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:45.362910  6240 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.363093  6188 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:45.333491  6001 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.363902  6240 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.332250  6096 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:45.364899  6240 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:45.365127  6096 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.338812  6244 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.058s
I20260709 12:22:45.366603  6244 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:45.367019  6244 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.367906  6244 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.368505  5914 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.369076  6244 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:45.338722  5884 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.370021  5528 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.370419  6089 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.331100  6184 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:45.353026  6272 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:45.359109 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.366240  6096 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.360351  6172 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:45.340767  6016 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.378686  6184 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:45.369012 27480 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 25 to 26, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 26 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:45.379988 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:45.340778  5405 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:45.357707  5927 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.387882 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.327008  6140 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.378947  6272 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:45.412331 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:45.412955 27662 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:45.418753 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.392700 27737 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:45.371289  5884 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.371819  6269 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:45.384980 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.395627 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.448704 27658 raft_consensus.cc:2410] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 35 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:22:45.419283 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.453318 27586 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 29 because replica has last-logged OpId of term: 28 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
W20260709 12:22:45.453886 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74293.5us
I20260709 12:22:45.419859  6103 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.021s
I20260709 12:22:45.420183 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.454757 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:45.420934  6096 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:45.456077 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.421099 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.459978 27588 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:22:45.421891 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.461330 27585 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:45.423085 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.423825 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.462863 27581 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:45.421893  6241 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.004s
I20260709 12:22:45.427682  5967 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:45.428535 26413 debug-util.cc:398] Leaking SignalData structure 0x7b0800378c80 after lost signal to thread 27702
I20260709 12:22:45.426200  6141 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.430435  6001 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:45.434175  6189 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:45.387079  6011 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.435230  6104 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:22:45.436060 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002df780 after lost signal to thread 26411
W20260709 12:22:45.486490 28358 debug-util.cc:398] Leaking SignalData structure 0x7b080023d780 after lost signal to thread 27521
I20260709 12:22:45.388109 27729 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:45.436655  6140 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:45.296250 27481 rpcz_store.cc:269] 0709 12:22:16.381833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.418120 (+ 36287us) service_pool.cc:224] Handling call
0709 12:22:45.258644 (+28840524us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:43.104238 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:43.104957 (+   719us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:43.105016 (+    59us) write_op.cc:435] Acquired schema lock
0709 12:22:43.105036 (+    20us) tablet.cc:662] Decoding operations
0709 12:22:43.125708 (+ 20672us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:43.125794 (+    86us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:43.125825 (+    31us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:22:43.240232 (+114407us) tablet.cc:701] Row locks acquired
0709 12:22:43.240282 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:22:43.240475 (+   193us) write_op.cc:270] Start()
0709 12:22:43.240625 (+   150us) write_op.cc:276] Timestamp: P: 1783599763240409 usec, L: 0
0709 12:22:43.240650 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:22:43.415310 (+174660us) log.cc:844] Serialized 6738 byte log entry
0709 12:22:44.003660 (+588350us) op_driver.cc:464] REPLICATION: finished
0709 12:22:44.007207 (+  3547us) write_op.cc:301] APPLY: starting
0709 12:22:44.007342 (+   135us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:44.050051 (+ 42709us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:44.051153 (+  1102us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:44.068454 (+ 17301us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:44.069043 (+   589us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0709 12:22:44.069118 (+    75us) write_op.cc:312] APPLY: finished
0709 12:22:44.070652 (+  1534us) log.cc:844] Serialized 156 byte log entry
0709 12:22:44.073046 (+  2394us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:44.073568 (+   522us) write_op.cc:454] Released schema lock
0709 12:22:44.073678 (+   110us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:44.074645 (+   967us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:44.074679 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7128431,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":3241,"apply.run_cpu_time_us":29595,"apply.run_wall_time_us":67261,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75187,"num_ops":17,"prepare.queue_time_us":116031,"prepare.run_cpu_time_us":64108,"prepare.run_wall_time_us":762135,"raft.queue_time_us":21344,"raft.run_cpu_time_us":3470,"raft.run_wall_time_us":4049,"replication_time_us":762835,"thread_start_us":497308,"threads_started":4,"wal-append.queue_time_us":539444,"wal-append.run_cpu_time_us":23640,"wal-append.run_wall_time_us":1114299}]]}
I20260709 12:22:45.422962 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.438351 27736 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:45.454075  6089 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:45.466372 27580 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:45.468284 27662 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:45.459304 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:45.478276 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.418856  6190 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.445797  6016 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.495391  6189 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:22:45.495560 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72706.3us
I20260709 12:22:45.486790  6147 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:45.509936  6016 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.510643  6147 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:45.512038 27580 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:45.513036  6189 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.514063  6189 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.514199  6272 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.516795 27580 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260709 12:22:45.506395 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.509465 28358 debug-util.cc:398] Leaking SignalData structure 0x7b080033be20 after lost signal to thread 27613
I20260709 12:22:45.519814 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:45.523692 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.486634  6197 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:45.486272  6001 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:45.486950  5991 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:22:45.530167 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002ed780 after lost signal to thread 27688
W20260709 12:22:45.544322 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002ed6c0 after lost signal to thread 27702
W20260709 12:22:45.530571 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 173058us
I20260709 12:22:45.515657  6272 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:45.539851 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.539976 27589 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:45.548621 27588 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:45.547103 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:45.540642 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.541276 27581 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:45.540596 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.445087  5927 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.547999 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:45.550555 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08003e32a0 after lost signal to thread 27762
I20260709 12:22:45.550694 27585 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: [31.17-32.18]
I20260709 12:22:45.517475  6189 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:45.552754 28358 debug-util.cc:398] Leaking SignalData structure 0x7b0800295060 after lost signal to thread 27763
W20260709 12:22:45.554188 28358 debug-util.cc:398] Leaking SignalData structure 0x7b0800102060 after lost signal to thread 28361
I20260709 12:22:45.548408  5405 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:45.529135  5967 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:45.563740  5967 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.564463  6190 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.565655  5927 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:45.565795  5967 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:45.566447 27580 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:45.568646 27584 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:45.572795 27585 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:45.573724 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.575148 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:45.577759 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.543848  6001 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:45.584481 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:45.585554 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.614217 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.620505 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:45.615011 27586 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:45.621393 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
I20260709 12:22:45.640614 27588 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:45.646222 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91369.5us
I20260709 12:22:45.577224  5528 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:45.563126  5405 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.568583  6191 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:22:45.640064 27480 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:45.661717 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.668406 27731 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:45.657440 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45298.4us
I20260709 12:22:45.540354  6147 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.671162 27660 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:45.655224 27737 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
W20260709 12:22:45.660128 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4aa1523b9af84c3db99ba394e2b9aa9a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:45.620193  6188 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:45.612469  5927 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:45.679265 27580 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:22:45.642167  6220 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.679939 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.681010 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:45.582093  6242 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.681771 27658 raft_consensus.cc:2376] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33. Re-sending same reply.
I20260709 12:22:45.689854 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:45.667279  5803 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.611603  6197 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.703853 27732 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:45.674651  6188 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.709138 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:45.721287 27585 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:45.740484 27736 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:45.712448  6188 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:45.757575 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.705618  6016 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.739521 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.749096 28329 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:22:45.720613  6190 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.705745  5884 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:45.757529  6141 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:45.706516 27737 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:45.798683 27737 raft_consensus.cc:606] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:45.764681 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:45.766454  6269 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.762907  6011 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.006s
I20260709 12:22:45.766501 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.768975 27658 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:45.771816 27660 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:45.772482  6242 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:45.772408 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.773563 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:45.705917  5803 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:45.775357 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.699678  5927 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.773816 27480 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 25, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 25 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:45.709692  6156 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.777819  6124 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:45.808202 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:45.828773 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:45.778931  6197 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:45.833344 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.836169 27582 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:45.838111 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.736024  6220 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.814893  6016 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:22:45.849886 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85273.7us
I20260709 12:22:45.850157 27730 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:45.850775 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:45.842448 27736 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:45.848855 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:45.855110 27729 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:22:45.856499 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.857124 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 131616us
I20260709 12:22:45.738068  6001 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.820722  6241 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:45.841584 28329 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:45.783434  6190 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:45.778151  6099 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.227s
I20260709 12:22:45.778002  6105 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.197s
I20260709 12:22:45.837672  6157 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:45.872752 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.809590  6271 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:45.877553 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:45.878587 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.892831 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:45.895244 27738 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.003s
I20260709 12:22:45.895766 27738 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:45.898289 27585 raft_consensus.cc:2368] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:45.905515 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:45.908411 27658 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:45.882123 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:45.915501 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.873099  6016 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:45.919718 27586 raft_consensus.cc:2410] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
W20260709 12:22:45.920469 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44339.2us
I20260709 12:22:45.877460  6172 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:45.925500 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:45.926168 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:45.928159 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.962301 27737 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:22:45.930212 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:45.928212 27480 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 19 to 22, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 22 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:45.947294 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.961637 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:45.969933 27585 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:22:45.979889 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:45.984169 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:45.969558  6244 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.600s	user 0.010s	sys 0.002s
I20260709 12:22:45.961061  5405 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:45.971735  6133 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:45.995366 27655 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:45.989838 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:45.997555 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80012.6us
I20260709 12:22:45.989610 27630 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:22:46.002655 27630 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:22:46.003228 27662 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
W20260709 12:22:46.009430 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.997079  6296 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260709 12:22:45.981823  6148 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:45.984344  6305 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:46.013332 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:46.032730 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.033887 27655 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:46.003191  6133 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:46.038035 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.015753 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:46.042141 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:46.021780  6244 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:46.036561 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:46.021667  6148 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:46.053908 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:46.033535  6172 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:46.056398 27586 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:46.061571 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:46.062862 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:46.064051 27731 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
W20260709 12:22:46.067729  6301 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:22:46.067951 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.068884 27655 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:46.069795 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.038597  6001 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:46.043463  6133 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:46.061784 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:46.071058 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.073642 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.077603 27729 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:46.064253 27480 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 19 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 25 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:46.079857 27588 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:46.067618  6156 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.037499  6016 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.084728 27580 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:46.085170 27738 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:46.088207 27738 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.089174 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:46.090910 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:46.092937 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103202us
I20260709 12:22:46.095798 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.089730 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:46.065974  6148 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:46.098846 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:46.100257 27584 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.523s	user 0.011s	sys 0.000s
I20260709 12:22:46.111624 27584 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:46.080530  5967 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:46.089594  6011 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:46.134456 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.136287 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.075261  6271 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:46.087087  6105 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:46.080828  6172 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:46.096426  6298 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.160s
I20260709 12:22:46.133999 27736 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:46.187644  6295 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:22:46.187469  6199 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:22:46.188010  6229 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:46.189693 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:46.191718 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 103690us
I20260709 12:22:46.176731 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.177794 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.197757 27737 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:46.145946 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.198992 27731 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 36
W20260709 12:22:46.138484 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44315.5us
I20260709 12:22:46.191448  6322 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:46.195433 27655 raft_consensus.cc:2368] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33.
W20260709 12:22:46.200909  6303 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:22:46.146618 27738 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.141512 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:46.205642 27480 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 20 to 22, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 22 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:46.144224  6310 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 12:22:46.148524  6189 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.572s	user 0.011s	sys 0.000s
W20260709 12:22:46.146384  6309 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:22:46.137790  6272 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.586s	user 0.008s	sys 0.003s
W20260709 12:22:46.145637  6315 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 12:22:46.212795 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76869.3us
I20260709 12:22:46.131804  5803 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.214560 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.192801  6105 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.223675 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:46.230348 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.227958 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:46.238137 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61495us
I20260709 12:22:46.230891 27586 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:22:46.233888 28329 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:22:46.274856 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:46.275627  6304 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 12:22:46.276028 27655 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:46.275684 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53718us
I20260709 12:22:46.205430  6322 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:46.143249  6227 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.078s
W20260709 12:22:46.288373 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58497.1us
I20260709 12:22:46.230527 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.284929 27480 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 16 to 18, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 18 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:22:46.287873 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:46.228636  6317 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
W20260709 12:22:46.277208  6295 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:46.296453 27702 leader_election.cc:400] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33.
I20260709 12:22:46.314883 27702 leader_election.cc:403] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:22:46.296301 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:46.315652 28335 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:46.215956  6229 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.298065 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.298126 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.232749  6269 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:46.320398 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.321043 27664 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:46.218065  5927 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:46.221889  6270 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:46.230003  6105 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:46.317646 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.325205 27657 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:22:46.290406  5405 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:46.227975  5528 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.086s
I20260709 12:22:46.310456 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:46.294677  6189 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.280162  6199 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:22:46.324012 27737 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:46.329525 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.319053 28332 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:46.336298 27731 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:46.307572  6227 consensus_queue.cc:1243] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 31.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:22:46.340610 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.326954 27480 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 27 to 30, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 30 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:46.340775 28336 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
W20260709 12:22:46.342455 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67718.9us
I20260709 12:22:46.343369 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.345885 28334 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:22:46.347846 27736 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:46.348811 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.350162 27731 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:46.349653 27588 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:46.351584 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.352149 27737 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:46.329209  5991 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:46.352404  5405 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.360984  5405 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.353343  6270 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:46.352942  6096 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.894s	user 0.009s	sys 0.004s
I20260709 12:22:46.369313 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:46.354209  6321 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:22:46.373095 27736 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:22:46.375049  6320 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 12:22:46.376575 27736 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.357905  6227 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:46.380738 27731 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:46.380090 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.331920  5927 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:22:46.383955  6229 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:46.386296 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103623us
I20260709 12:22:46.387390 27704 leader_election.cc:400] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:46.387671 27704 leader_election.cc:403] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:46.375335  6306 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.069s
I20260709 12:22:46.390108  5528 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:46.392731 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:46.395383 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:46.396430 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:46.401938  6124 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:46.402508  6124 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:46.378059  5991 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:46.404291  5991 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:46.407869  6311 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
W20260709 12:22:46.409765  6316 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
W20260709 12:22:46.411502  6314 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:22:46.412195 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.412832 27730 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 23.
I20260709 12:22:46.413798 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.414369 27730 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:46.414932 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.416865 27589 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
W20260709 12:22:46.419845  6326 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.131s	user 0.000s	sys 0.002s
I20260709 12:22:46.375646  6272 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:46.427538 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.428810 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.429335 27659 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:46.430666 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.431958 27659 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:46.432534 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.435727  6322 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.438225 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.475682 27664 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:46.475186 28335 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:46.438891 27737 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:46.363226  6270 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.327565  6172 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:46.440899 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.014s	sys 0.029s
I20260709 12:22:46.445739 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4612) took 1028 ms. Trace:
I20260709 12:22:46.488324 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1922) took 1045 ms. Trace:
I20260709 12:22:46.345410  6189 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:46.449199  6188 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.640s	user 0.008s	sys 0.003s
I20260709 12:22:46.449748 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.495775 27658 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 37 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
W20260709 12:22:46.453337 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.456842 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.458717  5528 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:22:46.377877  6184 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.971s	user 0.008s	sys 0.004s
I20260709 12:22:46.464468  6148 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:46.465396 28292 leader_election.cc:336] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Timed out: RequestConsensusVote RPC to 127.25.202.3:41211 timed out after 2.574s (SENT)
I20260709 12:22:46.468478  6241 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:46.469306  6191 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:46.416618 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.415140 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.474634 27655 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:46.479995  6272 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:46.482359 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.482975 27584 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:46.532841 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:46.533540 27584 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:46.534540 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:46.484575  6306 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:22:46.535647  6306 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:46.475019  6220 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:46.488322 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.014s	sys 0.029s
I20260709 12:22:46.488559 27583 rpcz_store.cc:276] 0709 12:22:45.417699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.447959 (+ 30260us) service_pool.cc:224] Handling call
0709 12:22:45.463780 (+ 15821us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.678863 (+215083us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:45.859037 (+180174us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.052658 (+193621us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.130035 (+ 77377us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.133335 (+  3300us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:46.135801 (+  2466us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.135975 (+   174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.136048 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.445438 (+309390us) raft_consensus.cc:1685] finished
0709 12:22:46.445472 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.445656 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17044,"prepare.queue_time_us":27229,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":91,"raft.queue_time_us":22755,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":18768,"spinlock_wait_cycles":12095360,"thread_start_us":500662,"threads_started":3,"wal-append.queue_time_us":220324}
I20260709 12:22:46.488912 27582 rpcz_store.cc:276] 0709 12:22:45.442690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.442829 (+   139us) service_pool.cc:224] Handling call
0709 12:22:45.443089 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.836615 (+393526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.483196 (+646581us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:46.488272 (+  5076us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":903568640}
I20260709 12:22:46.499017 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:46.511682 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 222661us
W20260709 12:22:46.485855  6302 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.438s	user 0.000s	sys 0.003s
I20260709 12:22:46.535235 27584 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:46.539541  6190 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.539834  5528 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:46.530721  6191 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:46.540820 28331 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:22:46.556732 27587 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:46.541013  6272 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.601374 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4636) took 1081 ms. Trace:
I20260709 12:22:46.502816 27736 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.547736 27734 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:46.541899 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:46.548287 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.562700  6191 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.565455  6227 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33.
I20260709 12:22:46.566793  6011 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:46.567677 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102327us
I20260709 12:22:46.571563 27585 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 20. (term mismatch)
I20260709 12:22:46.573228  6124 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.576248 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.577437 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.578464 27738 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:46.519249  6241 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:46.580837 27583 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
W20260709 12:22:46.581496 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.639930 27736 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:46.640574 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:46.582701  6148 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.582620  6156 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:46.520843  6188 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.585731 27589 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [31.18-33.19]
I20260709 12:22:46.527539  6184 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.527390  6271 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:22:46.586845  6220 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:22:46.591919 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.668058 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.560104  5528 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.611783 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.601933 27581 rpcz_store.cc:276] 0709 12:22:45.519907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.570613 (+ 50706us) service_pool.cc:224] Handling call
0709 12:22:45.571130 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.619907 (+ 48777us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:22:45.662100 (+ 42193us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.544700 (+882600us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.561042 (+ 16342us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.561237 (+   195us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:46.561286 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.561373 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.601100 (+ 39727us) raft_consensus.cc:1685] finished
0709 12:22:46.601128 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.601303 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":779,"prepare.queue_time_us":41406,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":3739,"raft.queue_time_us":35778,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":27782,"thread_start_us":845554,"threads_started":1,"wal-append.queue_time_us":599026}
I20260709 12:22:46.617492 27737 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:46.677747 27737 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.669523 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:46.622687 28335 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [28.19-30.20]
I20260709 12:22:46.632129 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.627463 27585 pending_rounds.cc:85] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 19
I20260709 12:22:46.634521 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.635414 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:46.636229 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.637059 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.638019 27733 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:22:46.644411 28330 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:22:46.649945  6300 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.864s	user 0.000s	sys 0.004s
I20260709 12:22:46.650867  6238 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:22:46.625569  6244 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.563s	user 0.008s	sys 0.002s
I20260709 12:22:46.664115 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.664201 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.687542 27664 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:46.664909 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.687615 27582 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:46.666082 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.676084 27738 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:22:46.632091  6011 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:46.680903 28329 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:46.681718 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:46.682109 27659 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:46.685849 27581 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:46.689465 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:46.689751 27738 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 16
I20260709 12:22:46.688186 27730 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:46.660221  6191 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:46.695277 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1885) took 1321 ms. Trace:
I20260709 12:22:46.692467 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.692925 27661 raft_consensus.cc:2410] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:22:46.693323 28332 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:46.680366 27585 pending_rounds.cc:107] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 36.20
I20260709 12:22:46.696645 27585 raft_consensus.cc:2889] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:46.695150 27729 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:22:46.697096 27583 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: [31.17-32.18]
I20260709 12:22:46.690284 27738 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:22:46.689003 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:46.697683 27738 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:46.698364 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 318322us
I20260709 12:22:46.698908 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:46.702997 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.695822 27656 rpcz_store.cc:276] 0709 12:22:45.373619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.373726 (+   107us) service_pool.cc:224] Handling call
0709 12:22:45.391204 (+ 17478us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:45.395140 (+  3936us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.395184 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:45.411822 (+ 16638us) log.cc:844] Serialized 52 byte log entry
0709 12:22:45.439799 (+ 27977us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18001fb400. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:45.661041 (+221242us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:45.661097 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.661192 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.485449 (+824257us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b24002ac128. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:22:46.695066 (+209617us) raft_consensus.cc:1685] finished
0709 12:22:46.695086 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.695222 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":442,"raft.queue_time_us":6483,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5382,"spinlock_wait_cycles":66819200,"thread_start_us":221044,"threads_started":1,"wal-append.queue_time_us":585339}
I20260709 12:22:46.706784 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.704938 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.708096 27738 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:46.706255 27735 raft_consensus.cc:2376] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39. Re-sending same reply.
I20260709 12:22:46.711179 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.714054 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:46.716681 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.717463 27731 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:46.712242 27733 raft_consensus.cc:2376] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31. Re-sending same reply.
I20260709 12:22:46.718714 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:46.708421 27737 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.719552 27664 raft_consensus.cc:2410] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 31 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:22:46.719569 27663 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:46.723242 27736 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
I20260709 12:22:46.692994  6244 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:22:46.730767 27660 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:22:46.731513 27659 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 24. Current term is 25. Ops: []
I20260709 12:22:46.727281 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.723732 27736 pending_rounds.cc:85] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 19
I20260709 12:22:46.743296 27736 pending_rounds.cc:107] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:22:46.743644 27736 raft_consensus.cc:2889] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:46.664240  6190 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:22:46.733567 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.727567 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:46.708056  6332 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.741889 27737 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:46.742017 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:46.735380 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:46.750180 27731 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: [32.18-32.18]
I20260709 12:22:46.746093 27735 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:46.719988  6238 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:46.733671  6133 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.636s	user 0.011s	sys 0.000s
I20260709 12:22:46.755302 28332 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
W20260709 12:22:46.755910 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50873.5us
I20260709 12:22:46.759140  6332 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.761965  6332 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:46.762143 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4aa1523b9af84c3db99ba394e2b9aa9a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:46.763897 27736 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:22:46.766522 27660 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:22:46.767035 28337 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:46.757215 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.772373 27737 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:22:46.773074 27663 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:46.774394 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.740034  6104 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:46.740229  6242 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.782032 27735 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:22:46.745124  6272 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:46.782541  6184 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:46.786711 27655 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:46.786686 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:46.789443  6270 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:46.790076  6270 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.790627  6270 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:46.804247 27661 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.20-35.20]
I20260709 12:22:46.811425 27663 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:46.811810 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:46.814427 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.782708  6124 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:46.783880  6016 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:46.783443  6104 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.819612 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.819854 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.723838  6306 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.804045  6140 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:22:46.823197 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 80543.1us
I20260709 12:22:46.773940  6133 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.833778 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.834425 27660 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:46.836387 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:46.836490 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 209237us
I20260709 12:22:46.823594  5991 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:46.843029 27729 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:46.805877  6242 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.837612 27735 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:46.756158  5528 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.827091  6104 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:22:46.847672 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.854380 27729 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:46.831451  6272 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:46.863276 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:46.864033 27585 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:46.869015 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.878960 27735 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:46.818807  6331 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.194s
I20260709 12:22:46.824456  6105 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.886512 27660 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:46.786782  6241 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:46.887383 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.866489  6188 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.869604  6272 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:46.902679 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:46.920188 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.894497  6343 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:22:46.892369  6331 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 33.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:46.893784 27663 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:22:46.889779  5528 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:46.918006 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:46.933308 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:46.927981 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:46.902052  6272 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:46.936190 28332 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:46.940183 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.922641  6348 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:46.940856 27735 raft_consensus.cc:2376] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26. Re-sending same reply.
I20260709 12:22:46.944386 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:46.973184 27731 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:22:46.946835 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:46.847241  6238 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:46.948623  6324 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 12:22:46.954691 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:46.941596  6199 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.960124  6133 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.923780  6196 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:22:46.944789  6156 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:46.961118 27735 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:46.999939 27735 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:46.964932  6348 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:46.967283  6105 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:46.968338 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:47.032090 27588 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:47.032912 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4627) took 1258 ms. Trace:
I20260709 12:22:46.989487 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:46.999704  6156 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:47.026422  6345 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:22:46.976377  6229 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:22:46.987218 27585 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:47.033385 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.033209 27588 rpcz_store.cc:276] 0709 12:22:45.774562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.351947 (+577385us) service_pool.cc:224] Handling call
0709 12:22:46.352429 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.027550 (+675121us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54000297a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:22:47.032622 (+  5072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.032857 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":278302080}
W20260709 12:22:47.034157 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44738.3us
W20260709 12:22:46.997282  6336 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
W20260709 12:22:47.037545 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 218017us
I20260709 12:22:47.040306 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:46.973131  6104 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.000523  6005 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:22:47.043008 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.042222 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.045841 27735 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.030810  6240 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.048055 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:47.037040  6156 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:47.037040  6133 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:47.050931 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.061638 27735 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:47.043910  6105 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.044062  6140 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:47.044435  6005 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:47.000089  6199 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.067253  6011 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.068050  6005 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:47.052367  6240 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.069813  6011 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.070508  6011 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.071571 27658 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:47.071661  6240 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.071907  6348 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.062428  6242 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.073694 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:47.074548 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.050561  6271 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:47.076529 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.044667  6270 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.063369  6016 raft_consensus.cc:2764] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:22:47.070972  6011 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.072536  6105 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.072263  6199 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:47.085516  6140 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:47.085955  6306 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:22:47.105702 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56582.9us
I20260709 12:22:47.109203 28333 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:47.109225 27659 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:47.119632 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:47.120623 27581 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:47.101958  6306 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:47.104411  6133 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:47.124960  6306 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:47.132453  6104 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.151742  6011 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:47.152211  6197 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 31 current term: 31 is higher.
I20260709 12:22:47.156100 27581 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
W20260709 12:22:47.156021  6357 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:22:47.160833 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.169723 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
I20260709 12:22:47.128019  6133 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.129321  6140 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.173486  6105 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:47.179147  6360 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:22:47.159142  6369 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:22:47.183420  6271 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:47.184445  6271 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.186398  6005 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.188645 27660 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:47.158471  6272 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.189955 27660 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:47.196355 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.196688 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:47.190801  6272 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:47.200992 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61570.6us
I20260709 12:22:47.201565 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:47.176167  6270 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.207440 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:47.208650 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:47.213728  6374 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:47.218032  6156 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:47.222286 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 172507us
I20260709 12:22:47.223716 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.244625  6156 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.246654  6005 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:47.248675 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.237077 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:47.256155 27735 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:22:47.261979  6375 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:47.262358 27581 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [32.20-36.22]
I20260709 12:22:47.262864  6375 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:47.267382 27588 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:47.273051 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:47.275400 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63284.5us
I20260709 12:22:47.263593  6375 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:47.283094 27737 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:22:47.285437 27735 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:22:47.283833 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.270857  6270 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:47.289311 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.290102  6270 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:22:47.307582  6368 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:22:47.307952  6372 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:22:47.287082  6272 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:22:47.311093  6270 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:47.293361  6374 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:22:47.323979  6364 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.015s
I20260709 12:22:47.327873  6372 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.328512  6372 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.331748 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4631) took 1521 ms. Trace:
I20260709 12:22:47.332235 27587 rpcz_store.cc:276] 0709 12:22:45.809852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.415950 (+606098us) service_pool.cc:224] Handling call
0709 12:22:46.416344 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.612061 (+195717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.331490 (+719429us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54001883d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:47.331685 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307154176}
I20260709 12:22:47.333839  6374 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:47.334295  6374 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:47.332026  6156 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:47.336367  6223 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:47.336459  6156 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:47.336853  6223 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:47.337503  6156 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.337741  6223 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.338690  6223 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:47.338776  6156 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:47.331058  6372 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.333448  6306 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.345474  6306 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.347865 27588 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:47.347124 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:47.353034 27585 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:47.343348  6374 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.363708  6374 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:47.369028  6133 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.369724  6133 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.375617  6379 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.411592  6148 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.411971  6379 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.414561  6189 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.415307  6379 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:47.412228  6148 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:47.416182  6227 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 31 current term: 31 is higher.
I20260709 12:22:47.417115 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4459) took 1646 ms. Trace:
I20260709 12:22:47.417763  6148 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.420897  6270 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.421172  6375 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.417479 27661 rpcz_store.cc:276] 0709 12:22:45.771059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.710412 (+939353us) service_pool.cc:224] Handling call
0709 12:22:46.787706 (+ 77294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.809929 (+ 22223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.416910 (+606981us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b540022a450. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:47.417046 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136562048}
I20260709 12:22:47.428149  6133 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:47.431600  6341 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.647s	user 0.004s	sys 0.000s
I20260709 12:22:47.437968 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4466) took 1390 ms. Trace:
I20260709 12:22:47.438210  6270 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.445115 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.445797 27584 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 24.
I20260709 12:22:47.439194 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3708) took 1210 ms. Trace:
I20260709 12:22:47.447983 27655 rpcz_store.cc:276] 0709 12:22:46.228956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.809003 (+580047us) service_pool.cc:224] Handling call
0709 12:22:46.809434 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.809740 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:22:46.899197 (+ 89457us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.438694 (+539497us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.438852 (+   158us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:47.438909 (+    57us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:47.438964 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.439031 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.439142 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":298888,"prepare.queue_time_us":56856,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":299497,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":2578,"spinlock_wait_cycles":28825344,"thread_start_us":315323,"threads_started":2,"wal-append.queue_time_us":276522}
W20260709 12:22:47.449625 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 503092us
I20260709 12:22:47.439461  6298 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:22:47.439709 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.450208  6298 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:47.450433 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:47.450991 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.451924 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:47.456009 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:47.457026 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:47.457113 27661 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 12:22:47.473680 27661 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 13
I20260709 12:22:47.473879 27661 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:22:47.474112 27661 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:47.457463 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:47.475685 27663 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:47.461913 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:47.458886 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.463078 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4472) took 1316 ms. Trace:
W20260709 12:22:47.441524 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.452260 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4621) took 1748 ms. Trace:
I20260709 12:22:47.498919 27580 rpcz_store.cc:276] 0709 12:22:45.703590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.734761 (+ 31171us) service_pool.cc:224] Handling call
0709 12:22:46.080765 (+346004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.089962 (+  9197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.452086 (+1362124us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b540006f750. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:47.452208 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":782780288}
I20260709 12:22:47.451138  6223 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:47.457950 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 205218us
I20260709 12:22:47.507012 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:47.493196 27659 rpcz_store.cc:276] 0709 12:22:46.146298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.732677 (+586379us) service_pool.cc:224] Handling call
0709 12:22:46.749712 (+ 17035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.118383 (+368671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.462862 (+344479us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:47.463022 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826121600}
W20260709 12:22:47.510010 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 58140.1us
I20260709 12:22:47.515210 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1898) took 1080 ms. Trace:
I20260709 12:22:47.515713 27658 rpcz_store.cc:276] 0709 12:22:46.434448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.034873 (+600425us) service_pool.cc:224] Handling call
0709 12:22:47.044530 (+  9657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.072080 (+ 27550us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.514960 (+442880us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:47.515142 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61744512}
I20260709 12:22:47.469060 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:47.517969 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49018.7us
I20260709 12:22:47.513199 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:47.489022 27585 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 30 because replica has last-logged OpId of term: 28 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
I20260709 12:22:47.522156 27586 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:47.523278 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:47.524188 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
W20260709 12:22:47.524242 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.461591  6370 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.283s	user 0.000s	sys 0.003s
I20260709 12:22:47.439704  5884 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.455502  6332 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:47.441093  6096 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.536024 27587 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:47.439782  6323 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:47.475188  6375 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.514938  6331 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:47.537284 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:47.441660  5803 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:47.540611 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.547660 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:47.553249 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.455654  5967 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.086s
I20260709 12:22:47.459040  6197 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.082s
I20260709 12:22:47.557227 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:47.533587 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 74773.7us
I20260709 12:22:47.559504 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.516568  6227 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:47.548602 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:47.552078 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:47.561066 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:47.460517  6104 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.080s
W20260709 12:22:47.543339 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.538859  6229 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:47.560182 27730 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:47.561908 27659 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:47.564549 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:47.565078 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.521626  5528 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:22:47.536036  6380 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:47.552547  6011 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:47.451280  6298 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.543877  6323 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:47.577416 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.581848 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.581943 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:47.582346 27733 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:47.578495 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:47.581220 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.584695 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.585500 27736 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:47.587263 27737 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:47.580299 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.589524 27734 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:47.590767 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:47.592001  6197 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:47.438401 27656 rpcz_store.cc:276] 0709 12:22:46.047135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.714121 (+666986us) service_pool.cc:224] Handling call
0709 12:22:46.740438 (+ 26317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.744209 (+  3771us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:22:47.045394 (+301185us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.126775 (+ 81381us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.436949 (+310174us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b1800262f00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:47.437386 (+   437us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.437556 (+   170us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:47.437604 (+    48us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:47.437647 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.437715 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.437906 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":335095,"prepare.queue_time_us":80043,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":2034,"raft.queue_time_us":312019,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":1154,"spinlock_wait_cycles":2959232,"thread_start_us":308834,"threads_started":2,"wal-append.queue_time_us":136199}
I20260709 12:22:47.595409 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.595564  6238 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:47.597518 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.597833 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:47.592480  6197 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:47.589359  6332 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:47.562553  6242 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:47.541616  5884 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:47.604836 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 323897us
I20260709 12:22:47.562973  6005 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:47.610661 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.566728  6229 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.609653 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:47.571264  6380 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.549872  6096 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.577204  6244 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:47.581557  6240 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:47.604358  6223 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:22:47.601457  6241 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:47.613014  6331 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:47.606161  6332 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:47.559517  5803 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.611475 27731 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:47.622074  5991 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:47.615448  6229 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:47.619179  6104 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:47.620952  6380 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:47.630507 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.616882  6298 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:47.622936  6348 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:47.621999  6242 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:47.684645 27664 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:47.639562 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:47.653290 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.639312 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.689105 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:47.688499 27731 raft_consensus.cc:2376] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32. Re-sending same reply.
I20260709 12:22:47.697013 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.692461 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
W20260709 12:22:47.700785 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 103031us
I20260709 12:22:47.697682 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:47.626221  6244 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.701821 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:47.709014 28333 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:47.695554  6157 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:22:47.709622 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.639766  6148 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.744012  6148 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:47.746482 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.710425 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 112983us
W20260709 12:22:47.747023 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 107554us
I20260709 12:22:47.748631 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:47.693390  6306 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.721961  5803 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.724428  6244 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.724969  6105 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:47.727548 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.728477 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:47.762507 27735 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:47.729346 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:47.608700  6359 consensus_queue.cc:1175] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3f90188458da46f6ba138d5f82c55b69" responder_term: 30 status { last_received { term: 29 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:47.731977  6157 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.693655  6196 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:22:47.739464  5884 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:47.739941  6242 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.749871 27584 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:47.751412 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.752789 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:47.701691  6348 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:47.701262  5991 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:47.763576 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:47.764729 27585 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 23. Current term is 24. Ops: []
I20260709 12:22:47.632895  6323 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.735709 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:47.635145  6197 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:47.785010 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 194335us
I20260709 12:22:47.790939 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:47.771178  6360 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:47.808866 27735 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:22:47.761775  6238 raft_consensus.cc:2764] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:22:47.762065  6105 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.760172  6332 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.841290 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:47.864208 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4470) took 1773 ms. Trace:
I20260709 12:22:47.864739 27733 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
W20260709 12:22:47.873083 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109593us
I20260709 12:22:47.872373 27664 rpcz_store.cc:276] 0709 12:22:46.090436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.726825 (+636389us) service_pool.cc:224] Handling call
0709 12:22:46.745364 (+ 18539us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.687792 (+942428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.863889 (+176097us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b5400343854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:47.864123 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2159406464}
I20260709 12:22:47.770521  6157 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:47.855786 27734 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:47.770866  6331 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:47.762509  6387 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:47.889812 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:47.889928 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82180us
W20260709 12:22:47.901995 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 166410us
W20260709 12:22:47.937740 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 146903us
W20260709 12:22:47.925047 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.874876  6105 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:47.899513  6381 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:47.828812  6096 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.943284 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:47.948383 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:47.942057 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52316.1us
I20260709 12:22:47.949978  6229 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.888348  6375 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:47.899633  6223 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:22:47.811113  6104 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.131s
W20260709 12:22:47.953837 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82261us
I20260709 12:22:47.995004 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:47.954945  5803 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:47.763593  6306 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:47.946501  6372 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:22:47.987576 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:47.991959 27661 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:22:47.996608 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50269.7us
I20260709 12:22:48.010069 27737 raft_consensus.cc:2368] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35.
I20260709 12:22:47.998826 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:47.965188 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:47.990864  6197 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:48.016626 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77877.6us
I20260709 12:22:47.944482  6241 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.058s
I20260709 12:22:47.994882  6133 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:47.868431  6348 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:47.994302  6331 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:47.999703  6375 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:48.023782 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58702us
I20260709 12:22:48.028467 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:47.999409  6374 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:48.028688 27735 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:22:48.043450 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:48.044075 27735 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:48.029227 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:48.030224 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:48.045178 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:48.032047 27582 raft_consensus.cc:2410] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:22:47.869158  5991 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:48.034889 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.031589  6104 consensus_queue.cc:1243] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 32.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:22:48.036422 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:48.031826  6244 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:48.040983  6229 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:48.044318  6372 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:22:47.999969  6381 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:48.046307 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:48.048254 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.049822 27729 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:48.051198 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:48.020255  6148 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:47.994012  6387 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:22:48.057298 27734 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:22:48.044916 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:48.032047  6005 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:22:48.066286 27738 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:22:48.069561 27660 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 21. (term mismatch)
I20260709 12:22:48.089320 27660 pending_rounds.cc:85] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 20
I20260709 12:22:48.089663 27660 pending_rounds.cc:107] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 31.21
I20260709 12:22:48.089910 27660 raft_consensus.cc:2889] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:48.055810  6223 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.092451  6223 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.094740  6223 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:48.095376  5884 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:48.097018  6005 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:22:48.097688  6005 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:22:48.072614 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56033.9us
W20260709 12:22:48.101780 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.103447  6244 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:48.104513 27659 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:22:48.105930  6244 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:48.106487  6244 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:48.106689  6403 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:22:48.108156 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:48.109210 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.109290 27658 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:48.073026 27729 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:48.073885 27663 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:22:48.075285 27735 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:48.084966 27661 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
W20260709 12:22:48.087436 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66202.1us
I20260709 12:22:48.116243 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:48.098796  6005 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.118924 27701 leader_election.cc:400] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33.
I20260709 12:22:48.053704  6133 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:48.070183 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:48.121413 27664 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:22:48.122884 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:48.127978 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:48.059172  6172 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.026s	sys 0.020s
I20260709 12:22:48.131856 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:48.133644 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:48.119645  6220 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:48.138315 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:48.054154  6374 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.140262  6157 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.177775 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:48.133520  6172 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.026s	sys 0.020s
I20260709 12:22:48.140609  6229 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:48.146771  6223 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:22:48.149287 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112961us
I20260709 12:22:48.057850  6380 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.137941  5528 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:48.240226  6380 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:48.152398  6323 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:48.157382  6016 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:48.243674  6016 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.158667 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:48.159430 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:48.169960 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.246708 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.246986 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 87689.9us
I20260709 12:22:48.247876 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:48.248060  5528 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:48.141610  6133 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.249250 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:48.202298 27588 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:22:48.202488 27584 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:48.126995  6399 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:22:48.150045  6389 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:22:48.280759 27585 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:48.208946  6157 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:48.209049 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:48.182541  6244 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.213389  6381 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.216567  6223 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:48.218871  6306 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:48.220397 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:48.203259 27735 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
W20260709 12:22:48.232105 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.170966  6379 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:48.210242 27580 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:22:48.338307 27580 pending_rounds.cc:85] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 17
I20260709 12:22:48.338696 27580 pending_rounds.cc:107] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:22:48.338986 27580 raft_consensus.cc:2889] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:48.283502 28328 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 24. Current term is 25. Ops: [24.19-24.19]
I20260709 12:22:48.197135  6331 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.345705 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:48.269852  6399 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:48.346693 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:48.349849 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 140858us
I20260709 12:22:48.367741 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:48.289376  6199 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:48.276134  6104 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:22:48.304417  6223 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:48.382084 28336 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:22:48.379073 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:48.276650  6220 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.075s
W20260709 12:22:48.396250 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.337575  6379 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:48.424654 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:48.419143 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:48.284636  6229 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.097s
I20260709 12:22:48.423203  6140 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.007s
I20260709 12:22:48.430415  6199 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:48.431607  6140 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:48.432520  6140 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:48.436429 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:48.439558  6374 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:48.203184  6375 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:48.439949 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65782.4us
I20260709 12:22:48.440959 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:48.442360  6016 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:48.443216  6016 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:48.443681  6016 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.444099  6016 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:48.439723  6140 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:22:48.447911  6406 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260709 12:22:48.448788  6141 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:48.450332  6399 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:48.451604 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:22:48.451720  6399 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:48.377300  6390 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.077s
I20260709 12:22:48.454448  6379 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.369768  6331 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.439628  6306 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:48.461095  6197 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:48.463706  6387 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:22:48.478974  6379 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:48.509020 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1908) took 1036 ms. Trace:
I20260709 12:22:48.511011  6242 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:48.512234  6242 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:22:48.513721 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 145217us
I20260709 12:22:48.514016  6104 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:48.514919  5803 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:22:48.515352  5803 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:48.516089  6402 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:22:48.516508  6402 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:48.517586  6402 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:48.519738 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 232443us
I20260709 12:22:48.520967  6323 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:48.521673 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:48.523253 27659 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:48.509656 27662 rpcz_store.cc:276] 0709 12:22:47.473057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.690622 (+217565us) service_pool.cc:224] Handling call
0709 12:22:47.708180 (+ 17558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.708490 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:22:47.810251 (+101761us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.283851 (+473600us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.508076 (+224225us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b18004e19c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:48.508307 (+   231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:48.508394 (+    87us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:22:48.508470 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.508572 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.508937 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26156,"prepare.queue_time_us":59559,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":109,"raft.queue_time_us":1336,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":5031,"spinlock_wait_cycles":3789696,"thread_start_us":518195,"threads_started":2}
I20260709 12:22:48.527886  6188 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:48.528702  6188 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.525863  6189 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:48.526060  6401 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:48.525187 27628 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:48.532867  6401 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.526316  5884 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.536126  5884 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:48.457273  6105 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.001s
I20260709 12:22:48.522413  6096 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.002s
I20260709 12:22:48.456250  6229 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.547492 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.504092  5528 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:48.543865 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:48.555238 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 114384us
I20260709 12:22:48.449268  6140 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
I20260709 12:22:48.557996  6011 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:48.554278  6241 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:48.545967  6189 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:22:48.608680 27554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.042s	sys 0.106s
I20260709 12:22:48.565783  6011 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:48.609176 27554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.042s	sys 0.107s
W20260709 12:22:48.608951 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 360265us
I20260709 12:22:48.542080  6105 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.615774  6011 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.616904  6105 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:48.612370 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:48.610584 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.006s	sys 0.004s
W20260709 12:22:48.618954 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.006s	sys 0.004s
W20260709 12:22:48.619899 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 573621us
W20260709 12:22:48.610675 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.004s	sys 0.005s
W20260709 12:22:48.622210 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.005s	sys 0.005s
W20260709 12:22:48.625945 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.606762  5528 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:22:48.610354  5991 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.007s	sys 0.004s
I20260709 12:22:48.617403  6011 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:48.613859  6270 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.008s	sys 0.001s
I20260709 12:22:48.608338  5967 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.557253  6229 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.634014 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:48.634898 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.610538  6401 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:48.678289 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:48.678783 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:48.678833 27729 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:22:48.679694 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:48.678973 27580 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:48.612826  6192 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.534855  6223 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.686214 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:48.686554  6331 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:48.623744  6105 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:22:48.687659 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.691846 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:48.632525  5991 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.008s	sys 0.004s
I20260709 12:22:48.713225  6229 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:48.713511  6401 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:48.713541  6223 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:48.717692  6223 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:48.718312  6223 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.719832  6189 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:22:48.721689 27580 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 25.
W20260709 12:22:48.676106  6270 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.008s	sys 0.002s
I20260709 12:22:48.631666  5528 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:48.633088  6220 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 33.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:48.633124  6390 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.637228  6271 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:48.715016  6229 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.015s
I20260709 12:22:48.734359 28330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "723b2024e4414796ba50ece654596156"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:22:48.732703 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:48.734792 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:48.739786 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:48.741916 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:48.743697 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:48.767082 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:48.786405 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:48.762676 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:48.796950 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.789755  6105 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:22:48.877463 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.012s	sys 0.044s
W20260709 12:22:48.882750 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.012s	sys 0.044s
W20260709 12:22:48.881403 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 442593us
I20260709 12:22:48.683878  5967 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:48.884949 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 336594us
W20260709 12:22:48.894115 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.886005 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 636790us
W20260709 12:22:48.886941 27732 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.016s	sys 0.043s
I20260709 12:22:48.889196 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:48.895229 27732 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.017s	sys 0.043s
I20260709 12:22:48.896937 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1735) took 1186 ms. Trace:
I20260709 12:22:48.791301  6401 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:48.730443  6189 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.154s
I20260709 12:22:48.790017  6104 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:48.790174  6241 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:48.945607  6197 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:48.945873  6241 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.946179  6197 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:48.946514  6241 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:48.884450  6105 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:48.903435 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:48.887085  6335 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.685633 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.914034 28333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s	user 0.058s	sys 0.067s
W20260709 12:22:48.959079 28333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.059s	sys 0.067s
I20260709 12:22:48.920895 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:48.909694  6401 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.909898  6375 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260709 12:22:48.932322  5528 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:48.936087  6424 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
I20260709 12:22:48.943991  6425 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:48.996814  6425 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:48.997997  6425 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:48.944561  6426 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:49.000881  6426 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.945045  6374 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:48.897398 27732 rpcz_store.cc:276] 0709 12:22:47.710170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.717587 (+  7417us) service_pool.cc:224] Handling call
0709 12:22:47.739867 (+ 22280us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:47.740398 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:47.740442 (+    44us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:47.741347 (+   905us) log.cc:844] Serialized 78 byte log entry
0709 12:22:48.025932 (+284585us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:48.347982 (+322050us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.895614 (+547632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.896569 (+   955us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.896646 (+    77us) raft_consensus.cc:1685] finished
0709 12:22:48.896665 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.896865 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":109648,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":142428,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":213,"thread_start_us":1141880,"threads_started":3,"wal-append.queue_time_us":470645}
I20260709 12:22:49.005798  6416 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:49.006398  6416 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:49.007593  6404 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.947s	user 0.014s	sys 0.005s
I20260709 12:22:49.007308  6416 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.008319  6244 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:49.008850 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1737) took 1289 ms. Trace:
I20260709 12:22:49.009222 27736 rpcz_store.cc:276] 0709 12:22:47.719963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.723514 (+  3551us) service_pool.cc:224] Handling call
0709 12:22:47.723982 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:47.724436 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:47.724478 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:47.725075 (+   597us) log.cc:844] Serialized 26 byte log entry
0709 12:22:47.807162 (+ 82087us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:47.807216 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.807293 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.006989 (+1199696us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:49.008573 (+  1584us) raft_consensus.cc:1685] finished
0709 12:22:49.008595 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.008790 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6791,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":260675456,"thread_start_us":81625,"threads_started":1,"wal-append.queue_time_us":269106}
I20260709 12:22:49.010687  6196 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:48.907522 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:48.725525  6192 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.728751  6331 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.155s
I20260709 12:22:48.887264  6415 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.965432 28292 leader_election.cc:400] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35.
I20260709 12:22:48.951576 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:49.101053 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 149539us
I20260709 12:22:48.732156  6390 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:48.972587 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69345.4us
I20260709 12:22:48.754263  6271 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:48.916013  6323 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:48.995858  5528 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:48.968732  6401 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:49.090147 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 79555.2us
I20260709 12:22:48.947271  6197 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:49.117578 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152329us
W20260709 12:22:49.132196 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.091111  6428 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.132532 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.133225 28331 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
W20260709 12:22:49.136986 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.956442  6105 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:48.970222  6335 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:49.139642 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.141175 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:49.126926  6425 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:49.117200  6374 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:49.118175  6223 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.128417  6415 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.127754  6323 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.135468  6229 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.167680  6229 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.168279  6323 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:49.170102  6229 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.170748  6229 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:49.146600  6298 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
I20260709 12:22:49.151718  6375 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.023s
I20260709 12:22:49.131222  6364 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.162537  6197 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:49.175966  5803 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.149242  6412 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:22:49.197597 27586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.006s	sys 0.070s
W20260709 12:22:49.203061 27586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.006s	sys 0.070s
W20260709 12:22:49.151299  6016 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.707s	user 0.008s	sys 0.000s
I20260709 12:22:49.177219  6306 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:49.184175  6229 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:49.212543 28337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.002s	sys 0.028s
W20260709 12:22:49.232753 28337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.003s	sys 0.030s
I20260709 12:22:49.206503 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1939) took 1033 ms. Trace:
I20260709 12:22:49.202965  6412 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:49.212256  6133 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.018s	sys 0.013s
I20260709 12:22:49.247803 27586 rpcz_store.cc:276] 0709 12:22:48.173039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.178472 (+  5433us) service_pool.cc:224] Handling call
0709 12:22:48.178861 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.179092 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:48.618726 (+439634us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.205918 (+587192us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.206164 (+   246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.206216 (+    52us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:49.206260 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.206323 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.206444 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8920,"prepare.queue_time_us":456356,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":517,"spinlock_wait_cycles":6494976,"thread_start_us":981668,"threads_started":2}
W20260709 12:22:49.268224 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 361182us
I20260709 12:22:49.177888  6223 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.208544  6306 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:22:49.265233  6005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.080s	sys 0.159s
I20260709 12:22:49.151777  6335 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.277426  6223 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:49.273164  6133 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.018s	sys 0.013s
I20260709 12:22:49.273085 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:49.264796  6220 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.345423 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.208125  6016 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.346261 27655 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:49.348146  6016 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.270560  5803 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:49.351744 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.352481 27661 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:49.354193 27663 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:22:49.230862  6364 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:49.357111 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 594491us
I20260709 12:22:49.357435  6364 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.268261  6412 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:49.357864 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:49.260138  6229 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.301394 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.259378  6346 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.342545 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.359558  6346 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.343577 27660 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:22:49.285364  5967 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.362339  6346 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:49.325500  6005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.184s	user 0.081s	sys 0.159s
I20260709 12:22:49.365901 27664 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:49.372432 27736 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:22:49.365841 27733 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:49.365639 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:49.365989 28330 raft_consensus.cc:606] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:49.367511 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94556.9us
I20260709 12:22:49.374553 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:49.368352  5528 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:49.376448 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:49.377019 27662 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:49.152268  6428 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.325073  6324 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.296523  6431 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.336831  6189 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.385771 28328 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:22:49.340781  6011 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:49.398483 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.406836  5528 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:22:49.408748 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50964.9us
W20260709 12:22:49.409046 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 283161us
I20260709 12:22:49.410020 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:49.393867  6192 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.384536  6346 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.415896 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:49.415151 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:49.394429  6390 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:49.417707  6430 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
I20260709 12:22:49.420905  6270 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:22:49.416834  6374 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:49.432608  6449 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:49.434214  6147 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:49.474918 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.476892  6332 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:49.476627  6412 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.477880 28336 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:49.434387 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:49.414145  6271 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.372016  6220 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.435518 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:49.415153  6385 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:49.494562  6385 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.394870  6324 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.455279  6428 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.497009  6324 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.399372  6431 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.410064  6011 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.404836  6189 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.420214  6346 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.438300  6432 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.431779  6016 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:49.365622  6306 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.428112  6437 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.484851  6229 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.422401  6439 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:49.486186 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:49.512844  6439 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:49.439759  6420 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:22:49.506937  6432 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.513844  6439 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.514405 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1601) took 1411 ms. Trace:
I20260709 12:22:49.420475  6364 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:22:49.491254  6148 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:22:49.491951  6441 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:22:49.492062  6444 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:49.448208  6223 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.493139  6220 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.493954 27732 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
W20260709 12:22:49.454151  6435 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:22:49.495057  6385 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:49.439924  5967 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:22:49.453940  6241 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.507s	user 0.007s	sys 0.002s
I20260709 12:22:49.499046  6431 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.506513 27736 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
W20260709 12:22:49.508297 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47445.6us
I20260709 12:22:49.510219  6011 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.510342  6189 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.510877  6306 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:49.511976  6346 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.447000  6446 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.066s
I20260709 12:22:49.518664  6432 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.545437 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.546209 27662 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:49.547348 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.548002 27662 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:49.548928 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.549549 27662 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:49.550318  6374 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.551503  6133 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.552093  6133 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.447124  6445 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.066s
W20260709 12:22:49.488385 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54094us
I20260709 12:22:49.514766 28333 rpcz_store.cc:276] 0709 12:22:48.103124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.103284 (+   160us) service_pool.cc:224] Handling call
0709 12:22:48.103648 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.103954 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.103978 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.104512 (+   534us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.959625 (+855113us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:48.959687 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.959786 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.514188 (+554402us) raft_consensus.cc:1685] finished
0709 12:22:49.514211 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.514347 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35062,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":29672,"thread_start_us":854698,"threads_started":1,"wal-append.queue_time_us":1105883}
I20260709 12:22:49.502266  6244 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:49.484287  6331 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:22:49.516391  6412 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.567358  6133 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.537926 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:49.568315 27660 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:49.539209 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.570812 27663 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:49.571420 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.571686 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.572203 27584 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:49.543231 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.574609 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:49.542053 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.522358  6323 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.575603 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:49.565541  6372 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.429s	user 0.018s	sys 0.029s
I20260709 12:22:49.576292 27658 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:49.576488  6372 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.446s	user 0.018s	sys 0.030s
I20260709 12:22:49.576978  6372 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:49.577242  6323 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.567238  6331 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:49.579391  6331 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:49.580191  6331 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.581355  6331 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:49.559942 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:49.566166  6244 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.583720  6016 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:49.583922  6148 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:22:49.584190  6016 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:49.566075 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.584810  6016 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:49.575615 27661 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:49.585513  6148 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:49.586583 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.587389 27661 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:22:49.575625  6133 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:49.588251  6133 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.588665  6133 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:49.560566  6431 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.543651 27656 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 18. (term mismatch)
I20260709 12:22:49.589460  6431 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.586478  6346 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:22:49.590271 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.590045 27656 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 17
I20260709 12:22:49.591647 27656 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 32.18
I20260709 12:22:49.591946 27656 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.592051 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1598) took 1906 ms. Trace:
I20260709 12:22:49.592481 28337 rpcz_store.cc:276] 0709 12:22:47.685469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.685668 (+   199us) service_pool.cc:224] Handling call
0709 12:22:47.700555 (+ 14887us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:47.716127 (+ 15572us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:47.716180 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.635012 (+918832us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b2c000471f8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:22:48.635680 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.233301 (+597621us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:49.233358 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.233458 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.591771 (+358313us) raft_consensus.cc:1685] finished
0709 12:22:49.591792 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.591992 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9669,"raft.queue_time_us":12708,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":8373,"spinlock_wait_cycles":68790144,"thread_start_us":521633,"threads_started":1,"wal-append.queue_time_us":619827}
W20260709 12:22:49.594054 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.594682  6402 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:49.594982  5967 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.595511  5967 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.596701  6401 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.597592  5967 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.597904 27655 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:22:49.598233  5967 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.572320 27663 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:22:49.600399  6332 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.600934  6332 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.602749 27657 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 25 because replica has last-logged OpId of term: 24 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:22:49.602943  6220 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.603511  6220 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.598749  5967 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.606068  5967 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.517118  6437 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.609427  6437 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:49.610666  6422 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:22:49.585529 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.612716 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:49.615384 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.616106 27736 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:22:49.562755 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 416664us
I20260709 12:22:49.617506 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.618212 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.584226  6244 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.618973 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.578104  6372 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.621309 27583 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:49.622231 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.625257 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:49.632512 27660 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
W20260709 12:22:49.627538  6459 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:49.628135 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.627811  6105 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.628816 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.638123 27581 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:49.629453 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.536612  6241 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.616715 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.639523  6244 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:49.633631 27584 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 31 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 15.
I20260709 12:22:49.642386 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.642720  6244 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.643242  6244 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:49.644984  6241 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.647035 27659 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:49.647640 27664 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:49.647569  6431 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.649973  6105 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.652498 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.653623 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.654753 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.655742 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.655949  6332 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.656759  6415 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.657202 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.657846 27736 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:49.658826 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:49.660097  6427 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:22:49.660573  6427 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:49.661509  6427 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.662725  6241 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:49.663405  6241 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.664665 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.638684 27588 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:49.640030 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:49.639973 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.627149 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.665870 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:49.666039 27580 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:49.662839  6427 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:49.664860 27738 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 19. (term mismatch)
I20260709 12:22:49.666601 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:49.673682 27738 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:22:49.638473 27660 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:49.672137 27661 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:49.667181 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:49.674067 27738 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 33.19
I20260709 12:22:49.675125 27738 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.676367 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:49.677008 27738 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:49.635617 27658 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:49.680474 27656 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:49.613893  6401 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.682844 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:49.684458 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.684603 27736 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:49.685465 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.686100 27660 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:49.686923 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.678524 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:49.687517 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:22:49.690500 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.690519 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:49.693634 27663 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.21-34.21]
I20260709 12:22:49.695636 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.696328 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:49.697331  6446 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:49.697768  6446 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.698563 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.699397 27732 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:22:49.705848 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:49.698173  6446 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:49.706209 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:49.679342  6412 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:49.683246  6431 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:49.710896 27738 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:49.707020 27734 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
W20260709 12:22:49.804291 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98532.9us
I20260709 12:22:49.802613 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:49.802973 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 113090us
I20260709 12:22:49.752779 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:49.682113  6364 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.803145 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:51028 (request call id 7) took 1231 ms. Trace:
I20260709 12:22:49.694960  6445 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:49.703867  6332 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:49.812531 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:49.812999 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.812736 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.806442 27735 rpcz_store.cc:276] 0709 12:22:48.478972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.479176 (+   204us) service_pool.cc:224] Handling call
0709 12:22:49.710281 (+1231105us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34e735 00007f0e9a3469fa 00007f0e9f30e4c9 00007f0e9888ac54 00007f0e9888abe2 00007f0e9888ab11 00007f0e9888aa8c 00007f0e98889d42 00007f0e9868348c 00007f0e986828c6
0709 12:22:49.710455 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":957250432}
I20260709 12:22:49.817370 27588 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:49.818004 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:49.818560 27584 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
W20260709 12:22:49.827803 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 131581us
I20260709 12:22:49.830941 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:49.831151 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:49.803452  6379 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.832198 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:49.835597 27735 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: [29.19-31.20]
I20260709 12:22:49.839717 27736 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:49.808266  6105 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:49.805732  6272 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.841275 27658 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:49.838845 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:49.856845 27735 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: [39.23-39.23]
I20260709 12:22:49.861971 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.811724  6445 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.810287  6332 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.864202 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:49.813242  6387 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.824771  6401 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.828088  6346 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.875102 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:49.895915 27588 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:22:49.827999  5991 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.880653 27734 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:49.875839 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.897279 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:49.876279 27735 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 37 index: 22. (term mismatch)
I20260709 12:22:49.903358 27735 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 21
I20260709 12:22:49.903720 27735 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 36.22
I20260709 12:22:49.904038 27735 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:49.906437 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 78840.2us
I20260709 12:22:49.908370 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:49.809726  6431 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.863723  6197 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:49.914181 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:49.915035 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.911628 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.914031  5528 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:22:49.916694  5528 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.916826 27658 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: [35.19-37.20]
I20260709 12:22:49.917227  5528 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:22:49.920708 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:49.880149  6399 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:49.942121 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.943887 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:49.881093  6298 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.880874  6229 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:49.881652  6445 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:49.928443 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.929257 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.879946  6332 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:49.925276  6223 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:22:49.932237  6454 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:49.932613  6448 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.251s
I20260709 12:22:49.933142  6453 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:49.933821  6429 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.935760  6271 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:22:49.863956  5967 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:22:49.938738  6241 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.949237  5528 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:49.951807 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:49.901001  5884 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.953544 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:49.902249  6401 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.961766 27658 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:22:49.855583  6379 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.964032  6223 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:49.827950  6364 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:49.956686 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:49.914575  6197 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.875017  6272 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.010416  6272 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.012888  6364 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.987649  6322 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:22:50.013665 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.014310 27729 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:50.015266 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.016881 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.017438 27729 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:50.017482  6431 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.018599 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.019196  5528 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:49.934877  6244 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.021483  6244 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.022917 27585 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:22:50.026409 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.027240 27730 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 35 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:22:50.029996 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.031339 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.032017 28337 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:49.986483  6453 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.033962 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:49.989400 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.906144  6425 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:49.912519  6270 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.070s
I20260709 12:22:50.044020  6270 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:49.905931  6346 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.044404  6270 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:50.045233  6270 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.046022  6346 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:49.897795  6387 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.047148  6413 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:50.048152  6387 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.048977 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.049638 27581 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:50.050222  6220 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:49.963217 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:49.960040  6229 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:49.970324  6454 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.051683 27588 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
W20260709 12:22:49.995371 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.051991  6464 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:22:50.002467 27471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61199.9us
I20260709 12:22:49.909056  5991 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.053718  6425 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:50.054512  5991 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.055452  5528 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:50.019232 27729 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:50.037835 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.055954  5528 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.056761 28333 raft_consensus.cc:2410] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:22:50.005447 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:50.057291 27734 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:50.050622 28329 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:50.007601  6450 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:50.008209  6426 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.069821 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.008594  6197 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:50.018376  6133 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.956554  6399 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.052137  6454 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:50.018599 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:49.993281  6331 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.038543 27585 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:50.006845 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:49.964138  6298 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:49.985536  6429 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.060061 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.061211 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.000605  6223 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:22:50.059464 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:50.051684  6229 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:50.075652 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:50.046530  6270 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:49.932096  6452 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:22:50.086725 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 79972.8us
I20260709 12:22:50.091171 27587 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 24: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:50.092109 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:49.999768  6241 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.075s
I20260709 12:22:50.095098 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.054670  6425 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.096719 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:50.104941  5528 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:50.101281 28337 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:50.107133 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.108560 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.109748 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.119772 27734 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:50.121552 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.080956  6450 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:50.122500 27583 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:50.123085 27589 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:22:50.125218 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.125663 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:50.124614 27553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.4:39321 timed out after 1.967s (SENT) [suppressed 16 similar messages]
I20260709 12:22:50.128994 28336 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:50.095718  6105 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:50.077852  6220 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.043s
I20260709 12:22:50.100348  6452 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.102739  6428 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:50.128470  6427 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:50.083240  6387 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:22:50.180337 28332 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:50.107561  6346 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:50.082572  6412 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:50.174397 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.175815 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.191159 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.183841  6105 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.193931  6452 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:50.194751 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46136 (request call id 2) took 1461 ms. Trace:
W20260709 12:22:50.196040  6462 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.335s	user 0.001s	sys 0.000s
I20260709 12:22:50.198520  6453 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:50.200233  6016 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.615s	user 0.006s	sys 0.005s
I20260709 12:22:50.099193  6331 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.204074  6448 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.204368  6379 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.204867  6372 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:50.211795  6133 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:50.226173 28337 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:50.225864 28333 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:50.228987 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:50.235158 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60839.7us
W20260709 12:22:50.192638  6385 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.656s	user 0.011s	sys 0.000s
I20260709 12:22:50.234313 28330 rpcz_store.cc:276] 0709 12:22:48.733227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.733395 (+   168us) service_pool.cc:224] Handling call
0709 12:22:50.194525 (+1461130us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54001a1dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a342853 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:50.194632 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1451157248}
I20260709 12:22:50.248970 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.173647  6271 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.061s
I20260709 12:22:50.199088  6446 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.218381  6346 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:50.233546  6105 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:50.172446  6364 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:22:50.255514 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.259582 27581 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:22:50.260080 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.251694  6298 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.266808 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:50.300266 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44806.6us
I20260709 12:22:50.301518 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.294443  6346 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:50.303879  6469 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:50.307854 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.250061  6016 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.254305  6448 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.321421 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:50.329468 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:50.334903  6470 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:22:50.330289 27586 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:50.304369  6469 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:50.343789 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.299227  6401 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:50.353971 28335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1600) took 2328 ms (client timeout 1967 ms). Trace:
W20260709 12:22:50.360810 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59354.1us
W20260709 12:22:50.363125 28335 rpcz_store.cc:269] 0709 12:22:48.025538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.025831 (+   293us) service_pool.cc:224] Handling call
0709 12:22:50.353671 (+2327840us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:50.353841 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4187381504}
I20260709 12:22:50.269711  6385 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.259406  6372 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.363534 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.366865 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.372804 27736 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:50.373652 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.379267 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.379963 27736 raft_consensus.cc:2368] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:50.380661 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:50.383626 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.336767  6241 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:50.409554 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:50.366518  6460 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:50.364094  6469 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:50.422616 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.428659 27736 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:50.429713 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.431929 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1688) took 2085 ms. Trace:
I20260709 12:22:50.432361 28328 rpcz_store.cc:276] 0709 12:22:48.346694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.347449 (+   755us) service_pool.cc:224] Handling call
0709 12:22:48.347721 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.395292 (+1047571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.431694 (+1036402us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003b4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:50.431868 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379532032}
I20260709 12:22:50.412654  6399 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.456197 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3728) took 1108 ms. Trace:
I20260709 12:22:50.456638 27655 rpcz_store.cc:276] 0709 12:22:49.347679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.350382 (+  2703us) service_pool.cc:224] Handling call
0709 12:22:49.350766 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.598367 (+247601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.455299 (+856932us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400344254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:50.456134 (+   835us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":567778688}
I20260709 12:22:50.382802  6387 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:50.311571  6412 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.459574 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:50.293751  6450 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.421435  6241 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:50.461953 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 337339us
I20260709 12:22:50.383162  6271 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:50.427711  6460 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:50.464300 27736 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:22:50.471490 27736 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:22:50.368011  6445 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.485013 27735 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: [33.24-33.24]
I20260709 12:22:50.466540  6241 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:50.472537  6460 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:50.461256  6387 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.510591 27731 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:22:50.486860  6401 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.520126  6331 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.524438  6364 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:50.532285  6332 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.553s	user 0.005s	sys 0.005s
W20260709 12:22:50.566850  6480 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
W20260709 12:22:50.568107  6494 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:22:50.558588  6016 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.549623  6446 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:50.562670  6346 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.574556  6385 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.549400  6429 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.537880  6401 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.547492  6332 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.588428 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:50.591403 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:50.587778  6385 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.592139  6147 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:50.592643  6147 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:50.592482  6385 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.593061  6147 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:50.594723  6385 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.595168  6416 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:50.595741  6416 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.597882  6416 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.592121 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4487) took 1055 ms. Trace:
I20260709 12:22:50.599663 27664 rpcz_store.cc:276] 0709 12:22:49.537208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.537544 (+   336us) service_pool.cc:224] Handling call
0709 12:22:50.591920 (+1054376us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:50.592056 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":169140864}
I20260709 12:22:50.587642  6401 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:50.589076 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 179628us
I20260709 12:22:50.576931  6372 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.603492 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:50.590085  6197 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.004s
I20260709 12:22:50.572938  6448 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.599896  6496 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:50.628593  6141 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:22:50.629179  6141 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:50.630002  6141 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.631211  6141 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:22:50.631964  6489 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.150s	user 0.002s	sys 0.001s
W20260709 12:22:50.633467  6223 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.522s	user 0.009s	sys 0.000s
I20260709 12:22:50.633917  6223 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.637187  6445 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.612687  6496 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:50.638182  6496 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:50.642510 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.628719  6493 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:50.643699 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.644369 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:50.588846  6469 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.651639  6401 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:50.591820  6229 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.654891 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.612581  6197 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.589493  6271 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:50.651219  6272 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:50.652016  6415 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:50.654399  6401 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.668541  6387 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:50.670910  6332 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:50.672943 27584 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:50.678349 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.679139 27584 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 25.
I20260709 12:22:50.680431 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:50.682184 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:50.685209 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.676761  6415 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.678557  6401 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:50.676528  6372 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.684603  6417 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:22:50.688807  6141 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:22:50.720754 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.691557  6105 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.725927 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3876) took 1426 ms. Trace:
I20260709 12:22:50.726744  6105 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.691059  6229 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:50.735745 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1748) took 1109 ms. Trace:
I20260709 12:22:50.736610 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.737215 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3732) took 1096 ms. Trace:
I20260709 12:22:50.737558 27656 rpcz_store.cc:276] 0709 12:22:49.640993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.641131 (+   138us) service_pool.cc:224] Handling call
0709 12:22:49.680269 (+ 39138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.692278 (+ 12009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.737033 (+1044755us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b54001f0250. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:50.737163 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77635968}
W20260709 12:22:50.698704 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 573051us
I20260709 12:22:50.703428 27731 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:22:50.744396 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.745299 27731 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
W20260709 12:22:50.706148 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 439396us
I20260709 12:22:50.707533 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:50.733237 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:50.760432 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.748334 27581 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:50.762567 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.762567 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.763394 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.768199 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:50.768651 28328 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:50.724090 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1745) took 1098 ms. Trace:
I20260709 12:22:50.769560 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:50.769673 27735 rpcz_store.cc:276] 0709 12:22:49.625436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.911284 (+285848us) service_pool.cc:224] Handling call
0709 12:22:49.911745 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.485457 (+573712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.723851 (+238394us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54000a4bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:50.724035 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1317728000}
I20260709 12:22:50.771467 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.772158 27735 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:50.773183 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:50.763540 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:50.755947 27702 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:50.773819 27735 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:50.764567 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.756356 27552 leader_election.cc:400] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:50.776551 27552 leader_election.cc:403] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:50.726313 27737 rpcz_store.cc:276] 0709 12:22:49.299086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.299215 (+   129us) service_pool.cc:224] Handling call
0709 12:22:49.299640 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.379550 (+ 79910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.379579 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.387929 (+  8350us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.823213 (+435284us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:49.823252 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.725571 (+902319us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:50.725612 (+    41us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.725655 (+    43us) raft_consensus.cc:1685] finished
0709 12:22:50.725668 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.725871 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16174,"raft.queue_time_us":25458,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":19980,"spinlock_wait_cycles":145521536,"thread_start_us":404098,"threads_started":1,"wal-append.queue_time_us":502783}
I20260709 12:22:50.781683 28335 raft_consensus.cc:2368] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31.
I20260709 12:22:50.782521 28337 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:22:50.782967 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:50.782564 27737 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [38.20-38.20]
I20260709 12:22:50.732107  6271 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.792383 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.797039 28337 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:22:50.706847  6454 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.603s	user 0.007s	sys 0.004s
W20260709 12:22:50.702479  6270 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.000s
I20260709 12:22:50.790701 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:50.816833 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:50.794533 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.798542 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.794018 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.800467 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.792970 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.820458 27664 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:22:50.736166 27730 rpcz_store.cc:276] 0709 12:22:49.626331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.027817 (+401486us) service_pool.cc:224] Handling call
0709 12:22:50.028094 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.028316 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:50.230219 (+201903us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.631713 (+401494us) spinlock_profiling.cc:243] Waited 9.48 ms on lock 0x7b24002c5b38. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:22:50.735189 (+103476us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.735416 (+   227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.735461 (+    45us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:50.735508 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.735571 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.735692 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":355332,"prepare.queue_time_us":56542,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":66,"raft.queue_time_us":322061,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":200,"spinlock_wait_cycles":21796224,"thread_start_us":339425,"threads_started":2,"wal-append.queue_time_us":225129}
I20260709 12:22:50.822252 27657 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:22:50.698872  6426 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:50.706715  6399 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:50.724058  6141 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:50.834336 27730 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:22:50.693645  6415 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:50.699440  6220 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 28.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:22:50.818439 27662 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:50.703739  6417 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:50.813655 27735 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
W20260709 12:22:50.792220  6452 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.544s	user 0.005s	sys 0.007s
I20260709 12:22:50.692888  6223 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.710443  6188 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.700625  6197 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:50.711818  6244 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.808851 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:50.819679 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.828678 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:50.868202 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48616.2us
I20260709 12:22:50.868876 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:50.706408  6496 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.692606  6346 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.830499 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.803232  6501 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:50.835589 27737 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:50.706393  6477 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:50.782908  6192 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:50.837858 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.791393  6105 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:50.802992  6229 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:50.834656  6141 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:50.704989  6416 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.123s
I20260709 12:22:50.861217 27732 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:50.842382  6417 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:50.693326  6147 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.795665  6148 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32.
I20260709 12:22:50.807730  5884 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.830499 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:50.868341 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.831439  6385 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:50.702827  6372 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.827477  6332 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:50.791880  6196 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:22:50.832304 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.831028  6426 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.874123 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.831205  6399 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.875692 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:50.838568  5967 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:50.876973 27737 raft_consensus.cc:606] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:50.877445 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:50.860488 27735 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:50.882684 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.791796  6469 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:50.862246  6482 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.056s
I20260709 12:22:50.821420  6454 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.977383 28328 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:22:50.866254  6270 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.887825 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.892894 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.878523  6220 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:50.982877 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95134.3us
I20260709 12:22:50.981755 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:50.984259 27737 raft_consensus.cc:614] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:50.983649 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.980551 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.876157  6244 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.881395  6223 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:50.866101  6452 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:50.885991 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77262.3us
I20260709 12:22:50.975723 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:50.976982 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101377us
I20260709 12:22:50.895996 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.974674 27735 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:50.884316  6387 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:22:50.979425 27585 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 31 because replica has last-logged OpId of term: 28 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
I20260709 12:22:50.875797  6500 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:51.003908 27735 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.006843 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:50.884441  6229 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:50.985263 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:50.941908  6426 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:50.897146  6473 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:50.895629  6385 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:50.987366 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:50.895666 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:50.992714  6469 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:50.972488  6399 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:50.890872  6105 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:50.997723 27662 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:50.998900 28334 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.23-32.23]
I20260709 12:22:50.975873  6448 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:50.884689  6192 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.898154  6241 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.007702 27661 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:50.987210  6429 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:51.009687 27657 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:50.944197  6346 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:22:50.937093  6449 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.095s
I20260709 12:22:50.986930  6104 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.985654  6196 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:50.894734  6460 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:50.990010  6332 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:51.098254  6448 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:50.992399  5991 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:51.015980 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:50.982062 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.019248 27735 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:51.020697  6504 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:22:51.021191  6506 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:22:51.025452  6148 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.139458 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:51.017819  6473 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:51.027894  6505 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:22:51.028929  6244 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.029778 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4403) took 1279 ms. Trace:
I20260709 12:22:51.000969  6374 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:51.018635  6501 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:50.998443  6417 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:50.988384  6270 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.005482  6381 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:22:51.031445 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.164130 27587 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:51.033038 28331 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:51.036763  6401 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.040464  6498 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:51.041183  6507 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:51.044504 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.048394  5967 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:51.126066  6104 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.051342  6372 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.040480  6385 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:50.975672  5884 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:51.069835 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.978229  6474 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:51.101279 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:51.008450  6500 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:51.102133 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47934us
I20260709 12:22:51.012429  6229 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:51.113214 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:51.064020  6469 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:51.123229 27664 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:22:51.126127 28337 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
W20260709 12:22:51.127669 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 127895us
I20260709 12:22:51.106024  6429 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:51.131603 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4703) took 1170 ms. Trace:
W20260709 12:22:51.127620 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 91749.4us
I20260709 12:22:51.160782  6148 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.148285 27735 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 38 index: 20. (term mismatch)
I20260709 12:22:51.158165 27657 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:51.125845  6332 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.128159  6196 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.131542  6504 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.149475  6473 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:51.173697 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.049547 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3886) took 1105 ms. Trace:
I20260709 12:22:51.162299  6374 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:51.164132  6496 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.162245  6501 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:51.247474  6504 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:51.257692  6501 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.261351  6496 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:51.262591  6196 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.160063 27734 rpcz_store.cc:276] 0709 12:22:49.750237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.184017 (+433780us) service_pool.cc:224] Handling call
0709 12:22:50.193788 (+  9771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.215800 (+ 22012us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:22:50.610769 (+394969us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.704195 (+ 93426us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.704246 (+    51us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:50.704289 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.029554 (+325265us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54002e86d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:51.029594 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.029724 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":426516,"prepare.queue_time_us":403359,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":41552,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":46,"spinlock_wait_cycles":49949056,"thread_start_us":52680,"threads_started":1,"wal-append.queue_time_us":223477}
I20260709 12:22:51.263907 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:51.264562 27734 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:51.265416  6192 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.265599 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:51.266002  6229 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.266776 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:51.267045  6335 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:22:51.267505  6335 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:51.267091 27738 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:51.215477  6450 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.205359 27729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:51.270565 27729 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:51.270946 27729 raft_consensus.cc:614] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:51.216568 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.244948  6332 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:51.209239 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.216861 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:51.206581 27736 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:51.238826 27580 rpcz_store.cc:276] 0709 12:22:49.960907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.995300 (+ 34393us) service_pool.cc:224] Handling call
0709 12:22:49.995624 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.055295 (+ 59671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.055342 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.055384 (+    42us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:50.055414 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.131241 (+1075827us) spinlock_profiling.cc:243] Waited 59.5 ms on lock 0x7b540020cbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:51.131316 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.131526 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136799104}
I20260709 12:22:51.194099 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4705) took 1090 ms. Trace:
I20260709 12:22:51.238938 27735 pending_rounds.cc:85] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 19
I20260709 12:22:51.282914 27735 pending_rounds.cc:107] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 37.20
I20260709 12:22:51.283260 27735 raft_consensus.cc:2889] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:51.285030 27735 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:22:51.242329  6429 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:22:51.293031 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76253.6us
I20260709 12:22:51.280521 27582 rpcz_store.cc:276] 0709 12:22:50.104048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.104259 (+   211us) service_pool.cc:224] Handling call
0709 12:22:50.104588 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.120995 (+ 16407us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:22:50.739498 (+618503us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b180011c980. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:22:50.739620 (+   122us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.988262 (+248642us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.988700 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.988839 (+   139us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:50.988859 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.988908 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.193826 (+204918us) raft_consensus.cc:1685] finished
0709 12:22:51.193916 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.194044 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21986,"prepare.queue_time_us":112961,"prepare.run_cpu_time_us":177,"prepare.run_wall_time_us":177,"raft.queue_time_us":2708,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":26589,"spinlock_wait_cycles":31229312,"thread_start_us":363918,"threads_started":2,"wal-append.queue_time_us":370749}
I20260709 12:22:51.247929 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4644) took 3014 ms. Trace:
I20260709 12:22:51.248328 27733 rpcz_store.cc:276] 0709 12:22:49.943888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.461062 (+517174us) service_pool.cc:224] Handling call
0709 12:22:50.475285 (+ 14223us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.700512 (+225227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.700543 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.701023 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.883914 (+182891us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:22:50.883954 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.884028 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.049274 (+165246us) raft_consensus.cc:1685] finished
0709 12:22:51.049299 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.049472 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":224852,"raft.queue_time_us":225942,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":182524,"threads_started":1,"wal-append.queue_time_us":317880}
I20260709 12:22:51.251014 27588 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:51.251029 27658 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:51.192143  6461 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.116s	user 0.005s	sys 0.001s
I20260709 12:22:51.207013  6244 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:51.310096 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.163785  6105 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:51.190801 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 141975us
I20260709 12:22:51.192472  6104 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.213126  6241 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.244761  6469 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:51.327400  6452 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.310905  6517 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:51.298344  6454 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.267843  6374 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.329810  6374 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:51.271692 27586 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:51.180027  6448 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.291138  6450 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.334669  6385 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.197209  5884 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.201262  6401 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.205749  6460 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.211869  6270 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:51.246613 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 128667us
I20260709 12:22:51.242218  6147 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.291780 27736 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: []
I20260709 12:22:51.190990  6500 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:51.259095  6501 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:51.298717 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.295894 27589 rpcz_store.cc:276] 0709 12:22:48.176750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.176888 (+   138us) service_pool.cc:224] Handling call
0709 12:22:48.210967 (+ 34079us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.184175 (+1973208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.179991 (+995816us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:51.190798 (+ 10807us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4396738304}
I20260709 12:22:51.303732 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.269276 27734 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:51.350184 27580 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:51.350371 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.173381  6498 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.194801 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:51.336160  6450 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.339020 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.374099 27588 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:22:51.339366 27729 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:22:51.374644 27729 pending_rounds.cc:85] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:22:51.374966 27729 pending_rounds.cc:107] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:22:51.375290 27729 raft_consensus.cc:2889] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:51.189002  5967 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.345393 27586 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:51.350992 27585 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:51.338066  6426 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.245389  6474 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.071s
I20260709 12:22:51.360013 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.388386 27589 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:22:51.369401  6148 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.380674 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:51.396663 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.367105 27580 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
W20260709 12:22:51.372087  6512 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:22:51.348671 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:51.340196 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:51.404649 27580 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:22:51.430156 27586 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
W20260709 12:22:51.430395 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49812us
I20260709 12:22:51.391208 27589 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:51.430720 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1597) took 1402 ms. Trace:
I20260709 12:22:51.431150 28331 rpcz_store.cc:276] 0709 12:22:50.028574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.033627 (+  5053us) service_pool.cc:224] Handling call
0709 12:22:51.430533 (+1396906us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:51.430664 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2295115520}
I20260709 12:22:51.407554  6522 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:51.431640 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1617) took 1310 ms. Trace:
I20260709 12:22:51.431777 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:51.416683  5967 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.341547  6244 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.445322 28334 rpcz_store.cc:276] 0709 12:22:50.121420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.121553 (+   133us) service_pool.cc:224] Handling call
0709 12:22:50.138815 (+ 17262us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.075743 (+936928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.431376 (+355633us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54002fab54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:51.431575 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1977415040}
I20260709 12:22:51.467283 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:51.471050 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:51.421922  6332 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.408463 27734 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:51.437105 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:51.462931 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 122807us
W20260709 12:22:51.441454 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 92915.2us
I20260709 12:22:51.380786  6229 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.485180 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:51.486271 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49301.5us
I20260709 12:22:51.486748 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:51.440011  6428 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:51.476540  6188 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:51.464854  5884 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:51.475862  6519 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:51.522673 27587 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:51.338714  6335 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.482564  6523 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:51.538998 27737 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:51.496258  5967 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:51.498208  6428 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:51.540442 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 283869us
I20260709 12:22:51.427306  6469 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.503778  6450 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:51.486290  6104 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.052s
W20260709 12:22:51.553035 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.555140 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:51.548048 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:51.544744  6415 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.680s	user 0.007s	sys 0.005s
I20260709 12:22:51.553020 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1921) took 1515 ms. Trace:
I20260709 12:22:51.538434  6401 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:51.562922 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:51.542130  6519 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:51.581362 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.571452  6229 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:51.562935 27663 rpcz_store.cc:276] 0709 12:22:50.037112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.089395 (+ 52283us) service_pool.cc:224] Handling call
0709 12:22:50.089774 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.093715 (+  3941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.093744 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.094190 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.740846 (+646656us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b18003ea880. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:50.872659 (+131813us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:50.872864 (+   205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.872957 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.544321 (+671364us) raft_consensus.cc:1685] finished
0709 12:22:51.544351 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.552929 (+  8578us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":231,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":8335,"spinlock_wait_cycles":4802432,"thread_start_us":131585,"threads_started":1,"wal-append.queue_time_us":1041893}
I20260709 12:22:51.550633  6241 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:51.558779  6104 consensus_queue.cc:1243] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 33.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:51.551163  6523 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:51.602380  6192 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:51.618160  6192 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.618511  6523 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.619127  6452 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.558473  6450 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.619722  6452 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.620813  6460 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:51.621256  6460 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.621356  6192 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.596274  6401 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.621922  6452 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.622404  6450 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.605331  6241 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:51.600427  6519 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.606166 27589 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:51.607808  6229 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.630923  6229 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.608392  6428 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.23, Last appended by leader: 23, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.632000  6428 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:51.587211  6473 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.610865  6244 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.612093  6498 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.561344  6335 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:51.568107  6148 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:22:51.615692 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52882.5us
I20260709 12:22:51.635622  6473 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.544946  6188 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:22:51.636555  6385 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.637003  6188 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:51.637187 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.637642  6188 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:51.639447  6532 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:22:51.639751 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:51.617384  6399 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.512s	user 0.011s	sys 0.000s
I20260709 12:22:51.642341 27589 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 19
I20260709 12:22:51.642729 27589 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 33.20
I20260709 12:22:51.642995 27589 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:51.621696  6460 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:51.624450 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:51.572618  6415 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.588140  6522 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 40.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.041s
I20260709 12:22:51.609735  6426 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:51.648931 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:51.654678 28334 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
W20260709 12:22:51.653734 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 84217.1us
I20260709 12:22:51.656173 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:51.660571 28336 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:22:51.638827  6188 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.672086  6188 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:51.673332  6415 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.673069  6141 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.674145  6141 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:51.675345 27584 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:51.662002 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:51.676774 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:51.664183 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:51.641239  6399 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.665751  6332 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.667285 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:51.667892  6500 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.668857  6426 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:51.680809 28329 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37.
I20260709 12:22:51.627110  6241 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:51.674998  6148 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:51.663024  6473 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260709 12:22:51.689678  6527 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:51.695900 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:22:51.707401 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:51.707664 27736 raft_consensus.cc:606] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:51.708014 27736 raft_consensus.cc:614] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:51.708258 27733 raft_consensus.cc:2410] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 38 because replica has last-logged OpId of term: 34 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 20.
I20260709 12:22:51.696846 28328 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:51.698650 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:22:51.703114 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:51.710872 28334 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:51.715425 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.702081  6385 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:51.728713 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:51.738094 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.701510  6220 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:51.706342  6241 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:51.682360  6498 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:22:51.745241  5967 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:51.746340  6148 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.741111  6401 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.756052 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:51.756925 27733 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:51.758015 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:51.758558 27733 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:51.759836 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.760179 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.760527  6399 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.761154 27582 raft_consensus.cc:2410] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:22:51.761919  6469 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.762511  6415 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.763341 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:51.764575 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:51.766811 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.767091 27553 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37.
I20260709 12:22:51.767583 27553 leader_election.cc:403] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
I20260709 12:22:51.773934 28328 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:51.779376 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:51.780031 27731 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:51.775995 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:51.788375 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.804876 27730 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:22:51.808643 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.810869 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:22:51.722724  5884 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:51.750910  6429 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.763032  6415 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.782835  6399 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:51.814345 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 329212us
I20260709 12:22:51.764616  6500 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.796584  6527 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:22:51.780498  6220 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:51.823604 27733 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:22:51.825251 27583 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:51.826274 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.831188 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:51.832660 27733 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:51.810738  6385 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:51.814991  6223 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.815245  5967 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:51.814756  6520 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37.
I20260709 12:22:51.841065 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:51.850143 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.857508 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.858424 27582 raft_consensus.cc:2368] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37.
I20260709 12:22:51.872395 27733 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:51.886668 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
W20260709 12:22:51.887163 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 332145us
I20260709 12:22:51.888095 27731 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:22:51.850453 27628 leader_election.cc:400] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31.
I20260709 12:22:51.850971  6426 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:51.843902  6271 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.828028  6527 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.907529  6527 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:51.907658  6519 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.871452  6500 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:51.908869  6500 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:51.910004  6500 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.911546  6426 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.914829 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:51.915438  6504 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.658s	user 0.012s	sys 0.000s
I20260709 12:22:51.916145 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:51.916347 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.917191 27580 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:51.917970  5991 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:51.918430  5991 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:51.918655  6429 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.854022  6223 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.920878 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3916) took 1164 ms. Trace:
I20260709 12:22:51.921954  6223 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:51.923823 27584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 1957) took 1249 ms (client timeout 1643 ms). Trace:
W20260709 12:22:51.924438 27584 rpcz_store.cc:269] 0709 12:22:50.674216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.680136 (+  5920us) service_pool.cc:224] Handling call
0709 12:22:51.923564 (+1243428us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:51.923744 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2286086528}
I20260709 12:22:51.927119  6415 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.897293 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:51.827836  6399 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.921373 27738 rpcz_store.cc:276] 0709 12:22:50.756116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.866186 (+110070us) service_pool.cc:224] Handling call
0709 12:22:50.866568 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.269147 (+402579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.920563 (+651416us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54002e95d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:51.920801 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920690944}
I20260709 12:22:51.891106 27628 leader_election.cc:403] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:51.865931  5967 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:51.931445 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81144.2us
I20260709 12:22:51.928452 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.933272 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:51.937036 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:51.938156  6538 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:51.893306  6452 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:51.863607  6540 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.084s
I20260709 12:22:51.944793 27580 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:51.915917  6504 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.945647 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:51.947464 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:51.930086  6426 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:51.935106  6514 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:22:51.954470 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:51.929834  6105 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
W20260709 12:22:51.950129 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 183226us
I20260709 12:22:51.930451  6271 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.889168  5884 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.971233 27581 raft_consensus.cc:2368] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39.
I20260709 12:22:51.934576  6510 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.020s
I20260709 12:22:51.965260 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:51.932314  6241 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.952286  6188 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:51.962062  6133 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:51.977787  6335 raft_consensus.cc:2764] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election decision vote started in defunct term 27: won
I20260709 12:22:51.979390  6133 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:51.982556  6133 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:51.981096 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:51.980286  6241 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:51.985229  6133 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:51.952354  6372 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:22:52.001493 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47522.3us
I20260709 12:22:52.008116 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:52.010151 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45020.2us
I20260709 12:22:52.017112 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:52.022159 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:52.017977 28334 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:52.025487  6426 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:52.026057 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:52.027459 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:52.027726  6188 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:52.028379  6188 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:52.028395  6241 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:22:52.029302  6241 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:52.029601  6188 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.030500  6244 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:51.960371  6526 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:22:51.961237  6540 consensus_queue.cc:1243] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 40.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:22:51.947764  6452 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.035456 28330 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.23-32.23]
I20260709 12:22:52.036844 27704 leader_election.cc:400] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37.
I20260709 12:22:52.039768 27704 leader_election.cc:403] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:51.934904  6220 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:51.940181  6385 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:52.045413 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 177748us
I20260709 12:22:52.045452 27658 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: []
I20260709 12:22:52.047168 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:52.047262 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.047415 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96851.6us
I20260709 12:22:52.023010  6401 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:52.053635  6401 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:52.054100  6401 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:52.043350  6241 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.014s
I20260709 12:22:52.006728  6372 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:52.060243  6452 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:52.060020  6372 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:52.061408  6547 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:22:52.062711  6399 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.062872  6522 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:52.051532 27738 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:22:52.052075 27731 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: [33.24-33.24]
I20260709 12:22:52.052982 27730 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:22:52.061106  5991 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.061337  6372 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:52.066428 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.067273 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:52.067849 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.068629 27730 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:52.080279 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.081243 27587 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:52.068671 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.044039  6428 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:22:52.037276  6526 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:52.086056 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.086846 27730 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:52.063392  6522 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:52.088018  6513 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.088234  6541 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:52.088636  6513 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.088804  6541 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.090509 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.090890  6513 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:52.093662 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.089018  6104 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:52.094133 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.091624  6322 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.095428 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.095871  6322 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.096537 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.097595 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.098109 27659 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:52.098655  6322 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.092432 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.091871  6541 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:52.092501 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.101112 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.102362 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.104943 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.105908  6332 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:22:52.106426 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.107143  6332 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:22:52.108104 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.108989 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.109333 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.109910 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:52.110533  6332 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:22:52.111125  6541 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:52.111284  6332 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:22:52.111814  6541 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:22:52.094749  6104 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:52.116284 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:52.116477 27659 raft_consensus.cc:2410] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 33 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 23.
I20260709 12:22:52.117638  6104 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.118618 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.118598 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.120889 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.122884 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.123893 27659 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:52.121206 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.137267 27659 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:52.139871 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.143136 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.142756 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.140220 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.144476 27659 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:52.155809 27659 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
W20260709 12:22:52.164155 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54889us
I20260709 12:22:52.165270 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.166007 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.166167 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.167006 27659 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:52.167716 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.168314 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:22:52.170627 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.173249 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.074501  6525 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:52.034669  5967 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.074220  6399 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:52.190073 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:52.185115  6452 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:52.184937  6332 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:52.187924  6220 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:22:52.189770  6554 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:22:52.197121  6525 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.231879 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:52.152370  6552 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:52.064143  6374 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.109s
I20260709 12:22:52.153937  6540 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:52.170455  6547 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:22:52.232185 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:52.345732 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 113664us
I20260709 12:22:52.186926 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.190774 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.189908 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.199523 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.348227 27658 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:22:52.187573 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.349438 27730 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:52.233148 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:52.240507 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1749) took 2558 ms. Trace:
I20260709 12:22:52.350718 27732 rpcz_store.cc:276] 0709 12:22:49.685495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.067489 (+381994us) service_pool.cc:224] Handling call
0709 12:22:50.080131 (+ 12642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.886652 (+806521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.240213 (+1353561us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:52.240439 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1795833600}
I20260709 12:22:52.242038 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 1955) took 1568 ms. Trace:
I20260709 12:22:52.352018 27583 rpcz_store.cc:276] 0709 12:22:50.673673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.721838 (+ 48165us) service_pool.cc:224] Handling call
0709 12:22:50.722139 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.836890 (+1114751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.241813 (+404923us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400186fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:52.241980 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2530116352}
I20260709 12:22:52.352600 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:52.243803 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1758) took 1356 ms. Trace:
I20260709 12:22:52.353292 28330 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:52.353440 27737 rpcz_store.cc:276] 0709 12:22:50.887377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.000964 (+113587us) service_pool.cc:224] Handling call
0709 12:22:52.243608 (+1242644us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:52.243745 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938746112}
I20260709 12:22:52.356843 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:52.359295 27587 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:52.359774 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:52.363988 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:52.364818 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.364907 28334 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:22:52.287477  6482 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:52.380628 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:52.246817  6504 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.248287  5991 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.261123  6517 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:22:52.173405 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:52.270283 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3754) took 1444 ms (client timeout 1580 ms). Trace:
I20260709 12:22:52.435585 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.436305 27659 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:22:52.175681 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
W20260709 12:22:52.436034 27664 rpcz_store.cc:269] 0709 12:22:50.825791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.839517 (+ 13726us) service_pool.cc:224] Handling call
0709 12:22:52.270010 (+1430493us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400344c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:52.270201 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":418946304}
I20260709 12:22:52.438429 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.437433 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.272249  6448 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:52.274405  6450 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:52.154578  6526 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:22:52.281927 27735 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:52.481873 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:52.283967 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93972.2us
I20260709 12:22:52.482682 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.483171 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.483414 27658 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:52.284042 27661 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24.
I20260709 12:22:52.286342 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.336143 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:52.342535  6525 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:52.286492  6452 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.369954 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:22:52.370139 27587 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:52.245626  6527 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.368778 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.437564  6133 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:52.404007 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:52.499955 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:52.505745 28328 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:52.406452 28331 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:22:52.408500 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:52.364998  6399 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.363878  6375 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:52.404896  6415 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:22:52.333425  6552 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:52.432538 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.432811 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.336853  6541 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:52.439213 27662 raft_consensus.cc:2368] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:22:52.343928  6271 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.469151 27732 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
W20260709 12:22:52.531092 27581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4671) took 1771 ms (client timeout 1962 ms). Trace:
I20260709 12:22:52.490397 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:52.531621 27581 rpcz_store.cc:269] 0709 12:22:50.759984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.942501 (+182517us) service_pool.cc:224] Handling call
0709 12:22:52.530941 (+1588440us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b5400088ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:52.531045 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1938959104}
I20260709 12:22:52.532274 27661 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:52.494513 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:52.495242 27738 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:52.501737 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 133080us
I20260709 12:22:52.536224 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:52.392854 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 159783us
I20260709 12:22:52.434513  6517 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:52.361511  6540 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.550352 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:52.509135 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.467901  6482 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:52.481313  6450 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:52.513406 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.553736 27733 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:52.553884 27704 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:52.508635  6460 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.845s	user 0.011s	sys 0.000s
I20260709 12:22:52.571602  6460 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.577271  6428 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37.
I20260709 12:22:52.578107 27731 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:52.520344  6375 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.579599  6375 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:52.580798  6452 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:52.581459  6452 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:52.523638  6133 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:52.584122  6133 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.585129  6517 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.586253  6517 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:22:52.587800 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.504040  6566 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:52.588853 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.502952  6197 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:22:52.589898  6514 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.634s	user 0.010s	sys 0.000s
I20260709 12:22:52.590327  6197 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:52.590838  6197 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.590941  6482 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:52.561079 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3750) took 1770 ms (client timeout 1917 ms). Trace:
I20260709 12:22:52.561939 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1760) took 1315 ms. Trace:
W20260709 12:22:52.592521 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43728.4us
W20260709 12:22:52.594858 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.595777 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.596364 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.593129 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:52.593464 27660 rpcz_store.cc:269] 0709 12:22:50.790954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.793641 (+  2687us) service_pool.cc:224] Handling call
0709 12:22:52.560846 (+1767205us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:52.561009 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2384717184}
I20260709 12:22:52.580807  6527 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:52.603737 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.593550 27729 rpcz_store.cc:276] 0709 12:22:51.246079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.291573 (+ 45494us) service_pool.cc:224] Handling call
0709 12:22:51.291834 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.384939 (+ 93105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.561666 (+1176727us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54002ea4d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:52.561874 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108471552}
I20260709 12:22:52.564726 27655 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:22:52.530076  6515 consensus_queue.cc:1175] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 26 status { last_received { term: 26 index: 18 } last_committed_idx: 17 last_received_current_leader { term: 26 index: 18 } }
I20260709 12:22:52.570645  6322 raft_consensus.cc:2764] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:22:52.588675  6566 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.613936 27729 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:22:52.545621 27661 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 33 because replica has last-logged OpId of term: 32 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 22.
I20260709 12:22:52.615967 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.616765 27661 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:52.613252 27655 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:52.618356 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.620651 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:52.621268 28328 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:52.618939 27732 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:52.617368 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.622725 27660 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 24.
I20260709 12:22:52.623665 27664 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:52.624074 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.624437 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.624989 27659 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:52.626021 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.553917  6525 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:52.619359 27655 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.636126 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.637804 27732 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:52.590368  6514 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31.
I20260709 12:22:52.591300  6197 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:52.592428  6223 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:22:52.643999  6188 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.074s	sys 0.124s
I20260709 12:22:52.619643  6454 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:22:52.650483 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:52.668568 27732 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:52.571113  6450 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.554454  6541 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.621325 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:52.659210  6401 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.005s
I20260709 12:22:52.662710 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:52.671305 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.663944 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:52.656394  6525 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:52.675179 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:52.664580  6188 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.075s	sys 0.125s
I20260709 12:22:52.647562 27655 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:52.683440 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.688539 27586 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:52.690779 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:52.686399 27589 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:52.705122 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.713902 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:52.717626 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.721526 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.035s	sys 0.044s
W20260709 12:22:52.722414 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.035s	sys 0.044s
I20260709 12:22:52.724304 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.725111 27661 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:52.687526  6454 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 33.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:52.648705  6552 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:52.715662  6529 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.911s	user 0.006s	sys 0.006s
I20260709 12:22:52.734308 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:52.743738 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.688036  6401 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.749493  6244 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:22:52.749243 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 254800us
I20260709 12:22:52.742969 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:52.756986 27655 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:22:52.733886  6450 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:52.755652 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:52.752585  6525 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.762815 27663 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:52.736264  6332 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:22:52.755306  6372 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.667s	user 0.012s	sys 0.005s
I20260709 12:22:52.761031 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.759491 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.758165 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:52.773671 28331 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 22. (term mismatch)
I20260709 12:22:52.761632 28328 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:52.721197  6188 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.029s
I20260709 12:22:52.780354 28331 pending_rounds.cc:85] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 21
W20260709 12:22:52.780634 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.755169  6519 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.780293 27660 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:52.760692  6552 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:52.786576 27655 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:22:52.790365 27659 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:22:52.783094 28331 pending_rounds.cc:107] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 34.22
I20260709 12:22:52.791024 28331 raft_consensus.cc:2889] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:52.791764 27659 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:52.789927 27655 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:22:52.786875 27661 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:22:52.795675 27661 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
W20260709 12:22:52.787285 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 262280us
W20260709 12:22:52.799154 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.786751 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:22:52.803516 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.804018 27661 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:22:52.772665  6399 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:52.809420 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 168487us
I20260709 12:22:52.806449 27655 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
W20260709 12:22:52.771401  6538 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 40 current term: 40 is higher.
I20260709 12:22:52.777495  6220 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:52.776845  6566 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.785218  6568 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:52.785344  6428 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:22:52.820966 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 267146us
I20260709 12:22:52.812757  6372 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:22:52.851514  6372 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:52.853067 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4672) took 2092 ms. Trace:
I20260709 12:22:52.852766  6372 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 28.23, Last appended by leader: 23, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.855672 27662 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.21-34.21]
I20260709 12:22:52.857515 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:52.853475 27588 rpcz_store.cc:276] 0709 12:22:50.760233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.379977 (+619744us) service_pool.cc:224] Handling call
0709 12:22:51.380290 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.437993 (+ 57703us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:51.710352 (+272359us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.631453 (+921101us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b18004a4940. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a
0709 12:22:52.640000 (+  8547us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.640234 (+   234us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.640294 (+    60us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:52.640387 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.852184 (+211797us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b18004a4940. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:22:52.852831 (+   647us) spinlock_profiling.cc:243] Waited 57.4 ms on lock 0x7b54000889d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:52.852865 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.853006 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2614,"prepare.queue_time_us":406024,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":7661,"raft.queue_time_us":202,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":13660,"spinlock_wait_cycles":183766272,"thread_start_us":403771,"threads_started":2,"wal-append.queue_time_us":269092}
I20260709 12:22:52.862972  6576 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:52.863827  6576 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.864784  6522 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.866111  6576 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:52.866660  6522 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:52.825335  6577 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.867762  6577 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:52.871614  6571 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:22:52.873740  6532 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:52.874326  6532 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:52.875449  6532 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.839962 27704 leader_election.cc:400] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39.
W20260709 12:22:52.877558  6572 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:52.880095 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.883343 27662 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:52.884835 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:22:52.840257  6197 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.842559  6272 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.843798  6148 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:52.844274 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:52.844856  6428 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.849083  6460 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.876096 27587 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:52.870455  6577 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:52.916290  6541 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.803963  6519 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.925817  6519 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.927969  6568 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:52.888433 27656 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:52.888191 27588 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:22:52.903877 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59669.3us
W20260709 12:22:52.826749  6564 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:22:52.905576  6496 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:52.817914  6220 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.907035 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:52.931705 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:52.938736 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.939523 27656 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:22:52.939615 28337 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 26.
I20260709 12:22:52.939610 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:52.824855  6540 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.888963  6223 raft_consensus.cc:2764] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election decision vote started in defunct term 38: won
I20260709 12:22:52.934064 27581 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:52.949411 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:52.904665  6452 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:52.835985 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:52.950405 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:52.934711  6569 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.930773  6272 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.937811  6148 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.952142 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:52.940343  6133 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.952769  6569 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:52.930038  6390 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:22:52.902191  6105 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.052s
I20260709 12:22:52.957651  6385 raft_consensus.cc:2764] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20260709 12:22:52.937034  6541 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:52.956988 27733 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:52.945703  6519 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:52.943614  6460 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:22:52.944327  6401 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:52.953706  6496 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:22:52.991379 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.996466 27733 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:22:53.002668 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 157877us
I20260709 12:22:52.975625  6271 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.014128 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.015034 28331 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:52.980793  6519 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:52.979180 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 232510us
I20260709 12:22:52.954617  6416 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:22:52.976976 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.019622 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.019845 27733 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:22:53.031306 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:53.025180 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:53.026230 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.029039 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:53.018890  6573 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:22:53.009073  6570 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.251s	user 0.001s	sys 0.000s
I20260709 12:22:53.044644 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:53.046761 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.048902 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:52.962714  6148 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:52.965287  6133 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.006742  6523 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:53.036156  6448 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.039701  6197 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.040870  6220 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.041021  6510 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:53.009485  6496 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:53.042187  6272 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.035531  6519 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:52.993584  6541 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.053733 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:53.060935 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:53.004920  6385 consensus_queue.cc:1059] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:22:53.090974 27733 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:22:53.054911  6416 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:53.054986  6523 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:53.030102  6271 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:53.091293 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60126.4us
I20260709 12:22:53.103838 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:53.116629 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62975.2us
I20260709 12:22:53.105576 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:53.104628 27587 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:22:53.105136 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.161305 27586 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:53.061380  6473 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:22:53.100836 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.091549  6220 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:22:53.173307  6428 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.175153  6148 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.062498  6448 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.177829  6448 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.178651  6416 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.179368  6482 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.012s
W20260709 12:22:53.103013 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:53.153131  6550 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 12:22:53.183863 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.094175  6197 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:53.060627  6581 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:22:53.062783  6540 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:53.063020  6510 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:53.159240 27588 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 39 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 20.
I20260709 12:22:53.163116 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:53.103281 27580 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260709 12:22:53.191854 27580 pending_rounds.cc:85] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 13
I20260709 12:22:53.192252 27580 pending_rounds.cc:107] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260709 12:22:53.192673 27580 raft_consensus.cc:2889] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:53.158577 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 180460us
I20260709 12:22:53.196384 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:53.196789 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.199148 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.088737  6566 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260709 12:22:53.091841  6272 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:53.060770  6585 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.119s
I20260709 12:22:53.200742  6220 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:22:53.154659  6582 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:22:53.187568  6526 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.680s	user 0.016s	sys 0.000s
W20260709 12:22:53.095340  5967 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.885s	user 0.003s	sys 0.024s
I20260709 12:22:53.186376 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.146495  6188 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:53.187691 27589 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:22:53.173313  6244 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.012s
I20260709 12:22:53.185485  6592 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:53.152093  6241 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.041s
W20260709 12:22:53.219738 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.188724  6473 consensus_queue.cc:1243] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 24.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:22:53.200800  6272 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.204497  6401 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:22:53.220329  5967 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.005s	sys 0.024s
W20260709 12:22:53.237349  6375 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.657s	user 0.006s	sys 0.006s
I20260709 12:22:53.237762  6510 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.275043 27734 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:53.238725  6401 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.204218  6582 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.192461  6197 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:53.248672 28337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1612) took 1612 ms (client timeout 2070 ms). Trace:
W20260709 12:22:53.364738 28337 rpcz_store.cc:269] 0709 12:22:51.636757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.639440 (+  2683us) service_pool.cc:224] Handling call
0709 12:22:53.248435 (+1608995us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:53.248588 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2987278592}
I20260709 12:22:53.266592 28331 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:22:53.267606 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:53.368264 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.271884  6417 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:53.273082  6133 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:53.357089 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 197134us
W20260709 12:22:53.371290 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 157661us
I20260709 12:22:53.377141 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:53.243070  6244 consensus_queue.cc:1243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 29.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:22:53.388358 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:53.220537  6526 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.274292  6426 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.360420  6401 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:53.361052  6220 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.412952 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:53.370229  6582 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:53.376180  6569 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.370961  5967 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.411507  6510 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:53.222621  6496 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.422775 27657 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:22:53.421150  6566 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:22:53.427902  6590 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
I20260709 12:22:53.399580  6482 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:53.229398  6592 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.429067  6581 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:53.376366  6585 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.403718  6428 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.487712 27587 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:22:53.415930  6426 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.421396  6531 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:22:53.353775  6375 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:53.418625  6595 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:22:53.521584 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.228714  6523 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:53.437839  6580 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.370s	user 0.001s	sys 0.000s
I20260709 12:22:53.482674 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.484577 27730 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:53.484920 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.438167  6390 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:22:53.545260  6375 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.545372  6454 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:53.477990  6581 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:22:53.549624 27663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1922) took 1604 ms (client timeout 1643 ms). Trace:
W20260709 12:22:53.550115 27663 rpcz_store.cc:269] 0709 12:22:51.945203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.945341 (+   138us) service_pool.cc:224] Handling call
0709 12:22:53.549452 (+1604111us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:53.549562 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1629217536}
I20260709 12:22:53.550442  6581 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.551918  6581 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:53.552266  6600 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:22:53.555789 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.559628 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3782) took 1011 ms. Trace:
I20260709 12:22:53.560752  6272 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.560029 27662 rpcz_store.cc:276] 0709 12:22:52.548589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.886233 (+337644us) service_pool.cc:224] Handling call
0709 12:22:52.886577 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.904133 (+ 17556us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.904165 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.941064 (+ 36899us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.423169 (+482105us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:53.423227 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.423604 (+   377us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.559407 (+135803us) raft_consensus.cc:1685] finished
0709 12:22:53.559428 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.559572 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17062,"raft.queue_time_us":19301,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":481759,"threads_started":1,"wal-append.queue_time_us":546107}
I20260709 12:22:53.562228  6526 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.562966 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:53.563647 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3778) took 1159 ms. Trace:
I20260709 12:22:53.561612  6581 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.564026 27660 rpcz_store.cc:276] 0709 12:22:52.403992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.623829 (+219837us) service_pool.cc:224] Handling call
0709 12:22:53.563491 (+939662us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400291654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:53.563592 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":357878656}
I20260709 12:22:53.564301  6581 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.452482  6148 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:22:53.566270  6148 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.566603  6581 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.566810  6148 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:53.570796 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4410) took 1489 ms. Trace:
I20260709 12:22:53.474224  6482 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:53.571890  6482 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:53.573751  6449 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:22:53.574141  6449 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:53.571203 27734 rpcz_store.cc:276] 0709 12:22:52.081310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.081733 (+   423us) service_pool.cc:224] Handling call
0709 12:22:52.244146 (+162413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.275499 (+1031353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.569806 (+294307us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:53.570617 (+   811us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b54000a50d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:53.570742 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2561376384}
I20260709 12:22:53.578888 27659 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:53.432334 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:53.522158 27580 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:53.425045  6220 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.523561  6428 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.563843 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:53.489742  6567 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.521988 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:53.584290 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:53.524880 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.586790 27658 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
W20260709 12:22:53.520864  6450 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.758s	user 0.007s	sys 0.005s
I20260709 12:22:53.590185 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.591030 28328 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:53.437670  5967 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:53.592010 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 159748us
I20260709 12:22:53.592391 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:53.601274 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79355.1us
I20260709 12:22:53.602353 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:53.526193  6271 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:53.618785 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.603387  6375 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:22:53.632179 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.624711 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:53.634227 27656 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:22:53.634691 27658 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:53.627682 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.636111 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.631624 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:53.636937 27656 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:53.629842 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.630502 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.638315 27662 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:22:53.575017  6449 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.492233  6585 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.637830 27736 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 31 because replica has last-logged OpId of term: 29 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 23.
I20260709 12:22:53.631544 27663 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:22:53.605010  6579 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:53.624822 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:53.606057  6566 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:53.605758  6608 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:53.658470 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.668951 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:53.669773 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45045us
I20260709 12:22:53.674045 27731 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:53.670559 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.672135 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.677160 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.675659 27730 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:22:53.678129 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:53.679273  6579 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.002s
I20260709 12:22:53.683516 27734 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:22:53.591140  6567 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.685701 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.686311 27730 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
W20260709 12:22:53.686862 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.684300 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:53.687736 27737 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:53.688223 27658 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:53.689263  6188 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.689742  6188 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.668419  6566 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.690759  6566 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:53.691998  6188 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.694958  6452 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.673087  6608 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:53.695611  6608 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:53.695554  6452 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.697800  6452 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.665400  6375 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:53.676828  6449 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:53.675889  6448 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:53.705173 27736 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:53.677922  6603 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:53.705463 27552 leader_election.cc:400] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 27. Current term is 28.
I20260709 12:22:53.715739 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:53.679394  6450 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.711772 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.706152 27737 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:53.717458  6531 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:22:53.719414  6567 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.687759 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.721725 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:53.677407  6541 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.683012  6517 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:53.723739 27736 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
W20260709 12:22:53.726512 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3779) took 1210 ms (client timeout 1519 ms). Trace:
W20260709 12:22:53.733402 27664 rpcz_store.cc:269] 0709 12:22:52.516217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.588548 (+ 72331us) service_pool.cc:224] Handling call
0709 12:22:53.726339 (+1137791us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b5400229054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:53.726460 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71784192}
I20260709 12:22:53.733779 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:53.714349  6552 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.906s	user 0.009s	sys 0.004s
W20260709 12:22:53.732774 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.736114 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:53.704980  6197 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:53.740159 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.741612 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:53.743553 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:53.748423 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:53.757100 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:53.758062 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:53.759435 28328 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:53.761296 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:53.762991 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.764153  6517 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:22:53.764720  6517 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:53.766088  6517 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.725696  6603 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.773653  6531 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:53.774183  6608 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.742282  6496 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:53.775288  6527 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:22:53.735157  6474 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:53.764671  6598 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:22:53.795562 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.798424 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.787097 28328 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:53.746495  6271 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:53.746140  6016 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:53.791698 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1923) took 1829 ms. Trace:
W20260709 12:22:53.780100 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.732913  6448 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:22:53.785521  6531 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:53.760921  6552 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:53.816867 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.817183 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:53.817612 27730 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:53.817937 27661 raft_consensus.cc:2376] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 24. Re-sending same reply.
I20260709 12:22:53.731403  6450 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.816360 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:53.800058 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 116935us
I20260709 12:22:53.821936 27655 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:53.800796 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.822971 27736 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 39 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 20.
I20260709 12:22:53.823239 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:53.823839 27655 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:53.824008 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.824955 27663 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:53.804736 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.804395 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:53.803222 27657 rpcz_store.cc:276] 0709 12:22:51.962343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.962508 (+   165us) service_pool.cc:224] Handling call
0709 12:22:51.962884 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.450526 (+1487642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.791477 (+340951us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:53.791631 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3356876160}
I20260709 12:22:53.807570 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.787390  6460 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:22:53.828491  6460 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:53.828629 27658 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:53.829386  6460 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.830521 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.782572  6426 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.831414  6426 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:53.787832  6592 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.795315  6220 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:53.856562  6603 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.858556 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:53.801524  6474 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.865279  6523 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:53.787845  6390 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 12:22:53.788398  6375 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.869207  6454 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:53.869757  6454 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:53.876511  6591 raft_consensus.cc:3055] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:53.877148  6591 raft_consensus.cc:740] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:53.878288  6591 consensus_queue.cc:260] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.902061 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4422) took 1421 ms. Trace:
I20260709 12:22:53.902655 27735 rpcz_store.cc:276] 0709 12:22:52.481159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.481568 (+   409us) service_pool.cc:224] Handling call
0709 12:22:53.901810 (+1420242us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54000a55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:53.901984 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1535674624}
I20260709 12:22:53.909228 27737 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:22:53.837440 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.839428 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.840099 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:53.912309 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:22:53.841408 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:53.835094  6616 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.008s
I20260709 12:22:53.835098  6241 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:53.844508  6532 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:22:53.845722 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:53.922017 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 132149us
W20260709 12:22:53.850275  6519 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.733s	user 0.007s	sys 0.001s
I20260709 12:22:53.927984 27664 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:22:53.850507  6585 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.850666  5967 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:53.851670  6614 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:22:53.853405  6608 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.938361 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.854871 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:53.969022 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 114299us
I20260709 12:22:53.806731  6016 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:53.854592  6482 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.971517 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.816463  6531 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:53.881955  6474 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.882782 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.883014 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:53.796505  6428 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:53.933929  6519 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.802372  6517 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:53.793787  6515 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:22:53.896920 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 115689us
I20260709 12:22:53.974299 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:53.897722 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
W20260709 12:22:54.051378 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 77168.3us
I20260709 12:22:53.913525 27657 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:53.923092 28336 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:22:53.923267 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:53.922323 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:53.926481 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 86455.2us
I20260709 12:22:54.055269 27734 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:54.055967 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.057628 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:53.929601 28331 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
W20260709 12:22:53.930747 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 182417us
I20260709 12:22:53.854723  6526 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:22:53.941882 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.985801  6016 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:53.870839  6454 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:53.933580  6591 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:53.953270 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.954768 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.959769 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.098743 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.934554  6538 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:53.936349  6601 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.100284  6473 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.100471  6601 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.100905  6473 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.103134  6473 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:53.845225  6148 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:53.925568  6241 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:53.974586 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.975667 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:53.983811  6541 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:53.984691 27733 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:53.886122  6582 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.103291  6601 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.105707  6148 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.044996 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3938) took 1457 ms. Trace:
I20260709 12:22:54.047976 27732 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:54.114974 28334 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:54.112432  6541 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.115948 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.116202  6541 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:54.116925 27732 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:22:54.117765 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:54.118435 27732 raft_consensus.cc:2368] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38.
I20260709 12:22:53.932889  6566 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:54.124116 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.119205 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.124372  6566 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:54.125205 27732 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 39 because replica has last-logged OpId of term: 38 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:22:54.074167  5884 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:54.064632 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:53.944696 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.131772 27730 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:22:54.060354 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.085068 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.133102 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:54.133637 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48641.4us
I20260709 12:22:54.134756 27581 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:22:54.094552 27583 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:54.137609  5884 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:54.138358  5884 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.066746  6375 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:22:54.076679  6522 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.209s	user 0.012s	sys 0.005s
I20260709 12:22:54.119975 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.120023 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.149664 27586 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:54.110355 27582 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
W20260709 12:22:54.073443  6401 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.651s	user 0.008s	sys 0.007s
I20260709 12:22:54.061417 27657 raft_consensus.cc:1217] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Deduplicated request from leader. Original: 21.17->[22.18-25.19]   Dedup: 22.18->[25.19-25.19]
I20260709 12:22:54.152493 27657 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 25. Current term is 26. Ops: [22.18-25.19]
W20260709 12:22:54.142964  6593 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:22:54.049048  6515 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:54.139122 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 78878.7us
I20260709 12:22:54.114480  6582 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:54.139672 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75224.2us
W20260709 12:22:54.148952  6510 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.699s	user 0.007s	sys 0.008s
I20260709 12:22:54.139210 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.163241 27662 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:22:54.141654 27735 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:22:54.067420  6513 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:54.107667  6241 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.084054  6482 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:54.145946 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.179118 27732 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:54.149659 27585 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:54.150657 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:54.153594 28329 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
W20260709 12:22:54.144965  6580 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:22:54.147572  6417 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.766s	user 0.011s	sys 0.001s
I20260709 12:22:54.156912 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:54.183965 27583 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:54.157480 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:54.142810  6413 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:54.146984  6566 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:22:54.161451 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:54.115218 27729 rpcz_store.cc:276] 0709 12:22:52.588149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.620609 (+ 32460us) service_pool.cc:224] Handling call
0709 12:22:52.621010 (+   401us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:52.638948 (+ 17938us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.638995 (+    47us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:52.639871 (+   876us) log.cc:844] Serialized 78 byte log entry
0709 12:22:52.836742 (+196871us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:22:54.041370 (+1204628us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.043691 (+  2321us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.044634 (+   943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.044720 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.044765 (+    45us) raft_consensus.cc:1685] finished
0709 12:22:54.044782 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.044933 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":936830,"prepare.queue_time_us":977676,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":22739,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":14421,"thread_start_us":418889,"threads_started":2,"wal-append.queue_time_us":351410}
I20260709 12:22:54.140278 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.166973 28336 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:22:54.167672 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:54.194679 27581 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:54.194957 28330 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:54.161957  6515 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:54.144207  5884 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:54.169773 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:54.170581 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.173342  6611 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:54.152392  6375 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:54.168617 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.146837  6619 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.046s
W20260709 12:22:54.225829 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 435541us
I20260709 12:22:54.183892  6474 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.158960  6401 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.151196  6592 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.043s
I20260709 12:22:54.220438 27664 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:54.221427 28330 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:54.190784 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.195698  6510 raft_consensus.cc:2764] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election decision vote started in defunct term 36: won
W20260709 12:22:54.199694 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 150163us
W20260709 12:22:54.257750 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67026.3us
I20260709 12:22:54.258641 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.249812 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.217450  6566 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:54.220913  6567 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:22:54.255967  6241 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.261866  6513 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:54.247722  6474 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.271865  6519 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:54.294512  6627 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 12:22:54.299518  6566 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.277925  6541 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.306572  6531 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.309020  6531 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:54.345703 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118917us
I20260709 12:22:54.353350 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.357446 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.362435 27580 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:54.208945 27581 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:22:54.411790  6401 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.290566  6375 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.201835 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.414002  6519 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:54.366935 27589 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:54.380057 28328 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:54.382726 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:54.417466 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.420368 27587 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 21. (term mismatch)
I20260709 12:22:54.420980 27587 pending_rounds.cc:85] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 20
I20260709 12:22:54.421314 27587 pending_rounds.cc:107] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 36.21
I20260709 12:22:54.421651 27587 raft_consensus.cc:2889] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:54.414048 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:54.410125  6148 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:22:54.289283  6241 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.424566 27664 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:54.437325 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1640) took 2009 ms. Trace:
I20260709 12:22:54.437889 28329 rpcz_store.cc:276] 0709 12:22:52.427990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.428151 (+   161us) service_pool.cc:224] Handling call
0709 12:22:52.428479 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.180959 (+1752480us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.436678 (+255719us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54005a00d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:54.437083 (+   405us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3966622464}
I20260709 12:22:54.440263  6513 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:54.442314  6390 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:54.443871  6244 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:54.444576  6244 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:54.443404  6390 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.446398  6244 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.452345 27658 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:22:54.453408 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:54.427836  6611 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:54.427276 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:54.460125  6611 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.462142  6611 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:54.463413  6450 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:54.459661  6390 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:54.468601 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111470us
I20260709 12:22:54.469611  6532 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:54.470095  6532 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.470381  6241 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.471493  6496 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:54.472474  6372 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:54.472926  6372 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:54.470486  6532 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:54.479884 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.486809  6519 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:54.486987  6448 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:54.487759  6519 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 32.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:22:54.488420 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1773) took 1001 ms. Trace:
I20260709 12:22:54.488898 27733 rpcz_store.cc:276] 0709 12:22:53.486524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.486986 (+   462us) service_pool.cc:224] Handling call
0709 12:22:54.487951 (+1000965us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540029fc54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:54.488160 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059669376}
W20260709 12:22:54.489339 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62666.4us
I20260709 12:22:54.490764  6619 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:22:54.490988 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:54.471004 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:54.474416  6372 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.504868  6390 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:54.505411 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.506601 27583 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:54.506753  6390 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:54.507505  6390 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:54.512234 27658 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:22:54.512864  6241 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.509183  6390 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.513928  6241 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.517089  6241 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.518460 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.523262 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.529793 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.497741 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:54.560880 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:54.564158 27658 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:54.565943 28335 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 19
I20260709 12:22:54.566371 28335 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 32.20
I20260709 12:22:54.566669 28335 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:54.575455 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.575656 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.583714 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:54.595409  6133 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:22:54.603693 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 680346us
I20260709 12:22:54.604242 28333 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
W20260709 12:22:54.606137 28333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1625) took 3718 ms (client timeout 2239 ms). Trace:
W20260709 12:22:54.606582 28333 rpcz_store.cc:269] 0709 12:22:50.887392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.892561 (+  5169us) service_pool.cc:224] Handling call
0709 12:22:54.605861 (+3713300us) spinlock_profiling.cc:243] Waited 3.24 s on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:54.606028 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7459131648}
W20260709 12:22:54.613696 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 354992us
I20260709 12:22:54.615962 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.618391  6244 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:54.619886  6510 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:54.620330  6510 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:54.621542  6510 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.623147  6510 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:54.629817  6531 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.630003 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:54.631067 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67429.5us
I20260709 12:22:54.630731  6531 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.632634 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:54.633845 27664 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:54.635417 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.635557  6644 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:54.635903  5884 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.638262  5884 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.638500 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.639539 27582 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:54.642578  6504 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:22:54.635583 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:54.652895  6454 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:22:54.655757 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 164974us
I20260709 12:22:54.656486 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.657300 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:54.658663 27659 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:22:54.659049  6133 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:54.657575 27655 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:54.661362  6482 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.661999  6482 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.663916  6428 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:54.664407  6428 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.664821  6428 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:54.664916  6460 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:54.665424  6460 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:54.679013 27664 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:54.682600  6374 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.683302  6374 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.683967  6517 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:22:54.685995  6569 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:54.686383  6569 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:54.687525  6569 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.688809  6474 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.690865 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.693490  6541 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.694500  6390 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:54.695461  6241 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:54.696648 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4424) took 2147 ms. Trace:
I20260709 12:22:54.700270 27481 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 30, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 30 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:54.701642  6482 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:54.702191 27481 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 35. New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:54.703552 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:54.706096  6415 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:54.706575  6415 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:54.706776  6372 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:54.707011  6415 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:54.707211  6271 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:22:54.707962  6413 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:54.708402  6413 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:54.708526  6271 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:54.709069  6271 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:54.709518  6413 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.711202  6271 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.714421  6515 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.716673 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46140 (request call id 9) took 3986 ms. Trace:
I20260709 12:22:54.724300 28332 rpcz_store.cc:276] 0709 12:22:50.730247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.743389 (+ 13142us) service_pool.cc:224] Handling call
0709 12:22:54.655194 (+3911805us) spinlock_profiling.cc:243] Waited 3.44 s on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34e735 00007f0e9a3469fa 00007f0e9f30e4c9 00007f0e9888ac54 00007f0e9888abe2 00007f0e9888ab11 00007f0e9888aa8c 00007f0e98889d42 00007f0e9868348c 00007f0e986828c6
0709 12:22:54.716447 (+ 61253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7906727168}
I20260709 12:22:54.742039  6413 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:22:54.742374 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62389.2us
I20260709 12:22:54.743044 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:54.743687  6585 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.744457  6585 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.745826  6579 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:54.746490  6579 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:54.746749  6585 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:54.747524 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.747895 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.748291  6401 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.749044  6401 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.750550  6531 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:54.751894 27585 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:22:54.697042 27738 rpcz_store.cc:276] 0709 12:22:52.548948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.558523 (+  9575us) service_pool.cc:224] Handling call
0709 12:22:52.558877 (+   354us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:52.746810 (+187933us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:22:53.772434 (+1025624us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.043087 (+270653us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.043263 (+   176us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:54.046082 (+  2819us) log.cc:844] Serialized 52 byte log entry
0709 12:22:54.046264 (+   182us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:22:54.050948 (+  4684us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.493084 (+442136us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b180033c760. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:54.571631 (+ 78547us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b180033c760. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:54.689485 (+117854us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b180033c760. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9e21c58f 00007f0e9725b571 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd
0709 12:22:54.695729 (+  6244us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.696013 (+   284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.696154 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.696204 (+    50us) raft_consensus.cc:1685] finished
0709 12:22:54.696218 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.696422 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2279652,"prepare.queue_time_us":1753733,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":3712,"raft.queue_time_us":662302,"raft.run_cpu_time_us":635,"raft.run_wall_time_us":91625,"spinlock_wait_cycles":61012864,"thread_start_us":568931,"threads_started":3,"wal-append.queue_time_us":375162}
I20260709 12:22:54.754171 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.754400 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.754752  6415 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.755097 27585 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:54.755232 27587 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
W20260709 12:22:54.756605 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62833.3us
I20260709 12:22:54.666620  6460 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:54.784782 27480 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 237) took 71686 ms (client timeout 15000 ms). Trace:
I20260709 12:22:54.762610 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.796582 27730 raft_consensus.cc:1240] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:54.748193  6374 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.763034  6515 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.798038 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:54.798358 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.799690  6496 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:54.800127  6496 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.800727  6496 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:54.716456  6271 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:54.766283 27481 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 31, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 31 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:54.767710  6532 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.774186 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:54.774550 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:54.775024  6223 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.775727  6655 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.775908  6482 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.776510  6654 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.776612  5967 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.827512 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:22:54.777671  6652 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:22:54.775746  6649 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.779744  6415 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:54.779675  6616 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.775146  6569 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:54.780499  6220 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.759783  6579 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.784034  6385 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:22:54.786252 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96463.6us
I20260709 12:22:54.759756 28331 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:54.787685 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:54.788740  6522 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.789745  6460 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:54.790668 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:54.792076 27701 leader_election.cc:336] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.506s (SENT)
I20260709 12:22:54.800240 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.807420 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:54.799989 27584 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:54.679590  6426 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.829118 28335 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:54.830801 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:54.843663 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:54.832402 27481 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 29 to 32, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:54.852943 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55031.5us
I20260709 12:22:54.824836  6585 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.830751  6413 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:54.842675  6401 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:54.852973  6515 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:22:54.879338  6515 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:54.879451  6426 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.880015  6515 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:54.881888  6426 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:54.883637  6649 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.886510 27737 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:22:54.887400  6522 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.887669  6585 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.888383  6385 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.888926  6385 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:54.890045  6585 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:54.890654  6585 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:54.900357 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:54.875417 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132154us
I20260709 12:22:54.876044  6644 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:22:54.875823 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4aa1523b9af84c3db99ba394e2b9aa9a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:54.891549 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:54.902401 27658 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:54.848369  6223 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.857267  6611 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.873065  6616 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.834129  5967 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.902001 27730 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:54.894738 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:54.913108 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:54.970782 27472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95202.5us
I20260709 12:22:54.915066 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:54.973954  6568 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:54.974340 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 200294us
I20260709 12:22:54.890060  6522 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.862536  6482 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:54.975299  6616 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:54.975620 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:54.976691 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:54.977097  6482 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:54.977772  6482 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:54.978133  6482 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.978379  6426 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:22:54.978538 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 204064us
I20260709 12:22:54.978267 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:54.974576  6415 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:54.979974 27730 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:54.983095 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:54.903438  6644 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:54.997097 28329 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:22:54.978850  6426 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.978977 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:55.003309 28332 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:55.000427 27659 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:54.909444  6428 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.007855  6504 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.004s
I20260709 12:22:55.012260  6632 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.012948  6632 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.013774  6428 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.003579  6648 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:55.004293  6566 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:22:55.031764  6415 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.032738  6611 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.033983  6415 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.039832  6532 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.040432 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:55.041160 27735 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
W20260709 12:22:55.041925  6016 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.888s	user 0.011s	sys 0.007s
I20260709 12:22:55.042385  6016 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.044564  6016 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:54.867568  6655 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.048806  6655 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.048943 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:54.970593  6450 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.048s
I20260709 12:22:55.054011  6428 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.054309  6647 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:55.007434  6650 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.869647  6220 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.054843 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.054024  6450 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.055756 27582 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:55.008352 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:55.019902 27733 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:55.020401 27734 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:22:55.020520  6646 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.337s	user 0.000s	sys 0.003s
I20260709 12:22:55.057025  6220 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.021020  6632 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:55.021273 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 229380us
I20260709 12:22:55.055311  6650 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.055858  6450 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.896209  6654 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:54.996789  6644 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:55.056859 27737 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:54.978567  6482 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:22:55.070508 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.074821 28330 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:22:55.015097  6426 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:55.078940 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:55.054811  6647 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:55.011082  6649 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.054207 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 7274fd8a2115405c9b78c61214bff736 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:51028
W20260709 12:22:55.083473 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110324us
I20260709 12:22:55.093277 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.077332  5967 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.054793  6428 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.104162  6223 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.104065  6647 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.104878 27702 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:55.117733 27702 leader_election.cc:403] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:22:55.107424 27735 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:22:55.109052  6654 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.000495  6585 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.117515 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.110939  6650 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.125959 27481 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 29 to 35. New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:55.130445 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:55.126861 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119518us
I20260709 12:22:55.133014 27730 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:55.134692 27732 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:22:55.137959  6197 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:55.139636 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:55.128703 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107915us
I20260709 12:22:55.131516 27737 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
W20260709 12:22:55.145865 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 174256us
W20260709 12:22:54.798375 27480 rpcz_store.cc:269] 0709 12:21:43.098624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.103493 (+  4869us) service_pool.cc:224] Handling call
0709 12:22:54.784512 (+71681019us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:46.813018 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:46.996982 (+183964us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:47.035507 (+ 38525us) write_op.cc:435] Acquired schema lock
0709 12:22:47.036974 (+  1467us) tablet.cc:662] Decoding operations
0709 12:22:47.127833 (+ 90859us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:47.128033 (+   200us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:47.128083 (+    50us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:22:47.131274 (+  3191us) tablet.cc:701] Row locks acquired
0709 12:22:47.131357 (+    83us) write_op.cc:260] PREPARE: finished
0709 12:22:47.141350 (+  9993us) write_op.cc:270] Start()
0709 12:22:47.162183 (+ 20833us) write_op.cc:276] Timestamp: P: 1783599767137378 usec, L: 0
0709 12:22:47.162229 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:22:47.163119 (+   890us) log.cc:844] Serialized 3610 byte log entry
0709 12:22:47.401849 (+238730us) op_driver.cc:464] REPLICATION: finished
0709 12:22:47.743480 (+341631us) write_op.cc:301] APPLY: starting
0709 12:22:47.743633 (+   153us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:47.856144 (+112511us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:47.857348 (+  1204us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:48.235098 (+377750us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:48.285589 (+ 50491us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:22:48.303061 (+ 17472us) write_op.cc:312] APPLY: finished
0709 12:22:48.428844 (+125783us) log.cc:844] Serialized 89 byte log entry
0709 12:22:48.452044 (+ 23200us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:48.452237 (+   193us) write_op.cc:454] Released schema lock
0709 12:22:48.452316 (+    79us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:48.535543 (+ 83227us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:48.535596 (+    53us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":495907,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":291909,"apply.run_cpu_time_us":18877,"apply.run_wall_time_us":759471,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41747,"num_ops":9,"prepare.queue_time_us":103026,"prepare.run_cpu_time_us":39736,"prepare.run_wall_time_us":450205,"raft.queue_time_us":79326,"raft.run_cpu_time_us":26222,"raft.run_wall_time_us":152403,"replication_time_us":239411,"thread_start_us":284854,"threads_started":4,"wal-append.queue_time_us":98905,"wal-append.run_cpu_time_us":13292,"wal-append.run_wall_time_us":2199859}]]}
I20260709 12:22:55.125641  6647 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:55.148456 27481 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 32 to 37, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 37 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:55.095600  6538 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.152s
I20260709 12:22:55.150581 27735 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 25.
I20260709 12:22:55.167495 27737 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:22:55.170353  6428 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.119155  6515 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.173837 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:55.175697 27735 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:55.173869  6271 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.181331  6271 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.183382 27735 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:55.185701 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.186854 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.148654  6650 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:55.189224 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.190160 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:55.190476 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.189101 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:55.163652  6223 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.194983 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.166967  6241 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:55.198319 27737 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 32 because replica has last-logged OpId of term: 29 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
I20260709 12:22:55.155031  6016 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:55.192893 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.611s (SENT) [suppressed 8 similar messages]
I20260709 12:22:55.205767  6271 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.206727 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:55.207615 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:55.208298 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46490us
I20260709 12:22:55.209021 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:55.199447 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.203711 27481 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 26 to 30. New cstate: current_term: 30 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:55.214871 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:55.216419 27661 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 26: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:55.214207 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:55.219080 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.219462 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.220009  6585 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.220216 27581 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:55.220695  6450 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.218031 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:55.223230 27664 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:55.191733  6670 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:55.224555 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.225183 27481 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 18 to 22, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 22 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:55.200531  6197 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:55.226903 27660 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:22:55.207230  6385 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.230907 27662 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:22:55.241887 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.236299 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:55.244380  6133 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:55.244518 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:55.222131  6569 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:22:55.247282 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 54413.9us
W20260709 12:22:55.244503 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 89007.9us
I20260709 12:22:55.252537 27589 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:22:55.245715 27481 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 30 to 40, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 40 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:55.204535  6223 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.257256 27585 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:55.233811  6197 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.262591 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.259033 27659 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:55.245970  6531 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:22:55.243274  6241 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:22:55.266978 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:55.244226  5884 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.268909 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:55.267832 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:55.211843  6390 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:22:55.230628  6016 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.274179 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:55.286736 27660 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
W20260709 12:22:55.274591 27701 leader_election.cc:336] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.734s (SENT)
I20260709 12:22:55.265405  6197 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:55.269362  6611 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:22:55.264722  6133 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:55.268000  6531 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.296518 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:55.301894 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98931.2us
I20260709 12:22:55.284441  6522 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:55.317950 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.282119  6450 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:55.345245 27661 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:55.344177 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:55.346879 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:55.300506  6666 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:22:55.350698 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:55.351305 27589 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:55.350832 27481 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 31 to 33, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 33 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:55.352805 27481 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 23 to 33. New cstate: current_term: 33 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:55.355268 27658 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:55.356725 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:55.359580 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:55.360745 27660 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:55.252454  6670 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.294291  6241 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:55.365006 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 7274fd8a2115405c9b78c61214bff736 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
W20260709 12:22:55.306983  6372 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.600s	user 0.003s	sys 0.010s
I20260709 12:22:55.305773  6531 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:22:55.379160 27471 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111357us
I20260709 12:22:55.381085 27658 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:55.362310  6611 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:55.301247  6654 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:22:55.303417  5884 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.373889  6585 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.410076 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:55.374830  6650 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:22:55.423507 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118716us
I20260709 12:22:55.428875 27589 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:55.415876  6515 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.432123 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:55.386683  6450 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:55.442224 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 179454us
I20260709 12:22:55.427361  6667 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:55.432164  6390 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.026s
I20260709 12:22:55.431398  6372 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:22:55.456475  6385 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.492453 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.492564 27735 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
W20260709 12:22:55.489261 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 376464us
I20260709 12:22:55.425030  6585 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.504347  6670 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.505054  6670 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:55.506450  5884 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.507737  6668 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.044s
W20260709 12:22:55.508438 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98553.4us
I20260709 12:22:55.509310 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:55.513794 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.514448 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.515450 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:55.515697 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.517071  6681 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:55.520064 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:55.521744  6665 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 12:22:55.500934 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:55.523277 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:55.524361 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.461971  6148 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:22:55.549149  6223 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.507200  6683 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:55.518872  6496 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.718s	user 0.014s	sys 0.000s
I20260709 12:22:55.445595  6671 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.236s
I20260709 12:22:55.558120 27660 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:55.560120 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.522471  6016 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.524793  6654 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:55.521955  6515 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.536594  6649 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:55.559660  6460 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.701s	user 0.009s	sys 0.004s
W20260709 12:22:55.572535 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71681us
I20260709 12:22:55.575734 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.555737  6650 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:55.462168  6133 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.493993  6241 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.489660  6611 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.588166 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:55.593060 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73833.9us
W20260709 12:22:55.603521 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100355us
I20260709 12:22:55.567199  6148 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:55.608477 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:55.555629  6385 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:22:55.620250 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.620945 27580 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:55.565140  6683 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.623967 27588 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:22:55.618392 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:55.569473  6644 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.635937 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:55.633373 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.566327  6496 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.637181 27583 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
I20260709 12:22:55.634357 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:55.588882  6016 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
I20260709 12:22:55.638723 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:55.615381  6566 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:55.620824  6223 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:55.605158  6482 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.593422  6515 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.656628 28337 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:22:55.658591 27588 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:22:55.649302  6569 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:55.650547  6686 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.670009 27734 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:55.673923 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.677667 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53546.6us
I20260709 12:22:55.671664 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:55.671394 28332 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:22:55.665962  6670 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:55.660679  6566 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:55.686935  5884 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 12:22:55.690196 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.668699  6650 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.705324 27588 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:55.705930 27627 leader_election.cc:400] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38.
I20260709 12:22:55.707193 27627 leader_election.cc:403] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
I20260709 12:22:55.708811 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:55.711108 27701 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.396s (SENT)
I20260709 12:22:55.711755 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:55.666311  6687 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:55.685931  6679 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:55.716727 27657 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:55.717222  6692 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:55.719408  6683 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.720196  6650 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.721884  6496 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.724241  6644 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.724593  6691 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:55.727720  6428 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:55.752996  6689 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.753089  6428 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:55.753834 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.753760  6689 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.754523 27585 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:55.756702  6683 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:22:55.758571  6372 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.758688  6691 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:55.759136  6372 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.759892  6691 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.760869  6482 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.761404  6372 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:55.729339  6426 consensus_meta.cc:287] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.000s
I20260709 12:22:55.747673 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:55.712445  6685 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:22:55.736681  6692 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:55.715191  6668 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:55.762717 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:22:55.768108 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 133841us
W20260709 12:22:55.756826  6669 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.535s	user 0.002s	sys 0.000s
I20260709 12:22:55.735775  5884 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38.
I20260709 12:22:55.736092  6679 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:55.779138 27583 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:55.688211  6686 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.754547  6272 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:55.788794 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:55.788271 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.780995 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:55.745414  6681 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.749835  6632 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:55.789475  6569 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:55.766041  6532 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.768126  6685 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:55.809957  6532 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.810585  6685 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:55.812268 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 117283us
I20260709 12:22:55.714169  6148 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.812608  6532 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.818787  6522 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:55.819419  6522 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:55.822201  6659 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.799s	user 0.002s	sys 0.000s
I20260709 12:22:55.765884  5967 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:22:55.824329  6515 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.753378  6566 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.823119  6568 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.778017  6692 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:55.828035 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.829035 28330 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
I20260709 12:22:55.806630 27587 raft_consensus.cc:2368] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
W20260709 12:22:55.792177  6647 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.627s	user 0.011s	sys 0.000s
W20260709 12:22:55.854171 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47842.9us
I20260709 12:22:55.807489 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:55.762537  6670 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.776993  6401 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.759889  6385 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:55.791858  6679 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:55.802109  6644 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:55.857231 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.777287  6426 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.866786 28337 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:55.866613 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61655.7us
I20260709 12:22:55.867624 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.878259 27584 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:55.854893  6685 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:22:55.884963 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:55.856895 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 144940us
I20260709 12:22:55.793956  6272 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:55.868891  6668 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:22:55.855252  6686 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.870648  6644 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:55.858351  6522 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.880348  6401 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.884330  6532 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.012s
W20260709 12:22:55.896481  6197 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.552s	user 0.006s	sys 0.005s
I20260709 12:22:55.871620  6385 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.890044  6689 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.863272  6681 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.896018  6644 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:55.883831  6415 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.017s
I20260709 12:22:55.878201  6510 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.022s
W20260709 12:22:55.931996 27661 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.001s
I20260709 12:22:55.932677 27661 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:55.871814  6568 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.901854  6585 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.924320  6686 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:55.952301  6701 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:22:55.953131 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65595.2us
W20260709 12:22:55.943154  6581 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.007s	sys 0.007s
W20260709 12:22:55.955538  6581 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.008s	sys 0.007s
I20260709 12:22:55.955729  6698 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:55.956208  6390 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:55.956517  6698 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:55.957203  6698 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:55.957628  6426 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.946803  6272 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:55.960361  6272 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:55.967587 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:55.971889  6390 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:55.972640  6390 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.975847 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:55.976783  6704 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:22:55.978189  6271 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:55.978858  6568 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.903275  6531 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.980322  6670 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.981865  6531 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:55.956602  6650 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:55.984784  6415 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:22:55.986140  6686 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:55.987381 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:55.987722  6696 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:22:55.975716  6697 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.013s
I20260709 12:22:55.989496  6390 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.993137 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:55.996935  6686 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.997545  6271 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:55.923692  6504 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.026s
I20260709 12:22:56.000823  6385 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.001368  6699 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.067s
I20260709 12:22:56.002172  6699 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 29.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:22:55.994524  6698 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.004599  6401 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.005208 27587 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:56.006311  6698 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.007122 27587 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:55.924769  6197 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.000810 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:56.015280 27580 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:22:55.945125  6482 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:56.019587 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 149415us
I20260709 12:22:56.029408 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.030248 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
W20260709 12:22:56.031260  6682 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.408s	user 0.000s	sys 0.001s
I20260709 12:22:56.032166 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.034063 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.034170 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:56.034868 27588 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:56.040112 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:56.041611  6702 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:56.042001  6702 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:56.042439 27588 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
W20260709 12:22:56.043015  6707 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:22:56.042506  6702 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:56.045094 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.045965  6697 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:56.046782  6697 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:56.047528  6697 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:56.047847  6697 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.048166  6697 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:56.051659 28330 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:56.058265  6697 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.061126  6702 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:56.062655  6705 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.137s	user 0.001s	sys 0.002s
I20260709 12:22:56.014020  6197 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:56.083555 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.114511 27660 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:22:56.116478 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:55.993244  6681 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:55.924422  6569 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.125169 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.125921 27659 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:22:56.146126 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 161342us
I20260709 12:22:56.147414 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:56.128859  6687 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:56.025910  6482 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.160516 27659 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:56.139851  6611 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:56.153506  6687 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:56.208482 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61204.3us
I20260709 12:22:56.154013  6413 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.053s
I20260709 12:22:56.215408 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.216094 27659 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:56.221606  6375 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:22:56.221784 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.223192 27735 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:22:56.223609 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.224787 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.225414 27655 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:56.208256  6687 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:56.225813  6482 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.226327 27660 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:22:56.213150  6454 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:56.227913  6454 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.230093 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.237905  6197 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:56.238715  6710 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.239192 27660 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:56.239308  6710 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.240480 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.240864 27704 leader_election.cc:400] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:56.241183 27704 leader_election.cc:403] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:22:56.241142 27660 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:56.241691  6710 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.242033  5967 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:56.244240 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.245342 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.246407 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.247092 27663 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:56.239651  6197 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:22:56.247970 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:56.249042 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.250028 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:56.250557 27589 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:56.252215 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.253322 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.254220 28332 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:22:56.255985 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.256759 27729 raft_consensus.cc:2368] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:22:56.257687 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.258452 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:56.259070 28336 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:22:56.259291 27588 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 38 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:22:56.260097 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.260200 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.260768 28328 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:22:56.261050 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:56.261401 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.261634 27588 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:56.259536  6538 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:56.250512 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:56.261968 28336 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:22:56.251179 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:56.282109 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.282655  6197 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:22:56.282761 27731 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:56.242466 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.283252 27661 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:56.288884 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:56.287472 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.288395  6450 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.265827  6709 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.298480  6454 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:56.265676  6717 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:22:56.263811  6708 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:22:56.302469  6244 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:56.302901  6244 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:56.303141 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.303857 27732 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:56.304764 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.305367 27732 raft_consensus.cc:2368] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34.
I20260709 12:22:56.306051 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.307890 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3810) took 1121 ms. Trace:
I20260709 12:22:56.308269 27657 rpcz_store.cc:276] 0709 12:22:55.185965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.193755 (+  7790us) service_pool.cc:224] Handling call
0709 12:22:55.206567 (+ 12812us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.736464 (+529897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.307666 (+571202us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b5400290c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:56.307814 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1032542592}
W20260709 12:22:56.311806  6706 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.356s	user 0.001s	sys 0.002s
I20260709 12:22:56.311915 27584 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:22:56.313200 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3969) took 1160 ms. Trace:
I20260709 12:22:56.313674  6675 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:22:56.313596 27730 rpcz_store.cc:276] 0709 12:22:55.152570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.156088 (+  3518us) service_pool.cc:224] Handling call
0709 12:22:55.156405 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.768771 (+612366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.768818 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.768879 (+    61us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:55.768953 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.312936 (+543983us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400317c54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:56.312986 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.313111 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393627008}
I20260709 12:22:56.314687  6675 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:22:56.315088  6675 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:56.315212  6244 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.316620  6244 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:56.315960  6675 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:56.317178  6718 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.061s	user 0.002s	sys 0.000s
I20260709 12:22:56.317870  6675 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:22:56.317938 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.266129  6538 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:56.319442  6538 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.321035  6538 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:56.271337 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:56.321946  6714 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.090s	user 0.000s	sys 0.002s
I20260709 12:22:56.322293  6691 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:22:56.272768 28335 raft_consensus.cc:3060] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:56.322914  6691 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:56.268392 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:56.324000  6691 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:56.325435  6700 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.423s	user 0.001s	sys 0.000s
W20260709 12:22:56.325695 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.326476 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 3977) took 1093 ms. Trace:
I20260709 12:22:56.326835 27734 rpcz_store.cc:276] 0709 12:22:55.232932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.306883 (+ 73951us) service_pool.cc:224] Handling call
0709 12:22:56.326316 (+1019433us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:56.326418 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":718944512}
W20260709 12:22:56.327670 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56452.3us
I20260709 12:22:56.298107  6709 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.328724 28331 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:56.329780  6709 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.271724  6579 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:56.330998 28334 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:22:56.330965  6579 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.331535  6372 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:56.331686  6016 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:22:56.332085  6372 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.333228  6579 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.333690 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.333897 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:56.278924  6644 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.334343 27660 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:56.334743  6016 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:56.335309  6016 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.335671  6016 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:56.336031  6644 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.280654  6702 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.291601 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:22:56.319466 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.338241 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:56.275452 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.338920 27664 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:56.338510 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.285032 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:56.339936 27734 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 25: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:56.339202 27733 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:56.340318 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.290779 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.280685 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.337147 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.340454 28335 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:56.343091 27730 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:22:56.274808 27589 raft_consensus.cc:2468] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:22:56.345512 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:56.273942 27588 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:56.283077  6197 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.350071  6197 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:22:56.344244 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:56.352192 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67271.4us
I20260709 12:22:56.348110 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:56.295543 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.360963 27588 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:56.343171 27731 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:22:56.364533 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.365610 27731 raft_consensus.cc:2368] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34.
I20260709 12:22:56.414299 27664 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:22:56.385917 27735 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:56.414752 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:56.419103 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:56.419633 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77925.6us
I20260709 12:22:56.420604 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.426120 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.426146 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.428165 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:56.415091 27704 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:56.420560 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3973) took 1260 ms. Trace:
I20260709 12:22:56.437422 28331 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:56.434357 27704 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:22:56.406543 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111033us
W20260709 12:22:56.521564 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 101065us
I20260709 12:22:56.523674 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.523083 27731 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:56.441529 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.440961 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.462080 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.529436 27735 raft_consensus.cc:2368] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40.
I20260709 12:22:56.531934 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:56.532254 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52856.7us
I20260709 12:22:56.415602  6450 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:56.435094 27737 rpcz_store.cc:276] 0709 12:22:55.160267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.227106 (+ 66839us) service_pool.cc:224] Handling call
0709 12:22:55.227494 (+   388us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:55.228221 (+   727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.228265 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:55.233332 (+  5067us) log.cc:844] Serialized 52 byte log entry
0709 12:22:55.434106 (+200774us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:55.496223 (+ 62117us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.520876 (+ 24653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.521006 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.270605 (+749599us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b180031dd40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:56.340849 (+ 70244us) raft_consensus.cc:1685] finished
0709 12:22:56.420303 (+ 79454us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.420488 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":127833,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":23729,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":14732,"spinlock_wait_cycles":26320128,"thread_start_us":188149,"threads_started":2,"wal-append.queue_time_us":329396}
I20260709 12:22:56.531559 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.271683  6681 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:56.543893 27734 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:22:56.417757  6460 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:56.524062 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.211462  6611 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.277276  6692 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.572499 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.573580 27661 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:56.542201 27730 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:22:56.479475  5967 raft_consensus.cc:2804] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:56.550196 27656 raft_consensus.cc:2368] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:56.532878  6686 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.275915  6679 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.429320  6454 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:22:56.385707  6667 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:56.546527  6450 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.541785  6522 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.532879  6372 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.542546  6415 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:22:56.583062 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59055.9us
W20260709 12:22:56.585134 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61533us
I20260709 12:22:56.556530  6401 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.558112  6460 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:56.592759 27583 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:56.566648  6579 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.577988  6617 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:56.619272 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.620649 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.277282  6428 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.623692 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:56.631963 28292 leader_election.cc:400] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34.
I20260709 12:22:56.587180  6569 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:56.557605  6681 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.589260  6519 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:56.595459  6450 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:56.643791  6689 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:22:56.659135 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.622131  6697 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.619164  6415 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:56.630857  6460 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:56.639830  6569 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.631228  6667 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:56.679765 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.679240 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.658062  6689 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.676889  6415 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:56.683564  6689 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:56.661070  6375 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:22:56.601178  6686 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:56.685595  6721 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:22:56.608076  6454 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:22:56.686949  6720 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:22:56.666163  6687 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.623569  6522 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.661939  6428 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:56.681068 27656 raft_consensus.cc:2376] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29. Re-sending same reply.
W20260709 12:22:56.693337  6716 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:22:56.691229 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.643937  6519 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:56.704418  6712 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.315s	user 0.001s	sys 0.000s
I20260709 12:22:56.632251  6372 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:22:56.705135  6569 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:56.708544 27661 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
W20260709 12:22:56.708849  6722 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:22:56.709491  6686 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.710487 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:56.651191  6617 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:56.758585  6692 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.760092 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.760391  6522 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:56.761386  6681 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:56.630532  6401 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:56.769446 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:56.847818 27729 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:22:56.764875 27655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.013s	sys 0.009s
I20260709 12:22:56.773929 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:56.770556 27656 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
W20260709 12:22:56.848524 27655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.014s	sys 0.009s
I20260709 12:22:56.711977  6454 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.712705  6687 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:56.644375  6579 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:56.745786  6519 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:56.710541  6667 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:56.847355  6244 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.530s	user 0.012s	sys 0.001s
I20260709 12:22:56.852203 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:56.841318 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 130907us
I20260709 12:22:56.669876  6449 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.421s
I20260709 12:22:56.806070  6697 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:56.864264 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.864454 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.885901 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.866880 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:22:56.892810 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.893484 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.896127 27586 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.557s	user 0.010s	sys 0.000s
I20260709 12:22:56.896656 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
W20260709 12:22:56.862838  6722 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:56.907017 27655 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:22:56.925529 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.931773 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:56.933642 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.935933 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.939132 27656 raft_consensus.cc:2368] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:56.936759 27662 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:56.932968  6541 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:22:56.981266 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:56.984067 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:56.987449 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:56.981966 27659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.008s	sys 0.018s
W20260709 12:22:56.990350 27659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.009s	sys 0.018s
I20260709 12:22:56.995525 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.938571  5967 raft_consensus.cc:697] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:22:56.986825 27660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.000s	sys 0.027s
I20260709 12:22:57.006469 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:57.007426 27656 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
W20260709 12:22:57.008277 27660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.001s	sys 0.027s
I20260709 12:22:56.942453  6611 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:22:56.917085  6697 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:57.011243 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:22:57.011641 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:57.012600 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 388951us
I20260709 12:22:57.013474 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.013271 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:57.014763 27738 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:56.977429  6679 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.020442 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.024488 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:56.987026  6541 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:57.026252 27661 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
W20260709 12:22:57.036422 28332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.002s	sys 0.039s
W20260709 12:22:57.036916 28332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.002s	sys 0.039s
I20260709 12:22:57.037709 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:57.038038 27733 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.608s	user 0.007s	sys 0.003s
W20260709 12:22:57.039923 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.004s	sys 0.036s
W20260709 12:22:57.040472 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 606173us
I20260709 12:22:57.051787 27733 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:57.046274 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.054100 28337 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:57.055513 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.056267 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:22:57.065302 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.070971 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.031675  6611 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.068122 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:57.075584 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:57.035717  6522 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.061357  6713 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:57.082187 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.082283  6522 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.082880 27737 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:57.083912 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:57.067226  6675 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.749s	user 0.014s	sys 0.000s
I20260709 12:22:57.079829 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.077435  6739 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.016s
W20260709 12:22:57.084652  6538 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.763s	user 0.011s	sys 0.000s
I20260709 12:22:57.120376 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.086975  6375 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:22:57.128782  6733 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
I20260709 12:22:57.088794 27662 raft_consensus.cc:2368] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:57.137218 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.149384 27627 leader_election.cc:400] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:22:57.162108 27627 leader_election.cc:403] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:57.162400 27659 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:57.077870  6372 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.046s
I20260709 12:22:57.087793  6611 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.128165  6496 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:22:57.165628 27656 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:22:57.243059 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93854.8us
I20260709 12:22:57.246105 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.155267  6375 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:57.259286 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 106740us
I20260709 12:22:57.119225  6415 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.131744  6401 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.249389 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.237752 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:57.254586 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.154978  6679 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:57.267890 27737 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:22:57.268826 27662 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:22:57.266013 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 211513us
I20260709 12:22:57.269820 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.270033 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:57.270512 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.262766 27729 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
W20260709 12:22:57.266070 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 123563us
W20260709 12:22:57.249135  6723 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.321s	user 0.001s	sys 0.000s
I20260709 12:22:57.276929 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.278723 27729 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:22:57.280004 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.260808  6579 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:57.281483 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:57.263273  6729 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
W20260709 12:22:57.265612  6731 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260709 12:22:57.277690 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:57.263917  6735 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
W20260709 12:22:57.264503  6754 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
W20260709 12:22:57.281157  6730 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.337s	user 0.001s	sys 0.000s
I20260709 12:22:57.284801 27627 leader_election.cc:400] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40.
I20260709 12:22:57.123538  5967 consensus_queue.cc:237] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.317660  6738 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:22:57.317727 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.318497  6738 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:57.319155 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.319484  6738 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.320355 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.321695 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:56.951290  6617 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.322528 27662 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
I20260709 12:22:57.266528  6375 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:57.324267  6753 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:22:57.263295  6332 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.325246  6332 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.325692 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 1966) took 1031 ms. Trace:
I20260709 12:22:57.326045 27660 rpcz_store.cc:276] 0709 12:22:56.294194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.335002 (+ 40808us) service_pool.cc:224] Handling call
0709 12:22:56.335307 (+   305us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:56.335650 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.335677 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:56.336249 (+   572us) log.cc:844] Serialized 52 byte log entry
0709 12:22:57.009475 (+673226us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:57.009528 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.009627 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.325476 (+315849us) raft_consensus.cc:1685] finished
0709 12:22:57.325498 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.325637 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1256,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"thread_start_us":672803,"threads_started":1,"wal-append.queue_time_us":785263}
W20260709 12:22:57.273855  6732 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260709 12:22:57.263624  6538 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:22:57.281569  6726 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.375s	user 0.000s	sys 0.005s
I20260709 12:22:57.297931  6764 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.340619  6764 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.315047 27729 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: [30.17-32.18]
I20260709 12:22:57.265842  6496 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:57.268868  6737 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:57.327809 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:57.297792 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.306871 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:57.328992 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59039.4us
I20260709 12:22:57.362573 27659 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:57.314940  6687 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.284471  6681 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:22:57.332099  6724 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20260709 12:22:57.276889  6743 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:57.342644 27655 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:22:57.344748 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.298236  6566 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:57.203125  6541 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:57.355422  6460 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.672s	user 0.010s	sys 0.002s
I20260709 12:22:57.355870  6761 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:57.358500 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4733) took 1090 ms. Trace:
I20260709 12:22:57.277536  6679 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.453097 27583 rpcz_store.cc:276] 0709 12:22:56.267677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.268731 (+  1054us) service_pool.cc:224] Handling call
0709 12:22:56.269066 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.631032 (+361966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.358223 (+727191us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54000bd6d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:57.358413 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":743875968}
I20260709 12:22:57.362895 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:22:57.457481 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94706.9us
I20260709 12:22:57.362948 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.365098  6496 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:57.366688  6744 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:22:57.369057  6531 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:22:57.373121  6569 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.565s	user 0.009s	sys 0.003s
I20260709 12:22:57.373629 28292 leader_election.cc:400] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34.
I20260709 12:22:57.489228 28292 leader_election.cc:403] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:22:57.375702 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3837) took 1032 ms. Trace:
I20260709 12:22:57.338039  6750 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.387392 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.383832  6697 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.369009  6713 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:57.394292 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1672) took 1035 ms. Trace:
W20260709 12:22:57.394960  6759 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.129s	user 0.002s	sys 0.001s
I20260709 12:22:57.396528  6415 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.399610 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3821) took 1183 ms. Trace:
W20260709 12:22:57.401453  6428 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.712s	user 0.010s	sys 0.001s
I20260709 12:22:57.403564 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3831) took 1167 ms. Trace:
I20260709 12:22:57.404393 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.404332 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.412573 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:57.429345 27737 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:57.511445 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.330323 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.506003 27658 rpcz_store.cc:276] 0709 12:22:56.215847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.215959 (+   112us) service_pool.cc:224] Handling call
0709 12:22:56.221872 (+  5913us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.222266 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.222297 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.222972 (+   675us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.566537 (+343565us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:56.566577 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.566655 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.399390 (+832735us) raft_consensus.cc:1685] finished
0709 12:22:57.399412 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.399550 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3985,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":343164,"threads_started":1,"wal-append.queue_time_us":596360}
W20260709 12:22:57.431964 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87331.4us
I20260709 12:22:57.510748 28333 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:22:57.391726  6450 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.712s	user 0.010s	sys 0.005s
I20260709 12:22:57.339718  6332 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.437997 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:57.518769  6197 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.168s	user 0.011s	sys 0.003s
I20260709 12:22:57.442502 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3990) took 1004 ms. Trace:
W20260709 12:22:57.431964  6016 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.096s	user 0.013s	sys 0.000s
I20260709 12:22:57.435000  6644 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:57.445583  6541 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:57.519584  6016 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.456398  6460 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.447726  6611 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:22:57.447865  6689 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.764s	user 0.011s	sys 0.001s
I20260709 12:22:57.448132  6375 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.450394  6522 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:57.464876 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:57.468828 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.452908  6761 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:57.493269 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119811us
I20260709 12:22:57.527019 27660 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:57.489987 27664 rpcz_store.cc:276] 0709 12:22:56.342816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.425840 (+ 83024us) service_pool.cc:224] Handling call
0709 12:22:57.375522 (+949682us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54000e6fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:57.375624 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1043125120}
I20260709 12:22:57.446004  6764 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.538846 28337 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:57.505316 27661 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:57.430913  6687 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.513553  6745 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.501281 28329 rpcz_store.cc:276] 0709 12:22:56.359264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.359424 (+   160us) service_pool.cc:224] Handling call
0709 12:22:56.426679 (+ 67255us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:56.484498 (+ 57819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.524305 (+ 39807us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:56.556374 (+ 32069us) log.cc:844] Serialized 52 byte log entry
0709 12:22:56.854586 (+298212us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:22:56.854632 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.854709 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.393985 (+539276us) raft_consensus.cc:1685] finished
0709 12:22:57.394017 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.394213 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":109,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":80478,"thread_start_us":261909,"threads_started":1,"wal-append.queue_time_us":478109}
I20260709 12:22:57.513746 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:57.515686 27737 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:22:57.516897 28334 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:57.508697 27657 rpcz_store.cc:276] 0709 12:22:56.235990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.334767 (+ 98777us) service_pool.cc:224] Handling call
0709 12:22:56.335090 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.335486 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.335516 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.336021 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.546172 (+210151us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:56.546217 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.546278 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.403367 (+857089us) raft_consensus.cc:1685] finished
0709 12:22:57.403392 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.403510 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1201,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":209773,"threads_started":1,"wal-append.queue_time_us":504345}
I20260709 12:22:57.548780 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.549183 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.549964 27657 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:22:57.550293 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.551597  6566 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:57.552448  6566 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:57.554062 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:57.557067 27581 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:57.446004 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.494141  6569 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.519434 27735 rpcz_store.cc:276] 0709 12:22:56.438176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.544634 (+106458us) service_pool.cc:224] Handling call
0709 12:22:56.558389 (+ 13755us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.880774 (+322385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.880811 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.917127 (+ 36316us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.123808 (+206681us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:57.123886 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.124030 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.442288 (+318258us) raft_consensus.cc:1685] finished
0709 12:22:57.442308 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.442448 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34136,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":27356,"thread_start_us":440808,"threads_started":2,"wal-append.queue_time_us":456624}
I20260709 12:22:57.519281  6197 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.524534 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:57.522660 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.490726  6737 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:57.540809 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.561167 27658 raft_consensus.cc:2410] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:22:57.561757 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4556) took 1085 ms. Trace:
I20260709 12:22:57.562161 27658 rpcz_store.cc:276] 0709 12:22:56.475912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.540489 (+1064577us) service_pool.cc:224] Handling call
0709 12:22:57.561684 (+ 21195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.539808  6332 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:22:57.531147 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:57.460999  6757 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:57.557040 27664 raft_consensus.cc:1217] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Deduplicated request from leader. Original: 33.21->[35.22-35.22]   Dedup: 35.22->[]
I20260709 12:22:57.571905 27664 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:22:57.572463 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4544) took 1148 ms. Trace:
I20260709 12:22:57.572764 27664 rpcz_store.cc:276] 0709 12:22:56.424342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.556621 (+1132279us) service_pool.cc:224] Handling call
0709 12:22:57.556922 (+   301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.572244 (+ 15322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.572413 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.573333 28333 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:22:57.545498 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.557740 27659 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:22:57.433730  6675 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:22:57.551628 27657 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:22:57.501165  6713 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:57.510946  6702 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34.
I20260709 12:22:57.576524 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3838) took 1233 ms. Trace:
I20260709 12:22:57.562697 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:57.505035  6750 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.521926  6450 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.582755  6742 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:57.582880 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.583381  6742 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:57.577067  6538 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:22:57.584452  6742 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.585853  6687 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.586638  6687 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:57.564695 27658 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:22:57.587806 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4546) took 1115 ms. Trace:
W20260709 12:22:57.588526  6736 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.570s	user 0.002s	sys 0.000s
I20260709 12:22:57.541750  6761 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:57.589382 27729 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: [28.23-31.24]
I20260709 12:22:57.590168 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.590817 27655 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:22:57.569208 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.591611 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3994) took 1034 ms. Trace:
I20260709 12:22:57.591711 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:57.592020 27729 rpcz_store.cc:276] 0709 12:22:56.557042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.348767 (+791725us) service_pool.cc:224] Handling call
0709 12:22:57.349185 (+   418us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:57.589771 (+240586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.591367 (+  1596us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54000a73d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:57.591551 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551714688}
I20260709 12:22:57.592423  6522 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:57.563377 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4563) took 1085 ms. Trace:
I20260709 12:22:57.579591 27657 rpcz_store.cc:276] 0709 12:22:56.343048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.550848 (+1207800us) service_pool.cc:224] Handling call
0709 12:22:57.551214 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.575745 (+ 24531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.576465 (+   720us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.593480  6460 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.593686 27662 rpcz_store.cc:276] 0709 12:22:56.478279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.522326 (+1044047us) service_pool.cc:224] Handling call
0709 12:22:57.563263 (+ 40937us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.577991 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3993) took 1034 ms. Trace:
I20260709 12:22:57.592407 27655 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 26.
I20260709 12:22:57.595710 27730 rpcz_store.cc:276] 0709 12:22:56.543304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.574448 (+ 31144us) service_pool.cc:224] Handling call
0709 12:22:56.574872 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.868161 (+293289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.868212 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.923534 (+ 55322us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.086401 (+162867us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:57.086437 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.086509 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.394217 (+307708us) spinlock_profiling.cc:243] Waited 899 us on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:57.577709 (+183492us) raft_consensus.cc:1685] finished
0709 12:22:57.577730 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.577927 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":89424,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":29563,"spinlock_wait_cycles":2068352,"thread_start_us":350658,"threads_started":2,"wal-append.queue_time_us":260125}
I20260709 12:22:57.597743 27655 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 22
I20260709 12:22:57.598030 27655 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 35.23
I20260709 12:22:57.598111 27730 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:22:57.598294 27655 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:57.598780  6197 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:57.581504 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.528465  6689 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.601876 27732 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 42: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:57.583427 27735 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:22:57.588364  6016 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.566066 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:22:57.580885 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122598us
I20260709 12:22:57.565017  6332 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:57.588172 27658 rpcz_store.cc:276] 0709 12:22:56.472694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.562975 (+1090281us) service_pool.cc:224] Handling call
0709 12:22:57.564410 (+  1435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.587511 (+ 23101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.587749 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.601195 27729 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: [37.21-37.21]
I20260709 12:22:57.607933 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.563117  6681 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:57.587083  6687 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:57.578032  6611 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:57.584342  6450 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.616581 27701 leader_election.cc:400] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:57.565553  6737 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.644122 27661 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:22:57.622548 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4449) took 1061 ms. Trace:
I20260709 12:22:57.624643  6569 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.585889  6750 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:57.648538 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 65734.1us
I20260709 12:22:57.644981 27733 rpcz_store.cc:276] 0709 12:22:56.561631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.402876 (+841245us) service_pool.cc:224] Handling call
0709 12:22:57.403212 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.403478 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.403502 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.403993 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.520794 (+116801us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:22:57.577273 (+ 56479us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.577546 (+   273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.577618 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.622282 (+ 44664us) raft_consensus.cc:1685] finished
0709 12:22:57.622309 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.622473 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":55491,"prepare.run_wall_time_us":31,"raft.queue_time_us":77,"spinlock_wait_cycles":24324736,"thread_start_us":116390,"threads_started":1,"wal-append.queue_time_us":174382}
I20260709 12:22:57.647429 27659 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:22:57.614073  6332 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:22:57.645138 27729 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:57.560024  6745 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.650424 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.652168 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.653321 27735 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:57.664165 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.662976 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.650806 27658 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:22:57.655073  6713 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.643443  6538 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:57.641206  6611 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.646392  6401 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.669405 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:57.666661 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.673035 27701 leader_election.cc:400] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36.
I20260709 12:22:57.680065 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.652362  6566 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.683136 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.679934 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.689677 27733 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:22:57.692010 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:57.692878 27659 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:57.667156  6482 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:57.671278  6531 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:57.671252  6569 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:57.675823  6617 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:22:57.676460  6611 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:57.678620  6450 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:57.694514  6579 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:22:57.694366  6710 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:57.696079  6738 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:57.698964  6750 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:57.701967  6569 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.701469  6531 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.707806  6713 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:57.693465  6401 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.713594  6579 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.714095  6710 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.725103  6531 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:57.694640  6566 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.714603  6375 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:57.726686 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 43659.3us
I20260709 12:22:57.748538 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.724542  6569 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:57.726583  6697 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:57.738355  6689 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.727844  6784 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:22:57.717378  6687 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.725929  6709 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.018s
W20260709 12:22:57.744325  6773 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:57.745981  6737 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.755894 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.759291 27659 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:22:57.730933  6710 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:57.729241  6579 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:22:57.762939 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61452.9us
W20260709 12:22:57.754549  6519 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.851s	user 0.010s	sys 0.001s
I20260709 12:22:57.753940  6750 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:57.770144 27657 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:22:57.773262 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.778379  6745 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:57.780906 27701 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 26. Current term is 27.
W20260709 12:22:57.781807  6777 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:22:57.782284  6738 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:57.782354 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3988) took 1394 ms. Trace:
I20260709 12:22:57.753453  6775 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:22:57.788981  6772 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:22:57.789320  6775 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.791940  6775 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.792001  6679 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:57.793419 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 3995) took 1226 ms. Trace:
I20260709 12:22:57.796381  6755 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.797348 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:57.797472  6755 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.753492  6741 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.803169 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4558) took 1326 ms. Trace:
I20260709 12:22:57.804296 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:57.805044 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.783849  6519 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.802968 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:57.805786 28332 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:57.803907  6755 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.795917 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:57.789036 27731 rpcz_store.cc:276] 0709 12:22:56.388147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.541327 (+153180us) service_pool.cc:224] Handling call
0709 12:22:56.575586 (+ 34259us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.575987 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.576016 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.586603 (+ 10587us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.316399 (+729796us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b1800307300. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:22:57.574929 (+258530us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:22:57.574969 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.575028 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.661341 (+ 86313us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b1800307300. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b557 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:57.782143 (+120802us) raft_consensus.cc:1685] finished
0709 12:22:57.782175 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.782300 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48012,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":14281,"spinlock_wait_cycles":31694464,"thread_start_us":258222,"threads_started":1,"wal-append.queue_time_us":1019326}
I20260709 12:22:57.803606 27656 rpcz_store.cc:276] 0709 12:22:56.476546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.014511 (+537965us) service_pool.cc:224] Handling call
0709 12:22:57.803015 (+788504us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54000e4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:57.803114 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":322892032}
I20260709 12:22:57.793860 27734 rpcz_store.cc:276] 0709 12:22:56.566943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.443278 (+876335us) service_pool.cc:224] Handling call
0709 12:22:57.443648 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.443998 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.444028 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.444510 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.560508 (+115998us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:57.670351 (+109843us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.670557 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.670643 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.788376 (+117733us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:22:57.793075 (+  4699us) raft_consensus.cc:1685] finished
0709 12:22:57.793198 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.793351 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19350,"prepare.queue_time_us":163831,"prepare.run_cpu_time_us":1469,"prepare.run_wall_time_us":3050,"raft.queue_time_us":5835,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":6055,"spinlock_wait_cycles":7161216,"thread_start_us":206890,"threads_started":2,"wal-append.queue_time_us":181297}
I20260709 12:22:57.821118 27583 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:57.779291  6737 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:57.824244 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:57.829602 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:57.825018 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.813297  6460 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:57.831718  6709 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:22:57.834563  6460 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.836807  6460 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.751909  6538 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.817454  6450 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:57.812153  6244 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.816196  6689 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:57.803607  6741 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:57.850752 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.851854 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:57.874611 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.819934  6671 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.831960  6510 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:57.853109  6727 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.142s	user 0.004s	sys 0.009s
I20260709 12:22:57.887112 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.824998  6679 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.891270 27734 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:57.889987 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.892575 27729 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:22:57.851387  6745 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:57.893735 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:57.863468  6738 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:57.848821  6401 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.856873  6566 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.846171  6519 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:57.897691 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:57.898461 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68903.8us
I20260709 12:22:57.940500 27701 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:57.940963 27701 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:57.840322  6496 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.841122  6611 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.858076  6697 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.879370  6737 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.896133  6510 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.882484  6709 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:57.910665 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.886807  6244 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.940052 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
W20260709 12:22:57.956317 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.973369 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:57.892278  6687 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.070s
I20260709 12:22:57.960736 27735 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:57.883483  6761 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.965935 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:57.883452  6689 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.969388 27734 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 34 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 20.
I20260709 12:22:57.902947 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.902606  6738 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:57.897276  6679 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.890529  6671 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:57.965878  6745 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:57.977317 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.981218 27731 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:22:58.008888 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:57.974865 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:57.963866  6697 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:57.984618 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1789) took 1697 ms (client timeout 1800 ms). Trace:
I20260709 12:22:57.884492  6775 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:22:58.003404 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63087.9us
I20260709 12:22:57.969897  6737 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:57.994993  6738 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:57.977876  6709 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.009099 27729 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
W20260709 12:22:58.025862  6793 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:22:58.029855  6689 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.035192 27735 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:22:58.037923  6244 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:58.008280 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 181264us
W20260709 12:22:58.011235 27732 rpcz_store.cc:269] 0709 12:22:56.283185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.305846 (+ 22661us) service_pool.cc:224] Handling call
0709 12:22:57.977695 (+1671849us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001d29d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:57.980968 (+  3273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2925489920}
I20260709 12:22:57.966527  6496 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.022403 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.043295 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.049880 27732 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260709 12:22:58.050328  6790 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:22:58.043856  6745 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:22:58.052003  6761 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.010077  6579 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.053222  6791 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:22:58.053391 28334 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:58.053844  6737 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.043712  6741 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.042939  6510 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.054536  6579 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.007727  6332 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:58.056260  6541 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.536s	user 0.012s	sys 0.000s
I20260709 12:22:58.056836  6332 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.060334  6689 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.060949  6689 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.064301 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.064474 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3829) took 1829 ms. Trace:
I20260709 12:22:58.066151  6016 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:58.065897  6531 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.067638 28333 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
W20260709 12:22:58.067714  6758 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.803s	user 0.000s	sys 0.003s
I20260709 12:22:58.066700  6016 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.015226  6775 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.042263  6776 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:22:58.087478 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65153.5us
I20260709 12:22:58.064906 27663 rpcz_store.cc:276] 0709 12:22:56.235442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.285571 (+ 50129us) service_pool.cc:224] Handling call
0709 12:22:56.285996 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.706902 (+420906us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:56.774265 (+ 67363us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.678272 (+904007us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b180021af60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:22:57.725988 (+ 47716us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.726069 (+    81us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:57.726133 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.063108 (+336975us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b180021af60. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:22:58.064251 (+  1143us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:58.064299 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.064418 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":62423,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":83,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":6638,"spinlock_wait_cycles":998987264,"thread_start_us":47475,"threads_started":1,"wal-append.queue_time_us":1011082}
W20260709 12:22:58.091792 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.039055  6710 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.084877  6775 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.096858 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.081750  6689 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.086259  6776 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.082063  6579 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.022s
I20260709 12:22:58.115736 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:58.106200  6776 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:58.080312  6569 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.133098 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.113816 27583 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:22:58.125330 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.125571 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.126484 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.127821 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.134400 27659 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:58.128533 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.130669 27729 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:22:58.130791 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:58.131995 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4547) took 1659 ms. Trace:
I20260709 12:22:58.131784  6755 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:22:58.135725 27664 rpcz_store.cc:276] 0709 12:22:56.472997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.573500 (+1100503us) service_pool.cc:224] Handling call
0709 12:22:57.573793 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.625588 (+ 51795us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:22:57.724032 (+ 98444us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.956987 (+232955us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.957177 (+   190us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.957224 (+    47us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:57.957276 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.060645 (+103369us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b18003380e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:58.131371 (+ 70726us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b18003380e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:22:58.131784 (+   413us) spinlock_profiling.cc:243] Waited 51.5 ms on lock 0x7b54001e46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:58.131826 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.131944 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9433,"prepare.queue_time_us":133708,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":10841,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":15803,"spinlock_wait_cycles":140873088,"thread_start_us":268230,"threads_started":2,"wal-append.queue_time_us":261840}
I20260709 12:22:58.137995 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.133850 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.138845 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.138950 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.139166 27588 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:58.140202 27664 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.144248 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.147876  6740 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:22:58.148394  6740 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.148753  6740 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.152891  6710 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.154150  6531 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.158195 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:58.164008 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 189259us
I20260709 12:22:58.164273 27735 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 42: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:58.132979  6332 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:58.164811 27588 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:22:58.116736  6667 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.169523 27659 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:22:58.172995 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:58.174809 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:58.175907 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.176944 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.177945 27659 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.177088 27664 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.179255 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.176306 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:58.156414  6784 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:58.182415 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:58.184864  6611 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:58.186065  6800 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:58.188426 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4790) took 1415 ms. Trace:
I20260709 12:22:58.188486  6496 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.159605  6737 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:58.194404 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:58.196290 27581 rpcz_store.cc:276] 0709 12:22:56.773446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.773625 (+   179us) service_pool.cc:224] Handling call
0709 12:22:58.188208 (+1414583us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54001888d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:58.188338 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1599603456}
I20260709 12:22:58.196069 27589 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:58.197096 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:58.197813 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:58.200021 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 84527.8us
I20260709 12:22:58.199535 27732 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:22:58.200176 27735 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:22:58.202907 27588 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:58.201876 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.202910 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:58.189224  6538 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:58.159072  6791 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:22:58.182554  6016 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.206238  6538 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:58.181999  6709 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 40.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.038s
I20260709 12:22:58.228628  6538 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:58.231799 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.223719 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.217978 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:58.233300 27664 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:58.233354 27588 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:22:58.226140 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.182128  6807 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:22:58.223722 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.164592  6738 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.244995  6738 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.228684  6709 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:22:58.245278 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.245656  6709 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:58.246516 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.246604  6709 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.248214  6709 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:22:58.252678 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.253908 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.224905  6541 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:58.254479 28335 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:58.255034  6541 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:58.226859 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.256274 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.189837  6784 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:58.262176  6784 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.241313  6679 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.198588  6802 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:22:58.209371  6496 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.263747  6775 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.264101  6496 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.265887  6775 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.257524 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:58.256050 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:58.254856 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:58.214839  6740 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.257354 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.269189 27588 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.254473 27732 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [38.22-38.22]
I20260709 12:22:58.270726 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:58.258236 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.225109 27659 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:58.258767 27589 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:22:58.241396 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.255540  6541 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:58.259828 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.260197 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.260455 28337 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:22:58.189595  6667 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.272887 27586 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
W20260709 12:22:58.276492 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59546.6us
I20260709 12:22:58.283418 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.284894 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.288393 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.254513  6569 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.240391  6807 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.291697 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:58.286408 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.293864 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.296501 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.280905  6811 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:22:58.299966  6811 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:58.300102 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.266458  6496 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.302182  6667 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:22:58.302560  6725 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.376s	user 0.002s	sys 0.009s
I20260709 12:22:58.303896 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.305369 28331 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:58.286382  6675 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:22:58.306155  6675 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:58.306524  6675 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:58.306717 28331 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:58.300871  6740 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.280699  6737 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:58.311896  6737 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:58.312399  6737 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:58.316402 27583 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:58.294732  6454 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:58.317261  6691 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.317770  6691 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.318873 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1794) took 1424 ms. Trace:
I20260709 12:22:58.319962  6691 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.300441  6811 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:58.327972 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:58.329020 27736 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:22:58.333877  6697 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.324021 27656 raft_consensus.cc:2376] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33. Re-sending same reply.
I20260709 12:22:58.298906  6807 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:22:58.336853  6743 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:58.337453  6743 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.338414 27581 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:22:58.339898 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.340801 27658 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:58.342326 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.343461 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.344403 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.345449 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.348956  6745 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.349509  6745 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.297238  6738 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:58.351934  6740 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:58.352782  6738 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:58.353171  6738 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:58.353304 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:58.354046 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.354306  6372 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:22:58.354331 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.355242  6496 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:58.355526  6372 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:58.355942  6372 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:58.355836  6496 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.356741 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.356910  6372 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.358139  6496 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.358881  6740 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:58.358253  6372 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:58.363706 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:58.362473  6738 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.365702  6738 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.368477  6617 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:22:58.369369 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.370777 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.374217 27588 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:22:58.358271 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1735) took 2243 ms. Trace:
I20260709 12:22:58.369495  6761 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:58.383153 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:58.333909 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:58.387254 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:58.374359  6332 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:58.360828 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4579) took 1462 ms. Trace:
I20260709 12:22:58.389401 28330 rpcz_store.cc:276] 0709 12:22:56.115041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.115224 (+   183us) service_pool.cc:224] Handling call
0709 12:22:56.242755 (+127531us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.243120 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.243151 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.243616 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.369010 (+125394us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:22:56.369067 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.369157 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.419411 (+1050254us) raft_consensus.cc:1685] finished
0709 12:22:57.419441 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.357962 (+938521us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54005a3550. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:22:58.358184 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21925,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":90,"spinlock_wait_cycles":256159616,"thread_start_us":124909,"threads_started":1,"wal-append.queue_time_us":801070}
I20260709 12:22:58.375233 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.397383 27660 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:22:58.375916 28333 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:22:58.379911  6745 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.382930  6671 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.319257 27737 rpcz_store.cc:276] 0709 12:22:56.895467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.548701 (+653234us) service_pool.cc:224] Handling call
0709 12:22:57.548974 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.669899 (+120925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.669951 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.670010 (+    59us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:22:57.670052 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.318625 (+648573us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:58.318678 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.318815 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277522176}
I20260709 12:22:58.383555 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4580) took 1445 ms. Trace:
I20260709 12:22:58.384804 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:58.326025 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:58.388691 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.398829 27587 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:58.416463 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:58.418689 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:58.345968 27658 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:58.388862 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.390194  6761 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:58.395088  6791 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:22:58.424353 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:58.429827 27735 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.425137 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.414034 27655 rpcz_store.cc:276] 0709 12:22:56.938060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.597247 (+659187us) service_pool.cc:224] Handling call
0709 12:22:57.597557 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.598921 (+  1364us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:22:57.698348 (+ 99427us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.324942 (+626594us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b18004631e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a
0709 12:22:58.325310 (+   368us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.325500 (+   190us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:58.325937 (+   437us) log.cc:844] Serialized 26 byte log entry
0709 12:22:58.375635 (+ 49698us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:22:58.375876 (+   241us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.376055 (+   179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.376197 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.383338 (+  7141us) raft_consensus.cc:1685] finished
0709 12:22:58.383361 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.383489 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81190,"prepare.queue_time_us":155987,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":115,"raft.queue_time_us":3002,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":191,"spinlock_wait_cycles":7519360,"thread_start_us":152812,"threads_started":2,"wal-append.queue_time_us":231411}
I20260709 12:22:58.395812 27662 rpcz_store.cc:276] 0709 12:22:56.898108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.594416 (+696308us) service_pool.cc:224] Handling call
0709 12:22:57.594728 (+   312us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:57.595090 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.595132 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:57.595749 (+   617us) log.cc:844] Serialized 52 byte log entry
0709 12:22:57.676132 (+ 80383us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:22:58.328580 (+652448us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b1800477760. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:22:58.328724 (+   144us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.360136 (+ 31412us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.360400 (+   264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.360476 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.360616 (+   140us) raft_consensus.cc:1685] finished
0709 12:22:58.360632 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.360772 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55519,"prepare.queue_time_us":90267,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":39262,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":6321,"spinlock_wait_cycles":6723456,"thread_start_us":89175,"threads_started":2,"wal-append.queue_time_us":106264}
I20260709 12:22:58.431455 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:58.433143 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 107197us
I20260709 12:22:58.430121 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.432128 27581 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:58.436655 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.438755 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.439513 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.439839 27655 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:58.440388 27737 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:22:58.440851 27738 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.441989 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.437676 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.404394  6519 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:22:58.445046 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.446965 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.460191 27660 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
W20260709 12:22:58.510905 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65947.4us
I20260709 12:22:58.511721 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.514583 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:58.520128 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:58.526633 27738 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:58.528110 27660 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:22:58.529340 27660 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:58.443779  6454 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:22:58.534042 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.521329 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.451691  6579 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:58.545725 27658 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:22:58.403949  6667 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:22:58.423993  6496 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:22:58.446722  6671 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:58.424542  6691 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.093s
I20260709 12:22:58.444695  6740 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:22:58.536895  6332 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:22:58.574807 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.429023  6776 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.429989  6541 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.558995 27655 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:58.445008  6791 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:58.534873  6531 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:22:58.510814  6681 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.071s
I20260709 12:22:58.610723 27735 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:22:58.538751  6745 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:22:58.621376  6745 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:58.628669  6791 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.635609  6332 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.636387  6332 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.641072  6776 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.651479 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:58.654116 27655 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 25.
I20260709 12:22:58.658993 27735 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:22:58.648303  6675 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:22:58.661547  6814 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:22:58.662418  6811 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.663492  6675 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.666141 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:58.597761  6803 raft_consensus.cc:2764] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election decision vote started in defunct term 35: won
I20260709 12:22:58.577580  6761 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.668219 27659 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:22:58.650384  6681 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:22:58.665014  6811 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.649474  6802 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:58.658632  6332 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.724255 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.726102 27628 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:22:58.726558 27628 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:58.683108  6531 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:58.670394  6776 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.048s
I20260709 12:22:58.649202  6784 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.737097  6332 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:22:58.694931  6681 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:58.745363  6784 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:22:58.747489  6538 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.518s	user 0.010s	sys 0.000s
I20260709 12:22:58.748086  6538 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.728761  6761 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.749593 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4005) took 1305 ms. Trace:
I20260709 12:22:58.749979 27733 rpcz_store.cc:276] 0709 12:22:57.444270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.649412 (+205142us) service_pool.cc:224] Handling call
0709 12:22:57.667811 (+ 18399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.690341 (+ 22530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.749364 (+1059023us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:22:58.749535 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25462528}
I20260709 12:22:58.750589  6538 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.651288  6807 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.760571 27658 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:58.761116  6541 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.761210  6807 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.735924 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:58.643304  6743 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.738001  6644 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:22:58.762414  6531 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.764458  6697 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.667349  6737 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.754365  6681 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.777957 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.779417 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:22:58.785821 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 264536us
I20260709 12:22:58.776041  6822 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:22:58.790791 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 216063us
I20260709 12:22:58.791448 27732 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:58.768230  6538 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
W20260709 12:22:58.798426 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 383742us
I20260709 12:22:58.783293  6519 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:58.777433  6332 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.014s
I20260709 12:22:58.790287  6738 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:22:58.781471  6617 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.018s
I20260709 12:22:58.799592 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.809406 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:58.800221  6681 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:58.812579  6737 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:58.813525  6519 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.810619 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.790473  6671 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:58.839097 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:58.848610 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:58.806270  6372 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:58.855614 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:58.860598 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:22:58.862195 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 52909.5us
I20260709 12:22:58.862920 27658 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:58.854676 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:58.873565 27655 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:22:58.846798  6332 consensus_queue.cc:1243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 30.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:22:58.850116  6821 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:58.803824  6697 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:22:58.859130  6460 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:58.893271 27732 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:22:58.866917  6679 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:22:58.876061 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:58.861742  6691 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:22:58.863936  6685 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.898244  6775 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:58.899019  6671 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.846554  6454 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:22:58.906642 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.879520  6830 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:58.910504 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.434921 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:58.901218  6679 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:58.918177  6372 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:58.918805  6016 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:22:58.908680  6496 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:22:58.933952  6496 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:58.936182 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.903513  6775 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:58.907145  6460 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:58.904906  6691 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:58.940784 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:58.944034 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.944630 27585 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:58.935725  6679 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:58.934963  6496 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.904827  6685 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.964181  6685 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:58.937100  6830 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:58.973783 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.974006 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.974921 27588 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:22:58.975910 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.977038 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.978176 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.979144 27588 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:58.980007 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.981232 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.982676 27589 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:22:58.938588  6460 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.947223  6825 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:58.985351 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.946741  6791 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:22:58.986464 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.993799 27580 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:22:58.994295 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.996413  6691 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:58.998093  6829 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:22:59.000555 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.004570 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:58.987175  6372 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:59.007010  6372 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:22:59.008306 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.009816  6372 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.956525  6743 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:58.947580  6531 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.066s
I20260709 12:22:58.946559  6454 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:22:59.015542  6830 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.015987  6775 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.017515  6775 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:58.989491 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.991986 27582 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:22:59.019894  6791 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:58.992252 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.020954  6791 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:58.996822 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:58.970436  6821 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.065s
I20260709 12:22:59.033056  6837 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:22:59.034150  6832 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.034752  6832 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.034216 27734 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:59.037307 27734 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:22:59.037389 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:58.988422 27588 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:59.038743 27585 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:22:59.039750 27734 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.028940  6685 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:59.043565 27734 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:59.039266  6689 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:22:59.047628 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.050402 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:22:59.042878  6821 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 29.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:22:59.043640 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.055787 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.062326 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.042551  6832 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.064967 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.070899 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.083009 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.084403  6784 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:22:59.084910  6761 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:59.086725 27730 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:22:59.088629  6685 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:22:59.080852  6581 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:59.091718 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:59.092370 27736 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:22:59.093887 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:59.068711  6826 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:59.095736  6826 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:59.096652 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.097414 27731 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:22:59.098208 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:59.098716 27731 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:22:59.099671 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:59.099956  6685 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:22:59.096659  6826 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.101064  6803 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:59.101547  6803 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.063285  6825 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:59.106492 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.048550  6372 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.049s
I20260709 12:22:59.104465 27732 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:22:59.084312 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.111892 27738 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:59.113230 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.114909 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:59.117228 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:59.118045 27730 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:59.122066 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:59.104281 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:22:59.095101 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:22:59.125869 27733 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:59.104113 27734 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:59.130575 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.132737 27659 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:22:59.130489 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.105021  6581 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:59.139555 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87279.3us
I20260709 12:22:59.140878 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.138397 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.144827 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.145638  6828 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.146179  6828 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.147920 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:22:59.148377  6828 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:59.121634  6738 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:22:59.097414  6789 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:22:59.145645  6644 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:22:59.152765 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.166339 27734 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:22:59.155802  6828 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:22:59.157711  6737 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:22:59.168215 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:59.128316  6332 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:59.162926 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:22:59.163690 27731 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:22:59.141669  6807 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:22:59.136842  6822 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 30 current term: 30 is higher.
I20260709 12:22:59.128019  6837 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:59.136370  6776 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:59.217693 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1806) took 1056 ms. Trace:
I20260709 12:22:59.193830  6450 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:22:59.119213  6133 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.050s
I20260709 12:22:59.220168  6828 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:22:59.239024 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:22:59.220777 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52672.2us
I20260709 12:22:59.142355  6802 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.139635  6531 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.079s
I20260709 12:22:59.132079  6681 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.050s
I20260709 12:22:59.165267  6803 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.165350  6824 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:22:59.239745 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:59.250196 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.250856 27734 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:59.228238 27733 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
W20260709 12:22:59.240214  6843 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:22:59.222785 27736 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:22:59.241205 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.241689 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.222643  6776 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.203303  6541 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:22:59.142051  6761 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:22:59.230993 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.242311  6697 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:22:59.245421  6812 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:59.254928 27659 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:22:59.218282 27729 rpcz_store.cc:276] 0709 12:22:58.161136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.161337 (+   201us) service_pool.cc:224] Handling call
0709 12:22:58.161762 (+   425us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:58.162147 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.162179 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:58.178423 (+ 16244us) log.cc:844] Serialized 52 byte log entry
0709 12:22:58.481547 (+303124us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:22:58.568919 (+ 87372us) log.cc:844] Serialized 14 byte log entry
0709 12:22:59.055676 (+486757us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b18001a53c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:22:59.192327 (+136651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.192548 (+   221us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.192658 (+   110us) raft_consensus.cc:1685] finished
0709 12:22:59.217441 (+ 24783us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.217626 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":50116,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":3648,"raft.queue_time_us":52719,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":8832,"spinlock_wait_cycles":11138944,"thread_start_us":383973,"threads_started":2,"wal-append.queue_time_us":412886}
W20260709 12:22:59.246614  6839 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:59.194418  6496 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:22:59.286139 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.287459 27664 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:22:59.260771 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 112899us
W20260709 12:22:59.220029 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57222.6us
I20260709 12:22:59.289397 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:59.237067  6740 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.239491  6828 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.142151  6687 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.079s
I20260709 12:22:59.308969 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.150672  6581 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.176527  6332 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.221145  6852 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:22:59.270092 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:59.253306 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.281487  6697 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:22:59.278208  6691 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.221491  6667 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.046s
I20260709 12:22:59.191107 27730 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:22:59.242111  6821 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.028s
I20260709 12:22:59.252060  6824 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:59.297442 27701 leader_election.cc:400] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:59.295966 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.296370 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:22:59.268424  6835 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.289s	user 0.001s	sys 0.001s
I20260709 12:22:59.301844 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.326493 27701 leader_election.cc:403] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:22:59.268656  6776 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:59.219760  6837 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.308068  6828 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:59.312458 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.249257  6531 consensus_queue.cc:1243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 29.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:22:59.316833  6332 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:59.272981  6856 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:22:59.253628  6679 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.324366 27655 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:22:59.307341  6761 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:22:59.249130  6802 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.331665 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.326682 27663 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:22:59.322748  6697 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:22:59.333245  6846 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.331250  6691 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.364429 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.367929 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.372810 27588 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:59.381529 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49940.7us
I20260709 12:22:59.325071  6852 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:59.311149  6740 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.385030 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:22:59.394243 27655 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: [35.19-37.20]
I20260709 12:22:59.337457  6737 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:22:59.331349  6824 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:22:59.364923  6845 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:22:59.371524  6761 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.413709 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.410010 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:59.413913 27731 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:59.408656 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:59.398274 27656 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:22:59.404341 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:59.407282 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.416558 27588 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:22:59.426355 28337 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:22:59.398356 27655 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
W20260709 12:22:59.401600  6849 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:22:59.380919  6803 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.433439 27735 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [38.22-38.22]
I20260709 12:22:59.432602 28328 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:59.396999  6848 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:22:59.402104  6841 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:22:59.399994  6689 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:22:59.443779  6844 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
I20260709 12:22:59.444168  6689 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.447645 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1808) took 1061 ms. Trace:
I20260709 12:22:59.435494  6803 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.440343 27658 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 26. Current term is 27. Ops: []
I20260709 12:22:59.448788 27735 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 29. Current term is 30. Ops: []
I20260709 12:22:59.440510 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:59.450579  6803 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.408795  6579 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:59.455533  6579 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:59.456519  6579 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.457693  6579 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:59.460695  6679 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:59.464254  6581 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.465880  6740 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:59.466562 27729 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:22:59.468453  6837 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:59.471128  6852 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.474469 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:22:59.475916 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.421862  6761 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:22:59.396544  6846 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.430418  6566 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.055s
I20260709 12:22:59.485540  6846 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.485694  6566 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:22:59.486083  6566 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:22:59.420037  6822 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:59.486936  6822 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.488554  6821 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:22:59.489070  6821 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.489486  6821 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:22:59.490778  6802 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.491788 27658 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:22:59.492753 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.493817 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:22:59.440260  6842 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:22:59.448064 27737 rpcz_store.cc:276] 0709 12:22:58.385901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.512911 (+127010us) service_pool.cc:224] Handling call
0709 12:22:58.513260 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.680995 (+167735us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.681038 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:58.696201 (+ 15163us) log.cc:844] Serialized 26 byte log entry
0709 12:22:58.860791 (+164590us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:22:58.871385 (+ 10594us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.871981 (+   596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.872091 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.933287 (+ 61196us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b180041c320. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:22:59.443241 (+509954us) raft_consensus.cc:1685] finished
0709 12:22:59.447409 (+  4168us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.447581 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10193,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":94007,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":6911,"spinlock_wait_cycles":29777152,"thread_start_us":325994,"threads_started":2,"wal-append.queue_time_us":234300}
I20260709 12:22:59.497715 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.501187 27588 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:59.499295 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.497326 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:59.511490 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.515384 27581 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:22:59.519135 27588 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:22:59.519726 27588 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:59.520644 27588 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.529479 27588 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:22:59.497877  6822 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:59.486846  6566 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.532858  6824 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.556408  6824 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:59.559607 27730 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: [35.19-37.20]
I20260709 12:22:59.569373  6776 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.548166  6697 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.573874  6697 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.575572  6531 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.576117  6531 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.578186  6531 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.579190  6566 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:22:59.574779  6761 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.584954 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:22:59.556602  6850 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.095s	user 0.003s	sys 0.000s
I20260709 12:22:59.586477 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.588445 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:22:59.586870 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 178288us
I20260709 12:22:59.604053 27582 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:22:59.606523 27732 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:22:59.595098 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:59.620312 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.592531 27581 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:22:59.583132  6332 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.658406 27588 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 20. (term mismatch)
I20260709 12:22:59.661438 27588 pending_rounds.cc:85] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 19
I20260709 12:22:59.661796 27588 pending_rounds.cc:107] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 26.20
I20260709 12:22:59.662081 27588 raft_consensus.cc:2889] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:59.663995 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49398.2us
I20260709 12:22:59.630513  6827 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:22:59.673792 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.677471 27585 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:59.627902  6789 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:59.592972  6828 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.680338 27735 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.19-32.19]
W20260709 12:22:59.681357 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72304us
I20260709 12:22:59.603893  6821 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.686319 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.686575 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.690440 27731 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []
I20260709 12:22:59.691613 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:22:59.660773  6454 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:22:59.679899  6827 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.696758 27737 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:22:59.699057 27731 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:22:59.705730  6852 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:22:59.700656  6871 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.010s
I20260709 12:22:59.707701  6852 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.683996  6824 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.025s
I20260709 12:22:59.708911  6449 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:22:59.709326  6449 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:59.674314  6531 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.022s
I20260709 12:22:59.671013  6667 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:22:59.674595  6846 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.020s
I20260709 12:22:59.684093  6737 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.712401 27656 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []
I20260709 12:22:59.691398  6776 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.700258  6827 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:22:59.710129  6449 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.710016  6852 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:22:59.741858 27553 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:22:59.685451  6789 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.745237 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.716706  6644 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:22:59.737138  6865 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:59.733429  6860 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.750138 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.747849 27581 raft_consensus.cc:1275] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:59.754343 27580 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:22:59.755442 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.756407 27581 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:22:59.748158 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.741567  6846 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:22:59.741994  6449 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:22:59.759994 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.764894 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.779225 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:22:59.746395  6847 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:22:59.733443  6737 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.793884 27664 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:22:59.795629 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.788776 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:22:59.763526  6846 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:59.746145  6496 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:22:59.803007 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.801640  6745 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:22:59.806915  6861 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:22:59.793498  6450 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:22:59.793262  6761 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:59.776126  6579 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.027s
W20260709 12:22:59.825356  6872 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:22:59.791275  6848 consensus_queue.cc:1048] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:22:59.764034  6860 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.811630  6450 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.836392 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:22:59.863557 27663 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:22:59.831199  6846 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.841637  6450 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:59.870209 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.871235 27583 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:22:59.876067  6875 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:22:59.877190  6866 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:22:59.878230  6821 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.859875  6454 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:22:59.879914  6454 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.882085 28337 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:22:59.885462  6860 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.853570  6579 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.887279  6579 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.891955  6579 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.892168 27661 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:22:59.892606  6579 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:59.838742  6332 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:59.893222  6579 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.862550  6710 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:22:59.895810  6454 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.887988  6016 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:22:59.896813  6016 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:59.897869  6016 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.899024  6016 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:22:59.899472  6541 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.899948  6579 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.900004  6541 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.902593  6541 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:22:59.892599  6450 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.912199 28336 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:22:59.912845 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:59.917788 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.919363 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:22:59.922016 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61032.7us
W20260709 12:22:59.962500  6874 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:22:59.962679  6579 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:22:59.958115  6709 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.009s
I20260709 12:22:59.963833  6579 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:22:59.964455  6579 consensus_queue.cc:1243] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 24.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:22:59.965155  6709 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:22:59.965585  6709 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:22:59.966414  6709 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:22:59.967593  6789 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:59.968411  6332 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:22:59.969151  6332 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:22:59.972317 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.980306  6332 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:22:59.980885 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:22:59.982514 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:22:59.983075 28336 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:22:59.992087 28336 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:22:59.982417  6709 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:22:59.993217 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:22:59.957618  6675 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:00.000599  6450 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.003517  6372 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:00.004057  6372 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:22:59.998991  6737 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:00.007230 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:00.005076  6372 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:00.017226 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.018707 27655 raft_consensus.cc:1217] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Deduplicated request from leader. Original: 21.17->[22.18-27.20]   Dedup: 22.18->[25.19-27.20]
I20260709 12:23:00.022185 27655 pending_rounds.cc:85] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 18
I20260709 12:23:00.022451 27655 pending_rounds.cc:107] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 24.19
I20260709 12:23:00.020308 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.022694 27655 raft_consensus.cc:2889] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:00.020383 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.021186 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.024370 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:22:59.997710  6675 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:00.024798 27661 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:00.028286 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:22:59.995095  6789 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:23:00.035792 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:00.041956 27583 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:23:00.044442  6671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.004s	user 0.007s	sys 0.003s
I20260709 12:23:00.044317  6775 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:23:00.046082 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.046155  6775 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:00.048951 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.049654 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.050390 27588 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:00.025270  6894 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.030s
I20260709 12:23:00.043336 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:00.053160  6894 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:00.053493  6737 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:23:00.053673  6894 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.054132  6737 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:00.025604  6541 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.032s
W20260709 12:23:00.044858  6671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s	user 0.008s	sys 0.003s
I20260709 12:23:00.057541  6691 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.058343 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 1987) took 1686 ms. Trace:
I20260709 12:23:00.058740 27662 rpcz_store.cc:276] 0709 12:22:58.371672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.511447 (+139775us) service_pool.cc:224] Handling call
0709 12:23:00.058107 (+1546660us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54000e51d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:00.058265 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3551935232}
I20260709 12:23:00.059425  6824 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:00.062842  6824 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:00.063663  6372 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:00.063735  6824 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.065065  6824 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:00.065294  6826 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:00.066048  6826 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.060421  6675 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.042948  6802 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:00.068001  6675 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:00.067936  6802 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.070140  6802 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.055130  6737 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:00.070928 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48450.2us
I20260709 12:23:00.077505 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.078644 27588 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:00.077978  6826 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:00.067224  6827 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.123631  6826 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.018s
W20260709 12:23:00.151574 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95233.4us
W20260709 12:23:00.153488 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.168970  6828 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:00.215960 28333 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:00.218350 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:00.225983 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:00.227273 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:00.228348 28332 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:23:00.233984 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 198294us
I20260709 12:23:00.235814 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:00.289978 27732 pending_rounds.cc:85] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:23:00.290462 27732 pending_rounds.cc:107] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:23:00.290764 27732 raft_consensus.cc:2889] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:00.291746 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.292554 27731 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:00.293778 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.294714 27736 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:00.233263  6450 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.301620 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.305230 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.306294 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.306763 27730 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.28-41.28]
I20260709 12:23:00.306969 27733 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:00.308048 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.309630 27738 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:00.298432  6802 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:23:00.319779 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.302551  6822 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.318799 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 83048.9us
I20260709 12:23:00.323571 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:00.326809 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.332818 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.333402 27733 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:00.310531  6450 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.324451  6900 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:00.331477  6802 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:00.375161 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:00.356122  6802 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:00.376618  6802 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:00.385479  6893 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:23:00.348106  6828 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.392290  6823 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:00.327930  6822 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.393095  6823 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.394788  6822 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.392452  6789 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.395975  6789 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.396570 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.392331 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.399436 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:00.349723  6827 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.392213  6828 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:00.401081 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.432403 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.437989 27584 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:00.433307 27738 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:00.448917 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.449746 27656 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:00.465310 27663 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 29. Current term is 30. Ops: []
I20260709 12:23:00.468662 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:00.488729 27657 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 34 index: 22. (term mismatch)
I20260709 12:23:00.489277 27657 pending_rounds.cc:85] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 21
I20260709 12:23:00.489526 27657 pending_rounds.cc:107] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 33.22
I20260709 12:23:00.461731  6784 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:00.472601  6903 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:23:00.472491  6905 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:23:00.489813 27657 raft_consensus.cc:2889] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:00.507627 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 115376us
I20260709 12:23:00.508074 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:00.509059 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46140 (request call id 10) took 1364 ms. Trace:
I20260709 12:23:00.465672  6823 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:00.514353 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.516873 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:00.517527 27731 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:00.518908 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:00.516587 28329 rpcz_store.cc:276] 0709 12:22:59.124860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.125015 (+   155us) service_pool.cc:224] Handling call
0709 12:23:00.488544 (+1363529us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54000f8fa0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c5520 00007f0e9a34eb42 00007f0e9a34d800 00007f0e9a3469fa 00007f0e9f30e4c9 00007f0e9888ac54 00007f0e9888abe2 00007f0e9888ab11 00007f0e9888aa8c 00007f0e98889d42
0709 12:23:00.488776 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96628096}
I20260709 12:23:00.493783  6531 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:00.531438 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:00.531350 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.398294  6900 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.508685  6784 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:00.479278 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:00.560151 27731 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:00.560614 27733 raft_consensus.cc:606] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.561082 27733 raft_consensus.cc:614] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:00.561390 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.562772 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.563491 27731 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:00.545747 27734 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:00.542455 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.564069 27735 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:00.528448  6697 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:00.532537  6789 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.547489  6531 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.567010 27731 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:23:00.545383  6909 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.562986  6784 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:00.595472 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53091.9us
I20260709 12:23:00.597229 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.577062  6332 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:00.585243  6531 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:00.609184  6902 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:23:00.577664  6697 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.635478  6697 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.639978  6825 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:00.640529  6825 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:00.641587  6825 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.642813  6825 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:00.642977  6904 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
I20260709 12:23:00.627703  6743 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:00.611137  6789 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:23:00.650463 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 135356us
I20260709 12:23:00.620884  6761 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:00.648517  6579 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:00.651690  6789 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.653820  6857 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:00.667654  6802 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.679689  6789 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:00.594242  6909 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.657184  6450 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:00.655670  6709 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:23:00.693971 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.666064  6579 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:00.702106  6579 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:00.704109  6911 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:23:00.653721  6743 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.709762  6743 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.689412  6713 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.710537  6713 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.701560  6909 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:00.662317  6761 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.717507 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:00.720544 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 108017us
I20260709 12:23:00.697016  6828 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:00.723440 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.712999  6713 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.716676  6920 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:23:00.726482  6920 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:00.727562  6920 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:00.736290 27659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3855) took 1456 ms (client timeout 1738 ms). Trace:
I20260709 12:23:00.737840 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4028) took 1325 ms. Trace:
I20260709 12:23:00.738467 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:00.738265 27729 rpcz_store.cc:276] 0709 12:22:59.412376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.412492 (+   116us) service_pool.cc:224] Handling call
0709 12:22:59.412756 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.466942 (+ 54186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.737609 (+1270667us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b54001d24d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:00.737779 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123305856}
W20260709 12:23:00.736896 27659 rpcz_store.cc:269] 0709 12:22:59.279625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.280361 (+   736us) service_pool.cc:224] Handling call
0709 12:23:00.736043 (+1455682us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54000e5bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:00.736210 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378782592}
I20260709 12:23:00.742954 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:00.743570 27664 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:00.744915 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:00.745326  6689 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.748845  6926 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:23:00.749320  6894 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:23:00.749490  6926 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:00.750794  6926 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.754586  6846 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:00.741983  6832 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:00.764650  6832 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:00.765532  6832 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:00.767068 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.770110 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.776762 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.779062  6821 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.779805  6821 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.782375  6821 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.790158 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:00.795823 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 231086us
I20260709 12:23:00.798172 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:00.798492 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 250215us
I20260709 12:23:00.799486 27664 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:00.800218 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:00.801898 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.802532 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:00.802951 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.803396 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.803870 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.804133 27658 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
W20260709 12:23:00.812091 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.812453 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.813321 27584 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 40 because replica has last-logged OpId of term: 38 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 21.
I20260709 12:23:00.814572 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.816179 27729 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: [28.23-31.24]
I20260709 12:23:00.816642 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.816787 27732 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
W20260709 12:23:00.820247 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.820653 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:00.821408 27731 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:00.822435 27737 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:00.822510 27736 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:23:00.823745 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:00.823316 27733 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:00.824146 27738 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:23:00.823863 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.824985 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.825865 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.826401 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:00.828006 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.734707  6775 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:00.825889 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.830221 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.841620 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:00.844872 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:00.828698 27737 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:00.846621 27731 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:00.839433 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:00.847210 27734 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:00.854422 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:00.855650 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.856662 27737 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:00.831051 27664 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:00.822486  6828 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:00.872622 27734 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:00.862628 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.822423  6909 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:23:00.828400  6644 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:00.836007  6743 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:00.839592  6332 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:23:00.827103  6697 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:00.898420 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:00.910594 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65783us
I20260709 12:23:00.864686  6822 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:00.908902 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:00.900384 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:00.920867 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.862557  6775 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:00.911900 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:00.922197 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.933434  6789 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.898399  6745 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:00.937565 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:00.941350 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:00.944828 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:00.906474  6871 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:00.919062  6519 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:00.909979  6909 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:00.909914  6827 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.956543 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.954257  6821 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:23:00.923702  6828 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.927703  6776 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:00.940498  6675 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:00.936161  6743 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:00.935690  6531 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.906548  6644 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.940261  6745 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.952157  6713 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:23:00.957852  6802 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:00.939946  6822 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:00.962361  6375 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:00.923401  6372 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:23:00.943712  6918 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:00.958801  6891 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.022363  6891 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.023981  6824 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.024590  6824 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:00.972649  6675 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:00.968351  6827 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:00.954552  6926 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:00.925326  6697 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.033664  6697 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.977864 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:00.970235  6541 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:00.977406  6745 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:01.018941  6789 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.019984  6822 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.041851  6644 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.020805  6531 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.021415  6372 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:01.012529  6918 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.978924  6713 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:00.974308 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.015s	sys 0.018s
W20260709 12:23:01.066079 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 341014us
W20260709 12:23:01.066449 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.015s	sys 0.019s
I20260709 12:23:01.027732  6824 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.028501  6784 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.029712  6891 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.026046  6675 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:01.037060 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:00.960242  6871 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.073257  6713 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:01.090211  6871 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:00.973873  6828 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.091516  6824 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:01.045322 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67574.1us
I20260709 12:23:01.038489  6689 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:00.966535  6909 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.093432 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:01.098848 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:01.050030  6541 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:01.092248  6824 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.065932  6918 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:01.096002 27730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.026s	sys 0.025s
I20260709 12:23:01.049726  6372 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:01.076665  6538 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:23:01.107460 27730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.026s	sys 0.026s
I20260709 12:23:01.012182  6775 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.086889  6802 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:01.103106  6697 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:01.122601  6566 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:01.106017  6828 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.226027 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:01.225471 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:23:01.119153  6919 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.223218  6852 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.015s	sys 0.039s
I20260709 12:23:01.123883  6697 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.248405 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:01.123294  6566 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:01.233799 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.254532 28333 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:01.255682 27553 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:01.224947  6858 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.258345 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 356754us
I20260709 12:23:01.121574  6802 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:01.253214  6852 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.016s	sys 0.039s
I20260709 12:23:01.267956 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.255568  6697 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:01.265666  6859 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:23:01.276964  6927 log.cc:927] Time spent T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:01.267100  6858 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.284403  6826 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:01.285076  6826 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.286059  6858 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.286798  6858 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:01.251019  6538 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.024s
W20260709 12:23:01.267732  6579 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.565s	user 0.009s	sys 0.002s
I20260709 12:23:01.288352  6579 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.290925  6579 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.273108  6828 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:01.297991  6802 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.297911  6828 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.277004  6859 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:01.299687  6859 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:01.300545  6828 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.294916  6891 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.301466  6891 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.303756  6891 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:01.305454  6925 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 12:23:01.306646  6784 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.307502  6826 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.310240  6824 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.316216  6909 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.317245  6566 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.260416  6821 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:01.311111  6675 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.320611  6827 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.322530  6675 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.253396  6919 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.323812  6918 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.324844  6697 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.325418  6919 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.326049  6918 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.256367  6541 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.327276  6697 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.327955  6859 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.319608  6821 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.331156 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
W20260709 12:23:01.331815 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.331843 27734 raft_consensus.cc:606] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:01.332921 27734 raft_consensus.cc:614] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:01.333788 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.334924 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.332999  6821 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:01.251477  6372 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:01.342777 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 480298us
W20260709 12:23:01.346951 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.339478  6846 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.584s	user 0.010s	sys 0.002s
I20260709 12:23:01.359238  6372 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:01.365031 27589 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:01.366647 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1761) took 1016 ms. Trace:
I20260709 12:23:01.367133 28335 rpcz_store.cc:276] 0709 12:23:00.350415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.350659 (+   244us) service_pool.cc:224] Handling call
0709 12:23:00.351126 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.354699 (+  3573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.354727 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.383691 (+ 28964us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.760685 (+376994us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b1800479560. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:01.119841 (+359156us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:01.119896 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.119975 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.360132 (+240157us) raft_consensus.cc:1685] finished
0709 12:23:01.360159 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.366577 (+  6418us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19307,"raft.queue_time_us":4654,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":33443,"spinlock_wait_cycles":34160384,"thread_start_us":358909,"threads_started":1,"wal-append.queue_time_us":974175}
W20260709 12:23:01.371562 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 426818us
I20260709 12:23:01.355863  6828 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.397616 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:01.398460 27734 raft_consensus.cc:606] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:01.398969 27734 raft_consensus.cc:614] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:01.376328  6924 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:01.370920  6846 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:01.395303  6921 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.008s	sys 0.020s
I20260709 12:23:01.446120 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.447084 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.446830 28328 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:01.416708  6825 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.388576  6871 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:23:01.453992 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.453447 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:01.459226 27658 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:23:01.456835 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:01.459636 28333 raft_consensus.cc:2368] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42.
I20260709 12:23:01.460528 27658 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
W20260709 12:23:01.460595 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 358382us
I20260709 12:23:01.461544 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:01.462078 27658 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:01.462287 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:01.431681  6921 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.009s	sys 0.020s
I20260709 12:23:01.388156  6827 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:23:01.457157 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:01.388373  6891 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
W20260709 12:23:01.458433 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68819.5us
I20260709 12:23:01.463201 27656 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:01.391666  6828 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.472473 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.449048  6858 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:01.440647  6945 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.442098  6519 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:01.466531 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:01.466902 27628 leader_election.cc:400] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42.
I20260709 12:23:01.526113 27628 leader_election.cc:403] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:23:01.454972  6832 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:01.526726 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 59942.9us
I20260709 12:23:01.528559 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:01.513044 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63960.8us
I20260709 12:23:01.514277 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.529235 28336 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:01.517046 27584 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:23:01.529258 27659 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:01.453467  6824 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.535712  6944 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:01.536144  6935 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:23:01.447613  6846 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.536424  6519 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.529947 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:01.538127 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:01.540095 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73652.5us
I20260709 12:23:01.559509 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:01.528756  6945 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.464970  6825 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.552666 27655 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:01.472118  6929 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:01.551469 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:01.569216 27734 raft_consensus.cc:606] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:01.569571 27734 raft_consensus.cc:614] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:01.569468 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.564172 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:01.564550 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.431988  6924 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:01.565019 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 305505us
I20260709 12:23:01.565762 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2010) took 1162 ms. Trace:
I20260709 12:23:01.583304 27662 rpcz_store.cc:276] 0709 12:23:00.404249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.449089 (+ 44840us) service_pool.cc:224] Handling call
0709 12:23:00.449434 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.472424 (+ 22990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.472459 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.507977 (+ 35518us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.823471 (+315494us) spinlock_profiling.cc:243] Waited 8.56 ms on lock 0x7b1800344ce0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:01.256227 (+432756us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:01.256269 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.256331 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.557034 (+300703us) raft_consensus.cc:1685] finished
0709 12:23:01.557063 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.565680 (+  8617us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22656,"raft.queue_time_us":96824,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6217,"spinlock_wait_cycles":19683200,"thread_start_us":432571,"threads_started":1,"wal-append.queue_time_us":871249}
I20260709 12:23:01.540546  6644 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.528525  6859 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.558383  6852 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.563828 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:01.564770  6775 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:01.539510  6832 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.581496  6857 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.629841 27659 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:01.598678 27660 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:01.581884  6938 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.572937  6929 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.628317 27658 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:01.687099 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.572947  6947 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:23:01.548205  6824 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.686344  6852 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.556025  6944 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:01.681638 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 216304us
W20260709 12:23:01.680898 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81706.8us
I20260709 12:23:01.709934 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.709618 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.682380 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.715258 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:01.679596 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 115821us
W20260709 12:23:01.694199 27589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 2036) took 1646 ms (client timeout 1842 ms). Trace:
W20260709 12:23:01.665462  6937 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.305s	user 0.001s	sys 0.000s
I20260709 12:23:01.721684 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:01.610802  6566 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.696732  6948 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
W20260709 12:23:01.712248  6940 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.314s	user 0.001s	sys 0.000s
I20260709 12:23:01.725857 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4747) took 1981 ms. Trace:
W20260709 12:23:01.694397  6941 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:23:01.742172 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:01.742213 27585 rpcz_store.cc:276] 0709 12:22:59.744445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.744590 (+   145us) service_pool.cc:224] Handling call
0709 12:22:59.744994 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.751984 (+  6990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.752034 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.782067 (+ 30033us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.722745 (+940678us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b18004d5900. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:01.102592 (+379847us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:23:01.102637 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.102709 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.719403 (+616694us) raft_consensus.cc:1685] finished
0709 12:23:01.719429 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.725773 (+  6344us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15913,"raft.queue_time_us":143,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7309,"spinlock_wait_cycles":6241664,"thread_start_us":379663,"threads_started":1,"wal-append.queue_time_us":1557107}
I20260709 12:23:01.750098 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:01.751085 27585 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:01.718434 27589 rpcz_store.cc:269] 0709 12:23:00.047645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.049446 (+  1801us) service_pool.cc:224] Handling call
0709 12:23:01.694015 (+1644569us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540008a2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:01.694141 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3023571712}
I20260709 12:23:01.728237 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:01.733994 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 133007us
I20260709 12:23:01.735603 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.754875 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.665740  6951 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.737180 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:01.737305 27580 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:01.739854 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.705542  6910 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.755517 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:01.756012 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.757288 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:01.758929 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.699906  6761 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.686305  6929 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:01.687451  6923 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:01.765829 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:01.711381  6947 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.597215  6644 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.713658  6938 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.784759  6644 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.785064  6938 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.785959  6644 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:01.786587  6644 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.762887 27659 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:01.772696  6910 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.740943 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.765219 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:01.789397  6910 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.789911  6923 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.790340  6923 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:01.698719  6846 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:01.735359  6745 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.686s	user 0.008s	sys 0.003s
I20260709 12:23:01.742472  6566 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.791870  6745 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.793242  6566 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.794275  6745 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.766527 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:01.696910  6519 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.760613 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:01.796217 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:01.798408  6828 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.689316  6611 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.076s
I20260709 12:23:01.799530  6825 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.780375  6961 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.800526  6961 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.802070  6852 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.802623  6852 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:01.770241  6951 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.759579  6454 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:01.806357  6828 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.752822  6891 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:23:01.781356  6824 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.770500  6859 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:01.806933  6828 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.807936  6859 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:01.808043 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4496) took 1493 ms. Trace:
I20260709 12:23:01.808385  6859 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:01.695641  6857 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.810738  6828 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.806329  6951 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.812896  6857 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.782799  6947 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:01.747298  6918 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.053s
I20260709 12:23:01.803221  6852 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:01.808410 27730 rpcz_store.cc:276] 0709 12:23:00.314494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.314650 (+   156us) service_pool.cc:224] Handling call
0709 12:23:00.325463 (+ 10813us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.400208 (+ 74745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.400240 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.400720 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:23:01.113800 (+713080us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:01.113857 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.807674 (+693817us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:01.807718 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.807766 (+    48us) raft_consensus.cc:1685] finished
0709 12:23:01.807781 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.807984 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6419,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":18257,"spinlock_wait_cycles":170905856,"thread_start_us":663311,"threads_started":1,"wal-append.queue_time_us":907614}
I20260709 12:23:01.827042  6944 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.827621  6944 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:01.828073  6944 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.828780  6775 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:01.830137  6944 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:01.834179 27663 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:01.836903 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:01.845531 27656 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:01.803166  6961 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:01.837909  6713 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.699s	user 0.012s	sys 0.000s
I20260709 12:23:01.853615  6519 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.004s
I20260709 12:23:01.859424  6924 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.871953  6689 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.873665  6644 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:01.850834  6930 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42.
I20260709 12:23:01.875880  6871 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:01.884137  6945 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.838232  6332 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:01.869335  6891 consensus_queue.cc:1243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 39.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:01.870065  6900 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:01.902102 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.903314 27734 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:01.905737 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:01.906584 27730 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:01.907186 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.908644 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.909291 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:01.911232 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.897397 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.898563 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.895798  6924 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:01.901465 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.893648  6871 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:01.852910  6846 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:23:01.893783  6689 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.894145  6745 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.902040 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.850s (SENT) [suppressed 18 similar messages]
W20260709 12:23:01.894469 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 125929us
I20260709 12:23:01.896456  6822 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:01.913508 27737 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:01.876603  6713 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.915314 27663 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:01.916050 27732 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 26: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:01.906400  6900 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.923415 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.924244 27737 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:01.918361 27734 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:01.925434 27734 raft_consensus.cc:606] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:01.925845 27734 raft_consensus.cc:614] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:01.928000 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.929374 27588 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 33 because replica has last-logged OpId of term: 31 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 23.
I20260709 12:23:01.930265 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:01.930984 27732 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:01.932459 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:01.934185 27737 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:01.936511 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.937692 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.938663 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:01.944832 27738 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:23:01.946585 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.948370 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.950989 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:01.951665 27738 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:01.951103 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.955651 27587 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:01.956084 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.956503 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:01.938558 27582 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:01.902196  6332 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:01.960955 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.965428 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:01.921924 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:01.966166 27732 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:01.968634 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:01.972817 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:01.973817 27582 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:01.921466 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:01.917599  6822 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.917042  6924 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:01.987690 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.988417 27738 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
W20260709 12:23:01.982679 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61265.1us
W20260709 12:23:01.984939 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 226739us
I20260709 12:23:01.993547 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.993803 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.994304 27656 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:01.919476  6745 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.001991 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:02.002380 27737 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:02.002683 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:01.999006 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:02.003458 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.000496 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:01.991267 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:02.007546 27582 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:02.007984 27582 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:02.004539 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:02.008392 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.009083 27738 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:02.009397 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:01.980017  6697 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.960489  6832 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:02.016839 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:02.017304 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.012640 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:02.013531 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:02.008752 27582 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.025552 27582 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:02.014528 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.026760 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:02.027383 27661 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:02.014998 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:01.994472  6852 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:02.024662 28333 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:02.018502 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:01.955336  6962 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.087s
I20260709 12:23:01.935532  6900 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:02.032585 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 130561us
I20260709 12:23:02.034287  6978 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:02.132380 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:02.022159  6857 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:02.137616 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:02.031697  6802 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.007508  6970 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:02.014461  6791 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:02.031172  6611 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:02.162711 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:02.034546  6973 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:02.168793  6974 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.307799  6973 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:02.307994  6974 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.308815  6973 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.310832  6974 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:02.131870  6919 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:02.131847  6945 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:02.026950  6697 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.165824  6848 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:02.288055 27661 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:02.168623  6791 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:02.156499  6821 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.800s	user 0.010s	sys 0.001s
I20260709 12:23:02.154558  6978 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:02.169180 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 177975us
W20260709 12:23:02.254065  6931 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260709 12:23:02.303786 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 141115us
I20260709 12:23:02.355758 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:02.168889  6857 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:23:02.292263  6923 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.501s	user 0.014s	sys 0.001s
I20260709 12:23:02.279816  6531 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.320791 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:02.362289 27738 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:02.320158  6848 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.301317  6970 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:02.292456  6611 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:02.332574  6697 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.340490 27582 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 37 index: 22. (term mismatch)
I20260709 12:23:02.378436 27582 pending_rounds.cc:85] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 21
W20260709 12:23:02.342106  6972 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:23:02.342893 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:02.344666 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.344811 27660 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:02.357632 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.315364  6965 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:02.174973  6802 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.322965  6945 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.345597  6791 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:02.377676 27738 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:02.372743  6848 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:02.378762 27582 pending_rounds.cc:107] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 36.22
I20260709 12:23:02.347618  6919 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:02.357173  6689 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.355198  6978 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:02.382567 27729 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
W20260709 12:23:02.358632  6859 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.549s	user 0.008s	sys 0.002s
I20260709 12:23:02.395167 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.389755  6980 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:02.348811  6821 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.401968  6611 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.361059  6827 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.029s
W20260709 12:23:02.387769  6929 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.593s	user 0.005s	sys 0.004s
I20260709 12:23:02.380471  6964 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:02.398885 27582 raft_consensus.cc:2889] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:02.385301  6965 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:02.379724  6531 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.313416  6947 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.372356  6923 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.434217 27585 raft_consensus.cc:2368] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39.
I20260709 12:23:02.415741  6944 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:02.402508  6919 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.431629  6822 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:02.424265  6689 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:02.430691  6965 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:02.452679 27737 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:23:02.412421  6949 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.037s
W20260709 12:23:02.453346 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 110557us
I20260709 12:23:02.460023 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:02.413926  6859 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.461805 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.471614 28333 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:02.471815 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:02.469224  6920 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.429332  6974 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:23:02.440788  6900 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.437844  6944 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:02.498919 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.442278  6924 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:02.445034  6713 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.473158  6920 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.531795  6920 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:02.533522  6923 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:02.536818  6947 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.543586  6986 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:02.496624 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:02.546437 27581 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:23:02.462462  6962 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:02.489051 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.562999  6962 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.565917 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.566642  6802 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.490845  6961 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:02.567677  6961 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:02.497651  6977 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:23:02.568907  6961 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.570654 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:02.454878  6822 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:02.429513  6929 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.572206 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:02.574162  6929 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.501829 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.576020 27655 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:02.503852 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.538913  6531 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:02.579119 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.580889 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.504437 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.585162 27582 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:02.585492 27585 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:02.586117 27585 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:02.570914  6961 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:02.508009 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.464159  6332 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:02.512498 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 156763us
I20260709 12:23:02.590332 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:02.590839 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.592422  6962 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.516328  6945 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:02.517576 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:02.432776  6964 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.598850  6964 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.599956 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:02.521128  6372 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.151s	user 0.012s	sys 0.000s
I20260709 12:23:02.427374  6857 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.525435  6859 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:02.526846  6821 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:02.527168 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.514151  6944 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:02.547695 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.617136 27581 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:02.485133  6943 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:02.440448  6970 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.606029 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.609213 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:02.622799 28328 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:02.617648 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.620491 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.620627 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:02.624958 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.625272 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.625319 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:02.624960 28336 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 30.
W20260709 12:23:02.624653 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206825us
I20260709 12:23:02.587188 27585 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.627257 27585 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:02.632217 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 135707us
I20260709 12:23:02.638437 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.646396 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:02.651420  6945 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:02.651485  6519 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:02.652053  6945 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.652467  6945 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:02.659687 28328 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:02.609086  6974 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.661643 28335 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:02.644567 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.615109  6953 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:02.669013 28295 leader_election.cc:400] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39.
I20260709 12:23:02.670838 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.674763 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:02.679584 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 162100us
I20260709 12:23:02.682327 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:02.684177 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.685352 28335 raft_consensus.cc:2410] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 33 because replica has last-logged OpId of term: 31 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 23.
I20260709 12:23:02.687074 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.690095 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:02.681622 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:02.692220 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49556.3us
I20260709 12:23:02.694834 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.693589 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:02.630468  6822 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.699899 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.704859 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.702760 27585 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 19. (term mismatch)
I20260709 12:23:02.703933 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:02.707304 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.705665 27585 pending_rounds.cc:85] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 18
I20260709 12:23:02.711226 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.713510 27584 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:02.715517 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:02.616479  6929 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.077s
I20260709 12:23:02.711069 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:02.720382 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.720383 28335 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:02.680470  6953 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.718216 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.691251  6775 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:02.692824  6611 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:02.726111 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:02.692446  6859 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:02.726117 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:02.722965 27585 pending_rounds.cc:107] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 33.19
I20260709 12:23:02.697134  6920 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:02.729417 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:02.703264  6986 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:02.737167 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:02.704476  6921 raft_consensus.cc:2764] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election decision vote started in defunct term 36: won
I20260709 12:23:02.738879 27585 raft_consensus.cc:2889] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:02.679384  6981 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:23:02.737668 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.710021  6852 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:02.741667 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.724157  6541 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:02.720610  6919 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:02.666313  6949 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:23:02.750774 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:02.777179 27582 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:02.743150  6924 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.720819  6332 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:02.733371  6953 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:02.696918  6969 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:23:02.724660  6947 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.630847  6943 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.743752  6821 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:23:02.748909 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:02.731104  6951 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:02.706223  6538 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
I20260709 12:23:02.714773  6697 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:23:02.733891  6775 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:02.700030  6848 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.747406  6832 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:02.756117 27587 raft_consensus.cc:2410] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 38 because replica has last-logged OpId of term: 34 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 20.
W20260709 12:23:02.781410 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65974.1us
I20260709 12:23:02.711997  6822 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.748078  6986 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:02.827534 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101523us
I20260709 12:23:02.753152  6923 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.812624  6949 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:02.688010  6974 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.739169  6991 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.062s
I20260709 12:23:02.772610  6454 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:02.813433 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.814205 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.790907  6827 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:02.757688  6859 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:23:02.791584  6989 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:02.825804 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.758683  6978 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.828331 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:02.823407 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:02.824251 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75433.8us
W20260709 12:23:02.823170 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95568.7us
I20260709 12:23:02.806960  6951 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.743494  6372 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:23:03.002166 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:03.122529 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.823009  6775 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:03.020656  6953 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:02.980835  6891 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:02.990124  6375 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:02.997725 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2051) took 1550 ms. Trace:
I20260709 12:23:03.019265  6745 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.007987 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.879457  6871 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.205138  6955 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:03.203554  6960 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.010s
I20260709 12:23:03.213354  6871 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:03.206952  6375 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:03.213356  6955 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.214576  6955 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:03.214720  6375 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.215910  6375 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:03.216902  6953 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.206779  6891 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:03.036221 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 354661us
I20260709 12:23:03.218843  6891 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.219851  6891 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:03.019289  6944 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.144467  6825 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.792940  6947 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.084398  6852 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:23:02.898839  6993 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:23:02.896106 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.203096  6951 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:23:03.178429 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 484874us
W20260709 12:23:02.982731  6965 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.512s	user 0.008s	sys 0.000s
W20260709 12:23:03.012439  6791 consensus_meta.cc:287] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.609s	user 0.010s	sys 0.000s
I20260709 12:23:02.877552  6822 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.018312  6988 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:23:02.790468  6924 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.021471  6920 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:23:03.159189 27481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 133) took 18417 ms (client timeout 15000 ms). Trace:
I20260709 12:23:02.835206  6986 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:02.898015  6994 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:23:02.992058  6998 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:02.798938  6332 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.896842 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:02.903899  6990 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 12:23:02.794636  6969 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:23:02.986598  6943 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.013185 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:03.016917 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4514) took 1120 ms. Trace:
I20260709 12:23:03.281181 27731 rpcz_store.cc:276] 0709 12:23:01.896947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.897096 (+   149us) service_pool.cc:224] Handling call
0709 12:23:03.016693 (+1119597us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:03.016847 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238402304}
I20260709 12:23:02.885396  7000 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:02.878620 27589 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:23:03.017933  6979 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:03.283902  6979 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:03.285302  6979 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.841576  6832 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:03.288201  6832 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:02.877945  6974 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:02.899405 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:02.997637  6848 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.293485  7000 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:02.822758  6985 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:02.983059 28337 raft_consensus.cc:2368] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37.
I20260709 12:23:03.294577  7000 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.000527 27580 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:23:03.186851 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:02.975028 27584 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:03.297725  6332 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.297999 27733 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
W20260709 12:23:03.193912 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 370587us
I20260709 12:23:03.282846 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.286972  6979 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:03.187831  6857 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.011s	sys 0.000s
W20260709 12:23:03.196607 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 368328us
W20260709 12:23:03.349210  6857 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.011s	sys 0.001s
I20260709 12:23:03.201709 27587 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:03.183207  6930 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:03.186384  6827 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:03.362592 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.224332 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:03.225997 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 446699us
I20260709 12:23:03.191854  6989 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:03.363924  6989 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:03.229270 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:03.204306  6538 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:02.844684  6923 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.368257 27582 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:23:03.369705  6986 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.370033  6923 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.230482 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
W20260709 12:23:03.239974 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.011s	sys 0.006s
W20260709 12:23:03.372123 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.011s	sys 0.006s
I20260709 12:23:03.223183  6825 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.373205 27733 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
W20260709 12:23:03.229137  6945 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.000s
I20260709 12:23:03.243916 27729 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:03.244473 27735 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:23:03.204779  6454 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:03.228574  6961 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:03.236706  6372 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.238960  6541 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:03.245435  6988 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:03.194911 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:03.278031  6998 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.391549  6821 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:03.391242  6541 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.392004  6821 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.392446  6821 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:03.248888 27481 rpcz_store.cc:269] 0709 12:22:44.734658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.497452 (+762794us) service_pool.cc:224] Handling call
0709 12:23:03.151262 (+17653810us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:55.926543 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:55.988550 (+ 62007us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:55.988601 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:22:55.988622 (+    21us) tablet.cc:662] Decoding operations
0709 12:22:55.989844 (+  1222us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:55.989934 (+    90us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:55.989961 (+    27us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:22:55.991054 (+  1093us) tablet.cc:701] Row locks acquired
0709 12:22:55.991139 (+    85us) write_op.cc:260] PREPARE: finished
0709 12:22:55.991308 (+   169us) write_op.cc:270] Start()
0709 12:22:56.117270 (+125962us) write_op.cc:276] Timestamp: P: 1783599775991241 usec, L: 0
0709 12:22:56.121612 (+  4342us) op_driver.cc:348] REPLICATION: starting
0709 12:22:56.264579 (+142967us) log.cc:844] Serialized 4330 byte log entry
0709 12:22:57.618612 (+1354033us) op_driver.cc:464] REPLICATION: finished
0709 12:22:57.770652 (+152040us) write_op.cc:301] APPLY: starting
0709 12:22:57.783674 (+ 13022us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:58.027842 (+244168us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:58.027978 (+   136us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:58.297082 (+269104us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:58.301561 (+  4479us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0709 12:22:58.301908 (+   347us) write_op.cc:312] APPLY: finished
0709 12:22:58.302374 (+   466us) log.cc:844] Serialized 105 byte log entry
0709 12:22:58.302603 (+   229us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:58.302756 (+   153us) write_op.cc:454] Released schema lock
0709 12:22:58.302833 (+    77us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:58.307920 (+  5087us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:58.307958 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2547023,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":114436,"apply.run_cpu_time_us":20875,"apply.run_wall_time_us":543243,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50107,"num_ops":11,"prepare.queue_time_us":204982,"prepare.run_cpu_time_us":33990,"prepare.run_wall_time_us":707777,"raft.queue_time_us":175719,"raft.run_cpu_time_us":21569,"raft.run_wall_time_us":77443,"replication_time_us":1489916,"thread_start_us":536152,"threads_started":4,"wal-append.queue_time_us":620546,"wal-append.run_cpu_time_us":31928,"wal-append.run_wall_time_us":2418053}]]}
I20260709 12:23:03.309650 27730 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:03.311203 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:03.391572 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 196733us
W20260709 12:23:03.394516 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93564.5us
I20260709 12:23:03.394912 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:03.275418  7007 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.397238 27580 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:03.264230  6745 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.054s
I20260709 12:23:03.319419  6944 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.320335  6947 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:03.320402 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 307284us
I20260709 12:23:03.321453 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.321815  6969 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:03.322374 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.322570 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.323278 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.419993 27657 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:03.420599 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:03.421072 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.421433 27660 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:03.324411 27661 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:23:03.422811 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:03.423542 27661 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:03.325361 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.426522 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.326328 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:03.427704 27659 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:03.327314 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:03.406224 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:03.328287 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.329694  6978 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.293258 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:03.431080 27657 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:03.335078 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:03.340411 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:03.245733  6791 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.242599  6965 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.196852 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.449186 27736 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:23:03.340338  6924 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.362115  6827 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.455080 27660 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
W20260709 12:23:03.455374 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 49282.9us
I20260709 12:23:03.207113 27586 rpcz_store.cc:276] 0709 12:23:01.447164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.448288 (+  1124us) service_pool.cc:224] Handling call
0709 12:23:01.465035 (+ 16747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:01.590281 (+125246us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:23:01.597777 (+  7496us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.724801 (+1127024us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b180037f7e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:23:02.830093 (+105292us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:02.830148 (+    55us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:02.830198 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.997453 (+167255us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:02.997509 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.997657 (+   148us) inbound_call.cc:177] Queueing success response
0709 12:23:03.006184 (+  8527us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b180037f7e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
Metrics: {"prepare.queue_time_us":26891,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":223,"raft.queue_time_us":79,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":7442,"spinlock_wait_cycles":441235968,"thread_start_us":105072,"threads_started":1,"wal-append.queue_time_us":1246108}
I20260709 12:23:03.363040  6930 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:03.379940 27734 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:03.379119 27582 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:03.388231 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.313850  6955 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.404243  6998 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.403795  7007 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.400295  6538 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:03.411002 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:03.431993 28337 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:03.465158 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.459594 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46859.2us
I20260709 12:23:03.436988 27661 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:03.387840  6671 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:23:03.437873 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.404979  6133 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:03.386497  6961 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:03.409293  6579 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:23:03.454521 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119517us
I20260709 12:23:03.483505  6579 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.483970  6579 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:03.483897 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:03.458297 27659 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:03.456143 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:03.486143 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:03.486963  6924 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:03.487605  6924 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:23:03.488833 27702 leader_election.cc:400] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37.
I20260709 12:23:03.489185 27702 leader_election.cc:403] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:03.489584  6951 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:03.492801 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.495103 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:03.497761 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:03.497988  6965 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.498458 28333 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:03.468824  6454 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:03.458717 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 202570us
I20260709 12:23:03.386477  6709 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:23:03.386811  6945 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.468367 27737 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:03.400537  6372 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.470170 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.409051  6950 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.052s
I20260709 12:23:03.459362  7004 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:03.472939  6133 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:03.473771  6961 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.419015  6988 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.527302  6969 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:23:03.527854  6966 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:03.528213 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44404.2us
I20260709 12:23:03.574141 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:03.577229  6953 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:03.577548  6988 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:03.577641  6953 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:03.578579  6953 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.569558  6745 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.579782  6953 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:03.580109  6745 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.582307  6745 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.582088  6821 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.582711  6372 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.583364  6961 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:03.459182  7006 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.121s
I20260709 12:23:03.573058  6969 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.586251  6945 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.586521  6969 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:03.569558  7012 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.538717  6802 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.591770  6133 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.592621  6802 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.593664  6989 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:03.528615 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.595548 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.528716  6918 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:03.596355  6821 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:03.596709 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.596814  6918 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.565313  6859 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:23:03.529081  6566 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:03.530627  7009 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:03.467041  6827 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:03.599264  7009 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.599433  6966 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.602273 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.459914  6944 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:03.418226  6923 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.048s
I20260709 12:23:03.453960  6852 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.060s
I20260709 12:23:03.603922  6852 consensus_queue.cc:1243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 40.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:03.604437  6791 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:03.539085 27731 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:03.605165 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:03.606073 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.607076  6566 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:03.607514  6566 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:03.514503  6986 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.541329 28295 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:03.617596 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76334.2us
W20260709 12:23:03.617805 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.533656  6538 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:03.554004 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.565274 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.619725 27733 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:03.559759 27736 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:03.465852  6825 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:03.595220 27738 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:03.562901  6965 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:03.604377 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:03.518448 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.470969  6531 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:23:03.608203 27732 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:03.627805 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.628953 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.630647 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.634318 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.608469 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:03.609130 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.636274 27582 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:03.610378 27734 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:03.612152 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43515.8us
I20260709 12:23:03.614614 27580 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: [38.22-38.22]
I20260709 12:23:03.640638 27702 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:03.641007 27702 leader_election.cc:403] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
W20260709 12:23:03.642299 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68264.7us
I20260709 12:23:03.614732 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:03.622195 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.643370 27736 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:03.624198 28337 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
W20260709 12:23:03.539364 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 128481us
I20260709 12:23:03.644464 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.644930 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:03.645133 27583 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:03.623574 27655 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:03.625396 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:03.644512 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:03.616463  6566 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:03.651683  7001 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.652238  7001 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.652730  6944 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:03.653225  6944 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.654285  7001 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:03.606102  6858 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:23:03.594730  6802 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.645968 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:03.659621 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44989.1us
I20260709 12:23:03.677474 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.657403 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:03.662128 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.667500 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.682098 27731 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:03.682202 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73792.4us
I20260709 12:23:03.683797 27582 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:03.684867 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:03.685474 27734 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:03.686556 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.687407 27582 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:03.688536 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.690008 27655 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:03.623538  6745 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.710026  6910 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:23:03.710299  6745 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.711777 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.712481  6745 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.713493 27731 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:03.714293 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:03.602274  6579 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.717034  6579 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.717744  6929 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:03.718565  6969 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.719135  6929 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:03.720491  6929 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:23:03.721822  7004 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:03.712489 27581 raft_consensus.cc:2368] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:03.623111  6998 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37.
I20260709 12:23:03.723346 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.681979  6858 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:03.724089 27581 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:03.724848 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.727986 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.667028 27733 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:23:03.707979  6802 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:03.708364  6924 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:03.691413  6923 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:03.591776  7012 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.708755  7016 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:03.709287 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.782152 27586 raft_consensus.cc:2410] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 42 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 19.
I20260709 12:23:03.709626 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.693002  6541 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:03.724916 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.701571  6951 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.670567  7001 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:03.784226 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.784816 27729 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:03.785707 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.786744 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.694469 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:03.788053 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93670.9us
I20260709 12:23:03.625540  6921 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:03.788508  7016 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:03.788848  6921 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:03.789289  7016 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:03.789834  6921 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.790414  7012 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.692296 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.626426  6531 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:03.748824 27581 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:03.750458 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:03.792304 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.795311 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:03.796684 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.747872 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.754329 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.651697  6848 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:03.761559  6858 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.762660 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.755894  6924 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:03.821722  6825 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:03.764683  6989 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.768548  6966 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:03.679989  6372 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:23:03.773170  6775 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.549s	user 0.009s	sys 0.001s
I20260709 12:23:03.777053  6969 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.778076  6944 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:03.818986  7012 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:03.722924  6566 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.787262 27729 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:03.832453  6924 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:03.631937  6965 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.820843 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.823141 27736 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:03.853649 27733 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
W20260709 12:23:03.825258 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.854770 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.764038 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.856281 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.856671 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.831866 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.834784 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.857477 27730 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:03.823797  6827 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:03.794745  6802 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:03.822844  6910 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:03.780036  6955 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:03.842427 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.843658 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:03.823681  6745 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:23:03.805814 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58057.6us
I20260709 12:23:03.859369 27581 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 39 because replica has last-logged OpId of term: 37 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:23:03.844383  6825 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:03.846251  6858 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:23:03.868137 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.868378 27585 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:03.861434 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147211us
I20260709 12:23:03.859630  6944 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:03.875211 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.882068  6745 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:03.883361  6745 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.889151  6930 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.890255  6930 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:03.790412  7016 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.902053  6802 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.904791 27737 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:23:03.906437 27737 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:23:03.878211  6910 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:03.911619  6910 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.915642  6825 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.918911  6945 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:03.919353  6945 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:03.863700  6644 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.056s
I20260709 12:23:03.920403  6945 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.921938  6965 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:03.923264 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.881255 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.925381 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.953720 27581 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:03.954670 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:03.878983 27585 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
W20260709 12:23:03.869328 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 211980us
I20260709 12:23:03.945977  6944 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.957649  6538 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:03.920415  6644 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:03.958910  6644 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:03.952848 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.969072 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.971135 27657 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:03.927889 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:03.973760 27656 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 38. Ops: []
I20260709 12:23:03.860613  6827 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:03.930712 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:03.981505 27659 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:03.932296 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.983361 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.984196 27661 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:03.985339 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.986644 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.933457 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:03.933772 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.988265 27583 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:23:03.933879 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.935830 27735 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:23:03.936290 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.989475 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:03.936550 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:03.989986 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:03.937328 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.991514 28295 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:03.979137 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:03.937894 27731 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:03.941447 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:03.845559  6775 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.958175  6538 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:03.930349 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:03.879372  6891 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.659s	user 0.010s	sys 0.001s
I20260709 12:23:03.979223 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:03.997953 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 119453us
W20260709 12:23:04.003628 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.016227 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.020954 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.021863 27661 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 25. Current term is 26. Ops: []
I20260709 12:23:04.023269 27735 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:23:03.996081 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:03.978397 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.028486 27731 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:04.023394 27729 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 43.
I20260709 12:23:04.026232 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.006299 27552 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:04.024307 27655 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:23:04.029791 27731 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [37.21-38.22]
I20260709 12:23:04.026850 27588 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:04.031389 27731 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 39. Ops: [35.19-37.20]
I20260709 12:23:04.025650 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.034958 27656 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:04.036042 27661 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 27.
I20260709 12:23:04.038014 27731 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:04.039027 27729 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:23:04.039789 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
W20260709 12:23:04.040624 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.040896  7012 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:04.043874 27661 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:04.044660 27731 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:04.045264 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:03.959306  6944 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:03.978852  6671 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.049086 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:04.051523 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:04.000394  6827 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:04.025358  6978 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:23:04.053303 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 116862us
I20260709 12:23:03.982503  6372 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:04.053222 27664 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:04.053325 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:04.010044  6955 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:04.092351 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:04.093564 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:04.094671 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:04.095446 27659 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:04.024485  6991 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:04.086629 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:04.052903  6579 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:04.034461  6966 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.067849 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.089107  6671 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:03.869205  6531 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.050740  6859 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:04.085855  6969 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:04.141992 27656 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:04.139191 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:04.090906  6978 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.101375  6846 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:04.053082  6894 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.088s
I20260709 12:23:04.185567 27732 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:04.186841 27658 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:04.110620  6929 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:04.186964  6918 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.170773 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.172554 27662 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:23:04.192412 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:04.184506 27656 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [36.22-38.23]
I20260709 12:23:04.194145 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:04.194970 27656 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:04.185179  6671 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.135874  6372 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:04.141054  7001 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:04.161012  7017 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260709 12:23:04.154578  6989 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.195472 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:04.215647 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.054348  6961 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:04.050503  7023 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.155522  6951 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:04.154919  6891 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:04.152658  6969 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.274458 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:04.156160  6859 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.176445  6978 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:04.202724 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.285650 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:04.286150 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:04.286738 27656 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:04.213321 27661 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:04.180404  6846 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.164623  6579 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:04.223043 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.143647  7008 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.064s
I20260709 12:23:04.137699  6991 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:04.306553  7020 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260709 12:23:04.308000  6991 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.248945 27658 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:04.309477 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:04.231057  6566 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:04.251811 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.254274 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:04.255743 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:04.197230  7015 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.266113  6951 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.198715  6923 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:04.149382  6966 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.061153 27583 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:23:04.281991  6969 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:04.100993  6644 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.289129 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 348036us
I20260709 12:23:04.292753 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:04.293076 27737 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [38.23-38.23]
I20260709 12:23:04.295890 28336 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:04.303588  6846 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:04.304778  6579 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.28, Last appended by leader: 28, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.250592 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4526) took 1441 ms. Trace:
I20260709 12:23:04.311933 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:04.232090  6989 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.361224 27738 rpcz_store.cc:276] 0709 12:23:02.809275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.810155 (+   880us) service_pool.cc:224] Handling call
0709 12:23:02.810504 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.621286 (+810782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.250267 (+628981us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b540029f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:04.250514 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1804196096}
I20260709 12:23:04.314033 27658 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 31: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:04.106572 27659 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:04.108738  7024 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:04.319188 27656 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 43.
I20260709 12:23:04.325611 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:04.281831 28337 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:04.265154  6891 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:04.270838  7023 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.332135 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.213935  7001 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.313680  6825 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:04.409735 27655 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 39. Ops: [35.19-37.20]
I20260709 12:23:04.372651 27729 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: [38.22-38.22]
I20260709 12:23:04.329617  6951 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:04.380746 27658 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
W20260709 12:23:04.398240 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.401958 27734 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:04.334923 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:04.407469 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 320894us
I20260709 12:23:04.407567 27730 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:04.406960  7012 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:04.367805  7016 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:04.412784 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:04.415673 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80827.9us
W20260709 12:23:04.409852 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77810.3us
I20260709 12:23:04.380980  6891 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:04.418643  6998 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:04.420475  6945 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:04.457120 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 424912us
W20260709 12:23:04.427886  7026 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260709 12:23:04.431071 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.464139 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.431412 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.444140 27658 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:04.326469  7015 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.450803 27659 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:04.444165 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:04.452135  6955 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:04.445374 27657 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
W20260709 12:23:04.423065  7027 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:23:04.383273  6775 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.471925 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.492271 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.493728 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.496378  6947 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:04.496919  6947 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:04.497390  6947 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:04.430511  6923 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:04.500581  6923 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:04.442521  7016 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.501580  7016 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:04.509518 27729 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:23:04.511071 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.359428  6372 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.495075 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.444883  6991 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:04.471288  6998 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:04.533365  7003 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.429530  6825 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.536835  6454 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:23:04.188505  6791 consensus_queue.cc:1175] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 29 status { last_received { term: 28 index: 19 } last_committed_idx: 17 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 28. Current term is 29. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:04.542671  7016 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.546715 27704 leader_election.cc:400] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:04.574975 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.513947  6929 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:04.537426  7012 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.594847 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.519586  6991 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.598775 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:04.604854 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.605742 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.609181 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:04.611439 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75031.1us
I20260709 12:23:04.617208 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:04.621315 27584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.006s	sys 0.005s
W20260709 12:23:04.614603 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93500.3us
I20260709 12:23:04.550254  7021 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.626078 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:04.551823  6538 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:04.622576 27584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.006s	sys 0.005s
I20260709 12:23:04.603057  6454 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:04.612998  7012 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:04.561364  6961 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.072s
I20260709 12:23:04.641124 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:04.613844  6991 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:04.652436 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.653275  7045 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:23:04.654412  7034 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:23:04.681463  6848 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:04.681902  6848 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.682260  6848 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:04.654609  6825 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.623282  7035 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:04.612897  7039 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:04.562785  7003 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.612834  6929 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.038s
I20260709 12:23:04.666591  7016 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.669353  6961 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.669719  7015 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.632929  6921 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.037s
I20260709 12:23:04.683341  6825 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:04.688994  6825 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:04.672412  7029 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:23:04.673189  6454 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.686827  7003 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.661963  6944 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:23:04.687458 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46451.1us
I20260709 12:23:04.619244  6951 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.698110  6827 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.589s	user 0.008s	sys 0.003s
I20260709 12:23:04.687582  6929 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:04.695639  6859 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.699927  6745 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:04.700110  7015 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:04.705463  6989 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.696081  7055 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:04.753788  6924 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.892s	user 0.011s	sys 0.004s
I20260709 12:23:04.719156  7001 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.719957  7015 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.759397  7055 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.761493  6969 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.710398  6929 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:04.761554  6924 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.723731  7023 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.764371  7055 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.658643  6955 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.765344  7015 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.766453  6947 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.688266  6961 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.758654  6930 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.868s	user 0.012s	sys 0.000s
I20260709 12:23:04.638656  7021 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.772775  7021 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.699379  6998 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.18, Last appended by leader: 18, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.755642  6978 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.777244  6978 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.764060  6929 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.779884  6929 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:04.757517  6991 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.770996  6961 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.770996  6947 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.786094  6991 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.788364  6969 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:04.832082 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 383371us
W20260709 12:23:04.844976 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.768343  6924 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:04.892868 27659 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
W20260709 12:23:04.894791 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.708312  6827 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.898842 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.900581  6827 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.858652  6961 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:04.933436  6848 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:04.877324 27660 raft_consensus.cc:1217] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Deduplicated request from leader. Original: 33.20->[34.21-36.22]   Dedup: 34.21->[36.22-36.22]
I20260709 12:23:04.888325 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:04.901501 27735 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:04.906455  6947 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.908718  6945 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.910305 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4542) took 1342 ms. Trace:
I20260709 12:23:04.911497  6745 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.913043 27580 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:23:04.691259  6825 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.917054 27585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4790) took 1289 ms (client timeout 1608 ms). Trace:
W20260709 12:23:04.921196 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4634) took 1600 ms (client timeout 1850 ms). Trace:
I20260709 12:23:04.932453 27656 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
W20260709 12:23:04.876931 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.910053 27487 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:04.951628 27660 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: [34.21-36.22]
I20260709 12:23:04.978615  6848 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:04.980324  6947 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:04.983443 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44841.1us
I20260709 12:23:04.998760  6961 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.000897 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.985353 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:05.007460  6891 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.011s
I20260709 12:23:05.008088  6891 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:05.012513 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.015794  6891 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.017472  6825 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:05.024857 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4889) took 1412 ms. Trace:
I20260709 12:23:04.983268  6947 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:04.969573 27664 rpcz_store.cc:269] 0709 12:23:03.321013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.321148 (+   135us) service_pool.cc:224] Handling call
0709 12:23:04.920552 (+1599404us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b5400211e54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:04.920890 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1476843392}
W20260709 12:23:04.969187 27585 rpcz_store.cc:269] 0709 12:23:03.626868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.627591 (+   723us) service_pool.cc:224] Handling call
0709 12:23:04.916382 (+1288791us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:04.916711 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551790848}
I20260709 12:23:05.037850 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.038851  6531 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:05.043360 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:04.935117  7021 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.097s
I20260709 12:23:05.048082  6924 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:05.047943  6961 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.050810  6961 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.051998  6929 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:05.053010  6929 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.046720  6945 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:05.059821 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.039096 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:05.068475  6991 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:05.025424 27584 rpcz_store.cc:276] 0709 12:23:03.611924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.612093 (+   169us) service_pool.cc:224] Handling call
0709 12:23:03.612458 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:03.612893 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:23:03.865992 (+253099us) log.cc:844] Serialized 14 byte log entry
0709 12:23:04.985537 (+1119545us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.986343 (+   806us) log.cc:844] Serialized 26 byte log entry
0709 12:23:04.986559 (+   216us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:04.986595 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.986705 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.024368 (+ 37663us) raft_consensus.cc:1685] finished
0709 12:23:05.024395 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.024625 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4689,"prepare.queue_time_us":494629,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":59090,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":6083,"thread_start_us":1296653,"threads_started":3,"wal-append.queue_time_us":843560}
I20260709 12:23:05.069604  6991 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.074926  6991 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.075703  6978 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.080812  7051 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.081422  6978 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.081617  7051 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.083690 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.077417  7007 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:05.077770 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.067701  7067 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:05.104023  7033 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:05.107398  7067 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.103729 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.109127 27659 raft_consensus.cc:2410] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:23:05.109536  6929 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:05.110118  7067 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:04.965590 27731 rpcz_store.cc:276] 0709 12:23:03.567812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.093104 (+525292us) service_pool.cc:224] Handling call
0709 12:23:04.093450 (+   346us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:04.448133 (+354683us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.448160 (+    27us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:04.464557 (+ 16397us) log.cc:844] Serialized 78 byte log entry
0709 12:23:04.583729 (+119172us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:04.583782 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.583876 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:04.901912 (+318036us) raft_consensus.cc:1685] finished
0709 12:23:04.901981 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:04.909987 (+  8006us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":82202,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":9377,"thread_start_us":409565,"threads_started":2,"wal-append.queue_time_us":205837}
W20260709 12:23:05.084210 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.112872 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.115248 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.115356 27582 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:05.117112 27582 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:05.118477 27582 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:04.985118  6745 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.121436  6644 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:05.121973  6745 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:05.116130 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.086253  6827 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:05.087448 27733 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:23:05.123294  6827 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.089315 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.126376  6827 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.127287 27582 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:05.128379  6846 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.759s	user 0.015s	sys 0.011s
I20260709 12:23:05.128954  6846 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.130614 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.131567  6745 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:05.090233  7004 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.132125 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.132611  7004 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.134625 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.091672  7051 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:05.107134 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.105096 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:05.080384  6891 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.092695 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 107633us
I20260709 12:23:05.093060 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4065) took 1654 ms. Trace:
I20260709 12:23:05.095106 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4074) took 1618 ms. Trace:
I20260709 12:23:05.104146 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4551) took 1069 ms. Trace:
I20260709 12:23:05.110035 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.080065  7070 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.084769  6978 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.115622  6945 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:05.076395 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:05.092355  7015 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:05.137483 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.140971 27737 rpcz_store.cc:276] 0709 12:23:03.476922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.942952 (+466030us) service_pool.cc:224] Handling call
0709 12:23:03.943273 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.332979 (+389706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.094541 (+761562us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b54000a6ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:05.094912 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":803822464}
I20260709 12:23:05.146047 27659 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:23:05.105156 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:05.103277  6858 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.225s	user 0.002s	sys 0.021s
I20260709 12:23:05.147867  6846 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.153549 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:05.154007  7071 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.154960 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.155989 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.150945  7070 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.157583 27587 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:05.158149 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.145210  7015 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.158890 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:05.140648 27730 rpcz_store.cc:276] 0709 12:23:03.438522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.858090 (+419568us) service_pool.cc:224] Handling call
0709 12:23:03.858401 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.412334 (+553933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.092442 (+680108us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54000a41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:05.092838 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262371584}
I20260709 12:23:05.135190  7004 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.171998  6978 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:05.172500  6978 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:05.172966  6978 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:05.175495 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.176370 28335 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:05.176924 27657 raft_consensus.cc:2368] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:23:05.177696  6945 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.178480  6945 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.179787 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.180274 28336 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:23:05.180567 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:05.183676  7055 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:05.184917  6966 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.186955 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3889) took 1210 ms. Trace:
I20260709 12:23:05.187285 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:05.187587 27660 rpcz_store.cc:276] 0709 12:23:03.976980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.989676 (+ 12696us) service_pool.cc:224] Handling call
0709 12:23:04.876455 (+886779us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:04.975875 (+ 99420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.186690 (+210815us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x7b540022a950. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:05.186863 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1975114112}
I20260709 12:23:05.188870 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.189985 27731 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:05.189901 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:05.160516 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.193445 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3901) took 1090 ms. Trace:
I20260709 12:23:05.193692 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.194077 27661 rpcz_store.cc:276] 0709 12:23:04.102620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.102788 (+   168us) service_pool.cc:224] Handling call
0709 12:23:04.103064 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.303397 (+200333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.192820 (+889423us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54000e60d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:05.193210 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253021952}
I20260709 12:23:05.194675 27585 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:23:05.165410 27587 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:05.201512 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4076) took 1724 ms. Trace:
I20260709 12:23:05.167742  7015 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:05.168045 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 170210us
I20260709 12:23:05.168217  7071 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.212795  7070 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.213846  7071 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:05.214731  6823 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.215440  6823 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:05.170516 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 92966.6us
I20260709 12:23:05.219061 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.219208  6891 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.221894  6891 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.197049 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.224778 27588 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:23:05.225723 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:05.200603 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4063) took 1767 ms. Trace:
I20260709 12:23:05.226508 27733 rpcz_store.cc:276] 0709 12:23:03.433455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.854308 (+420853us) service_pool.cc:224] Handling call
0709 12:23:03.859257 (+  4949us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.123337 (+1264080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.200241 (+ 76904us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001d3dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:05.200535 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2822775680}
I20260709 12:23:05.228012 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.228785 27733 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:05.229864 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.230878 27657 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:05.163899 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.232344 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:05.233471  7028 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.720s	user 0.000s	sys 0.002s
I20260709 12:23:05.233937 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:05.196133 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:05.238698 27661 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:05.239655 27663 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:23:05.240775  6951 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.199275 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1845) took 1535 ms. Trace:
I20260709 12:23:05.242998 27734 rpcz_store.cc:276] 0709 12:23:03.664231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.843406 (+179175us) service_pool.cc:224] Handling call
0709 12:23:05.198835 (+1355429us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:05.199061 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1138058112}
I20260709 12:23:05.205598 27735 rpcz_store.cc:276] 0709 12:23:03.477411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.028764 (+551353us) service_pool.cc:224] Handling call
0709 12:23:04.029024 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.983378 (+954354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.201196 (+217818us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:05.201450 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1902486784}
W20260709 12:23:05.211090 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.222720 27580 raft_consensus.cc:2410] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
W20260709 12:23:05.245585 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.222895 27589 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:23:05.249742 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.251543  7067 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:05.252521  7067 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.253019  6965 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.253623  6965 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.254980  6823 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.255576  6846 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:05.256001 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.257061  6965 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:05.242607  7030 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.786s	user 0.004s	sys 0.001s
W20260709 12:23:05.257774  7031 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.839s	user 0.000s	sys 0.002s
I20260709 12:23:05.260304 27588 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: [39.21-39.21]
I20260709 12:23:05.261325 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4539) took 2425 ms. Trace:
I20260709 12:23:05.262329 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:05.262689 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.263005 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.263624 27585 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:05.261737 27736 rpcz_store.cc:276] 0709 12:23:02.835603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.858297 (+1022694us) service_pool.cc:224] Handling call
0709 12:23:03.858618 (+   321us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:03.859018 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:03.859054 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:03.861727 (+  2673us) log.cc:844] Serialized 52 byte log entry
0709 12:23:04.219147 (+357420us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:04.563302 (+344155us) log.cc:844] Serialized 14 byte log entry
0709 12:23:04.656350 (+ 93048us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:04.698342 (+ 41992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.698431 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.256733 (+558302us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:05.260979 (+  4246us) raft_consensus.cc:1685] finished
0709 12:23:05.261000 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.261164 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66093,"prepare.queue_time_us":435892,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":72529,"raft.run_cpu_time_us":2061,"raft.run_wall_time_us":24963,"spinlock_wait_cycles":47103488,"thread_start_us":638678,"threads_started":3,"wal-append.queue_time_us":547418}
I20260709 12:23:05.265513  6945 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:05.265877 27589 raft_consensus.cc:1217] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Deduplicated request from leader. Original: 37.19->[38.20-41.22]   Dedup: 38.20->[39.21-41.22]
I20260709 12:23:05.266498 27589 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 20
I20260709 12:23:05.266778 27589 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 40.21
I20260709 12:23:05.267490 27589 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:05.268770 27487 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 28 to 30, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 30 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:05.223694 27587 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:05.223105 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.231143 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.207669 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.263630 27664 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:05.274432 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.276165 27737 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:05.198002 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3904) took 1034 ms. Trace:
W20260709 12:23:05.283878 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 225) took 148369 ms (client timeout 15000 ms). Trace:
I20260709 12:23:05.284104 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:05.284551 27483 rpcz_store.cc:269] 0709 12:20:36.889712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:36.889996 (+   284us) service_pool.cc:224] Handling call
0709 12:23:05.259169 (+148369173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1277097,"reported_tablets":51}
I20260709 12:23:05.286185 27584 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:05.281567 27655 rpcz_store.cc:276] 0709 12:23:04.163273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.171429 (+  8156us) service_pool.cc:224] Handling call
0709 12:23:04.171829 (+   400us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:04.410240 (+238411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.197405 (+787165us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54000e65d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:05.197818 (+   413us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":546419072}
I20260709 12:23:05.255157  7067 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.295087 27582 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 18. (term mismatch)
I20260709 12:23:05.295799 27582 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 17
I20260709 12:23:05.141324 27738 rpcz_store.cc:276] 0709 12:23:04.034602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.362484 (+327882us) service_pool.cc:224] Handling call
0709 12:23:04.362932 (+   448us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:04.369285 (+  6353us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:23:04.649714 (+280429us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.005358 (+355644us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:05.081530 (+ 76172us) log.cc:844] Serialized 78 byte log entry
0709 12:23:05.081723 (+   193us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:05.082479 (+   756us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.097589 (+ 15110us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.097840 (+   251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.103720 (+  5880us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.103772 (+    52us) raft_consensus.cc:1685] finished
0709 12:23:05.103783 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.103964 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9705,"prepare.queue_time_us":277217,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":104,"raft.queue_time_us":12642,"raft.run_cpu_time_us":675,"raft.run_wall_time_us":18596,"thread_start_us":336387,"threads_started":2,"wal-append.queue_time_us":276902}
I20260709 12:23:05.298800 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.300758 27582 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 34.18
I20260709 12:23:05.301244 27582 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:05.298574 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47323.2us
I20260709 12:23:05.306478 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.317816 27587 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:05.318116 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46914.4us
I20260709 12:23:05.319496 27731 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:05.321902 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.323024 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.325280 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.324533 28335 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:23:05.328917 27588 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 22. (term mismatch)
I20260709 12:23:05.330935 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.334435 27734 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:23:05.334544 27487 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 21 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 25 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:05.337102 27487 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 24 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:05.340219 28329 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:23:05.341327  6745 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:05.345185  6923 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:05.346472  6923 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:05.346906  6923 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:05.325197  7033 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:05.348774 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.349768 27582 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:05.350519 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.351428 28330 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:05.352092 27585 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:23:05.352262 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:05.329468 27588 pending_rounds.cc:85] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 21
I20260709 12:23:05.354429 27588 pending_rounds.cc:107] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 38.22
I20260709 12:23:05.354693 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.354732 27588 raft_consensus.cc:2889] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:05.354916  7067 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.002s
I20260709 12:23:05.358271  7067 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:05.359123 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.359851 28334 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:23:05.360169 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 205430us
I20260709 12:23:05.359143 27584 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:05.368979 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.339272 27487 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:05.372165 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.367640 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:05.370412 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.375070 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.375931 27586 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:05.376554 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.377436 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.378377 27580 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: [38.22-38.22]
I20260709 12:23:05.380741 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:05.381524 27737 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:23:05.383841 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:05.384423  7009 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.384588 28332 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:05.386029 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:05.386488  7015 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.389195  7015 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:05.391690 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.393545  6745 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:23:05.394097  6745 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.394554  6745 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:05.395735 27730 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:05.398744 27730 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:23:05.405319  6823 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:05.405864  6965 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:05.406389  6965 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.406510 27662 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 39 because replica has last-logged OpId of term: 38 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 21.
I20260709 12:23:05.406836  6965 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:05.407967 27731 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:05.414801 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.416294 27734 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
W20260709 12:23:05.417641 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63133.8us
I20260709 12:23:05.418926  6891 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:23:05.419582  6891 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:05.420104  6891 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:05.421115  6978 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.422848 27581 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:05.424374 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:05.421175 27662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 3910) took 1253 ms (client timeout 1518 ms). Trace:
I20260709 12:23:05.427153 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:05.427527 27662 rpcz_store.cc:269] 0709 12:23:04.168232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.192113 (+ 23881us) service_pool.cc:224] Handling call
0709 12:23:05.420845 (+1228732us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400210a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:05.421081 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2789445504}
I20260709 12:23:05.428607 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.429437 27661 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:23:05.415518 27487 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 27 to 28, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 28 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:05.426911 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.433122 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.436280 27587 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:23:05.446524 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.447734 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.452035 27701 leader_election.cc:400] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:23:05.452526 27701 leader_election.cc:403] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:23:05.453737 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.454965  6998 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:05.454985 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4549) took 1470 ms. Trace:
I20260709 12:23:05.459820 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.459686 27487 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:05.461481 27487 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 19 to 28, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 28 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:05.462299 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.459048 27729 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 29 index: 23. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:23:05.467366 27583 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:05.435310  6978 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:05.483616 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.484509 27655 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:05.459410 27732 rpcz_store.cc:276] 0709 12:23:03.984939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.262524 (+277585us) service_pool.cc:224] Handling call
0709 12:23:04.262867 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.263452 (+   585us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:23:04.263494 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.264081 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.098502 (+834421us) spinlock_profiling.cc:243] Waited 7.52 ms on lock 0x7b1800354580. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:05.343425 (+244923us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:05.343474 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.343566 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.454360 (+110794us) raft_consensus.cc:1685] finished
0709 12:23:05.454391 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.454693 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":141,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":16406,"spinlock_wait_cycles":17298176,"thread_start_us":194887,"threads_started":1,"wal-append.queue_time_us":1187330}
I20260709 12:23:05.464738  6644 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.491597  6644 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.476599  7015 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:23:05.476770  7009 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.493788  7009 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.494064  6644 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.495582  6825 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:23:05.496119  7009 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:05.497014  7016 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:23:05.497296 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 78153.5us
I20260709 12:23:05.471206 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:05.499641 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:05.505559 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.507350 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:05.518175 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53187.8us
W20260709 12:23:05.518270 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48873.8us
I20260709 12:23:05.523280 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:05.534067 27587 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260709 12:23:05.542289 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.549769 27738 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:05.552062 27734 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
W20260709 12:23:05.550318 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77481.1us
I20260709 12:23:05.555457 27656 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
W20260709 12:23:05.554966 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:05.553738 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:05.561184 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.562152 27663 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:05.569759 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.570565 27580 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:05.548826  6978 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:05.578892 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.579980  7016 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:05.563787  6823 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:05.561136  6998 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34.
I20260709 12:23:05.585187 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:05.587622 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:05.590834 27663 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 27.
I20260709 12:23:05.578852  7067 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.601140 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:05.605347 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.605844 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.584331  6823 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.607447 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
W20260709 12:23:05.610484 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 183698us
I20260709 12:23:05.618119 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.619040 27582 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:05.620143 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:05.620927 27587 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
I20260709 12:23:05.580945  7016 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.030s
I20260709 12:23:05.625527 27584 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:05.627312 27583 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:05.591132  7071 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.033s
I20260709 12:23:05.625372 27487 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 25 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:05.628201 27738 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:23:05.614183  6823 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:05.635958  7056 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:23:05.636343 27734 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:23:05.639295 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:05.639643 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:05.641247  6846 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:05.641499 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.643000  6966 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:05.643369  7037 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.111s
I20260709 12:23:05.643826  6966 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:05.644696  6966 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:05.645896 27586 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:23:05.646211  6832 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:05.635192  6372 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.025s
I20260709 12:23:05.646699  6832 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:05.646910  7051 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:05.612378  6978 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.647194  6372 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:05.650405 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.622440  7069 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:05.649989 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:05.647955  6832 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.656090  6858 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:05.656060  7000 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:05.627408  6709 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:23:05.653594 27487 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 25 to 31, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 31 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:05.629935  7067 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.651383  6372 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:05.666558 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 196096us
W20260709 12:23:05.673849 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.739111 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.739591 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.629787  6891 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.653810  7069 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.747287 27663 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:23:05.748409 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.757162  6671 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:05.757682  6671 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.758023  6671 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:05.670862  7076 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.114s	user 0.004s	sys 0.000s
I20260709 12:23:05.669041  7037 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:05.697605  7016 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:23:05.764174 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 113820us
I20260709 12:23:05.669627  6832 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:05.752390  7069 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:05.705912  7055 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.058s
I20260709 12:23:05.666857  6709 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:05.779125 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.780233 27487 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:05.775143  7070 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:05.768105  7037 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:05.782202 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:05.771495  7016 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:05.783419 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.773236  6945 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:23:05.787585  6579 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:23:05.788594  7086 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:23:05.791450 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.798548  6823 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.805848 27580 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:05.806294  6745 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:05.807224 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 158405us
I20260709 12:23:05.811826 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:05.812407  6953 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.054s
I20260709 12:23:05.813602 27661 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:23:05.813529  6923 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:05.814966 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 148634us
I20260709 12:23:05.668417  6825 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.108s
W20260709 12:23:05.828738 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.830037 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47968.6us
I20260709 12:23:05.666874  6965 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.841463 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.670274  7006 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.174s
I20260709 12:23:05.824613  6709 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:05.827728  7016 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:05.837574  6923 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:05.826812  6579 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:05.830214  7067 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:05.878786 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95553.7us
W20260709 12:23:05.891556 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 79820.9us
I20260709 12:23:05.897552 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.847224  6531 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:23:05.900102 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 108762us
I20260709 12:23:05.856899  6745 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:05.857228  6951 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:05.860424  7006 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:23:05.880213  6846 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:05.890192  6644 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:05.877343  6966 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:05.907550 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:05.899830  7000 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:23:05.929033 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 96292.8us
I20260709 12:23:05.939399 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:05.912581  6846 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:05.921027  6966 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.945806 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:05.912235  7006 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:05.909893  6951 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:05.951100 27630 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:05.950521 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:05.943385  7045 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.934762  7071 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:05.935403  6978 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:05.964771 27487 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 29 to 30, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 30 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:05.950034  6644 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.016s
I20260709 12:23:05.919747  7004 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.034s
W20260709 12:23:05.993496 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.954068  6966 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:05.954433  7006 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:23:06.003595 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96153.4us
W20260709 12:23:06.006404 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60869.5us
W20260709 12:23:06.002032 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51602.5us
W20260709 12:23:06.014339 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 92168.8us
I20260709 12:23:06.008085 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:05.915239  6372 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:05.908080  7037 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:05.976048  7009 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:23:06.064217 27487 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 26 to 28, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 28 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:06.067473  6823 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.068326  6823 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:06.071659  6965 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:06.071122  7037 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:06.008780  6454 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.218s
I20260709 12:23:06.061904  6825 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:06.076020  6825 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:06.078385  7083 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:23:06.080099 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4557) took 1420 ms. Trace:
I20260709 12:23:06.080345  6579 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.082209  6579 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:06.080554 27735 rpcz_store.cc:276] 0709 12:23:04.660152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.245055 (+584903us) service_pool.cc:224] Handling call
0709 12:23:05.392648 (+147593us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:05.392909 (+   261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.407662 (+ 14753us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.407709 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.414544 (+  6835us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.744566 (+330022us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:05.744624 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.744711 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.079788 (+335077us) raft_consensus.cc:1685] finished
0709 12:23:06.079808 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.080033 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20068,"raft.queue_time_us":14423,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":52590,"spinlock_wait_cycles":67132800,"thread_start_us":328702,"threads_started":1,"wal-append.queue_time_us":433098}
W20260709 12:23:06.098246 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 274229us
W20260709 12:23:06.108209 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 215306us
I20260709 12:23:05.983453  7069 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.085785  6825 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:06.140977 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.048723  7045 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.230402 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:06.240748 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 116014us
I20260709 12:23:06.242825 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:06.243680 28333 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:06.132293  7009 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:06.247303 27487 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 28 to 31, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 31 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:06.165632  6923 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.083983  7071 consensus_queue.cc:1175] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 34 status { last_received { term: 34 index: 22 } last_committed_idx: 22 last_received_current_leader { term: 34 index: 22 } }
I20260709 12:23:06.271828 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.273000 27587 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 40 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
I20260709 12:23:06.267874 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:06.262064 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 151871us
I20260709 12:23:06.125734  6965 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.131s
I20260709 12:23:06.277153 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.278352 27582 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:23:06.279670 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.280560 27583 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:06.279070 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:06.295542 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.296135 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.325191 27584 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
W20260709 12:23:06.340607 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.345357 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 320860us
I20260709 12:23:06.128667  6846 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.278678  6910 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:23:06.354367 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 347150us
I20260709 12:23:06.108562  6951 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.244890  7015 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.095s
I20260709 12:23:06.361652 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.354615 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.368321 27581 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:23:06.343617  7009 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:23:06.369240 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 491823us
W20260709 12:23:06.371107 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 363115us
I20260709 12:23:06.373402 27587 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:06.377796 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.378005  6969 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:06.378705 27580 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:06.381608  7094 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:23:06.389483 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4086) took 1474 ms. Trace:
I20260709 12:23:06.389979 27736 rpcz_store.cc:276] 0709 12:23:04.923764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.408885 (+485121us) service_pool.cc:224] Handling call
0709 12:23:05.438275 (+ 29390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.502759 (+ 64484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.502804 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.563925 (+ 61121us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.906854 (+342929us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:05.906893 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.906962 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.389243 (+482281us) raft_consensus.cc:1685] finished
0709 12:23:06.389266 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.389417 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28057,"raft.queue_time_us":9760,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":24403,"thread_start_us":314525,"threads_started":1,"wal-append.queue_time_us":668715}
I20260709 12:23:06.396113  7069 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:06.403550 27487 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.404839  7045 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:06.406189  6951 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:06.328004  6923 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.410918  6923 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:06.355711  7098 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.413182  7098 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.370781  7009 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.415805  7009 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:06.417596  6644 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.417971  7102 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.418179  6644 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.418360  6825 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:06.377847  7080 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260709 12:23:06.420200  6709 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.561s	user 0.011s	sys 0.000s
I20260709 12:23:06.420660  6709 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.421866 28331 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:06.422976  6709 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.420663  6644 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.423880  6891 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:06.429585 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65308.6us
I20260709 12:23:06.431278 28333 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
W20260709 12:23:06.447870 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 88556.2us
W20260709 12:23:06.435169  7090 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.323s	user 0.001s	sys 0.000s
I20260709 12:23:06.418735  7102 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:06.478338 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.485028 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:06.490149 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.481709  6671 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.707s	user 0.009s	sys 0.002s
I20260709 12:23:06.480489  6891 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.004s
I20260709 12:23:06.471498  7098 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.503971 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:06.504139 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:06.508141 28333 raft_consensus.cc:2368] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:23:06.487352  6846 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:06.476271  7037 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.031s
I20260709 12:23:06.518059 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:06.514298 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.491029  6966 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.520532 28329 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: [37.26-37.26]
I20260709 12:23:06.520769 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:06.502965  6671 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.526744  7109 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:23:06.527112  6998 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:06.527477  7108 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:23:06.540761  6671 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.528209  6923 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:23:06.543222  7016 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.650s	user 0.011s	sys 0.000s
I20260709 12:23:06.543932  7016 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.546743  6372 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:06.553656  7086 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.630s	user 0.011s	sys 0.002s
I20260709 12:23:06.554157 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.554944 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4569) took 1504 ms. Trace:
I20260709 12:23:06.556360 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:06.555371 27733 rpcz_store.cc:276] 0709 12:23:05.050909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.460548 (+409639us) service_pool.cc:224] Handling call
0709 12:23:05.460920 (+   372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:05.476351 (+ 15431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.476386 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:05.488549 (+ 12163us) log.cc:844] Serialized 52 byte log entry
0709 12:23:05.549435 (+ 60886us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:05.573334 (+ 23899us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.573578 (+   244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.573655 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.553396 (+979741us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b24002c5aa8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e9729a09d 00007f0e99ecc10e 00007f0e99ee388c 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221
0709 12:23:06.554633 (+  1237us) raft_consensus.cc:1685] finished
0709 12:23:06.554729 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.554879 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45080,"prepare.queue_time_us":16091,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2236,"raft.queue_time_us":16102,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":16835,"spinlock_wait_cycles":28017536,"thread_start_us":60374,"threads_started":1,"wal-append.queue_time_us":166372}
I20260709 12:23:06.559095  6671 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:06.532148  7004 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:06.573611  7004 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:06.534003 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.517941  6965 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:06.535710  6644 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:06.535948  6891 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:06.536365  6985 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:23:06.605088 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71151.1us
I20260709 12:23:06.606671 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.608327 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.608990 27661 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:23:06.610788 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.561887  7112 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:06.612010 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.610163 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:06.617842 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.596859 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:06.529237 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:06.576001  7051 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.022s	sys 0.052s
I20260709 12:23:06.588711 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.589341  6965 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.590253  7111 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:06.633764 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.585803  7016 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:06.594362  7006 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.538s	user 0.006s	sys 0.003s
I20260709 12:23:06.574121  7009 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.604893  6923 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:06.610857 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.584107  6825 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:06.628197 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 99009.5us
I20260709 12:23:06.629189 27660 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:06.629828 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.634348 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.623083 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.574110  7004 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:06.641777 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:06.631708 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:06.630407  7051 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.022s	sys 0.052s
I20260709 12:23:06.649276 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.654073 27738 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:06.645113  6978 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:06.635339  6745 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:06.650991 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.651460 27657 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:06.651815 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.622951  7108 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:23:06.649384 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90841.2us
W20260709 12:23:06.678323 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.701815 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.657066 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.657713 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.635686  6985 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:23:06.700896 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.704545 27589 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 27. Current term is 28. Ops: []
I20260709 12:23:06.700817 27582 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:06.726939 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:06.730319 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.730671 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:06.731599  7111 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:23:06.732079  7111 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.732456  7111 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:23:06.735898 27582 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 39 index: 19. (term mismatch)
I20260709 12:23:06.658147  6978 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:06.737587  6644 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:06.727892  6951 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:06.738085  6644 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.738312  6966 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.738492  6644 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:06.739213  6832 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.923s	user 0.006s	sys 0.006s
I20260709 12:23:06.739733  6978 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.742033  6978 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:06.736720  6965 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:06.703239 27580 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:23:06.743655  7004 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.732157 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.744966  7004 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:06.745102  7118 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:06.748319  7102 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:06.751606  6832 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:06.753433  6671 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:06.753845  6671 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:06.755460 27663 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:06.657788  6923 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.658066 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.758677 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.658902 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.658080 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:06.704617 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.724309 27655 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:06.762638 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:06.752363  7088 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.843s	user 0.005s	sys 0.010s
I20260709 12:23:06.658349 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.759591 27582 pending_rounds.cc:85] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 18
I20260709 12:23:06.769435 27581 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:06.703780 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.773346 27582 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 35.19
I20260709 12:23:06.773694 27582 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:06.770165 27553 leader_election.cc:400] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:23:06.773921 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.774168 27553 leader_election.cc:403] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:06.773818 27730 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:06.779549 27582 pending_rounds.cc:107] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 37.20
I20260709 12:23:06.784700 27582 raft_consensus.cc:2889] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:06.779456  7000 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:06.743254  6579 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.660s	user 0.007s	sys 0.001s
I20260709 12:23:06.770834 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.781294 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:06.781701 27581 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:06.787675 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.785034 27734 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:06.792757 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.796258 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:06.786772 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.799265 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.754825  6671 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.800880 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:06.771983  7070 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.934s	user 0.051s	sys 0.051s
I20260709 12:23:06.762295  7102 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:06.805327 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:06.802117 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.649950  7006 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.772174  6965 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.807885 27657 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:06.808936 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.805794 27735 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 26.
W20260709 12:23:06.817862  7081 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:06.797518  7108 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:06.797762  7116 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:23:06.822387 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.822090  6965 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.824690  6965 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:06.804833  7000 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:06.827502 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 68946us
I20260709 12:23:06.793042  6825 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:23:06.819039  7116 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:06.829852 27662 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:06.832214  6671 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:06.813706  6910 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.869751 27664 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:06.829312 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:06.871718 27732 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:06.837733 27581 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
I20260709 12:23:06.839471 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.839977 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:06.805771  7070 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.052s	sys 0.052s
I20260709 12:23:06.841143 27737 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:23:06.877638 27657 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 39 index: 19. (index mismatch)
I20260709 12:23:06.793659  6745 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:06.801826  6832 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.841737 27660 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
W20260709 12:23:06.884138 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 68438us
I20260709 12:23:06.868417 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:06.882556 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 73733.6us
I20260709 12:23:06.884778 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.886703 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:06.887890 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:06.878151 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49943.1us
I20260709 12:23:06.841418  7108 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:06.847517 27655 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:23:06.875356  6965 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.899557 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:06.901849 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.908306 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:06.880872  7070 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:06.885418  7123 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.921809 27582 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:06.922679 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.924881 27737 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:06.881058  6579 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.035s
I20260709 12:23:06.896747  7016 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:23:06.900219  6825 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:06.939994 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:06.892724  6832 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.893067  6985 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:23:06.945267 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:06.884001  6910 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.896368  7116 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:06.959160 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.968591  7070 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.973891  6919 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.095s
I20260709 12:23:06.974588 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.975807 27585 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:06.976711 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.977303 27585 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:06.981989  6910 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:06.982698  6910 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:06.986379 27585 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:06.987073  7110 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:23:06.909328  6965 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.987543  7110 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:06.988693  7110 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:06.989830  6965 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:06.933257  7123 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.992599  7000 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:06.987449 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:06.994932  7098 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:06.828686  7102 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:06.997084  7108 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:07.001147 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59122.5us
W20260709 12:23:07.002117 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56927.9us
I20260709 12:23:06.944499  6998 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.023185 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.035466 27737 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:06.995699  6745 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.026041 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:06.998967  7124 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.955250  7016 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:06.959712  7009 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.051750 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.016773  6825 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.058681 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:07.026708  7098 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:06.983239  6910 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.079564 27586 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:23:07.079185  6891 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:07.086369  7118 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:23:07.087690 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.089643 27580 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:07.090488  7006 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.090870  6945 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.096582 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:07.099279 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:07.108975 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:07.111083 27588 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 18. (term mismatch)
I20260709 12:23:07.106135 27737 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 26.
I20260709 12:23:07.114851 27588 pending_rounds.cc:85] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 17
I20260709 12:23:07.112965 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.115581 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.116007 27660 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 40 because replica has last-logged OpId of term: 38 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 21.
I20260709 12:23:07.116313 27729 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:07.082020  6846 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.002s
I20260709 12:23:07.115259 27588 pending_rounds.cc:107] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 34.18
I20260709 12:23:07.062374  7112 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.118276 27588 raft_consensus.cc:2889] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:07.121861 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.122577 28331 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:23:07.123088  7006 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:07.123690  7006 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:07.125188 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.124827  7006 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.126940  7133 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:07.127509 27656 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:23:07.128016  7133 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:07.129123  7133 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.070333  7124 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.135701  6644 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.139183 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.142601 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.092829  7130 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:07.139914 27736 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:07.062805  6923 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.071058  7122 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:07.147866 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.081132  7009 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.154722 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.102538  6832 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.100286  6985 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.080847  6825 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.053625  6998 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.158816 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.171852 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.165215 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.146826  6891 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:07.175783 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.175868 27731 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:07.177104 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.149149  7130 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.119570  7015 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.050s
W20260709 12:23:07.189745 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 90595.7us
I20260709 12:23:07.138692  6945 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.202250 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.203532 27660 raft_consensus.cc:2410] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 44 because replica has last-logged OpId of term: 41 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 27.
I20260709 12:23:07.206573 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.155678  7112 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.184785  7130 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:07.169183  7009 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.224246 27731 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:23:07.224164 27664 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:23:07.175753  6985 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.230392 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.235870 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:07.216513  7138 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.237237  7138 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.243434  7136 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.244097  7136 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.183746  7016 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.247653  7123 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.249464  6945 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.242944 27660 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:23:07.247426  7131 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.078s
I20260709 12:23:07.226428  7000 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:07.256951  7135 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:07.257295  6910 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.257750  6985 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.258332 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.173619  7098 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.269676 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:07.270399 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.275672 27583 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:07.271910 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:07.280602 27588 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:07.285492 27729 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: [37.26-37.26]
W20260709 12:23:07.288550 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52739us
I20260709 12:23:07.262705  7136 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.257453  7138 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.295569 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:07.296684 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:07.297627 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.300469 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.288301  7051 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:07.311565 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:07.312115 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.313045 27583 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:07.314344 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.315886 27583 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:07.316062 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.317579 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.321756  7070 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.322021  7098 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:07.322600  7112 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.322409  7070 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.323627  6825 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:07.312654  7004 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.567s	user 0.013s	sys 0.000s
I20260709 12:23:07.324991  7070 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.325948 27661 raft_consensus.cc:2410] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 39 because replica has last-logged OpId of term: 38 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 21.
I20260709 12:23:07.328320 27661 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:07.329631  7134 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:07.325099  6372 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.778s	user 0.015s	sys 0.000s
I20260709 12:23:07.331485  7136 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.331841  6998 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.332152  7136 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.334424  7136 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.330271  7134 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.288321  7000 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.335712 27589 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 23. (term mismatch)
I20260709 12:23:07.336117 27589 pending_rounds.cc:85] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 22
I20260709 12:23:07.336364 27589 pending_rounds.cc:107] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 38.23
I20260709 12:23:07.336597 27589 raft_consensus.cc:2889] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:07.294377  6891 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.024s
I20260709 12:23:07.346459 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.354447 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.353360 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.355937 27581 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:07.362246 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.363411 27581 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:07.368359 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:07.374429 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.374652 27588 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:07.375017 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:07.375115 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:07.377915 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:07.369215  7111 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.636s	user 0.011s	sys 0.000s
I20260709 12:23:07.369616  7000 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.336603  7134 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.379969 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:07.371358  7140 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:23:07.385861 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.385865 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.360785  7070 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.395514 27660 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:07.383445 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:07.353652  6891 consensus_queue.cc:1243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 29.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:23:07.380921  6825 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:23:07.381523  7097 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:07.385960  7136 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.386507  7131 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.411684 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.405493  6825 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.395291  6846 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.025s
W20260709 12:23:07.447063 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 235138us
I20260709 12:23:07.409297  7097 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.447623 27581 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: [35.21-35.21]
I20260709 12:23:07.411334  7045 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:07.382115  7111 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.413559 27660 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
W20260709 12:23:07.453540  7149 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:07.461014  6919 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:07.462548  6671 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.588s	user 0.006s	sys 0.005s
I20260709 12:23:07.467410 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:07.468447  7102 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.471855 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.472590  6823 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:07.477725 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:07.538445  6825 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:23:07.480762  7142 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 12:23:07.480537  7116 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:07.481619  7146 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
W20260709 12:23:07.482182  7141 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:23:07.484498 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:07.485818  7147 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:23:07.448488  7133 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.451540  6372 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.512760 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.513180 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.514492 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:07.415779  7136 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.408629  7070 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.539256 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:07.578169 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100558us
I20260709 12:23:07.479727 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:07.579741 27657 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
W20260709 12:23:07.583335 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 116012us
I20260709 12:23:07.555874  7097 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:07.599135 27660 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 22. (term mismatch)
I20260709 12:23:07.607589 27660 pending_rounds.cc:85] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 21
I20260709 12:23:07.607985 27660 pending_rounds.cc:107] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 38.22
I20260709 12:23:07.608366 27660 raft_consensus.cc:2889] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:07.571451  7112 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.579352  6823 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:07.579643  7138 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.572620  7045 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:07.586203  7000 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.636564 27583 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:07.608383 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:07.638266 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:07.611090 27736 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:07.611569 28329 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
W20260709 12:23:07.608628 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69507.5us
I20260709 12:23:07.640558 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:07.592877  6671 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.044s
I20260709 12:23:07.579372  6919 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.652578 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:07.649458 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:07.631238  7145 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:23:07.632436  7109 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.543744  7131 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.631165  6998 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:23:07.663656 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.651945  7051 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:07.623816  6372 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.628536  7130 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.655159  6945 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:07.695267  6945 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.695725  6945 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:07.696599  6372 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.697654 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4689) took 1047 ms. Trace:
I20260709 12:23:07.698192 27656 rpcz_store.cc:276] 0709 12:23:06.653719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.658560 (+  4841us) service_pool.cc:224] Handling call
0709 12:23:07.697442 (+1038882us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400344754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:07.697577 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":831603968}
I20260709 12:23:07.701282  7124 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.645812  7000 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.704689  7000 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.705360 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:07.666076  7109 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.670204  7129 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:23:07.671254  7136 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.630847  7112 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.661896  7145 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.662971  6832 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:07.678493  6919 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.651146  7118 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:23:07.679391  7159 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:07.680073 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:07.758786 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:07.759914 27660 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:07.681267  7070 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.682682  7116 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.684635  7137 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:07.685925 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:07.686530  6910 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:07.689963 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.791574 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.791832  7131 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.792261 27661 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:07.690793 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4682) took 1043 ms. Trace:
I20260709 12:23:07.691143  7045 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.683158  7143 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:23:07.611032  7133 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.654616  7015 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:07.776422  6372 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.696658  7130 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.670387  6823 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:07.731087 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62456.1us
I20260709 12:23:07.741832 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.749475 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:07.751103  7112 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.741369  7145 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:07.754148  7109 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.725090  6953 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.739992  7102 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:07.793381 27655 rpcz_store.cc:276] 0709 12:23:06.647106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.930646 (+283540us) service_pool.cc:224] Handling call
0709 12:23:06.942376 (+ 11730us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.158072 (+215696us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.158720 (+   648us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.177378 (+ 18658us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.269619 (+ 92241us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:07.269655 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.269724 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.690583 (+420859us) raft_consensus.cc:1685] finished
0709 12:23:07.690601 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.690732 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86657,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":14997,"thread_start_us":244410,"threads_started":2,"wal-append.queue_time_us":169263}
W20260709 12:23:07.794027 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 141504us
I20260709 12:23:07.742511  6832 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:07.637861  7138 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:07.851608 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.840823  7070 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.870553 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:07.844884  7169 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.109s	user 0.001s	sys 0.002s
I20260709 12:23:07.821247  7129 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:23:07.883487 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.881191 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.886811 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.841092  7143 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.875455  7137 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:07.912890 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.914752 27735 pending_rounds.cc:85] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 22
I20260709 12:23:07.916972 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.915223 27735 pending_rounds.cc:107] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 29.23
I20260709 12:23:07.917941 27735 raft_consensus.cc:2889] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:07.913841 27582 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:07.922104 27662 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:07.887851  6846 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:07.926663 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:07.932704 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.931057 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.895426  7129 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:07.878309  6372 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:07.938572 27730 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:23:07.939157 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.939921 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:07.935302 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:07.899430  7143 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:07.941885 27663 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
W20260709 12:23:07.954655 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 2.097s (SENT)
I20260709 12:23:07.920063  7122 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:07.959758  7122 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.960650  7110 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:07.961980  7122 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.965955 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:07.966981 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.968374 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.969573 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.968572  6644 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:07.970669 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.971616 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.972908  7009 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:07.978480  7133 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:07.978552 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4838) took 1359 ms. Trace:
I20260709 12:23:07.890628  6953 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:07.979157 27589 rpcz_store.cc:276] 0709 12:23:06.618910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.726313 (+107403us) service_pool.cc:224] Handling call
0709 12:23:06.726780 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.344765 (+617985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.978311 (+633546us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b54000be0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:07.978495 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1399503232}
I20260709 12:23:07.980988 27589 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:23:07.991590 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.992260 27582 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:07.993381  6823 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:07.999398 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:07.999799 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4098) took 1363 ms. Trace:
I20260709 12:23:08.000831 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:07.972934 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4686) took 1324 ms. Trace:
I20260709 12:23:08.002286 27664 rpcz_store.cc:276] 0709 12:23:06.648660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.795910 (+147250us) service_pool.cc:224] Handling call
0709 12:23:07.972758 (+1176848us) spinlock_profiling.cc:243] Waited 72.6 ms on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:07.972874 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167043456}
I20260709 12:23:08.002856 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.003708  6985 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:08.004248  6985 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:08.005194 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:08.005760 27582 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:23:08.007391 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4692) took 1134 ms. Trace:
I20260709 12:23:08.007287  7137 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.008059 27657 rpcz_store.cc:276] 0709 12:23:06.872632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.894123 (+ 21491us) service_pool.cc:224] Handling call
0709 12:23:08.007211 (+1113088us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:08.007308 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1542313856}
I20260709 12:23:07.981020  6953 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.010128 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 2103) took 1443 ms. Trace:
W20260709 12:23:07.966022  7148 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.640s	user 0.002s	sys 0.000s
I20260709 12:23:08.010607 27584 rpcz_store.cc:276] 0709 12:23:06.566246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.575868 (+  9622us) service_pool.cc:224] Handling call
0709 12:23:08.009953 (+1434085us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b54001892d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:08.010075 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1509031808}
I20260709 12:23:08.011538 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4835) took 1712 ms. Trace:
I20260709 12:23:08.011790 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:08.011971 27587 rpcz_store.cc:276] 0709 12:23:06.299346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.412973 (+113627us) service_pool.cc:224] Handling call
0709 12:23:06.413299 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.413673 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.413706 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.153401 (+739695us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b2c000014c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:23:07.168577 (+ 15176us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.272059 (+103482us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:07.272096 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.272164 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.965286 (+693122us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:08.011355 (+ 46069us) raft_consensus.cc:1685] finished
0709 12:23:08.011376 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.011486 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8123,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":90089600,"thread_start_us":70075,"threads_started":1,"wal-append.queue_time_us":134888}
I20260709 12:23:08.015906 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:08.016767 27658 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
W20260709 12:23:08.018733 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52864.5us
I20260709 12:23:08.000394 27734 rpcz_store.cc:276] 0709 12:23:06.636366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.641579 (+  5213us) service_pool.cc:224] Handling call
0709 12:23:07.999582 (+1358003us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b54002e7cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:07.999725 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26716544}
I20260709 12:23:08.000404 27589 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:08.020355 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.024528 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:08.025226 27581 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:23:08.005446  6985 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.025893 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.027134 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.032989 27731 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:23:08.033396 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:08.033663 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.034509 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:08.065754 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.067435 27582 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:07.967140  7175 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.900092  7070 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.097361  7136 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:23:08.084323 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.084489 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50887.9us
I20260709 12:23:08.109977 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.074649 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:07.924023  6945 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.097956  7131 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:08.024652  7071 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.023983  6454 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:23:08.028507  6969 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:08.111367 27738 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:08.035475  7000 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:08.088177 27662 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:07.938771  6846 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.147889  7168 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:08.148638  7168 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:08.149181  7168 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:08.149389  6846 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.069206  7102 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:08.069496  6923 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:08.069037  7109 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:08.126835 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:08.127811 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:08.143182 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.152944 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:08.092772  7009 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.140303 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:08.091809  7116 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:08.165434 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.011647  6832 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:08.135634  7178 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:23:08.118999  7136 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:08.166349 28329 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:08.198051 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4683) took 1550 ms. Trace:
I20260709 12:23:08.200595  6372 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.198740 27659 rpcz_store.cc:276] 0709 12:23:06.647650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.000761 (+353111us) service_pool.cc:224] Handling call
0709 12:23:07.035573 (+ 34812us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:07.132576 (+ 97003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.132635 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:07.133386 (+   751us) log.cc:844] Serialized 52 byte log entry
0709 12:23:07.179519 (+ 46133us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:07.199618 (+ 20099us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.211609 (+ 11991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.691660 (+480051us) spinlock_profiling.cc:243] Waited 96.5 ms on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:07.691713 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.704871 (+ 13158us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b18003c12a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:08.180022 (+475151us) spinlock_profiling.cc:243] Waited 8.71 ms on lock 0x7b24002ac1b8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:23:08.197709 (+ 17687us) raft_consensus.cc:1685] finished
0709 12:23:08.197745 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.197959 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25619,"prepare.queue_time_us":19924,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":80908,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":14473,"spinlock_wait_cycles":244838272,"thread_start_us":45601,"threads_started":1,"wal-append.queue_time_us":122305}
I20260709 12:23:08.203343  6945 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.213766  7108 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.029s
I20260709 12:23:08.142832  6985 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:08.172720 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:08.234738 27731 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:23:08.153180  6910 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.153851  7133 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.237931  6823 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:08.094514  7097 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.102919  7175 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.254756  7175 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.256206 27589 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:23:08.257498  7070 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.265798  6832 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:08.272066  7138 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.248561  6823 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.292865  7051 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:08.293613  7051 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:08.294070  6823 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.295346  7000 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:08.295946  7000 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:08.296447  7000 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:08.304200 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4580) took 1519 ms. Trace:
I20260709 12:23:08.304759 27736 rpcz_store.cc:276] 0709 12:23:06.785017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.154861 (+369844us) service_pool.cc:224] Handling call
0709 12:23:08.303944 (+1149083us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:08.304122 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984016768}
I20260709 12:23:08.306195 27729 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:08.310357 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4858) took 1013 ms. Trace:
I20260709 12:23:08.310746 27583 rpcz_store.cc:276] 0709 12:23:07.296930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.373055 (+ 76125us) service_pool.cc:224] Handling call
0709 12:23:08.310178 (+937123us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54002ba2d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:08.310301 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":546372992}
I20260709 12:23:08.311298 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4939) took 1024 ms. Trace:
I20260709 12:23:08.311728 27580 rpcz_store.cc:276] 0709 12:23:07.287402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.495201 (+207799us) service_pool.cc:224] Handling call
0709 12:23:08.310307 (+815106us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.310490 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.310515 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.310557 (+    42us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:08.310585 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.310637 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.311118 (+   481us) spinlock_profiling.cc:243] Waited 788 ms on lock 0x7b54002ba2d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:08.311223 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1811773312}
I20260709 12:23:08.224277  7136 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:08.176697  7098 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:23:08.282171 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 141980us
I20260709 12:23:08.335793 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.135547  7071 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.152750  7131 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.329692 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:08.171111  7009 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:08.375730 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 478861us
W20260709 12:23:08.377288 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 422635us
I20260709 12:23:08.222144  7116 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:08.332059 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.432237 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.433444 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.433691  7116 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.434598 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.333745 27587 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: [35.18-35.18]
I20260709 12:23:08.330466 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:08.323499  7134 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.436590  7134 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.176342  6969 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.441452  6969 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.387817 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:08.445482 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57781.4us
I20260709 12:23:08.445973 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.431744 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:08.447825 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 117429us
I20260709 12:23:08.448858 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.138454  7129 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.462642 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.250690  7133 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.280028  6910 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.455735 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:08.354768  6965 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:08.473913 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:08.306564  7145 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.489347 28329 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:08.485308 27627 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:23:08.490450 27627 leader_election.cc:403] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:23:08.445096  7134 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.462250  6951 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:08.462759  7004 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:08.462949  6969 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:08.511073 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:08.594964 27580 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:08.469204  7037 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:23:08.563114 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 497388us
I20260709 12:23:08.562956 27729 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:23:08.484949  7175 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.480343  7070 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.492568  7122 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:08.510079  6372 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:08.591663 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2071) took 1005 ms. Trace:
I20260709 12:23:08.770239 27662 rpcz_store.cc:276] 0709 12:23:07.586234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.685619 (+ 99385us) service_pool.cc:224] Handling call
0709 12:23:08.591450 (+905831us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54001e55d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:08.591587 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":308424960}
I20260709 12:23:08.770726 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:08.678769 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.680601 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:08.584599  6745 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:08.592895  6945 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:23:08.797051 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118344us
I20260709 12:23:08.798183 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.510425  7109 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:08.822028  7109 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.687942  7122 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:08.783404 28294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.033s	sys 0.118s
W20260709 12:23:08.839428 28294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.034s	sys 0.118s
I20260709 12:23:08.786307  7097 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.677919  7037 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:08.813295 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:08.589314  7134 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:23:08.852473  7045 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:23:08.840348  7143 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.861s	user 0.009s	sys 0.012s
I20260709 12:23:08.853608  7143 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.854550 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.855509 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.856132 28333 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:08.856884  7183 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:08.857409  7183 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.857764  7183 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:08.859978  7009 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:08.860635  7037 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:08.863415 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.866106  7190 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.866878 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.866771  7190 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.869328  7190 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:08.869867  7153 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
W20260709 12:23:08.871083  7187 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.288s	user 0.000s	sys 0.000s
I20260709 12:23:08.873260  7185 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.874578  7186 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:23:08.770031  6951 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:08.875306  6951 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.856154  7189 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:23:08.818751 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.881316  7145 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.855726  7143 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.883288 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:08.885567 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:08.890296  6923 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.740s	user 0.007s	sys 0.002s
W20260709 12:23:08.781204 27734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.015s	sys 0.035s
W20260709 12:23:08.810114  7160 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:23:08.793287  6945 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.849391 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:08.897549 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48268.8us
I20260709 12:23:08.760973  7175 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.851183  7170 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:08.787706  6372 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.873981  7185 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.879128  6951 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:08.892016 28331 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.025s	sys 0.047s
I20260709 12:23:08.905465 28333 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
W20260709 12:23:08.905821 28331 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.025s	sys 0.047s
W20260709 12:23:08.892390 27734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.016s	sys 0.036s
I20260709 12:23:08.842584 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:08.814373  6891 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:08.832269  6978 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.899252 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.890806  6923 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:08.759958  7070 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.846278  7122 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:08.900786  6945 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:08.848384  7172 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:08.911904 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:08.919266 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:08.947778  7202 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.071s	user 0.005s	sys 0.000s
W20260709 12:23:08.953274  7168 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.803s	user 0.013s	sys 0.000s
I20260709 12:23:08.953855  7168 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.957856 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:08.928558 27583 raft_consensus.cc:2410] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 35 because replica has last-logged OpId of term: 33 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 16.
I20260709 12:23:08.921842 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.806052  6745 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:08.984975 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:08.984109 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62394.9us
I20260709 12:23:08.985880 27581 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:23:08.986300 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.844600  7144 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.903108  7189 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:08.910257  6953 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:23:08.911749  6825 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.314s	user 0.005s	sys 0.009s
I20260709 12:23:08.902323 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4115) took 1017 ms. Trace:
I20260709 12:23:08.940164  7133 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:08.984078 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:08.986812 27731 consensus_meta.cc:287] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.735s	user 0.012s	sys 0.000s
I20260709 12:23:09.020380 27731 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 27.
I20260709 12:23:08.987069 27589 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 35 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
W20260709 12:23:08.987206 27738 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.824s	user 0.008s	sys 0.004s
W20260709 12:23:08.989467 27733 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.017s	sys 0.021s
W20260709 12:23:09.021692 27733 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.017s	sys 0.021s
I20260709 12:23:09.021656 27738 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:09.022330 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4602) took 1057 ms. Trace:
I20260709 12:23:09.022833 27738 rpcz_store.cc:276] 0709 12:23:07.965199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.025494 (+ 60295us) service_pool.cc:224] Handling call
0709 12:23:09.022232 (+996738us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.993636 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.927173  7198 raft_consensus.cc:2764] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 36: won
W20260709 12:23:08.967664  7110 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.006s	user 0.009s	sys 0.004s
I20260709 12:23:09.004253 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2075) took 1416 ms. Trace:
I20260709 12:23:09.011411 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:08.933959  6372 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:09.011389 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.019s	sys 0.025s
W20260709 12:23:09.028559 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.019s	sys 0.025s
I20260709 12:23:09.028931 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:09.028985 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 553677us
W20260709 12:23:08.982829 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 398045us
I20260709 12:23:08.929702  7071 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:08.927021  7131 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:08.943588  7172 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:08.983103  7170 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:08.990058  7045 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:09.066700 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.020372 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:09.015952 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.028126 27657 rpcz_store.cc:276] 0709 12:23:07.587688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.009130 (+421442us) service_pool.cc:224] Handling call
0709 12:23:08.190376 (+181246us) spinlock_profiling.cc:243] Waited 6.53 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:08.190711 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.237490 (+ 46779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.237531 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.238049 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.581668 (+343619us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:08.583962 (+  2294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.584038 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.990471 (+406433us) raft_consensus.cc:1685] finished
0709 12:23:08.990501 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.004163 (+ 13662us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45992,"raft.queue_time_us":46100,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":90,"spinlock_wait_cycles":15019264,"thread_start_us":320520,"threads_started":1,"wal-append.queue_time_us":665737}
I20260709 12:23:09.038259 27738 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:09.039362 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:09.036466 28328 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:23:08.925446  6978 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:08.998782  7097 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:08.999030  6644 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:09.043215 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:08.932080  7185 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.056339 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:09.077682 28336 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:08.945447  7183 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.056993 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:09.061728 27660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.025s	sys 0.049s
W20260709 12:23:09.081075 27660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.025s	sys 0.049s
I20260709 12:23:09.033175  7189 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:09.004935  7108 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.055s
I20260709 12:23:09.075567 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.045109  6372 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:23:09.002902  6825 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.045101  7172 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:09.066301  7045 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.059602  6910 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:09.000202  7144 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.080513  7198 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:09.141235 28336 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:09.085017 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.006140 27735 rpcz_store.cc:276] 0709 12:23:07.884992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.885133 (+   141us) service_pool.cc:224] Handling call
0709 12:23:07.895665 (+ 10532us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:08.013669 (+118004us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.013716 (+    47us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:08.014570 (+   854us) log.cc:844] Serialized 78 byte log entry
0709 12:23:08.333059 (+318489us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:23:08.843634 (+510575us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.844418 (+   784us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.844759 (+   341us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.844886 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.844950 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.902011 (+ 57061us) raft_consensus.cc:1685] finished
0709 12:23:08.902039 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.902236 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":57804,"prepare.run_cpu_time_us":136,"prepare.run_wall_time_us":7026,"raft.queue_time_us":107658,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":256,"thread_start_us":790751,"threads_started":2,"wal-append.queue_time_us":449864}
I20260709 12:23:09.084040 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:09.374719 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 290814us
W20260709 12:23:09.153360  7136 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.798s	user 0.011s	sys 0.000s
I20260709 12:23:09.085990  7124 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:09.073789  7196 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.096038 28334 raft_consensus.cc:2368] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36.
W20260709 12:23:09.081146 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.084630 27628 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.000s	sys 0.009s
W20260709 12:23:09.383430 27628 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.000s	sys 0.009s
W20260709 12:23:09.383791 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 933840us
I20260709 12:23:09.153502 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4710) took 1381 ms. Trace:
I20260709 12:23:09.086879  7208 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.261s
I20260709 12:23:09.092828  7183 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.098724  6891 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:09.087402  7111 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:09.339749  7188 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.322s	user 0.000s	sys 0.001s
W20260709 12:23:09.341945  6832 consensus_meta.cc:287] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.021s	user 0.014s	sys 0.004s
I20260709 12:23:09.346180 27584 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 27: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:09.391227  6891 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.107163  7045 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:09.377584 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.391834  6891 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:09.077255  7097 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.392973  7097 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:09.394701  7196 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.396039 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.397239  7006 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:09.398806 27738 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:09.380141  7168 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:09.400913 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.401656 27738 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 30: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:09.099187  7170 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.405226  7170 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.406157  6745 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:09.379709 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 294765us
I20260709 12:23:09.391302 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4116) took 1506 ms. Trace:
I20260709 12:23:09.382863 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.341256  7137 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:09.407177 27737 rpcz_store.cc:276] 0709 12:23:07.885348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.885479 (+   131us) service_pool.cc:224] Handling call
0709 12:23:07.885788 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.920493 (+ 34705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.920523 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.933810 (+ 13287us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.342736 (+408926us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:08.342793 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.342892 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.391082 (+1048190us) raft_consensus.cc:1685] finished
0709 12:23:09.391111 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.391242 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18249,"raft.queue_time_us":14663,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7073,"thread_start_us":308290,"threads_started":1,"wal-append.queue_time_us":973231}
W20260709 12:23:09.409653  7112 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.014s	sys 0.014s
I20260709 12:23:09.421820  7192 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:09.421908  7112 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.014s	sys 0.014s
I20260709 12:23:09.422441  7192 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.133464  6372 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.274s
I20260709 12:23:09.424072  7198 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:09.424584  7198 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:09.424952  7129 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:09.425556  7198 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.421389  7144 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:09.427950  7199 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:09.431864 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1869) took 1306 ms. Trace:
I20260709 12:23:09.432237 27734 rpcz_store.cc:276] 0709 12:23:08.125537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.125714 (+   177us) service_pool.cc:224] Handling call
0709 12:23:08.147182 (+ 21468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.152836 (+  5654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.152868 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.179569 (+ 26701us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.908324 (+728755us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:08.908386 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.908485 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.427280 (+518795us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:09.431531 (+  4251us) raft_consensus.cc:1685] finished
0709 12:23:09.431557 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.431801 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25352,"raft.queue_time_us":13051,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":51530,"spinlock_wait_cycles":72887168,"thread_start_us":728337,"threads_started":1,"wal-append.queue_time_us":884288}
I20260709 12:23:09.434316 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.436414  7192 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:09.437325  7183 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:09.438197 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4946) took 1442 ms. Trace:
W20260709 12:23:09.438651  6985 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.191s	user 0.006s	sys 0.005s
I20260709 12:23:09.411654 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.440994 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1736) took 1010 ms. Trace:
I20260709 12:23:09.441254 27737 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:09.441358 28334 rpcz_store.cc:276] 0709 12:23:08.430999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.431165 (+   166us) service_pool.cc:224] Handling call
0709 12:23:09.440818 (+1009653us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:09.440941 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1449160960}
I20260709 12:23:09.411123  7136 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.443418 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4948) took 1446 ms. Trace:
I20260709 12:23:09.443768 27587 rpcz_store.cc:276] 0709 12:23:07.996653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.013979 (+ 17326us) service_pool.cc:224] Handling call
0709 12:23:08.014277 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.435452 (+421175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.443149 (+1007697us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54002b9dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:09.443358 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":734101376}
I20260709 12:23:09.444197  7136 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:09.420382  7137 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.453279 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2076) took 1865 ms. Trace:
I20260709 12:23:09.454283 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.455915 27734 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:09.453675 27660 rpcz_store.cc:276] 0709 12:23:07.588055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.013868 (+425813us) service_pool.cc:224] Handling call
0709 12:23:08.229726 (+215858us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:08.229984 (+   258us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.286651 (+ 56667us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:23:08.286707 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.354517 (+ 67810us) log.cc:844] Serialized 26 byte log entry
0709 12:23:09.081541 (+727024us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:23:09.081597 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.081687 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.453010 (+371323us) raft_consensus.cc:1685] finished
0709 12:23:09.453031 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.453221 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20885,"raft.queue_time_us":21139,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":144,"spinlock_wait_cycles":4472704,"thread_start_us":614282,"threads_started":1,"wal-append.queue_time_us":901172}
I20260709 12:23:09.457285 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.407070  7131 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:09.438591 27584 rpcz_store.cc:276] 0709 12:23:07.996187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.011548 (+ 15361us) service_pool.cc:224] Handling call
0709 12:23:09.437995 (+1426447us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54002bacd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:09.438136 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2726988544}
I20260709 12:23:09.421248  6825 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:09.494565 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 81075.6us
W20260709 12:23:09.451171  7122 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.517s	user 0.007s	sys 0.000s
I20260709 12:23:09.498347 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:09.500841 27704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.141s	user 0.015s	sys 0.022s
W20260709 12:23:09.501806 27704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.142s	user 0.015s	sys 0.023s
I20260709 12:23:09.503027 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:09.503355 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 1.14387e+06us
W20260709 12:23:09.504441 27701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.040s	user 0.014s	sys 0.025s
W20260709 12:23:09.504827 27701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.014s	sys 0.025s
I20260709 12:23:09.504348 27735 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:09.505448 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 1.04972e+06us
I20260709 12:23:09.506070 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.506286 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:09.510743 28334 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:23:09.500347  6998 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.181s	user 0.009s	sys 0.028s
W20260709 12:23:09.503079  6945 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.563s	user 0.011s	sys 0.001s
I20260709 12:23:09.514937 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:09.497356  7175 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:09.524588 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.520604 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.522114 27662 raft_consensus.cc:2376] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40. Re-sending same reply.
I20260709 12:23:09.501089  7137 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:09.526415 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.528477 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:09.527431 27657 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
W20260709 12:23:09.530808 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 438658us
W20260709 12:23:09.516592  6998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.202s	user 0.010s	sys 0.028s
I20260709 12:23:09.531776 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.551298 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.553525 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.555589 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.518998  7111 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.518749  7134 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.561604  7134 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.563859  7134 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.567549 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.569449 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:09.554543 27660 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:09.570103 27738 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
W20260709 12:23:09.554061  7037 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.691s	user 0.008s	sys 0.003s
I20260709 12:23:09.504918  7122 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.531471  6978 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:09.570565 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54835.8us
W20260709 12:23:09.575613 27703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.006s	sys 0.012s
W20260709 12:23:09.582073 27703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.160s	user 0.007s	sys 0.012s
W20260709 12:23:09.583889 27732 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.088s	user 0.012s	sys 0.040s
W20260709 12:23:09.584241 27732 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.088s	user 0.012s	sys 0.040s
I20260709 12:23:09.530099  6945 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.573375 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.519673  7108 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.586253 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:09.572633  7175 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:09.594218 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4586) took 2690 ms. Trace:
I20260709 12:23:09.589625  7015 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:09.561094  7137 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:23:09.601647 27658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.025s	sys 0.031s
W20260709 12:23:09.594547  7051 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.296s	user 0.009s	sys 0.005s
I20260709 12:23:09.571043  7111 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.582263  6825 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.025s
I20260709 12:23:09.612253 27732 rpcz_store.cc:276] 0709 12:23:06.897477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.318379 (+420902us) service_pool.cc:224] Handling call
0709 12:23:07.318749 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.319205 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:23:07.338088 (+ 18883us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.277412 (+939324us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b18001c0180. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:23:08.947077 (+669665us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:09.587132 (+640055us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.587201 (+    69us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:09.587259 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.588011 (+   752us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.588146 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":17528,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":6240,"raft.queue_time_us":356486,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":74,"spinlock_wait_cycles":22258432,"thread_start_us":1306785,"threads_started":2,"wal-append.queue_time_us":1538424}
W20260709 12:23:09.615370 27658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.025s	sys 0.031s
I20260709 12:23:09.578878  7037 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.584679  7097 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.622602 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:09.630342 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57091.4us
I20260709 12:23:09.624670 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2077) took 2036 ms. Trace:
W20260709 12:23:09.606859  7000 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.310s	user 0.008s	sys 0.007s
I20260709 12:23:09.633366 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.634514 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:09.638033 28330 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:09.638955 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2080) took 2040 ms. Trace:
I20260709 12:23:09.626664  7170 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:23:09.615532  6644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.004s	sys 0.039s
I20260709 12:23:09.644300 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:09.645571  7170 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:09.651674  7170 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:09.651950  6825 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:09.652405 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:09.653262  6825 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:09.653661  6825 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:09.654160  6825 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:09.631266  6978 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:09.659551  7122 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:09.645864  7224 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:09.660740  6644 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:09.661758  6644 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.662252  7196 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:09.666180  7215 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:09.615108  7108 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.642047 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:09.669340 28334 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
I20260709 12:23:09.632925  7102 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.636458  7137 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:09.639397 27661 rpcz_store.cc:276] 0709 12:23:07.598324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.229101 (+630777us) service_pool.cc:224] Handling call
0709 12:23:08.284845 (+ 55744us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.442526 (+157681us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:23:08.442904 (+   378us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.838130 (+395226us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.838207 (+    77us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:23:08.838284 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.638709 (+800425us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:09.638756 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.638894 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":147103,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":5792,"raft.queue_time_us":18303,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":45246,"spinlock_wait_cycles":367726720,"thread_start_us":376643,"threads_started":1,"wal-append.queue_time_us":562263}
I20260709 12:23:09.624594  7175 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:09.624965  7051 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.675961 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.631781 27658 rpcz_store.cc:276] 0709 12:23:07.588297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.017590 (+429293us) service_pool.cc:224] Handling call
0709 12:23:08.062375 (+ 44785us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.062752 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:23:08.951748 (+888996us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b18004b1660. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:23:08.951905 (+   157us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.624303 (+672398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.624359 (+    56us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:23:09.624408 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.624477 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.624613 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":322325,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":47,"raft.queue_time_us":2248,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":2024,"spinlock_wait_cycles":97190784,"thread_start_us":950759,"threads_started":2}
I20260709 12:23:09.680013 28336 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:09.675766  6978 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.681607  7122 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.681955  6825 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.682448  6978 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.682876 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:09.685434 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.685552  6978 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.683029  7122 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.682336 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:09.686193  6978 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:09.687026  6978 consensus_queue.cc:1243] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 38.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
W20260709 12:23:09.675915 27552 connection.cc:256] RPC call timeout handler was delayed by -1.3453600319994621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
I20260709 12:23:09.688107  7122 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.685563  6825 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:09.688632  7122 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:09.689088  7122 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:09.689186  6825 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.689913  6825 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.690968 27735 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:23:09.691816  6891 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.692718  6825 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:09.746659 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 240458us
W20260709 12:23:09.747404 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 233687us
I20260709 12:23:09.697300  7170 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.689904  7122 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.26, Last appended by leader: 26, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.702894  7137 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:23:09.702306 27628 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29.
W20260709 12:23:09.800148 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 97956.6us
I20260709 12:23:09.801746 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:09.706630 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:09.698978 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.691875  7175 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.680644  7102 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.646442  7134 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:23:09.694903  6945 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:09.634035  7000 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.724486 27585 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
W20260709 12:23:09.701565  7205 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.638s	user 0.008s	sys 0.004s
I20260709 12:23:09.704664  7108 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.674526  7224 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:09.774128 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 360424us
I20260709 12:23:09.823812  7224 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.826151 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:09.826453  7224 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.693715  7037 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:09.827694  7037 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:09.828109 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:09.808279 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 101726us
I20260709 12:23:09.813650 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:09.818521  7196 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:09.836506 27584 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 36
W20260709 12:23:09.813413 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4aa1523b9af84c3db99ba394e2b9aa9a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 30 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:09.806555 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:09.837278 27472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95662.5us
I20260709 12:23:09.837483 27582 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:09.807343  6832 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.797909  6969 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:09.839830 27663 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:23:09.843593 27731 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:09.818197  7102 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.844696 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:09.841536 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:09.842504 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 166561us
I20260709 12:23:09.845678 27586 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 28.
I20260709 12:23:09.848871 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:09.849922 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:09.850641 27580 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:09.818478  7134 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:09.853789 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.854533 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:09.855126 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:09.855760 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.857456 27553 leader_election.cc:400] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36.
I20260709 12:23:09.857797 27553 leader_election.cc:403] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:09.855839 27580 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:09.835541  7137 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:09.861065 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:09.862092 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:09.865233 27582 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:09.866196 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:09.867820 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.868693 27660 raft_consensus.cc:2410] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 40 because replica has last-logged OpId of term: 39 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 22.
I20260709 12:23:09.862469 27662 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:09.862071 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.916210 27586 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:09.840339  7196 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.846589  7037 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:09.846118  7211 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.934927 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.949143 27661 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:23:09.855180  7221 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:23:09.843252  7111 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.978312 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:09.979355 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:09.979688 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.981107 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:09.981812 27662 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:09.918179 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67949.7us
I20260709 12:23:09.858584  7134 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:09.851872  7192 raft_consensus.cc:2764] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:23:09.869349 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.939174 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:09.940397  7000 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:09.862219  7180 raft_consensus.cc:2764] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:23:09.849205  6969 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:09.945343  7172 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.845s	user 0.007s	sys 0.004s
I20260709 12:23:09.846328  6832 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.947427  7228 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:09.955921  7231 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:09.956189  7037 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:09.957243  7051 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:09.958664  6891 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.963121 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:09.967648  7123 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:09.994053  7172 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:09.998301  7123 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.000171  7172 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:10.001420  7123 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:09.918495  7196 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:09.969383  7170 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:09.971516 27584 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:10.007592  6372 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36.
I20260709 12:23:10.008342 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.008826  6832 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.009327 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
W20260709 12:23:09.972043 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 105917us
I20260709 12:23:09.971577 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:09.918905  7222 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:09.974021 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.974291 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:09.982571 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:09.982869 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 116437us
I20260709 12:23:09.979013 27583 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:10.019137 27661 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 41 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:23:09.995910 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.020143 27581 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:09.993271  7231 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.020190 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:10.023396  7231 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.023512 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.023953  6832 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:10.024405  6832 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:10.024538 27583 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.024863  6832 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:10.020049 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.027091 27583 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: [35.18-35.18]
I20260709 12:23:09.973614  7211 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.011873  7228 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.019403  7111 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:10.027432 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:10.030155 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.031230 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:10.048285 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.037760 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2117) took 1514 ms. Trace:
I20260709 12:23:10.049774 27585 rpcz_store.cc:276] 0709 12:23:08.524500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.585174 (+ 60674us) service_pool.cc:224] Handling call
0709 12:23:08.585538 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.819695 (+1234157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.037502 (+217807us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400186fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:10.037692 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2618834688}
I20260709 12:23:10.040197 27627 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:10.043690 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.028388 27657 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 38 index: 26. (term mismatch)
I20260709 12:23:10.052734 27657 pending_rounds.cc:85] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 25
I20260709 12:23:10.052953 27657 pending_rounds.cc:107] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 37.26
I20260709 12:23:10.052976 27582 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:10.053186 27657 raft_consensus.cc:2889] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:10.040529 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.028934  7175 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:10.067034 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:10.045197  7111 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.063258  7175 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.078797 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.079720 28336 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:23:10.072468 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50431.6us
I20260709 12:23:10.070464  7051 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:10.084870  7236 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.085450  7236 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.042131  7211 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.082477  7175 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.089355  7236 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.044618  7228 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.089426  7134 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.084172  7051 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.096581  7051 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.096354  7228 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:10.077808  7225 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:10.099635  7225 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.100013  7225 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.100062  7231 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.101581  7175 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.102614  7231 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:10.030911  7180 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.065s
I20260709 12:23:10.076315  7137 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:23:10.116685 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:10.131621 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.132579 27661 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:10.135411 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.137240 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.142513 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.143411 27664 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:10.083153  7111 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.153887 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.154922 27662 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:10.156808 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75165.7us
I20260709 12:23:10.070986  7170 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:23:10.159410 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.159795 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:10.160523 27664 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
W20260709 12:23:10.161125 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44577.1us
I20260709 12:23:10.162186 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:10.163511 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.109865  7175 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.175016 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.176225 27655 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:10.179145 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 107732us
I20260709 12:23:10.141181  7037 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.156394  7045 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.665s	user 0.012s	sys 0.000s
I20260709 12:23:10.180279 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:10.186260 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.191533 27657 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.166072  7102 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:23:10.193672 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 204474us
I20260709 12:23:10.200187 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:10.170877  7071 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:23:10.196909 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:10.205284 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:10.180529  7137 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:10.188783  7108 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:10.212832 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.214289 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.215562  7175 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:10.216113  7175 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:10.216599  7175 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:10.217898 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:10.218817  7180 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:10.221827  7180 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.222255  7180 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.219653 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:10.224753  7231 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:10.231125  7180 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.189136  7006 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:10.204931  7071 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:10.266377  7070 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:10.190589  7037 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.503001  7037 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.192354  7045 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.504715  7122 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.505034  7045 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:10.348654 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:10.443560 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.207391  7245 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:10.239151  7211 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:10.509192  7211 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.509701  7211 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:10.516758  7130 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:23:10.518786 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1803) took 1664 ms. Trace:
W20260709 12:23:10.519100 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 281129us
I20260709 12:23:10.519207 28330 rpcz_store.cc:276] 0709 12:23:08.854744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.856641 (+  1897us) service_pool.cc:224] Handling call
0709 12:23:08.856957 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.645115 (+788158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.518512 (+873397us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54005d93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:10.518700 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1791953664}
I20260709 12:23:10.522070  7136 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:10.522884  7136 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.528563 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.530006 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.530925 27732 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:10.549806 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.550599 27732 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 27: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:10.552170 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 342283us
I20260709 12:23:10.240356  6644 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:10.346045  7144 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:10.555109  7144 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:10.556471  7144 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.556761  7051 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.557756  7144 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:10.558238  7175 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.560395  7175 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.450783 27655 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:10.568858  7252 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:10.570705 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.444952  6969 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:10.445791  7193 log.cc:927] Time spent T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.112s	user 0.000s	sys 0.020s
I20260709 12:23:10.446261  7237 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.448089  7134 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:10.450080 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1877) took 1410 ms. Trace:
I20260709 12:23:10.586143 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.586326 27731 rpcz_store.cc:276] 0709 12:23:09.039871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.040043 (+   172us) service_pool.cc:224] Handling call
0709 12:23:09.040383 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.844189 (+803806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.331958 (+487769us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b540017e3d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:10.449996 (+118038us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1846570752}
I20260709 12:23:10.583511  7123 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:10.267856  7137 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.451095  7071 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.452943 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.453642 27661 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:10.456652 27657 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:10.457126  6832 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.459673  7015 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:10.447065  7111 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.474242  7198 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.479794  7097 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.459604  7196 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.664249 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:10.498179 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4708) took 2785 ms. Trace:
I20260709 12:23:10.499481  7110 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:10.643293  7015 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.532212  6965 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.532724 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 736483us
I20260709 12:23:10.532894 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.673982 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:10.533255 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 736602us
I20260709 12:23:10.675266  6965 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.678081  6965 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:10.533887 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.679699 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.680724 27738 raft_consensus.cc:2410] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 37 because replica has last-logged OpId of term: 35 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:23:10.681602 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.682868 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.534916  7136 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.535264 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.663301  7123 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.687700 27735 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:10.537091 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1870) took 2411 ms. Trace:
I20260709 12:23:10.689597  7123 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:10.689514 27733 rpcz_store.cc:276] 0709 12:23:08.125952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.126141 (+   189us) service_pool.cc:224] Handling call
0709 12:23:08.147657 (+ 21516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.434972 (+287315us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:23:09.032764 (+597792us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.838164 (+805400us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.536257 (+698093us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.536300 (+    43us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:23:10.536338 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.536900 (+   562us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54002a1054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:10.536939 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.537039 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":961298,"prepare.queue_time_us":352771,"prepare.run_cpu_time_us":8861,"prepare.run_wall_time_us":98600,"raft.queue_time_us":1711454,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":338,"spinlock_wait_cycles":660236672,"thread_start_us":1100169,"threads_started":3,"wal-append.queue_time_us":570690}
I20260709 12:23:10.692148 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.692950 27733 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:10.665225  7134 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:23:10.693939 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.694096  7134 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:10.694532  7134 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:10.537106 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 413313us
I20260709 12:23:10.696789  7196 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:10.538028 27734 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:23:10.672302  7110 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.697898  7110 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:10.698432 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.543001 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.703573 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.703949  7172 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:10.704984 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.706244 28331 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36.
W20260709 12:23:10.547446 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 365462us
I20260709 12:23:10.707917 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.709095 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:10.709455 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:10.552735  7225 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.710414 27662 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:10.711614 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.711910 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.712230 27662 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:10.262061  7244 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:10.712639 27661 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:10.712821  6998 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:10.713629  7244 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:10.713704  7207 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:10.444203  7180 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.714267 27655 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:23:10.715294  7249 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:10.715466  7207 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:10.715723 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.715902  7249 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.716419  7249 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:10.718395  7144 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:10.718461  6998 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:10.719100  7144 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:10.720074  7144 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:10.720710 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:10.721956  7172 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:10.722422  7172 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:10.722838  7172 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.723417 27730 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
I20260709 12:23:10.727008 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.729975  7225 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.730355  7111 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.611910 27737 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:10.644551 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63920.1us
I20260709 12:23:10.587409  7051 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:10.599056 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 398904us
I20260709 12:23:10.663568 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:10.663630 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.744194 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.601379 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.744835 27661 raft_consensus.cc:2376] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29. Re-sending same reply.
I20260709 12:23:10.745040 27731 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:09.386404 27656 rpcz_store.cc:276] 0709 12:23:07.772032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.320966 (+548934us) service_pool.cc:224] Handling call
0709 12:23:08.321437 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.321722 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:23:08.830232 (+508510us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.153083 (+322851us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.153156 (+    73us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:09.153213 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.153283 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.153430 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56311,"prepare.queue_time_us":260588,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":84193,"raft.queue_time_us":101249,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":40410,"thread_start_us":734919,"threads_started":2}
I20260709 12:23:10.569610  7252 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.669073 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.531617  7217 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:10.765429 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.765743 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.668704 27663 rpcz_store.cc:276] 0709 12:23:07.712338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.288409 (+576071us) service_pool.cc:224] Handling call
0709 12:23:08.338306 (+ 49897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.843258 (+1504952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.497814 (+654556us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b5400344254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:10.498060 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3452267904}
I20260709 12:23:10.766178 27655 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.766355 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.766875 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.767688 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.766368 27583 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:10.768424 27582 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.769182 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.770169 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.770814 27582 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.770920  7131 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:10.771763 27589 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:10.771842  7131 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.773231 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.773931 27658 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:10.774524  7211 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.757550  7134 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.695187 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.742038 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.743396 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:10.749163 27731 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:10.745620 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.755267 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.755636 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.756366 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.756456 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.756721 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.760053 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.714211  7244 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:10.706954 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.739655 27664 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:10.800104 27581 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.801697 27730 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:10.716428 27659 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:10.801755 27734 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 21. (term mismatch)
I20260709 12:23:10.807710 27734 pending_rounds.cc:85] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 20
I20260709 12:23:10.808005 27734 pending_rounds.cc:107] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 35.21
I20260709 12:23:10.808383 27734 raft_consensus.cc:2889] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:10.808640 27661 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:10.801786 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56238.4us
I20260709 12:23:10.575224  6644 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.815894 27583 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:23:10.800560 27586 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:10.808256 27656 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.818109 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.817884 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.818872 27664 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:10.817867 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.819211 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.816763 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:10.819839 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76553.2us
I20260709 12:23:10.820785 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.823724 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.778317 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.824132 27661 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:23:10.824384 27663 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:23:10.827462 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.828540 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.829057 27730 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:10.829485 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.816013 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:10.831689 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.832367 27734 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:10.832640 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.833137 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.833297 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.833926 27657 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:10.835570 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.836175 27731 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:10.836460 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.777199  6965 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.776854  6998 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:10.839927 28295 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:10.867560 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.842191 27735 raft_consensus.cc:2410] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 36 because replica has last-logged OpId of term: 35 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:23:10.868518 27583 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:10.870746 28293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.020s	sys 0.017s
I20260709 12:23:10.844035 27628 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36.
I20260709 12:23:10.833829 27734 raft_consensus.cc:2376] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38. Re-sending same reply.
I20260709 12:23:10.875617 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.801954  7097 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.587589  7237 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.848826 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.851641  7208 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:10.852066  6709 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:10.852300  6965 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.765970  7217 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.854688  7211 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:10.855722 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.909478 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:10.856465 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.857120 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:10.858407 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.859377 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.859544  7255 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:10.868492  7016 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.776074  7131 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:10.841310 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.869012 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:10.869886 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.821477  7111 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:10.843266 27662 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:23:10.876237 27735 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 29.
I20260709 12:23:10.874282 27628 leader_election.cc:403] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:23:10.882606  7248 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.876739 28293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.021s	sys 0.018s
I20260709 12:23:10.869396 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:23:10.877468 27655 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
W20260709 12:23:10.874132 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95893.9us
I20260709 12:23:10.867206 27664 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
W20260709 12:23:10.883785 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43978.3us
I20260709 12:23:10.901700 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:10.776952  7172 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.860571 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:10.911693 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 228) took 69591 ms (client timeout 15000 ms). Trace:
I20260709 12:23:10.909969 27582 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.777348  7253 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.149s
I20260709 12:23:10.896339  7056 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:10.953698 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.822805  7123 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.069s
I20260709 12:23:10.919462 27583 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 28: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.881258  7070 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:23:10.922250 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53305.4us
W20260709 12:23:10.976852 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 133035us
I20260709 12:23:10.923136 27661 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:10.968392 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.875092  7252 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:10.970249 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:10.979704 27657 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:10.863137  7110 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:10.972142 27733 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:10.973320 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:10.975191 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:10.978767 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:10.974418 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:10.975806 27587 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:23:10.978916 27732 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:10.979488 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:11.028366 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:11.030970 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:11.032738 27589 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:11.035771 27734 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:10.875451  6998 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:11.037381 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 135770us
I20260709 12:23:11.002988 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:11.038686 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:11.041144 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.064124  7248 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.089901  6998 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.028731  7192 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:23:11.037106  7133 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:23:10.967979 27487 rpcz_store.cc:269] 0709 12:22:01.320244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.321354 (+  1110us) service_pool.cc:224] Handling call
0709 12:22:01.381974 (+ 60620us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a350f3a 00007f0ea0106e9c 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc 00007f0e9e6c5392 00007f0e9868348c 00007f0e986828c6
0709 12:23:10.911613 (+69529639us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:06.775123 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:06.775853 (+   730us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:06.775914 (+    61us) write_op.cc:435] Acquired schema lock
0709 12:23:06.775955 (+    41us) tablet.cc:662] Decoding operations
0709 12:23:06.777531 (+  1576us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:06.777651 (+   120us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:06.777684 (+    33us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:23:06.779076 (+  1392us) tablet.cc:701] Row locks acquired
0709 12:23:06.779140 (+    64us) write_op.cc:260] PREPARE: finished
0709 12:23:06.779315 (+   175us) write_op.cc:270] Start()
0709 12:23:06.809694 (+ 30379us) write_op.cc:276] Timestamp: P: 1783599786779254 usec, L: 0
0709 12:23:06.830405 (+ 20711us) op_driver.cc:348] REPLICATION: starting
0709 12:23:06.959521 (+129116us) log.cc:844] Serialized 5924 byte log entry
0709 12:23:07.683811 (+724290us) op_driver.cc:464] REPLICATION: finished
0709 12:23:07.736774 (+ 52963us) write_op.cc:301] APPLY: starting
0709 12:23:07.736923 (+   149us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:08.248817 (+511894us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:08.248992 (+   175us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:08.878592 (+629600us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x7b180016cfc0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9a0870ec 00007f0e9a089af7 00007f0e9a08afb0 00007f0e9dd9f1bb 00007f0e9dde65d4 00007f0e9dc928da 00007f0e9dc953cb 00007f0e9dc94fd3 00007f0e9dd93e34
0709 12:23:09.657513 (+778921us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:09.657928 (+   415us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=0,mrs_lookups=0
0709 12:23:09.657997 (+    69us) write_op.cc:312] APPLY: finished
0709 12:23:09.658456 (+   459us) log.cc:844] Serialized 139 byte log entry
0709 12:23:09.916464 (+258008us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:09.941548 (+ 25084us) write_op.cc:454] Released schema lock
0709 12:23:09.941624 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:09.942169 (+   545us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:09.942206 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1203810,"reported_tablets":51,"spinlock_wait_cycles":108698880,"child_traces":[["op",{"apply.queue_time_us":52569,"apply.run_cpu_time_us":45309,"apply.run_wall_time_us":2205372,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66827,"num_ops":15,"prepare.queue_time_us":230940,"prepare.run_cpu_time_us":11200,"prepare.run_wall_time_us":604635,"raft.queue_time_us":153425,"raft.run_cpu_time_us":9113,"raft.run_wall_time_us":75091,"replication_time_us":853139,"spinlock_wait_cycles":205098752,"thread_start_us":844277,"threads_started":5,"wal-append.queue_time_us":685335,"wal-append.run_cpu_time_us":12367,"wal-append.run_wall_time_us":2299700}]]}
I20260709 12:23:10.980554  6531 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.117s
I20260709 12:23:11.092515 28328 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:11.091782 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.966079  7131 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:11.029227  7211 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36.
I20260709 12:23:11.093722  7131 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.094234  7211 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:23:10.982051 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:10.869387 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:11.096459 27583 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:11.096697 27738 raft_consensus.cc:606] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:10.981406 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:11.097393 27738 raft_consensus.cc:614] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:11.039686 27733 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:10.909834  7208 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:10.915243  7130 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.119s
I20260709 12:23:10.915457  6951 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.118s
I20260709 12:23:10.902694  7220 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.155s
I20260709 12:23:10.895711  7097 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.090879  7192 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:11.131302 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:11.111876  7248 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.106652 27658 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
W20260709 12:23:11.145485 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.102391 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:10.922755  7016 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:10.922843  7111 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.976939  7250 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:23:10.969996  6832 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.066560  7252 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:11.321062  7252 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:11.321552  7252 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:11.321142  7192 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.322283  7192 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:11.322511 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:11.243247 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 268921us
I20260709 12:23:11.323422 27580 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:11.325382 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.326367  7122 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.821s	user 0.011s	sys 0.000s
W20260709 12:23:11.244956 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 266242us
W20260709 12:23:11.242898 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 264757us
I20260709 12:23:11.327312 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:11.327776 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.327828  6965 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:11.328552  6965 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:11.328919  7056 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:11.153893  7219 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:11.333441 27657 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
W20260709 12:23:11.108035 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 105162us
I20260709 12:23:11.333864 27655 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:23:11.334548 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:11.121833 27583 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
W20260709 12:23:11.144681 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 284186us
I20260709 12:23:11.330811 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:11.342589 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:11.333477  7219 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:11.278265  7257 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:23:10.966694  7244 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.353494 27583 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
W20260709 12:23:11.326853  7015 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.001s
I20260709 12:23:11.094776 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:10.965790  7136 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.250468  6709 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:11.115592  7116 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.133s
W20260709 12:23:11.252838  7242 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:23:11.225037  7208 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:11.315344 27732 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:11.235397  6998 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:11.318699 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:11.320555  7097 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.102917  7070 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:11.130885  7231 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:11.325397 27580 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:11.347102 27587 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:23:11.349476 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:11.345804 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:11.423568 27658 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:23:11.093032  7110 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:11.364315 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 269621us
I20260709 12:23:11.346534  7219 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:11.346501  7263 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:23:11.330797  7111 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:11.428511 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79144.1us
I20260709 12:23:11.453292 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:11.454484 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:11.430828  7264 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 12:23:11.431123  7272 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.006s
W20260709 12:23:11.432329  7267 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:11.347999  7270 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:23:11.435374  6998 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.359591  7131 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:11.436766  7217 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.346151  7259 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.435410  6709 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:11.438005  7207 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:23:11.438820 27656 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.621s	user 0.001s	sys 0.005s
I20260709 12:23:11.438774  7231 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:11.346763  7257 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.439806  7172 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:11.440006  6945 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:11.350255  7220 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
I20260709 12:23:11.412002  6644 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.427285  7133 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.443280  7123 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:23:11.444058  7183 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:11.355228  7112 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.356652  7000 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:23:11.625000 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 294306us
W20260709 12:23:11.448982 27730 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.619s	user 0.007s	sys 0.003s
I20260709 12:23:11.449761 27736 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:11.361032  7051 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:11.460093 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:11.461154 27702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.000s	sys 0.014s
I20260709 12:23:11.412334  7225 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:11.618803  7183 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:11.627617  7112 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.634423  7183 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.631130  7208 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:11.636562 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 176558us
I20260709 12:23:11.601027  6832 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:11.461216 27586 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.631s	user 0.007s	sys 0.003s
I20260709 12:23:11.637424 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:11.637787 27586 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:11.425580  7122 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:23:11.360293  6965 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.092s
W20260709 12:23:11.469976 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 141455us
I20260709 12:23:11.470731 27734 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:11.475476 27658 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:11.406659  7015 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.482547 27656 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
W20260709 12:23:11.488912 27737 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.742s	user 0.009s	sys 0.001s
I20260709 12:23:11.490216  7180 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:11.479586  7131 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:11.479758  7270 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:11.493726  7257 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:23:11.601181 27589 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.002s
I20260709 12:23:11.671904 27589 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:11.672576 27734 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:11.491679  7259 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.502957  7070 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.481458  7207 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:11.501756 27614 heartbeater.cc:646] Failed to heartbeat to 127.25.202.62:43715 (9 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.25.202.62:43715 timed out after 15.000s (SENT)
I20260709 12:23:11.492120  7217 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:11.688763 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:11.538321  6709 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.689519 27587 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:11.626225 27730 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
W20260709 12:23:11.608250  7261 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.142s	user 0.006s	sys 0.000s
W20260709 12:23:11.627941 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 293451us
I20260709 12:23:11.692643 27730 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:11.692791 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:11.628583 27702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.001s	sys 0.014s
W20260709 12:23:11.608484  7249 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.891s	user 0.010s	sys 0.008s
I20260709 12:23:11.702128  7249 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.702934  7278 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:11.682189  7180 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.632187  7000 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:11.610374  7123 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:11.632459  7051 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.619258  7133 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.602622  6945 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:11.639741 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:11.668184 27737 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:11.659682 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:11.696656  7207 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:11.497768  7237 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.676153 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:11.677179  7045 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:11.756937  7045 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.759178  7045 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:11.759438 27737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1881) took 2316 ms (client timeout 2148 ms). Trace:
W20260709 12:23:11.759933 27737 rpcz_store.cc:269] 0709 12:23:09.413577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.334948 (+921371us) service_pool.cc:224] Handling call
0709 12:23:11.730138 (+1395190us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:11.761230  7144 consensus_meta.cc:287] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.041s	user 0.025s	sys 0.000s
I20260709 12:23:11.761727  7144 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:11.764525  7198 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.097s	user 0.011s	sys 0.001s
I20260709 12:23:11.764729  7180 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.669713  6832 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:11.765460 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 89467.2us
I20260709 12:23:11.766893  6832 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.678222  7110 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:11.767620  7110 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:11.768409  6531 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:11.769011  6531 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:11.768748  7110 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.769593  7279 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:11.770087  7110 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:11.769972  6531 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.679953 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:11.772848  7045 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:11.773494  7045 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.688422  7270 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.772897  7283 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.775197  7283 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:11.694223 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 815079us
I20260709 12:23:11.630602  7231 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:11.694537 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.779739 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.779894  7136 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:11.696682 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:11.780300 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:11.697647 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:11.781976  7000 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.631127  7252 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.783761 27663 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:11.636590  7112 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:11.708364 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.008s	sys 0.012s
W20260709 12:23:11.785034 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.008s	sys 0.012s
I20260709 12:23:11.668332  7277 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:11.785101  7123 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.670205  7137 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:11.787698  7281 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:11.788384  7281 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:11.789342  7281 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.792011  7133 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.793030  6825 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.684446  7183 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:11.731051 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:11.810268 27580 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:11.670830 27731 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
W20260709 12:23:11.737800 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98157.3us
I20260709 12:23:11.812510 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:11.682471  6644 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:11.754653  7217 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:23:11.795740 27582 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:23:11.797845 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.798422 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.793601  6825 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:11.823421 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 784619us
I20260709 12:23:11.818161  6945 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.818197  7283 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.819276  7015 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:11.824694 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:11.829869 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:11.833094 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:11.831183 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 138501us
I20260709 12:23:11.831843 27587 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:11.848310 27580 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:11.835825 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:11.833447 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:11.831416  7071 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:11.849113  7279 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:11.851030  7045 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.852205 27584 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:23:11.830500  7133 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:11.862061 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:11.878392 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:11.830519  7217 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:11.860368 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:11.937983 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59672.7us
W20260709 12:23:11.938476 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 78253.9us
I20260709 12:23:11.845098  7285 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.939622 27552 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:11.831413  7208 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:11.896696 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.837584  7122 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:11.943005 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 78975.2us
I20260709 12:23:11.944948 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:11.862345  7283 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:11.823614  6644 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.952194  7283 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:11.862562  6945 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:11.960867  7111 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:11.962553 27660 raft_consensus.cc:2368] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39.
I20260709 12:23:11.965121  7257 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.968281  7257 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.969666  7122 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.971482  7111 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.953807 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:12.045424 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91730.2us
I20260709 12:23:11.952723  7275 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:11.956647  7282 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:11.974717  7144 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:11.972298  7109 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.018s
I20260709 12:23:11.977213  6644 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:12.055043  7282 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:12.055545  7109 consensus_queue.cc:1243] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 39.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:23:11.953812  7285 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.056169  7282 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.941157  7287 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:11.978320 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4628) took 1198 ms. Trace:
I20260709 12:23:12.058808 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.058095 27735 rpcz_store.cc:276] 0709 12:23:10.780498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.445194 (+664696us) service_pool.cc:224] Handling call
0709 12:23:11.445518 (+   324us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:11.445863 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.445888 (+    25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:11.446399 (+   511us) log.cc:844] Serialized 52 byte log entry
0709 12:23:11.825381 (+378982us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:23:11.825419 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.825485 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.978081 (+152596us) raft_consensus.cc:1685] finished
0709 12:23:11.978101 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.978263 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24014,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":20774,"thread_start_us":378614,"threads_started":1,"wal-append.queue_time_us":528459}
I20260709 12:23:11.979166 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:11.980189 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:12.060115 27658 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:12.061101 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.061905 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.062520 27588 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:11.981036  7296 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:12.003656 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:11.938961  7051 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:11.939750  6965 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:23:12.001091 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.021498  7217 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:11.877645  7136 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:11.940625  7198 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:12.032745 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4144) took 1277 ms. Trace:
I20260709 12:23:11.943696  7237 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:23:11.976807  7283 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.060052 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.084080 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.063299 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.080354 27738 rpcz_store.cc:276] 0709 12:23:10.754934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.245493 (+490559us) service_pool.cc:224] Handling call
0709 12:23:11.261235 (+ 15742us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.261493 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:23:11.298744 (+ 37251us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.793509 (+494765us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b1800393040. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:23:12.032258 (+238749us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.032339 (+    81us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:12.032414 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.032502 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.032665 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":380,"prepare.queue_time_us":50857,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":683,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":10714,"spinlock_wait_cycles":38028672,"thread_start_us":237741,"threads_started":1}
I20260709 12:23:12.085117 27588 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:23:12.064369 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:12.067953 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88911us
I20260709 12:23:12.086467 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.064858 27589 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:12.087879 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.088574 27589 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:12.089381 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.090513 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.061928 28329 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 28.
I20260709 12:23:12.080192 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1887) took 1212 ms. Trace:
I20260709 12:23:12.091405 27733 rpcz_store.cc:276] 0709 12:23:10.867811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.680048 (+812237us) service_pool.cc:224] Handling call
0709 12:23:11.680384 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.680607 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:23:11.680909 (+   302us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.079802 (+398893us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.079858 (+    56us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:12.079907 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.079980 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.080127 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":977,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":398564,"threads_started":1}
I20260709 12:23:12.082417 27658 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:12.032563  7016 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:12.099704 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.041380  7131 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:12.108340 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.103116 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:12.112215 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.113054 27587 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:12.105418 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:12.106237 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:12.109930 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 106361us
I20260709 12:23:12.110342 27581 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:23:12.112972 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:12.113685 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.120829 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:12.120904 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:12.104931 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:12.119829 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.121822 27737 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:23:12.122766 27589 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:12.079943  7137 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:12.116026 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:12.078253  7231 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:12.117796 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.064406  6965 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:12.020421  7219 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.558s	user 0.010s	sys 0.003s
I20260709 12:23:12.100356  7270 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:12.111738  7138 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:12.122083 27587 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:12.079943  7287 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:12.094997  6998 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.135823  7000 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:12.152765 27628 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:12.153368 27628 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:12.128880 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:12.098038 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.104630  7296 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:12.118791  6644 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:23:12.110910  7130 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:12.134370  7225 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:12.108716  7275 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.035s
W20260709 12:23:12.177971 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 80005.1us
I20260709 12:23:12.195873 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.098960  7136 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.184412 27588 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:12.139725  7138 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.230718  6644 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:12.185508 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.144855  7270 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.231786  7138 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.187502 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:12.232492  7138 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:12.124918  7131 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.199005 27580 pending_rounds.cc:85] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 25
I20260709 12:23:12.202258  7237 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:12.203214  7292 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:23:12.246414 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:12.204022 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.204457  7134 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:12.207758  7287 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.211946 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4885) took 1347 ms. Trace:
I20260709 12:23:12.213042 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4142) took 1460 ms. Trace:
I20260709 12:23:12.254068 27731 rpcz_store.cc:276] 0709 12:23:10.752450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.147111 (+394661us) service_pool.cc:224] Handling call
0709 12:23:11.667610 (+520499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.820287 (+152677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.212853 (+392566us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54002e8bd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:12.212978 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1195989888}
I20260709 12:23:12.187500  7102 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.029s
I20260709 12:23:12.221894  6825 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:12.222684  7250 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.259708  7250 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:23:12.224378 27736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4121) took 1698 ms (client timeout 2157 ms). Trace:
W20260709 12:23:12.260665 27736 rpcz_store.cc:269] 0709 12:23:10.526142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.528260 (+  2118us) service_pool.cc:224] Handling call
0709 12:23:12.224193 (+1695933us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:12.224321 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2110156672}
I20260709 12:23:12.191099 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.149926  7144 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:12.237704 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:12.237706 27580 pending_rounds.cc:107] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 39.26
I20260709 12:23:12.262676 27580 raft_consensus.cc:2889] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:12.263155 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 59201.4us
I20260709 12:23:12.253032 27582 rpcz_store.cc:276] 0709 12:23:10.864270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.318414 (+454144us) service_pool.cc:224] Handling call
0709 12:23:12.211771 (+893357us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b54000bdbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:12.211889 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":886540672}
I20260709 12:23:12.263986 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:12.245102 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 115328us
I20260709 12:23:12.186753  7051 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:12.141746  7219 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.188802  7225 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:12.244422 27587 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:12.266180 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:12.279960 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:12.280534 27732 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:12.280978 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.268648 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:12.281919 27656 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:12.270183 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:12.274654 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.284440 27735 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:12.286651 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:12.246886  7292 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.287889 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:12.294138 27656 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:12.288385 27735 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:23:12.262003  7045 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.286194  7294 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.277935  6965 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.303681  7294 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.305774  7294 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.295221  7237 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:23:12.308338  7300 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:23:12.277462 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:12.309731 27735 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:12.310051  7208 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:12.296095 27655 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41. Ops: [40.22-40.22]
I20260709 12:23:12.310942  7208 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:12.311468  7208 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:12.312529  7110 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.000s
I20260709 12:23:12.312294  6644 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.312407  7208 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.313244 28330 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:12.314198  7208 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:12.298331 27733 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:12.299095 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:12.316092 27656 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:12.276330  7296 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:12.283361  7225 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:12.289260 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.199445  7130 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.292500 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:12.309818 27663 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 41 because replica has last-logged OpId of term: 40 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 21.
I20260709 12:23:12.299876  7259 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.319906 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:12.319391 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:12.341300 27657 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:12.344758 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:12.342572 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:12.346437 27655 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:12.342890 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:12.355537  7247 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:23:12.357273  7000 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.357813  7000 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.360200  7000 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:12.360610  7219 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.296132  7292 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:12.293812  6825 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:12.289794  7137 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:12.370705 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81511.8us
W20260709 12:23:12.353482  7298 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:23:12.371644  7315 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:12.372246  7312 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:12.372349  7295 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:12.384984 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:12.373967  7272 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:12.374024 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:12.375267  7303 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:12.365919  7198 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:12.376803 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4732) took 1520 ms. Trace:
W20260709 12:23:12.378213  7293 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:23:12.379177  7306 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:23:12.379838  6969 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.404012  7071 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.554s	user 0.000s	sys 0.010s
I20260709 12:23:12.403801  6969 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.406271  6969 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.382671 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:12.387584 27663 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:12.407502  7219 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:23:12.373396  7137 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.408016  7295 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:12.408946  7295 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:12.409025 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:12.409315  7295 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:12.410666 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:12.366415 27732 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:12.294196  6998 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 12:23:12.316603  7172 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 39.19, Next index: 20, Last known committed idx: 16, Time since last communication: 0.066s
I20260709 12:23:12.395810 27660 rpcz_store.cc:276] 0709 12:23:10.856106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.953350 (+ 97244us) service_pool.cc:224] Handling call
0709 12:23:12.376567 (+1423217us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54000c2954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:12.376734 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2275133184}
W20260709 12:23:12.396973 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 244369us
I20260709 12:23:12.385653  7247 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:12.396312  7312 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:12.462337 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:12.397631  7198 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:12.410120 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:12.428689 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.463145 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:12.411084 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c204963842941859d4313114fd92c93"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:12.437285 27655 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:12.442368 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.463982 27735 raft_consensus.cc:606] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:12.444795 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.445188 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.319113  7045 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.465286 28329 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:12.397701  6825 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:23:12.523524 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61236.6us
I20260709 12:23:12.392860  7272 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:12.454996 27660 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:23:12.528442 27582 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:23:12.533047  6825 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:12.534138  6825 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:12.529362 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119320us
I20260709 12:23:12.527524  7276 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.438086  7219 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:12.452651  7070 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.524861 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45874.3us
I20260709 12:23:12.537824 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:12.464694  7257 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.555377  7292 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.577293 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:12.578635 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.566033 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:12.537979  7272 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:12.452265  6969 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.120s
I20260709 12:23:12.557034  7219 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.536152  7144 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:23:12.586872 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49127.9us
I20260709 12:23:12.389423  7315 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.594995 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:12.599380  7323 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:23:12.599707  7130 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.468196  7247 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.603186 27580 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:12.410159  7295 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.605237  7219 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:12.603955  7131 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.613929  7016 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.613270  7270 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:12.624861  7144 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:12.579979  7070 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.570474  7276 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:12.639408  7183 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.830s	user 0.014s	sys 0.003s
I20260709 12:23:12.611246  6644 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:12.583837  7257 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.622427  7136 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.694875 28329 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: []
I20260709 12:23:12.688386 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.691805 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.683540 27662 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:12.649597  7045 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.625751  7287 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.060s
I20260709 12:23:12.679018  7144 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:12.738444 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44235us
W20260709 12:23:12.741778 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 126292us
I20260709 12:23:12.704869  7295 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:12.750206 27662 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:12.706291  6644 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:12.674598  7316 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.100s
I20260709 12:23:12.741649  7144 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:12.677013  7016 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.859676  7116 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:12.860186  7116 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:12.861173  7116 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.862139  7116 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:12.871937  6825 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:12.672765  7312 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.705142  7250 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:12.887832 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.898496  6825 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.990550 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:12.900422  7327 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:23:12.937093  6644 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:12.937944  7277 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.943642  7292 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.944845 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.945888  7250 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:12.946432 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.947597 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:12.948779 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.949435  6709 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:12.950450 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:13.053380 28329 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:12.950657  7278 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:12.985565  7276 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:12.856539  7045 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:12.954885 28336 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:23:12.956082  7315 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:12.865145  7331 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.957093 27587 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
W20260709 12:23:12.958647 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.065333 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.066200 27580 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:12.959261 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:12.959486  7285 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.962683 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:12.964432  7070 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:12.964944  7009 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:12.966554  7249 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:12.873389  7287 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:23:12.973232  7207 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.191s	user 0.012s	sys 0.002s
I20260709 12:23:12.973702 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:12.876730  7133 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.102s
I20260709 12:23:12.880805  6945 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.981205  7016 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:13.086359 27629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.002s	sys 0.006s
I20260709 12:23:12.983523  7296 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:12.983693  7325 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.335s	user 0.001s	sys 0.000s
I20260709 12:23:13.017287 27589 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:13.064178  7266 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:23:13.010378 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2100) took 2220 ms. Trace:
I20260709 12:23:13.087826 27661 rpcz_store.cc:276] 0709 12:23:10.789837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.869461 (+ 79624us) service_pool.cc:224] Handling call
0709 12:23:10.922956 (+ 53495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.978488 (+ 55532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.010113 (+2031625us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b54000e83d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:13.010296 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122279808}
I20260709 12:23:12.857618  7320 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:13.016170 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 4979) took 2175 ms. Trace:
W20260709 12:23:12.983789  7319 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.296s	user 0.000s	sys 0.001s
I20260709 12:23:13.023663 27731 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:12.997475 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:12.984019  7329 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
W20260709 12:23:12.984776 27552 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.694s (SENT) [suppressed 5 similar messages]
W20260709 12:23:13.045420 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.001585  7314 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.692s	user 0.003s	sys 0.000s
I20260709 12:23:13.046943  7316 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.052045 28328 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:12.952426  7247 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.057611  7278 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.009891  7252 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:12.983312  7257 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.067947 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.068120 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.074676 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.083009 27588 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 29: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:12.982199 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.069661  7045 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:13.090039 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 524110us
I20260709 12:23:13.051932  7277 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.092707 27585 rpcz_store.cc:276] 0709 12:23:10.840865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.841012 (+   147us) service_pool.cc:224] Handling call
0709 12:23:12.999279 (+2158267us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400088ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:13.016072 (+ 16793us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2776405376}
I20260709 12:23:13.079032  7287 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:13.149752 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:13.116514 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.119563 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 377068us
I20260709 12:23:13.120329 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.151840 27731 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:13.147289 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.153083 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.120430 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.091612  7296 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.131605  6825 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.085563  7207 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.142067 27729 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:13.089529 27629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.003s	sys 0.006s
I20260709 12:23:13.151041 27580 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:13.142413 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 284529us
I20260709 12:23:13.143891 27733 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:13.074241  7331 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.162478 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:13.172483 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.151722 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.154134 27736 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:13.158465 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:13.161774 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.087186  6945 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.155844 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:13.166126 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:13.158934 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.144325  7134 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.882s	user 0.008s	sys 0.004s
I20260709 12:23:13.180969 27580 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:23:13.191039 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.188040 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.193320 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.199297 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.106483  7320 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:13.202314 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.203378 27732 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:13.205108 27734 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:13.206281 27731 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:23:13.211599 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4638) took 1106 ms. Trace:
I20260709 12:23:13.207881 28331 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:13.212196 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.211974 27730 rpcz_store.cc:276] 0709 12:23:12.105066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.122157 (+ 17091us) service_pool.cc:224] Handling call
0709 12:23:12.122494 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.122812 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.122840 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.123286 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.280508 (+157222us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:12.280561 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.280634 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.000977 (+720343us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:13.204672 (+203695us) raft_consensus.cc:1685] finished
0709 12:23:13.204705 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.211536 (+  6831us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2721,"raft.queue_time_us":12464,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":5644,"spinlock_wait_cycles":11302272,"thread_start_us":137620,"threads_started":1,"wal-append.queue_time_us":179707}
I20260709 12:23:13.214996 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.215682 27664 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:13.216552 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.220131 27661 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:13.124316  7316 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.182950  6825 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:13.160998  7345 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:23:13.209919  7198 consensus_meta.cc:287] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.682s	user 0.006s	sys 0.005s
I20260709 12:23:13.231473 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.190124  6832 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:13.183832  7344 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:13.203418  7313 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.208715  7341 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.200963  7133 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:13.236276  7225 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.859s	user 0.006s	sys 0.006s
W20260709 12:23:13.246565 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 261745us
I20260709 12:23:13.245400 27581 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
W20260709 12:23:13.236089  7208 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.921s	user 0.014s	sys 0.002s
I20260709 12:23:13.190045  6709 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.049s
I20260709 12:23:13.229386  7334 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.233934  7244 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:13.235376  6825 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.267097 27664 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:23:13.271111 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.273492 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.276726 27738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1890) took 1174 ms. Trace:
I20260709 12:23:13.277187 27738 rpcz_store.cc:276] 0709 12:23:12.102172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.102334 (+   162us) service_pool.cc:224] Handling call
0709 12:23:12.108957 (+  6623us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.109363 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:23:12.149471 (+ 40108us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.704207 (+554736us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.737102 (+ 32895us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.737799 (+   697us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:23:12.737850 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.737925 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.887520 (+149595us) spinlock_profiling.cc:243] Waited 9.38 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:13.271229 (+383709us) raft_consensus.cc:1685] finished
0709 12:23:13.276537 (+  5308us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.276670 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27917,"prepare.queue_time_us":20054,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":35693,"raft.run_cpu_time_us":3787,"raft.run_wall_time_us":781231,"spinlock_wait_cycles":21568896,"thread_start_us":453734,"threads_started":2,"wal-append.queue_time_us":429143}
I20260709 12:23:13.258363  7344 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:13.290773 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 139144us
I20260709 12:23:13.292258 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.298005 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.274533  6825 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:13.248718  7198 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.253204  6832 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.267560  7225 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.247900  7287 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.316354  7344 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.264431  7341 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.318537  7341 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.266388  7133 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:13.319712  7340 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
W20260709 12:23:13.320730  7346 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:23:13.321764 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4740) took 1039 ms. Trace:
I20260709 12:23:13.322875  7351 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:13.323438  7351 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.323930  7249 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:13.324743  7249 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:13.326138  7351 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.326191 27733 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:13.328382  7316 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.331871  7198 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.333261  7296 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.334915  7045 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.322145 27658 rpcz_store.cc:276] 0709 12:23:12.282097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.282281 (+   184us) service_pool.cc:224] Handling call
0709 12:23:12.282751 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.283137 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.283179 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.286348 (+  3169us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.906491 (+620143us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b180045d000. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:13.160343 (+253852us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:13.160387 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.160453 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.321428 (+160975us) raft_consensus.cc:1685] finished
0709 12:23:13.321448 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.321702 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15815,"raft.queue_time_us":3722,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":22050,"spinlock_wait_cycles":4436864,"thread_start_us":143130,"threads_started":1,"wal-append.queue_time_us":974958}
I20260709 12:23:13.265429  7313 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.342374 27664 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:13.268651  7334 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.362884 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.370255 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4640) took 1264 ms. Trace:
I20260709 12:23:13.370705 27737 rpcz_store.cc:276] 0709 12:23:12.105570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.122564 (+ 16994us) service_pool.cc:224] Handling call
0709 12:23:12.122889 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.263616 (+140727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.263646 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.263685 (+    39us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:12.263715 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.369994 (+1106279us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.370051 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.370189 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323168640}
W20260709 12:23:13.376255 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78302.5us
I20260709 12:23:13.377978 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.379309  7345 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.379580  7295 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.577s	user 0.007s	sys 0.003s
I20260709 12:23:13.360589  7351 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:13.379869  7345 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.375869  7278 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:13.387204 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.396816  7354 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:13.389810 27658 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
W20260709 12:23:13.384684  7144 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.000s
I20260709 12:23:13.374092  6945 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.399894 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:13.399613  7344 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:13.403056  7341 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:23:13.404258  7144 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.405481  7225 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.406136  7331 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:13.406617  7144 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:13.387024  7272 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.003s
W20260709 12:23:13.397912  7219 consensus_meta.cc:287] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.746s	user 0.011s	sys 0.001s
I20260709 12:23:13.413195 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.415408 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.416126 27580 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:13.418118 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:13.417466 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:13.389624  7207 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.421909 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:13.389007  7351 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.401968  7320 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.455951  7345 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.460058 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4749) took 1092 ms. Trace:
I20260709 12:23:13.460570 27657 rpcz_store.cc:276] 0709 12:23:12.368054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.368245 (+   191us) service_pool.cc:224] Handling call
0709 12:23:12.390934 (+ 22689us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.637925 (+246991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.637988 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.638039 (+    51us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:12.638094 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.459832 (+821738us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.459874 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.459993 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":541870336}
I20260709 12:23:13.460058 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4747) took 1096 ms. Trace:
I20260709 12:23:13.463620 27655 rpcz_store.cc:276] 0709 12:23:12.363797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.365122 (+  1325us) service_pool.cc:224] Handling call
0709 12:23:12.368859 (+  3737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.463989 (+ 95130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.459846 (+995857us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b54001e46d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.459995 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156873984}
I20260709 12:23:13.464536  7287 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:13.467440  6825 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.469225 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4898) took 1381 ms. Trace:
I20260709 12:23:13.469928  6825 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.470770 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4752) took 1062 ms. Trace:
I20260709 12:23:13.471158 27662 rpcz_store.cc:276] 0709 12:23:12.408076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.410386 (+  2310us) service_pool.cc:224] Handling call
0709 12:23:13.470617 (+1060231us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b54001e41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:13.470720 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":556199296}
I20260709 12:23:13.472494 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37660 (request call id 67) took 1061 ms. Trace:
I20260709 12:23:13.469599 27589 rpcz_store.cc:276] 0709 12:23:12.087591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.125278 (+ 37687us) service_pool.cc:224] Handling call
0709 12:23:13.017029 (+891751us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.087524 (+ 70495us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.469044 (+381520us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b5400088ed0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:13.469167 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2050164480}
I20260709 12:23:13.472888 27735 rpcz_store.cc:276] 0709 12:23:12.410777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.410960 (+   183us) service_pool.cc:224] Handling call
0709 12:23:13.472091 (+1061131us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b5400048ba0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2c5520 00007f0e9a342652 00007f0e9f31060a 00007f0e988967d4 00007f0e98896762 00007f0e98896691 00007f0e9889660c 00007f0e988958c2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.472413 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5041920}
I20260709 12:23:13.473745 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:13.485248 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 67891.7us
W20260709 12:23:13.486481 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.490180  7334 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:13.481626  7360 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.063s
W20260709 12:23:13.496603 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83659.8us
I20260709 12:23:13.412510  7272 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.421267  7219 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.500065 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.501971 27580 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:13.508865 27589 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:23:13.505110 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:13.494058  6969 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:13.512581  7312 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:13.514619  7312 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.514986  7312 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:13.518412 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 4992) took 1467 ms. Trace:
I20260709 12:23:13.518805 27584 rpcz_store.cc:276] 0709 12:23:12.051377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.058045 (+  6668us) service_pool.cc:224] Handling call
0709 12:23:12.058367 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.084355 (+ 25988us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:23:12.235438 (+151083us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.365081 (+129643us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.365157 (+    76us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:12.365234 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.901609 (+536375us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:13.518187 (+616578us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b5400087ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.518236 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.518357 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16505,"prepare.queue_time_us":63029,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":21492,"raft.run_cpu_time_us":7609,"raft.run_wall_time_us":574998,"spinlock_wait_cycles":86082688,"thread_start_us":236793,"threads_started":2,"wal-append.queue_time_us":164112}
I20260709 12:23:13.520787  7272 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.521768 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.498607  7345 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.524729  7345 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.526185 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.530941 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:13.532130 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:13.532455 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.511050 27580 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:23:13.535864 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.536518 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.537277 27735 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:13.537323 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.538031 28328 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:13.538497 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.529636 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.513438  6969 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.541914 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.542778 27729 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:13.542937 28329 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:13.541613 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.543902 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:13.544626 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.544780 28330 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:13.545817 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.543242  7272 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:13.546456 28331 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:23:13.553220 27585 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:13.556407 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:13.559154 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.561872 27589 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:13.568178 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.569602 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:13.572360 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:13.573696 27729 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:13.575546 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.529934  6832 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.579481 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.550021  7353 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:13.584460  7207 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:13.585202  7207 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:13.585603  7207 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:13.543109  7219 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:13.543445  6969 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:13.560303  7270 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:13.595861  7313 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.597398  6951 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:13.598421  6951 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:23:13.598765  6951 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:13.602155 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.602890 27729 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:23:13.603204 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.604252 27585 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:13.605974 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.577819 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.608763 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:13.606552 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:13.600685  7172 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 19, Time since last communication: 0.039s
I20260709 12:23:13.616680  7345 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.616873  7172 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:13.617316  7172 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:13.618115  7172 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.602428 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:13.619287  7172 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:13.620613  7351 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.632220  6969 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:13.623593 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51347us
I20260709 12:23:13.623914  7009 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:13.632591  7285 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:13.634132 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.635084  7282 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.621143  7133 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:13.624454  6951 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.627642  7219 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:13.628015 28330 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:13.624081 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:13.641355 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:13.641851 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.644049 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:13.635572 27655 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41. Ops: []
W20260709 12:23:13.659308 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.648768 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.660701 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.661466 27662 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:13.662459 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.653327 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.653692 27658 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:23:13.628963 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.635586  7270 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:13.667794 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.668635 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.670244 27658 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:23:13.672408 27737 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:13.673676 27731 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:13.674346 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.667106 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:13.676247 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.676395 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:13.676863 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.677153 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:13.678313  7320 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:13.679054 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:13.679591 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4900) took 1214 ms. Trace:
I20260709 12:23:13.679976  7320 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:23:13.678943  7070 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.002s
I20260709 12:23:13.679948 27587 rpcz_store.cc:276] 0709 12:23:12.465056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.466691 (+  1635us) service_pool.cc:224] Handling call
0709 12:23:12.528977 (+ 62286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.064696 (+535719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.679358 (+614662us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54000889d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.679521 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984094976}
I20260709 12:23:13.684283 27662 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:13.685971 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.691823 27586 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:23:13.686820 27659 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:13.696138 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.696339 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.696847 28336 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:13.647547  7285 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.702579 27733 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:13.703656 27733 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:13.701138 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.655304  7130 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:13.693099  7237 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:13.673892  7219 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.675307  7133 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:13.715903  7133 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:13.715626  7137 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:13.716349  7133 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:13.719082  7247 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:13.715659  7219 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:13.724123  7247 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:13.726644 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:13.727970  7303 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:13.728415  7303 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:13.729082  7247 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:13.729583  7247 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.729238  7303 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.19, Last appended by leader: 19, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.729971  7247 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:13.730324  7303 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:13.710395 27737 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: []
I20260709 12:23:13.709522 27733 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.693279  7331 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:13.705039  6644 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:13.713227  6832 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:13.713708  7070 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.004s
I20260709 12:23:13.629662  7312 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.693902  7287 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:13.740753  7312 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.733210 27733 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:13.743824  7277 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:13.675666  7009 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:13.746330 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5009) took 1363 ms. Trace:
I20260709 12:23:13.749567 27660 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:23:13.751792  6969 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.700019  7351 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:13.655478  7282 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.705701  7285 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:13.701180  7345 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:13.752791 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.710234  7130 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:13.706467  6951 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:13.746721 27583 rpcz_store.cc:276] 0709 12:23:12.382443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.393165 (+ 10722us) service_pool.cc:224] Handling call
0709 12:23:12.403229 (+ 10064us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.384930 (+981701us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:23:13.398245 (+ 13315us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.658847 (+260602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:13.658903 (+    56us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:13.658952 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.745588 (+ 86636us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b18004ced00. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:23:13.746109 (+   521us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b540006f754. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.746156 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.746272 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16097,"prepare.queue_time_us":11573,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":16222,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":8988,"spinlock_wait_cycles":2273950208,"thread_start_us":242305,"threads_started":1,"wal-append.queue_time_us":308916}
I20260709 12:23:13.759807 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.708760  7270 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:13.703613  7316 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:13.705952  7112 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:13.763846 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:13.730955  7133 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.764482 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:13.764680 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:13.765466 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.767163 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:13.810317 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45714us
I20260709 12:23:13.772099 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.811982 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.781922 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.785398 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.812875 27658 raft_consensus.cc:2368] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41.
I20260709 12:23:13.785974 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:13.781606  7345 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.764439 27733 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:23:13.804613 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.763396  7131 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:23:13.805147 28332 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:13.781042  7331 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:13.796892  7248 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.797205 27657 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:13.804072 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.812695 27731 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:13.827517 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.810971 27660 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:13.813709 27655 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:13.820070 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.828847 27660 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:13.828933 27658 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:13.822585 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:13.798239  7244 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:13.836086 27660 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:23:13.836431 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:13.833413 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:13.816287  7070 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:13.838891  7070 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.800565  7342 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:13.840099 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:13.792341  7015 consensus_queue.cc:1059] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:23:13.830629  7333 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
I20260709 12:23:13.817921  7345 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:13.818547  7131 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:13.818051  7277 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:13.793156  7207 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:13.847388 27657 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:13.848425 27661 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:23:13.847437 27731 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
W20260709 12:23:13.853216 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.854708  7070 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:13.866360  7134 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:13.821162  7122 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:13.867647  7130 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:13.830961  7315 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:23:13.868340 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:13.986137 27581 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:13.869058 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.869798 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.987568 27655 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:13.988174 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:13.988837 27585 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:13.870518 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.871227 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.993235 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:13.994626 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.996098 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.996829 27588 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:13.997867 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.000168 27589 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:14.002105 27589 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:23:13.837419  7248 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.006719  7248 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:13.843618 27658 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:13.945652 27729 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:23:14.009269 27729 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:23:14.010807 27729 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
W20260709 12:23:13.946722 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.951504  7369 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 12:23:13.951994 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:14.013408 28330 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:13.952971  7367 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:23:13.952827 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:14.018196 28333 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:13.953595 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.954468 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:13.955152  7270 raft_consensus.cc:993] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:13.814673  7279 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.036s
I20260709 12:23:13.947207 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1885) took 3089 ms. Trace:
I20260709 12:23:14.021368 27734 rpcz_store.cc:276] 0709 12:23:10.857459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.674280 (+816821us) service_pool.cc:224] Handling call
0709 12:23:11.674524 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.210685 (+1536161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.871803 (+661118us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b5400316d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.947151 (+ 75348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3519424384}
I20260709 12:23:13.962059 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.023720 27580 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:13.815691 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:13.963766 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.025864 27584 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:23:13.965653 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2135) took 2264 ms. Trace:
I20260709 12:23:14.026669 27586 rpcz_store.cc:276] 0709 12:23:11.680702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.680836 (+   134us) service_pool.cc:224] Handling call
0709 12:23:11.681104 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.692296 (+2011192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.884517 (+192221us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:13.945114 (+ 60597us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4623721344}
I20260709 12:23:13.971364 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:14.027927 27582 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:13.972541 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2111) took 1526 ms. Trace:
I20260709 12:23:14.028548 27663 rpcz_store.cc:276] 0709 12:23:12.445861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.464035 (+ 18174us) service_pool.cc:224] Handling call
0709 12:23:12.526075 (+ 62040us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.528553 (+  2478us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:23:13.467156 (+938603us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b18003435a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf
0709 12:23:13.467301 (+   145us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.793695 (+326394us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.799059 (+  5364us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.799231 (+   172us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:23:13.799268 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.799355 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.871538 (+ 72183us) raft_consensus.cc:1685] finished
0709 12:23:13.972156 (+100618us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.972422 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108041,"prepare.queue_time_us":301824,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":4132,"raft.queue_time_us":110601,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":5713,"spinlock_wait_cycles":12783872,"thread_start_us":475767,"threads_started":2,"wal-append.queue_time_us":281479}
I20260709 12:23:13.976653 27733 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:23:14.035975 27580 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:14.038865 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:13.966264  7277 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.844950  7244 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:14.046451 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 230835us
I20260709 12:23:14.050274 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:14.048259 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 217052us
I20260709 12:23:13.999243 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:14.061287 27588 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:14.062171 28330 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:13.964211  7316 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:13.977763  7134 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:14.070813 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.066681 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:14.067224 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 116507us
I20260709 12:23:14.063912 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:14.085412 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:13.965410  7313 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.088s
I20260709 12:23:14.073928 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:14.088819 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:14.089160 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 144502us
I20260709 12:23:14.097781 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.098423 27729 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:13.979851  7252 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.084s
I20260709 12:23:14.049173  7123 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:14.049955  7277 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:14.141625 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 67761.5us
I20260709 12:23:14.142733 27628 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:14.051241  7266 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:14.067677  6832 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:14.065886  7259 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:14.071353  6825 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:13.986642  7122 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.144404 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:14.151887  7138 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:23:14.152555 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 91931.2us
I20260709 12:23:14.174459 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:14.152806 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 86201.6us
I20260709 12:23:14.130803  7198 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:14.153313  7295 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:14.071185  7231 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:23:14.158329 27581 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:14.072290  7316 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:14.178977 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:14.141139  7130 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:14.150285  6969 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:14.203629 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:14.186995 27585 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:14.197913 27587 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:14.102010  6945 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:14.210695 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:14.212209 27630 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:14.213171 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 147958us
I20260709 12:23:14.164064  6825 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:14.052312  7331 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.149655  7110 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.050s
I20260709 12:23:14.214787 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:14.224285 27588 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:14.225507 27580 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:14.194144  7315 raft_consensus.cc:2764] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:23:14.194003  7295 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:14.230131 27581 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 21. (term mismatch)
I20260709 12:23:14.197525  6644 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.210155  7137 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:14.233448 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:14.236058 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:14.176689  7341 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:23:14.213575  7237 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:14.275991 27581 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 20
I20260709 12:23:14.162949  6832 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:14.277014 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 437000us
W20260709 12:23:14.277949  7172 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.644s	user 0.008s	sys 0.003s
I20260709 12:23:14.214737  7320 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:14.278643  7244 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.226874  7133 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:14.232558  7353 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:23:14.303721  7368 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.236s	user 0.000s	sys 0.000s
I20260709 12:23:14.304158  7353 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:14.305575  7353 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.163421  7361 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:23:14.237548  7102 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:14.309064  7320 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:14.308890  7102 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.279488  7282 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.202026  7198 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:14.269775  7279 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:23:14.291687 27581 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 37.21
I20260709 12:23:14.218490  7070 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:14.315254 27581 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:14.293139 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57153us
I20260709 12:23:14.215760  7266 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:14.295924 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:14.231000  7270 raft_consensus.cc:1081] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:14.298652  7338 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.299930  6832 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:14.234120 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 162001us
W20260709 12:23:14.318217 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 220764us
W20260709 12:23:14.350613 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.228803  6825 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:14.322964 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:14.213351  6945 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:14.294071 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60725.5us
W20260709 12:23:14.333884 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.196898  7134 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.350854 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:14.289258  7295 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:14.312022  7109 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:23:14.351651 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.360508 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:14.361344 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:14.367856 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.291079  6823 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.053s
I20260709 12:23:14.313059  6969 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:14.337527  7133 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.340390  7278 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:14.371500 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48608us
I20260709 12:23:14.340186  7172 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:14.285652  6644 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.376215 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:14.344295  7231 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.380020 28330 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:23:14.353866  7207 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:14.421811 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48663.3us
I20260709 12:23:14.369323  7116 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:14.364029  7279 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:14.408980  6969 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:14.370484  7109 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:14.377627  7244 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.056s
I20260709 12:23:14.433725  7266 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.434865  7102 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.425194  7231 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:14.437789  7285 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.640s	user 0.009s	sys 0.000s
I20260709 12:23:14.438649  7122 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:14.440373  7112 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.635s	user 0.008s	sys 0.002s
I20260709 12:23:14.440995  7388 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:14.442332 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:14.422505  7278 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.445525  7387 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:14.445603  7207 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:23:14.414526  7219 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.690s	user 0.010s	sys 0.001s
I20260709 12:23:14.413229  7015 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.035s
W20260709 12:23:14.449084  7303 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.718s	user 0.008s	sys 0.004s
I20260709 12:23:14.375528  7138 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:14.355273  7338 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.409375  7133 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.433202  7116 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.374972  7137 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:23:14.456674  7372 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 12:23:14.457589  7375 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 12:23:14.374341  7374 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:23:14.474942 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.478753 27704 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:14.460963  6951 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.661s	user 0.012s	sys 0.000s
I20260709 12:23:14.422600  7377 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.369812  6832 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.091s
W20260709 12:23:14.498694 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56442.8us
I20260709 12:23:14.574393  6832 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:14.574785  6832 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:14.575631  6832 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.576761  6832 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:14.576990 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.577642 27664 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
W20260709 12:23:14.577917  7354 consensus_queue.cc:1175] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 34 status { last_received { term: 34 index: 20 } last_committed_idx: 20 last_received_current_leader { term: 34 index: 20 } }
I20260709 12:23:14.578758 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.473759  7109 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:23:14.507289 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129292us
I20260709 12:23:14.507478 27737 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:14.509285 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.498454  7112 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:14.506255  7331 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:14.504463  7220 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:23:14.481395  7198 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.512048  7116 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:14.543020  7278 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:14.544602  7133 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:14.499886  6825 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.549796  7138 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.551940 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:14.552295  7259 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:14.512168  7137 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.561394  6644 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:14.641866 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 410783us
I20260709 12:23:14.564409  7231 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.504245  7386 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.066s
I20260709 12:23:14.565398  7277 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.566525  7282 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:14.571786 27662 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 29 because replica has last-logged OpId of term: 28 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:23:14.572695  7338 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.573081  6969 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.19, Last appended by leader: 19, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:14.488274  7351 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.716s	user 0.009s	sys 0.002s
I20260709 12:23:14.579924 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.587498 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.586434 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.584825 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.667271 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.582762 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.590772 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.583307 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.606536  7395 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.609609 28336 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33. Ops: []
W20260709 12:23:14.611411 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 132769us
I20260709 12:23:14.667528 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:14.671891 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.614421 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:14.672745 27657 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:14.672992 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.615037  7015 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:14.672094 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:14.617357 28295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.013s	sys 0.013s
W20260709 12:23:14.675359 28295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.013s	sys 0.013s
W20260709 12:23:14.675704 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 705697us
I20260709 12:23:14.675827 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.617674  7380 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:14.618304  6709 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:14.624071  7237 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:14.624809 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.683211 27733 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:14.499269  7285 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.628069 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:14.635684 28331 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:14.504148  7219 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.624893  7393 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:14.646617  7320 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.648960 27734 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:14.685887 27736 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:14.536365  7316 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.612479 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:14.632974  7379 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.046s
I20260709 12:23:14.599150  7377 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:14.659449  7345 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.725s	user 0.010s	sys 0.002s
I20260709 12:23:14.641794  7137 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:14.686908 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:14.653586  7282 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:14.692149 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:14.693028 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80624.6us
W20260709 12:23:14.711340 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 83386.4us
I20260709 12:23:14.711877 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:14.699914 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65766.8us
I20260709 12:23:14.713445 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:14.579264  7131 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.670428  7351 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:14.676846 27658 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:14.720670 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:14.675026  7015 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:14.703133 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:14.695462  7320 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.721261 27658 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:14.704370 27657 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:14.707576  7331 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:14.665966  7231 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.030s
I20260709 12:23:14.674006 27587 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:14.715747 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:14.626096  7279 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.682842  6709 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:14.691490  7341 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:14.725683 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49277.4us
I20260709 12:23:14.682071  7312 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:23:14.698364  7133 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:14.928408  7133 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.928857  7133 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:14.738193  7247 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.008s	user 0.011s	sys 0.003s
W20260709 12:23:14.931049 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.744757 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:14.724619  7387 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:14.719663  7112 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:14.674571  7395 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.728361  7015 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:14.731963  7131 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:14.728595  7331 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.952962  7331 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:14.930384  7341 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.955641  7341 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:14.721098  7383 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:14.725193  7379 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 38.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.032s
I20260709 12:23:14.717907  7345 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.734573  7295 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:14.758529  7123 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.612s	user 0.011s	sys 0.000s
W20260709 12:23:14.768582 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 125693us
I20260709 12:23:14.728216  7109 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.733116  7403 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.049s
I20260709 12:23:14.786310  7386 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:14.791718  7400 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:23:14.792577  7316 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.796972 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4671) took 1004 ms. Trace:
W20260709 12:23:14.797132  7397 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:23:14.802237 27655 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7c204963842941859d4313114fd92c93"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:50014
I20260709 12:23:14.803593  6825 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:14.804167  7277 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:14.804894  7285 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.804800 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:14.810316  7137 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.982746 27659 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:14.814461 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:14.983047 28329 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:14.983660 28336 raft_consensus.cc:2410] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:23:14.816872  7272 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:14.991501  7137 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.991595  7272 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:14.992619  7272 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:14.818903  7371 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.643s	user 0.004s	sys 0.000s
I20260709 12:23:14.824139 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:14.944742  7247 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:14.827296 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:14.815095 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:14.833774 28332 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:14.835768  7282 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:14.783929  7342 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.838572 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:14.998343 27736 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:14.847903  7377 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:14.861249 27738 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:14.862250  7116 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:14.870098 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 197816us
I20260709 12:23:15.002306 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:14.875928  6965 raft_consensus.cc:3055] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:14.873194  6832 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.003929  6965 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:14.994715  7398 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.010s
I20260709 12:23:15.004156  6832 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.004985  7398 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:23:15.004856  6965 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.005944  6965 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:15.006331  6832 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:14.877596  6945 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:14.889843 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4652) took 1349 ms. Trace:
I20260709 12:23:14.882645 27729 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:15.007181 27735 rpcz_store.cc:276] 0709 12:23:13.540547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.540872 (+   325us) service_pool.cc:224] Handling call
0709 12:23:13.945439 (+404567us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:14.071270 (+125831us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:23:14.323547 (+252277us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.502884 (+179337us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.503639 (+   755us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.504298 (+   659us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:14.530742 (+ 26444us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.553086 (+ 22344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.553334 (+   248us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.761293 (+207959us) raft_consensus.cc:1685] finished
0709 12:23:14.761315 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.889620 (+128305us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b540029f750. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:14.889769 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":302685,"prepare.queue_time_us":80706,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":17390,"raft.queue_time_us":174203,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":44465,"spinlock_wait_cycles":859901568,"thread_start_us":304861,"threads_started":2,"wal-append.queue_time_us":308757}
I20260709 12:23:14.897466 27658 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:14.907934  7219 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:14.916023 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:14.721859 27737 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
W20260709 12:23:14.924316 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 212949us
I20260709 12:23:14.932145 27662 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:23:14.936941 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:14.972733  7070 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.027s
I20260709 12:23:14.931921 27661 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:14.948745 27664 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 41. Ops: []
I20260709 12:23:14.930949  7138 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:23:14.973714 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.972375 27732 rpcz_store.cc:276] 0709 12:23:13.792389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.946042 (+153653us) service_pool.cc:224] Handling call
0709 12:23:13.946392 (+   350us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:14.071551 (+125159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.071599 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:14.104264 (+ 32665us) log.cc:844] Serialized 52 byte log entry
0709 12:23:14.362061 (+257797us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:14.362117 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.362214 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.796738 (+434524us) raft_consensus.cc:1685] finished
0709 12:23:14.796761 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.796914 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":176484,"raft.queue_time_us":188639,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":12511,"thread_start_us":209435,"threads_started":1,"wal-append.queue_time_us":564912}
I20260709 12:23:14.979267 27581 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:14.813347  7320 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.029165 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:14.981885 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:15.029835 27731 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:14.987226  7109 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:14.987785 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:14.988041  6709 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:14.993958 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 169904us
I20260709 12:23:14.825757 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:15.042070 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 216380us
I20260709 12:23:14.971810  7360 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.997109 27663 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:15.003646  7405 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:15.005359  7398 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:14.912984  7134 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:15.011662 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4660) took 1443 ms. Trace:
I20260709 12:23:15.012809 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:15.017866 27585 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:14.938720 27588 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:23:15.018481 27662 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:15.025902 27660 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: [35.20-35.20]
I20260709 12:23:15.056509  7398 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.051641 27730 rpcz_store.cc:276] 0709 12:23:13.568277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.568404 (+   127us) service_pool.cc:224] Handling call
0709 12:23:13.568743 (+   339us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:13.573490 (+  4747us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.573528 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:13.741408 (+167880us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:23:13.742015 (+   607us) log.cc:844] Serialized 52 byte log entry
0709 12:23:13.975982 (+233967us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:14.051633 (+ 75651us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.485782 (+434149us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.486027 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.486102 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.620647 (+134545us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b18003dace0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:14.818323 (+197676us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:15.011459 (+193136us) raft_consensus.cc:1685] finished
0709 12:23:15.011488 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.011610 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123185,"prepare.queue_time_us":152323,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":4705,"raft.queue_time_us":189401,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":15732,"spinlock_wait_cycles":17559424,"thread_start_us":555298,"threads_started":2,"wal-append.queue_time_us":348442}
I20260709 12:23:14.998978  7272 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:14.998780  7037 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:15.030983 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.074905 27732 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:15.030433 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48603.9us
I20260709 12:23:15.077891 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:15.078809 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:15.026261 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:15.079695 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53551.6us
I20260709 12:23:15.080551 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:15.036530 27587 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:14.978104  7386 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.041229 27655 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:15.087016 27655 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:15.000397  7122 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:15.044031 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4673) took 1251 ms. Trace:
I20260709 12:23:15.098541 27737 rpcz_store.cc:276] 0709 12:23:13.792963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.836217 (+ 43254us) service_pool.cc:224] Handling call
0709 12:23:15.043819 (+1207602us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b5400318654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:15.043956 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540013312}
I20260709 12:23:15.047142 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2117) took 1230 ms. Trace:
I20260709 12:23:15.099893 27659 rpcz_store.cc:276] 0709 12:23:13.816603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.822283 (+  5680us) service_pool.cc:224] Handling call
0709 12:23:15.046968 (+1224685us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:15.047088 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2644574592}
I20260709 12:23:15.043267 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:15.047868 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:15.101240 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58088.8us
I20260709 12:23:15.050354 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:15.103077 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52799.9us
I20260709 12:23:15.104007 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:15.096376 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:15.114228 27664 raft_consensus.cc:1240] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:15.049003  7377 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:23:15.026727  7387 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:15.067734  7112 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:15.060675 27662 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:15.084018 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.040241  7285 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:15.089962 28331 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:15.093540 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:15.140784 27659 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:15.141252 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:15.064802 27663 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:15.095782 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:15.141829 27588 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:15.095679 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:15.115653 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:15.142664 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:15.002051  7342 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.051455  7405 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:15.025609  6945 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.051s
I20260709 12:23:15.125170 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:15.117509 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.143306 28333 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:15.074659  7282 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:15.084779  6825 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:15.083091  7037 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:15.307281  7375 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:23:15.062180  7360 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.053429  7331 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:15.159821  7411 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:23:15.149595  7351 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.139091  7387 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:15.167843  7219 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:15.116148  7277 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:15.115958  7278 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:15.125316  7380 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:15.139019  7220 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:15.126624  7122 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.113458  7378 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:23:15.123989  7259 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:15.045989  7381 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.095s
I20260709 12:23:15.161458 28330 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:23:15.335273  7405 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.125910  7395 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.140022  7285 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:15.168119  7401 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.199s
I20260709 12:23:15.159950  7399 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.202s
I20260709 12:23:15.343681  7037 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:15.169636  7404 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.213s
W20260709 12:23:15.369946  7420 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 12:23:15.364179  7387 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:15.371531  7219 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:15.371469  7278 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:15.398904  7220 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.412717  7342 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.398411  7345 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.416081  7386 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:15.416432  7331 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.354701  7383 consensus_queue.cc:1059] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 17, Time since last communication: 0.055s
W20260709 12:23:15.407611  7418 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 12:23:15.407200  7285 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:15.402724  7259 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:15.404193  7277 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:15.429337  7259 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:15.429941  7277 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.420861  7015 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.371871  7295 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.425729  7247 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.409855  7405 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:15.431640  7386 raft_consensus.cc:3055] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:23:15.434479  7386 raft_consensus.cc:740] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:15.435422  7386 consensus_queue.cc:260] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.370379  7351 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.424764  7278 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:15.415175  7116 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.402390  7380 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.426581  7220 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:15.444814  7380 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:15.450413  7422 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 12:23:15.483466 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 360115us
I20260709 12:23:15.498962 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:15.501773 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:15.503538 27580 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:15.514587 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.515380 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:15.124608 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:15.553012 27582 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:15.142668 27583 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:15.516409 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:15.518600 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:15.520074 27586 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:15.535482 27589 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:15.338760 27655 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.564486 27656 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:23:15.566716 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4750) took 3171 ms. Trace:
I20260709 12:23:15.568753 27656 rpcz_store.cc:276] 0709 12:23:12.394850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.395018 (+   168us) service_pool.cc:224] Handling call
0709 12:23:12.395351 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.565208 (+3169857us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.566254 (+  1046us) spinlock_profiling.cc:243] Waited 3.17 s on lock 0x7b54000e6ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:15.566607 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7285662976}
I20260709 12:23:15.576066 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 3998) took 1743 ms. Trace:
I20260709 12:23:15.576341 27655 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.576609 27660 rpcz_store.cc:276] 0709 12:23:13.832712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.844920 (+ 12208us) service_pool.cc:224] Handling call
0709 12:23:13.845250 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.055601 (+1210351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.575294 (+519693us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400343d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:15.575854 (+   560us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2705241344}
W20260709 12:23:15.339406 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.579457 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.207720 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52988.4us
I20260709 12:23:15.586233 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:15.586597 27478 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 25 to 27, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 27 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:15.589627 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.339721 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:15.593027 27730 raft_consensus.cc:606] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:15.593674 27730 raft_consensus.cc:614] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:15.596097  6825 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:15.597764  6969 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.598953  6969 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.599872  6965 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.341419 27732 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:15.601016  6965 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.601621  7037 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.602258  6969 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.602849  6969 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.604336  6965 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.604439  7037 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:15.583011 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.612208 27733 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
W20260709 12:23:15.613749 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 474559us
I20260709 12:23:15.615420 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:15.615629 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:15.608302  6969 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:15.634716 27657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.000s	sys 0.004s
I20260709 12:23:15.599116  6825 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:15.639566  7270 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:15.639922  6825 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.643829  7207 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.644918  7207 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.645928  6825 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.641309  7270 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:15.650324 27587 consensus_meta.cc:287] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.569s	user 0.010s	sys 0.017s
I20260709 12:23:15.650790 27587 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:23:15.442309  7341 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:15.651872 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66111us
I20260709 12:23:15.654273  7122 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.654663  7341 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:15.655473 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160776us
I20260709 12:23:15.655875  7381 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:15.656656  7381 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:15.657006 27552 leader_election.cc:400] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41.
I20260709 12:23:15.657472 27552 leader_election.cc:403] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Cancelling election due to peer responding with higher term
I20260709 12:23:15.658476  7198 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:23:15.659183 27736 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.660s	user 0.009s	sys 0.013s
I20260709 12:23:15.662243 27736 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:15.665977 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:15.666157 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51188.7us
I20260709 12:23:15.667726  7401 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:15.668536  7401 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.659746  7266 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.007s
W20260709 12:23:15.651202 27657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.000s	sys 0.004s
I20260709 12:23:15.666841  7433 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:15.658318  7381 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.681279 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:15.663807  7432 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.028s
I20260709 12:23:15.661337  6969 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:15.686347 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.426066  7219 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.660909  7351 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:15.682693 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.677244  7403 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.674683  7133 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.745s	user 0.010s	sys 0.001s
I20260709 12:23:15.681599  7320 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election lost for term 40. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41.
I20260709 12:23:15.673712  7315 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.651060  7270 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.704254 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:15.439157  7259 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.686551  7207 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:15.683283  7272 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.606s	user 0.007s	sys 0.002s
I20260709 12:23:15.719144  6825 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:15.671315  7401 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.690900 27478 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:15.730453 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:15.732168 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51166us
I20260709 12:23:15.734011 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:15.734545 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:15.735684  7435 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:15.728675  7133 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.683583  7395 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:15.731212 27478 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 24 to 25, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 25 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:15.738529  7395 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.739629  7315 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.742127 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.742836 27656 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:15.741142 27478 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:15.744680 27588 consensus_meta.cc:287] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.602s	user 0.005s	sys 0.018s
I20260709 12:23:15.745127 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:15.748399  7259 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.722659  7354 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.752115  7354 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.754688  7354 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.755409  7405 raft_consensus.cc:2764] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election decision vote started in defunct term 33: won
I20260709 12:23:15.756050 27731 raft_consensus.cc:2368] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:23:15.755627  7403 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.757208 27655 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: [39.21-39.21]
I20260709 12:23:15.757468  7133 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:15.758332  7403 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.758913 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.761277 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:15.761423 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:15.762318 27663 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:15.762472 27729 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:15.764103 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:15.764472 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.765519 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:15.671799  7404 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.071s
I20260709 12:23:15.764119 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.768677 27731 raft_consensus.cc:2368] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
W20260709 12:23:15.772476 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 83996.4us
I20260709 12:23:15.774103  7395 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:15.775651 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87894.4us
I20260709 12:23:15.780493 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1743) took 1328 ms. Trace:
I20260709 12:23:15.781072 28330 rpcz_store.cc:276] 0709 12:23:14.452401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.453348 (+   947us) service_pool.cc:224] Handling call
0709 12:23:14.453659 (+   311us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.401900 (+948241us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.779827 (+377927us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b5400520154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:15.780266 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1627126656}
I20260709 12:23:15.783367  7351 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:15.784153  7351 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.784677 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.786041 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.786511  7351 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.786701 27733 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:23:15.787022 27733 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:15.787931  7282 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.555s	user 0.013s	sys 0.000s
W20260709 12:23:15.789103 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55422.5us
I20260709 12:23:15.790911  7207 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:15.791860  7207 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.792294 27733 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.792505  7207 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.793445 27733 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:15.794265  7244 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:15.794688  7244 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:15.795055  7207 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:15.758810  7315 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.745564 27478 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 29, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 29 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:23:15.819284 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 303204us
I20260709 12:23:15.853442  7109 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:15.853435  7432 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.25, Next index: 26, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:15.854470  7109 consensus_queue.cc:1243] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 37.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:23:15.856010  7432 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.856727  7432 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.859416  7432 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:15.860208  7432 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:15.860622  7432 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:15.862167  7432 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.867378 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:15.806110  7270 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.877729 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.878201 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.884032 27655 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:23:15.884603 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:15.902510 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 94167.3us
I20260709 12:23:15.887553  7248 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.886690  7278 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:15.920516 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 53458.6us
I20260709 12:23:15.919027  7138 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.888307  7315 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.922389 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.927604 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53432.6us
I20260709 12:23:15.931910  7122 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:15.941597  7122 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.941970  7122 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:15.948576  7220 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.951750 27734 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:23:15.951300 27478 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 11 to 31, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 31 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:15.955482 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:15.956751 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68836.2us
I20260709 12:23:15.957599  7449 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:23:15.957931  7399 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:15.958046 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:15.958662  7399 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:15.959259  7399 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:15.959842  7131 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:15.960309  7248 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.960486  7131 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.962561  7248 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.962564  7131 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:15.959118  7444 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.005s
I20260709 12:23:15.965020  7278 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:15.965394  7438 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:15.966255  7438 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.966770 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2120) took 1345 ms. Trace:
I20260709 12:23:15.967177 27658 rpcz_store.cc:276] 0709 12:23:14.620771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.010617 (+389846us) service_pool.cc:224] Handling call
0709 12:23:15.010884 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.011083 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:23:15.011394 (+   311us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.863754 (+852360us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.938832 (+ 75078us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:15.966068 (+ 27236us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:23:15.966233 (+   165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.966380 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.966577 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":285073,"prepare.queue_time_us":5401,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":391342,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":9449,"thread_start_us":615438,"threads_started":3,"wal-append.queue_time_us":410070}
I20260709 12:23:15.973451 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.974324 27730 raft_consensus.cc:2368] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36.
I20260709 12:23:15.976893  7338 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.977558  7338 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.978102 27662 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:15.979305 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.981089  7345 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:15.981853  7345 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.957016 27661 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:15.984478 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.985162 27660 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:15.985296 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:15.985991 27661 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:15.985554  7315 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.987849 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.919410  7351 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.988641 27659 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:15.989328  7351 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:15.988248  7315 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.991842 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:15.980160 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.981199 27662 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:15.992586 27656 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:15.992865 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:15.980408 27655 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:15.993569 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.994900 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:15.956430  7138 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:15.996317 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:15.997175  7138 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:15.980564 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4779) took 1385 ms. Trace:
I20260709 12:23:15.975090 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:15.998469  7432 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:15.999001 27663 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:15.968778  7438 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.008162  7351 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.008528  7360 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.011529  7270 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.012174 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.013372 27733 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:16.014497 27738 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:16.028246 27655 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:23:16.025679 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:15.958669  7220 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:15.999162 27657 rpcz_store.cc:276] 0709 12:23:14.594699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.005818 (+411119us) service_pool.cc:224] Handling call
0709 12:23:15.006154 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.006517 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.006558 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.006981 (+   423us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.979835 (+972854us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b5400182520. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a2bf6b8 00007f0e9a34892b 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c
0709 12:23:15.979947 (+   112us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:23:15.980060 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.980166 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:15.980245 (+    79us) raft_consensus.cc:1685] finished
0709 12:23:15.980258 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.980495 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":361,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"spinlock_wait_cycles":106079872,"thread_start_us":665328,"threads_started":1,"wal-append.queue_time_us":627769}
I20260709 12:23:16.030228 27659 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:15.954838 27478 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 31, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 31 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.036844 27478 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 32, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 32 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.029630 27737 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
W20260709 12:23:16.039852  7285 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.007s
W20260709 12:23:16.040587  7387 consensus_meta.cc:287] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.608s	user 0.003s	sys 0.013s
I20260709 12:23:16.030283 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:16.041450 27737 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 19. Preceding OpId from leader: term: 41 index: 20. (index mismatch)
I20260709 12:23:16.041626  7285 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.026019  7122 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.048838 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:16.051626 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.052313 27658 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:23:16.060407 28330 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:23:16.063436  7122 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.064859  7070 raft_consensus.cc:2764] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election decision vote started in defunct term 36: won
W20260709 12:23:16.056006 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98069.5us
W20260709 12:23:16.067099 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.071687 27662 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:16.076745 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.085733 27583 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:16.086745 27656 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:16.038770 27478 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 29, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 29 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.103022 27733 raft_consensus.cc:606] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:16.103529 27733 raft_consensus.cc:614] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:16.096271 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.071213  7278 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.026s
I20260709 12:23:16.105846 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.109334 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.116154 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.117019 28331 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:16.056519  7387 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.101027  7270 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.119119 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.113652 28334 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:16.135551  7070 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:16.135911  7237 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:16.137326  7237 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:16.138206  7237 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:16.138895  7444 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:16.139286  7444 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.140331  7444 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.141543  7279 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:16.142280  7279 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.143503  7408 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:16.143908  7408 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:16.144524  7279 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.144919  7408 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.141713  7444 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:16.146215 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.138036  7395 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:16.137430  7070 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:16.145756  7285 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.170516 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.146937  7237 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.143244  7015 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.177583 27659 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:23:16.174993 27737 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:16.182283 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.182550  7452 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:23:16.184475  7320 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:16.185241  7320 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.190178 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:16.190796 27731 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:16.194257 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.194753  7387 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.195122 27735 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:16.160434  7433 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:16.172806  7435 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:16.209725 27731 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:16.159505  7345 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.216118 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.217476 27552 leader_election.cc:400] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:23:16.207556  7431 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:23:16.220197  7380 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.516s	user 0.011s	sys 0.003s
I20260709 12:23:16.211221  7070 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:16.224579  7454 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.201s	user 0.003s	sys 0.000s
I20260709 12:23:16.225835 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.225322  7431 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.207828  7122 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:16.200076  7279 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.029s
W20260709 12:23:16.233204 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.228853 27735 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:16.233707 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.240077 28331 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
W20260709 12:23:16.244964 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.240681 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.245661 27662 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:16.243703 27661 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:16.234522 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.237641 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.250209 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.248152 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.248648 27734 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:16.233583 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.231293  7070 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.262681 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:16.264988 27589 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:23:16.266973 27580 raft_consensus.cc:2410] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 34 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
I20260709 12:23:16.268314 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.268930 27588 raft_consensus.cc:2393] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:16.273238 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.278982 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:16.280581  7225 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.281279  7225 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.282223  7345 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:16.232371  7122 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.288744 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.290010 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.292028 27734 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 31 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:23:16.242892  7380 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.259876  7320 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.306207  7122 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.306919 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.242412 27478 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.308949 27478 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 31 to 34. New cstate: current_term: 34 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:16.310037 28333 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:16.310412 28333 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:16.297374 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.298532  7244 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.311241 28333 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.312075  7244 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:16.312222 28333 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:16.312976  7398 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:16.314601 27734 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 19. (term mismatch)
I20260709 12:23:16.314965 27734 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:23:16.300698 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.302760 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:16.290802 28334 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 29.
I20260709 12:23:16.232029  7431 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.307855 27731 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 42 because replica has last-logged OpId of term: 40 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 21.
I20260709 12:23:16.294746 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:16.318646 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.324311 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.327508 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.327696 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.315214 27734 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:23:16.335035 27582 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.329990 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:16.330466 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 226286us
I20260709 12:23:16.336243 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.310811 27478 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:23:16.331820 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.328622  7431 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:16.334329 27734 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.233631  7237 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.334395 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:16.332299 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.341221 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:16.338944 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.344211 27731 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:16.342849 27738 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:16.345902 27738 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:16.346086 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.346810 27589 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:16.347512  7070 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:23:16.350680 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.347143 27738 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:16.352792 27735 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:16.296418  7399 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:16.354277 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 248952us
I20260709 12:23:16.356545 27732 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:16.356580 27584 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:16.341531  7225 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.356200 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:16.359709 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 147605us
I20260709 12:23:16.360343 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.350519  7070 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.361359 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:16.364866 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.364967 28333 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 23. (term mismatch)
I20260709 12:23:16.367673 27580 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:16.357785  7266 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:16.365602 28333 pending_rounds.cc:85] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 22
I20260709 12:23:16.371866 27585 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:16.372356  7231 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.363305  7278 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:16.373368 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.375038  7387 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:23:16.380633 28333 pending_rounds.cc:107] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 36.23
I20260709 12:23:16.377220 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:16.378014  7131 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:23:16.348069  7320 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.015s
I20260709 12:23:16.376137  7383 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:16.366317  7070 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:16.370656 27729 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: [39.21-39.21]
I20260709 12:23:16.381625  7219 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:16.386032  7278 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.387667  7231 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.387787  7219 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.389624  7278 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.389817  7231 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.390516  7219 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.392244  7237 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.382371  7266 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:16.382606 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.381268 28333 raft_consensus.cc:2889] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.382233  7387 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.385763  7270 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:16.395680  7266 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.400956 27478 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 26, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 26 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:16.403406  7270 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.403895  7270 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:16.404930 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.407521  7387 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:16.396391 27586 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:16.418956 28329 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:23:16.421562 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61267.3us
I20260709 12:23:16.424474 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.425360 27586 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:16.426503 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.427248 27586 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:16.434571 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:16.435559 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58497.7us
I20260709 12:23:16.436012 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.457165 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5051) took 1316 ms. Trace:
I20260709 12:23:16.459729  7338 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:16.463897  7458 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
I20260709 12:23:16.457221  6709 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:16.471426 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.473201 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 68382.8us
I20260709 12:23:16.471200 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.472762 27583 rpcz_store.cc:276] 0709 12:23:15.140503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.556239 (+415736us) service_pool.cc:224] Handling call
0709 12:23:15.556787 (+   548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.086244 (+529457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.456844 (+370600us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54000898d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:16.457056 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1186739072}
I20260709 12:23:16.384245  7131 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.478452 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.483563 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:16.483860  7404 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 37 current term: 37 is higher.
I20260709 12:23:16.490868 27584 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
W20260709 12:23:16.493363 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 71010.9us
I20260709 12:23:16.493820 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.494180 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.495407 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:16.477996  7015 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:16.500133  7015 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.500582  7015 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.503926  7444 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:16.505300 27733 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:16.477701  7109 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:16.508595 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.477039  7278 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:16.478017  7225 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:23:16.520505 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.512540 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.522694 27587 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:16.527429 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:16.531888  7134 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:16.532655 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.535938 28329 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:16.402733 27478 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 25 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.524005 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.540971  7272 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:16.540285 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.517388  7393 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:16.506798  7383 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.029s
I20260709 12:23:16.543224 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:16.546350 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.552508 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44006.6us
I20260709 12:23:16.534334  7278 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.554880  6945 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:16.535199  7134 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.557034 27583 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 36 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:23:16.556896 27478 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.558113  7287 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:16.537009  7444 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:16.517756  7109 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.563791 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:16.570269 27582 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:16.571800 27584 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:23:16.547991  7219 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:16.549360  7380 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.577349  7219 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.564736  7220 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:16.587819 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.581234 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:16.590782 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 116572us
I20260709 12:23:16.588274 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.565228  6945 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:16.593497 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:16.593638 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.592973 27628 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:16.594628  7225 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:23:16.596714 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 43494.5us
I20260709 12:23:16.596956  7225 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.597361  7225 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:16.597553 27552 leader_election.cc:400] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34.
I20260709 12:23:16.597159  7278 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.598466 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.599598 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:16.600044  7444 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:16.600749  7383 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.600710  7444 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.601439  7383 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.603248  7109 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.607553  7408 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:16.614688  7386 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:16.575312  7393 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.564227  7381 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.053s
I20260709 12:23:16.617776  7393 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.579522  7398 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:16.618650  7398 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.619501  7295 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:16.619652  7398 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.622071 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.622326  7270 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.623560  6945 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:16.623538 27581 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:16.624257 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.625404 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4932) took 1530 ms. Trace:
I20260709 12:23:16.625785 27581 rpcz_store.cc:276] 0709 12:23:15.094599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.142435 (+ 47836us) service_pool.cc:224] Handling call
0709 12:23:16.625163 (+1482728us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:16.625324 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3234270592}
I20260709 12:23:16.622738 27583 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:16.623499  7220 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.586922  7219 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:16.637547 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:16.641367 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48460.7us
I20260709 12:23:16.638381 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.627185 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.640239 27582 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:16.638406  7272 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.645617  7272 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.646842 27587 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:16.655583 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.603109  7444 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.643571 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:16.657589 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.658432 27581 raft_consensus.cc:2410] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 42 because replica has last-logged OpId of term: 40 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 21.
I20260709 12:23:16.658667 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:16.659519 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.566694  7387 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:16.662746 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.663403  7387 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.594545  7131 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.664745  7393 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:16.668555 28330 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:23:16.670892 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:16.672257 28334 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:16.580523  7287 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:16.677497 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 127377us
I20260709 12:23:16.686728 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:16.686208 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:16.690074 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:16.697589 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:16.698061 27586 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:16.646750  7237 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:16.651015  7360 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:16.709682 27582 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 21. (term mismatch)
I20260709 12:23:16.710186 27582 pending_rounds.cc:85] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 20
I20260709 12:23:16.710445 27582 pending_rounds.cc:107] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 32.21
I20260709 12:23:16.710780 27582 raft_consensus.cc:2889] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.700009 28331 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.702229 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.702183  7380 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
W20260709 12:23:16.718647 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45270.3us
I20260709 12:23:16.715029 27584 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:23:16.725142 27583 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 19. Preceding OpId from leader: term: 41 index: 20. (index mismatch)
I20260709 12:23:16.680668  7287 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.693653  7431 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.034s
I20260709 12:23:16.701002  7444 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:16.729396 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.735728  6951 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:16.735993 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:16.737243  7431 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:16.709828  7383 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.709461  7360 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.686352  7225 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.744840 28333 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:23:16.751014 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.758616 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.758659 27583 raft_consensus.cc:1240] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: [30.20-30.20]
I20260709 12:23:16.744581  6951 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.718132  7313 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.060s
I20260709 12:23:16.749198  7360 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:16.745249  7431 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.772431 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.776307 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.777097 27584 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:23:16.731094  7070 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.732733  7266 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.773267 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:16.737520  7338 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.027s
I20260709 12:23:16.783980 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.788240  7443 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:16.796451 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:16.769320  7237 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:16.774001  6951 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:16.806543 27580 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:16.770169  7134 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.840637 27588 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 30: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:16.813987 27589 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:16.842311 27583 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:16.679384 27478 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 30 to 31, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 31 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:16.845933 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:16.847361 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74160.8us
I20260709 12:23:16.848310 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:16.852825 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55829.2us
W20260709 12:23:16.852092 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 111360us
W20260709 12:23:16.857892 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.859097 28292 leader_election.cc:400] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36.
I20260709 12:23:16.859504 28292 leader_election.cc:403] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:23:16.857046  7287 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:16.861169  7272 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:16.861346  7287 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.861732  7272 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.752388  7220 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.863219  7312 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:16.863543 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:16.864261 28336 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:16.868615 28328 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:16.863711  7312 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.869653  7312 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.870994  7313 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.870935 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.871703  7474 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.871577  7313 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.872251  7474 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.873366 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.873766  7313 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.874601  7474 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.875093  7444 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.875700  7444 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.876452  7225 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.846397  7285 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:16.877784  7444 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.878561  7070 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:16.801293  7231 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.057s
I20260709 12:23:16.880136  7231 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:16.880576  7231 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.881546  7231 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.862818  7272 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.883627 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.884670  7312 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.889691 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.889019 27478 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 29 to 33. New cstate: current_term: 33 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:16.890714  7312 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.876888  7445 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.891315 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.891678  7445 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.892390 27658 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:16.893167  7466 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:16.893911  7466 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.843685  7237 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:16.894779  7466 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.896396  7237 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.861368 28337 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I20260709 12:23:16.872468 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.888444 27589 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:16.899832 27733 raft_consensus.cc:2368] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
W20260709 12:23:16.886641 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 236130us
I20260709 12:23:16.861799  7287 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:16.907075 27552 leader_election.cc:400] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:16.908635 27552 leader_election.cc:403] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:16.909624 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:16.907579 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48692.7us
I20260709 12:23:16.911964 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.922765 28331 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:16.910411 27738 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:16.860235  7443 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:16.854882  7270 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:16.965422 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:16.930572 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.969506 27730 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:16.928830 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:16.933825  7465 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:23:16.915673  7431 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.928834  7470 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.036s
I20260709 12:23:16.973748  7470 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:16.974150  7470 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:16.962406  7070 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 42
I20260709 12:23:16.975031  7470 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:16.976455  7470 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:16.976514  7404 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:23:16.933859  6823 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:16.980773 27478 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 22 to 26, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 26 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:16.935307 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:16.936664  7445 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:16.936842 28330 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:23:16.937784 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:16.983788 27729 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:16.984320 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46631.7us
I20260709 12:23:16.984866  7443 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:16.985466 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:16.985839  7443 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.986699 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:16.987026  7225 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:16.988219  7137 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:16.989420  7137 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:16.990080  7137 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:23:16.990546  7137 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:16.939658 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.991863  7225 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:16.993237 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.993309  7225 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:16.994385 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.996060 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.940404 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.997730 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.998404 27659 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:23:16.947182 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.951037  7469 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.214s	user 0.003s	sys 0.000s
I20260709 12:23:16.953504 27662 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:23:17.001246 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.002475 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:16.961639 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.963585 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:17.003830  7137 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.004474 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.006186  7266 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.007485 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.992117 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:17.012388 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83668.9us
I20260709 12:23:16.916245  7392 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.931383  7408 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36.
I20260709 12:23:16.991796  7431 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:16.990767  6945 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.025215 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:17.032644 28331 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:17.026078 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:16.939054  7015 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:17.029748 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.063992 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:17.065326 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.066146 27662 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:17.022126  7444 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:17.024444  7313 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:17.034004  7285 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.096895 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.033802  7404 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:17.079150 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 314570us
I20260709 12:23:17.111832 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.112890 27662 raft_consensus.cc:2368] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:17.116389 27661 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:17.076313  7444 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.122054 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.128360 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:17.028703  7392 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.136704 27661 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:23:17.144112 27659 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:23:17.078974  7338 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.046s
I20260709 12:23:17.145967 27660 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: [40.21-40.21]
I20260709 12:23:17.147246 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:17.084096  7395 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:23:17.125137  7444 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:17.154670 27663 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:17.157437  7272 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:17.157600 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:17.157963  7272 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.158459  7272 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:17.160650  7381 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:23:17.161036  7467 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 12:23:17.161209  7381 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:17.161655  7381 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:17.169346  7272 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.178172  7272 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:17.178755  7381 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.179762  7116 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.189568  7445 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.189850  7116 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.190240  7445 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.193339  7381 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.194814  7445 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.208974 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.211810  6945 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.214373  7392 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.132496  7404 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.225121  7399 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:17.241504  7475 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:23:17.030985 27478 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 18 to 20, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 20 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:17.257900  7231 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:17.270277 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:17.121310  7270 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.026s
I20260709 12:23:17.121145  7285 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.283270 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:17.182514  6951 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.298293 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.299990 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:17.303603 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:17.261289  7134 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:17.309442 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:17.275916 28331 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
W20260709 12:23:17.285148 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 125804us
I20260709 12:23:17.305392 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:17.315924 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:17.324494 28337 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:17.325875 27660 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 30: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:17.328459 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.329036 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.329672 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:17.336670 27658 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:17.339844 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.300966  6945 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:23:17.356338 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.364888 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 506862us
I20260709 12:23:17.366930 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.371120 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:17.323515  7395 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:17.371930 27733 raft_consensus.cc:606] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:17.372727 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:17.314757  7134 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:17.375766 27733 raft_consensus.cc:614] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:17.370744 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 318667us
I20260709 12:23:17.377976 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.355576  7399 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:17.384181 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.383308 27702 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:17.420709 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.422113 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.422941  7313 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:17.423261  7285 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:17.424365  7482 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:17.424717  6951 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.368566  7122 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:17.427187 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:17.424407  7015 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.375872  7070 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 41. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:17.430061 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.375404  6945 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:17.417691  7474 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:17.450074  7480 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:17.433043  7207 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:17.441687  6823 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:17.452629  7134 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.454319  7134 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:17.259292 27478 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 33. New cstate: current_term: 33 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:17.435154  7387 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:17.455678  7387 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.434377  7312 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:17.467964  7404 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.468607  7386 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:23:17.470306  7287 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.540s	user 0.005s	sys 0.006s
I20260709 12:23:17.471437  7480 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:17.482980 27585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.025s	sys 0.040s
W20260709 12:23:17.506178 27585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.025s	sys 0.040s
I20260709 12:23:17.486097  7491 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:17.486373  7015 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.009s
I20260709 12:23:17.466974  7444 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.502435  7433 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:23:17.479784 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49793.1us
I20260709 12:23:17.527621 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5055) took 1160 ms. Trace:
W20260709 12:23:17.504601  7295 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.007s	sys 0.008s
I20260709 12:23:17.538534 27585 rpcz_store.cc:276] 0709 12:23:16.367007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.689431 (+322424us) service_pool.cc:224] Handling call
0709 12:23:16.699209 (+  9778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.750705 (+ 51496us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:23:16.938403 (+187698us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.527067 (+588664us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:17.527157 (+    90us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:17.527239 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.527356 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.527539 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":245295,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":42,"raft.queue_time_us":20878,"thread_start_us":709666,"threads_started":2}
I20260709 12:23:17.505277  7207 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:17.521978  7137 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.005s	sys 0.015s
W20260709 12:23:17.505966  7466 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.000s	sys 0.018s
I20260709 12:23:17.505110  7386 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:17.542248  7470 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.565s	user 0.005s	sys 0.008s
W20260709 12:23:17.549507  7295 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.007s	sys 0.008s
I20260709 12:23:17.522121  7266 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.559428 27580 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:17.541765  7433 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:17.506413  7287 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:17.556074  7137 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.009s	sys 0.024s
I20260709 12:23:17.560071  7386 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:17.555716  7392 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:17.555662  7466 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.000s	sys 0.019s
I20260709 12:23:17.570412  7266 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.565589  7491 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.587888 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4710) took 1487 ms. Trace:
I20260709 12:23:17.588371 27736 rpcz_store.cc:276] 0709 12:23:16.094409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.330730 (+236321us) service_pool.cc:224] Handling call
0709 12:23:16.331041 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.605667 (+274626us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.605702 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:16.605747 (+    45us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:16.605790 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.581096 (+975306us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54001d47d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:17.581151 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.581304 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":631020288}
I20260709 12:23:17.559892  7207 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.606993  7377 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.607756  7377 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.610131  7116 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:17.610352  7219 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.921s	user 0.009s	sys 0.000s
I20260709 12:23:17.596663  7480 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.583165  7392 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.617692 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:17.618906  7392 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:17.620326  7287 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.620690 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:17.589856  7491 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.621610 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5076) took 1011 ms. Trace:
I20260709 12:23:17.622102 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:17.622938  7491 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:17.623524  7131 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:17.613811  7219 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.628489 27735 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:17.591598  7285 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:17.631487  7433 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.604218  7404 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.621984 27580 rpcz_store.cc:276] 0709 12:23:16.610400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.814984 (+204584us) service_pool.cc:224] Handling call
0709 12:23:16.816116 (+  1132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.567423 (+751307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.621341 (+ 53918us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:17.621541 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1708742912}
W20260709 12:23:17.636894 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 201284us
I20260709 12:23:17.572945  6945 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.651837 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:17.679631 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.668977 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.681151 27585 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:17.685779 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:17.687352 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:17.689278 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:17.690369 27733 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 23. (term mismatch)
I20260709 12:23:17.690744 27733 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 22
I20260709 12:23:17.690954 27733 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 36.23
I20260709 12:23:17.691188 27733 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:17.691450 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:17.692867 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:17.610146  7377 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.700575 27584 raft_consensus.cc:2393] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:17.701149 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:17.693998 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.704319 27553 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:17.707425 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:17.717671 28334 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
W20260709 12:23:17.718041 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 148649us
I20260709 12:23:17.742987 27730 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 30. Current term is 31. Ops: [30.19-30.19]
I20260709 12:23:17.685859  7237 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:17.708159  7360 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.909s	user 0.009s	sys 0.002s
I20260709 12:23:17.720194 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:17.689589  7392 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:17.694597  7393 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:17.700276  7433 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:17.721362 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:17.698961  7491 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:17.723173 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:17.716343 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 338421us
I20260709 12:23:17.788995 27552 leader_election.cc:400] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:17.789978 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.725435 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:17.790767 27733 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:23:17.699602  7207 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.725847 28331 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:23:17.712172  6951 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:17.712042  7225 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:17.699096  7070 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:17.734316 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:17.714712  7313 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:17.732772 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:17.735217 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:17.801329 27580 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:17.735705 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:17.734319 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:17.651585  7404 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.715612  7443 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.027s
W20260709 12:23:17.778084 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51687.1us
I20260709 12:23:17.748561 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:17.778478 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:17.803843 27729 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:17.732757  7381 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.722528  7505 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:23:17.817229  7505 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:17.817438  7506 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:17.818593  7505 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.793673 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:17.742308  7220 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.000s	sys 0.017s
I20260709 12:23:17.794942 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:17.821254  7220 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.000s	sys 0.017s
I20260709 12:23:17.821484 27731 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:17.822086  7506 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:17.739266  7110 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.777567  7360 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.823513  7399 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:17.822944  7431 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.002s
I20260709 12:23:17.824854  7360 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:17.826215  7404 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.827473 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47076 (request call id 1910) took 1246 ms. Trace:
I20260709 12:23:17.828027 27737 rpcz_store.cc:276] 0709 12:23:16.580698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.589343 (+  8645us) service_pool.cc:224] Handling call
0709 12:23:17.827235 (+1237892us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:17.827390 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":605418368}
I20260709 12:23:17.828514  7266 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.808316  7379 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:17.810221  7504 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:23:17.811026  7493 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
I20260709 12:23:17.816118  7122 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:17.816056  7381 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.784682 27581 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:17.784785 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.853821 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:17.854781 27734 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:17.827397 28337 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:23:17.808838 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:17.829633 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 118440us
I20260709 12:23:17.745373  7444 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:17.847500 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.857643 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.848568 27729 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:17.848837 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:17.859946 27580 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:17.794786  7395 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:17.789067  7287 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:23:17.850148  7122 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:17.788445  7237 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:17.865104 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:17.867970 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:17.905889 27729 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:23:17.849740 27584 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:17.797019  7116 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.056s
I20260709 12:23:17.854014  7110 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.934262  7110 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.935647  7393 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.877427  7399 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:17.922441  7468 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:17.923571  7502 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
W20260709 12:23:17.924674  7231 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.666s	user 0.013s	sys 0.001s
W20260709 12:23:17.925047  7503 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:23:17.925158  7312 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:17.926157 28330 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:17.928196  7520 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:17.929883  7383 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:17.849715  7398 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:17.947362 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:17.939570  7443 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:23:18.072777 27581 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:23:17.953496 28333 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
W20260709 12:23:17.958035 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93051.7us
I20260709 12:23:17.936302  7393 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.971062 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:17.862367  7431 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:18.057541  7320 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.064028 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 126235us
I20260709 12:23:17.829594  7070 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:17.939272  7504 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:23:18.094489 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.095192 27581 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:17.861369  7466 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.200s
W20260709 12:23:18.066677 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 281980us
I20260709 12:23:18.069077  7383 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:17.950891  7387 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:17.864830  7404 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.201s
I20260709 12:23:17.831858  7392 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 16, Time since last communication: 0.080s
I20260709 12:23:17.850963  7379 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.065997  7520 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:17.938571  7295 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.149s
I20260709 12:23:18.100236 27627 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:18.068014  7287 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:23:18.091884  7237 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.112322 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5080) took 1473 ms. Trace:
I20260709 12:23:18.112630  6951 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:23:18.130046 27589 rpcz_store.cc:276] 0709 12:23:16.639410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.900148 (+260738us) service_pool.cc:224] Handling call
0709 12:23:16.900542 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.900982 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:23:17.395467 (+494485us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.584772 (+189305us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.604820 (+ 20048us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.604980 (+   160us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:17.605013 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.605091 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.112065 (+506974us) raft_consensus.cc:1685] finished
0709 12:23:18.112087 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.112238 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58717,"prepare.queue_time_us":105745,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":48,"raft.queue_time_us":204,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":17733,"thread_start_us":550439,"threads_started":2,"wal-append.queue_time_us":270717}
I20260709 12:23:18.132321 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.132890 27589 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:18.113354  7266 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.022s
I20260709 12:23:18.112725  7122 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.138751  7122 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:18.074813  7231 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.139679  7231 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.087085  6709 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:18.120658 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.056792  7468 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.108520  7519 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.091821  7110 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:18.121553  7312 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:23:18.098073  7320 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.121420  7131 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:18.161381  7312 consensus_queue.cc:1243] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 39.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:23:18.161967  7312 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:18.162436  7312 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:18.157208  7491 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.006s
W20260709 12:23:18.143347 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.164426  7278 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:18.165025  7468 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:18.163604  7312 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.166841  7312 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:18.166400  7520 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.167793  7520 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:18.168653  7320 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.168884  7070 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.169709  7393 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.170265  7131 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.164999  7278 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.152392  7519 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.174460  7278 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:18.175899  7514 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:23:18.152590  7287 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:18.177352  7287 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.177716  7287 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:18.149712 27581 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:18.087756  7398 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:18.180809  7134 consensus_meta.cc:287] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.726s	user 0.014s	sys 0.000s
I20260709 12:23:18.181567  7379 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.182689  7398 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:18.162067  7386 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.581s	user 0.006s	sys 0.003s
I20260709 12:23:18.184237  7386 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.174804  7131 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:18.185636  7510 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:18.186038  7510 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:18.187152  7386 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.158953  7110 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.174865  7519 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:18.200127 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.165798  7468 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:18.128208 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 248539us
I20260709 12:23:18.218340 27734 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:18.157737  7399 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:18.233415 27589 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:18.191145  7516 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.189473  7383 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.235428  7231 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.206496 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.249912 27581 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:18.245914 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:18.275961 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.276969 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:18.322053 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 171820us
I20260709 12:23:18.234517  7468 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.288426  7231 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.327721 27663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4814) took 1386 ms (client timeout 1704 ms). Trace:
I20260709 12:23:18.333395 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:18.330713  7527 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.005s
W20260709 12:23:18.334429 27663 rpcz_store.cc:269] 0709 12:23:16.937446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.995786 (+ 58340us) service_pool.cc:224] Handling call
0709 12:23:18.323546 (+1327760us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540022a954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:18.323668 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363122048}
I20260709 12:23:18.186846  7510 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.342448 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:18.343135 27737 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:18.323560  7399 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:18.350225  7510 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:18.353299  7433 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:23:18.354547  7110 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:23:18.354220  7433 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:18.363104  7468 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:18.365864 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:18.366489 27729 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:18.257350  7516 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.368378 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.365424  7399 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:18.344013  7231 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:18.394461 27630 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.1:41657 timed out after 1.617s (SENT) [suppressed 2 similar messages]
I20260709 12:23:18.417227  7399 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.416950  7433 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.418422  7231 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.389746 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.383312  7398 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.441632 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.447031 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.443728 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.446393 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.452839 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:18.459144 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 69517.8us
I20260709 12:23:18.450628 27583 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:18.468178 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.471478 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:18.419309  7433 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:18.515851 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.468343  7528 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:18.540863 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.553527 27583 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:18.541085 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:18.541366 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:18.541896 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:18.558717 27738 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:18.558843 27731 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:18.541136 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:18.542651 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:18.560494 27735 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:18.563351 27737 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:18.542117 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.455909  7444 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:18.562383 27584 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:18.464164  7231 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:18.432214  7070 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.456161  7527 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:18.564929 27586 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:23:18.569109 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 174681us
I20260709 12:23:18.571888 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.438879  7287 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.579725 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:18.542801  7408 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:18.591423 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.581637 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.588326 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.588863 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:18.561494  7392 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:18.590685 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:18.591288 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.586709 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:18.647241 27656 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:18.568311  7444 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.632983 27655 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
W20260709 12:23:18.638904  7531 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:23:18.567574  7520 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:18.575910  7320 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:18.575724  7231 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:18.669551  7122 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.001s
I20260709 12:23:18.651002  7408 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.672705  7408 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:18.588945  7070 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.654250  7444 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:18.645022 27660 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:18.591100  7527 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.691298 27737 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:18.684098 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.667719  7231 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:18.700614  6945 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:18.701215  6945 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.565393  7528 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.698050 27704 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.702699 27589 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:18.710747  7528 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.712425  7528 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.684778 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 203257us
I20260709 12:23:18.708235  7516 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:18.731815 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.688563  7313 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:18.453827  7398 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:18.716729  7496 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.110s	user 0.002s	sys 0.014s
I20260709 12:23:18.729224 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 4966) took 1004 ms. Trace:
I20260709 12:23:18.664638  7520 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.737423 27587 rpcz_store.cc:276] 0709 12:23:17.725165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.324635 (+599470us) service_pool.cc:224] Handling call
0709 12:23:18.332187 (+  7552us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:18.345213 (+ 13026us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.345256 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:18.348354 (+  3098us) log.cc:844] Serialized 52 byte log entry
0709 12:23:18.560785 (+212431us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:18.585106 (+ 24321us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.585390 (+   284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.585491 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.729010 (+143519us) raft_consensus.cc:1685] finished
0709 12:23:18.729029 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.729169 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18135,"prepare.run_wall_time_us":385,"raft.queue_time_us":455,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":60472,"spinlock_wait_cycles":10820224,"thread_start_us":169100,"threads_started":1,"wal-append.queue_time_us":254557}
I20260709 12:23:18.718905 27589 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:18.694131  7320 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.741032 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:18.733220  6823 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.719121  7278 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:18.763300 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.772616 27587 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:18.729266  6945 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:18.732635  7535 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:18.726202  7495 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.122s	user 0.011s	sys 0.033s
I20260709 12:23:18.774106  6823 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:18.784678  7498 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.165s	user 0.012s	sys 0.018s
I20260709 12:23:18.786181  6823 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.712889  7528 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.774226  7526 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:18.789269  7528 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:18.792001  7533 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:23:18.794978 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.795885 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:18.803018 27581 consensus_meta.cc:287] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.004s
I20260709 12:23:18.803380 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:18.806802 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 4964) took 1082 ms. Trace:
I20260709 12:23:18.807229 27581 rpcz_store.cc:276] 0709 12:23:17.724638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.193334 (+468696us) service_pool.cc:224] Handling call
0709 12:23:18.806722 (+613388us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:18.735435  7313 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:18.811179 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.820685 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.821559 27581 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:18.829649  7398 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.829769  7381 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:18.830061 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.830506 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.830953  7137 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.831663 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:18.832023  7524 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
I20260709 12:23:18.832314 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:18.833177 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.833680 27587 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:18.834158 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.834499 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.834723 27580 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:18.835032 27587 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 27.
I20260709 12:23:18.836557 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:18.838461  7408 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:18.838538  7520 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.838917  7408 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.839350  7408 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:18.799816  7535 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.841854  7535 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.842556  7535 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:18.842878  7535 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:18.831696  7137 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:18.846590 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:18.847226 28328 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:18.848714 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.849305 27580 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:18.850037 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.850683 27580 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:18.852720  7527 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:18.852211 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.772204  7278 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:18.830377 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.799206  7506 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.859411 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89690.7us
I20260709 12:23:18.855907 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.861675 27580 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:18.862532 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.862808 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.814599  7540 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.867018 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71195.1us
I20260709 12:23:18.863231 27738 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:18.843228  7535 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:18.868579 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.861342 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:18.872166 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:18.874594 27581 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:18.873426 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.857744  7543 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.860313  7278 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:18.876798 27580 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:23:18.936991 27734 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:18.926481 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.928397 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:18.953450 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:18.951288 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.967314 27738 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:18.967374 28295 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:18.941231 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:18.923237 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:18.971309 27581 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:18.972812 28336 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:18.901046 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:18.877709 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.877224 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:18.923048 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.982596 27662 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:18.980317 27587 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:18.942210 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.903468 27627 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:18.932652 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.984659 27663 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:18.988337 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:18.929944 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.013767 27655 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:18.923758  7494 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 1.303s	user 0.004s	sys 0.018s
W20260709 12:23:18.922664  7312 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.755s	user 0.013s	sys 0.001s
I20260709 12:23:19.018196 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:18.873080  7381 raft_consensus.cc:2764] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election decision vote started in defunct term 43: won
I20260709 12:23:19.018190 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.023018 27662 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:18.895725  7380 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.871039  7392 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:18.877518  7137 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.876438  7393 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:19.011611  7270 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.049213 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.012274  6823 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:18.864019  7506 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.020330  7070 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:18.874095  7540 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.019019  7545 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:19.069962 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.021816  6945 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.059489 27581 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:18.992082  7543 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.083546  7543 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.057906  6823 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.094267 27586 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:19.096078 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.096441 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:19.097225 28336 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:19.075838  7480 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.459s	user 0.023s	sys 0.014s
I20260709 12:23:19.054693  7380 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.100786 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:19.106022 27655 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:19.101672 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 200753us
I20260709 12:23:19.075059  7393 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:19.104981 27581 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 24. (term mismatch)
I20260709 12:23:19.107287 27581 pending_rounds.cc:85] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 23
I20260709 12:23:19.107611 27581 pending_rounds.cc:107] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 41.24
I20260709 12:23:19.107478 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:19.108394 27581 raft_consensus.cc:2889] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:19.109495 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206088us
I20260709 12:23:19.110231 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.059346  7231 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.061241  7270 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.092089  6823 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:23:19.102816  7480 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s	user 0.023s	sys 0.014s
I20260709 12:23:19.112643 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:19.112600 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54086.7us
I20260709 12:23:19.085618  7549 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:23:19.109191  7497 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.492s	user 0.015s	sys 0.008s
I20260709 12:23:19.128100 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.132754 27581 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:23:19.102367  7554 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:19.110827  7393 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:19.143141 27584 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:19.116288  7545 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:19.143939 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.193327 27586 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
W20260709 12:23:19.149605  7550 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:23:19.193396 27662 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:19.128880  7313 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:19.176993  7480 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:19.180574 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.180778 27581 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:19.186786  7070 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:19.190270  7548 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:23:19.191623 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 80970.2us
I20260709 12:23:19.209076 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.198331 28329 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:19.176054 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:19.202411 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95088us
I20260709 12:23:19.206715 28336 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:19.145993  7393 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:19.126861  7516 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.223264 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.234460 27663 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:19.234373 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4806) took 2309 ms. Trace:
I20260709 12:23:19.199220  7312 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:23:19.139137  7408 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.110852  7392 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.204609  7543 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:19.222980  7116 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.208209  7399 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.245505 27630 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:19.207748  7520 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.237015 27657 rpcz_store.cc:276] 0709 12:23:16.925045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.925164 (+   119us) service_pool.cc:224] Handling call
0709 12:23:17.249486 (+324322us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b5800091b98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223eee 000000000036eb39 00007f0e9f2971e0 00007f0e9f3600c4 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:17.249693 (+   207us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.250213 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.250253 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.250840 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.544091 (+293251us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:17.544134 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.544204 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.958294 (+1414090us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:19.234135 (+275841us) raft_consensus.cc:1685] finished
0709 12:23:19.234160 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.234303 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9474,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"spinlock_wait_cycles":146314752,"thread_start_us":292805,"threads_started":1,"wal-append.queue_time_us":366178}
I20260709 12:23:19.250736 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:19.252100 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 263841us
W20260709 12:23:19.254443 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.235651  7539 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.258033 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.244053  7398 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:19.247762  7543 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:19.212543  7070 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.263190 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:19.263789 27661 raft_consensus.cc:2393] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:19.255733  7313 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 38.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.016s
I20260709 12:23:19.272560  7408 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:19.310104 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47036.2us
I20260709 12:23:19.273346 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:19.282132  7433 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.730s	user 0.009s	sys 0.004s
I20260709 12:23:19.283524  7270 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.284394  7540 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.237164  7516 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.254943  7116 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.288663  7398 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:19.292845  7555 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:23:19.298564  7558 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:19.208989  7480 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:19.304498 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95524.5us
I20260709 12:23:19.312067 27630 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:19.295635 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0700688510000873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
I20260709 12:23:19.342216  7231 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.334257 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:19.358839  7529 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.716s	user 0.002s	sys 0.000s
I20260709 12:23:19.336423 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.352522 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.298714  7543 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:19.332446  7565 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:19.255564  7399 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.344343 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:19.344262 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.259939  7520 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.346355 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.338145  7433 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.348335 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.325585  7015 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:19.261482  7560 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.349717  7506 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.351570 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.366149  7558 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:19.316551  7545 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.343053  7398 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.380426 27661 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:19.352051 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:19.352211 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:19.354380 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.294661  7278 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.363386 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:19.363636 27734 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:19.262419  7539 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.366250 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.337078 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:19.365275  7499 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.747s	user 0.005s	sys 0.012s
W20260709 12:23:19.371739  7444 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.675s	user 0.010s	sys 0.003s
I20260709 12:23:19.379716 27581 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:23:19.378156  7070 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:19.378245  7270 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:23:19.386948 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.459936 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.397842 27655 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:19.397428 27737 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:19.404076 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52101.8us
I20260709 12:23:19.401880 27586 raft_consensus.cc:2376] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42. Re-sending same reply.
I20260709 12:23:19.474972 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:19.475764 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:19.384510  7015 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:19.386583  7535 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.503s	user 0.007s	sys 0.003s
I20260709 12:23:19.378417  7501 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.384868  7560 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.419308  7116 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:19.420861  7278 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.386392  7520 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:19.386633  6945 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.516317 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.387704  7380 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.388634  7558 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.386439  7516 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:19.423861 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57700.5us
I20260709 12:23:19.424458  7571 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:19.425333 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:19.441448  7539 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.361059  7537 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:19.443081 27588 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:23:19.444396  7399 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.379607  7565 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.446502  7480 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.379902  7568 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:19.383692  7433 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.414011  7506 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:19.530066 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.452145 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.452723 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.452994  7225 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:19.393416  7398 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.383284  7554 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:19.454849 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.455108 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.455961 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:19.468055 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 115327us
I20260709 12:23:19.469460 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2177) took 1831 ms. Trace:
I20260709 12:23:19.473333 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.482816 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.485031 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.512746 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.514529 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.484437  7474 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:19.520556 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95324.7us
I20260709 12:23:19.544719  7474 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.545135  7474 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:19.524541 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.514062  7501 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.527954  7565 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:19.532824  7568 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.553184  6823 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.555685 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:19.556294 27661 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:19.557221  7015 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:19.558466 28337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1767) took 1876 ms (client timeout 1855 ms). Trace:
I20260709 12:23:19.562109 28334 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38.
I20260709 12:23:19.563130 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.535292  7225 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:19.565124  7225 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.566483 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.566962 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.568194 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.569363 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.573828  7433 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.524789 27581 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:19.540943 27582 rpcz_store.cc:276] 0709 12:23:17.638100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.662502 (+ 24402us) service_pool.cc:224] Handling call
0709 12:23:17.705523 (+ 43021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.119701 (+414178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.119750 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:18.119810 (+    60us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:18.119859 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.469134 (+1349275us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54000bdbd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:19.469204 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.469384 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":951949056}
I20260709 12:23:19.531621 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:19.533366 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.478509  7444 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.491139  7535 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.583034  7070 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.583642  7116 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:19.584234  7116 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:19.584560 27659 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:19.585217  7116 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.586540  7444 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.576254  7219 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.526913  7380 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:19.540566 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:19.541361 27630 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:19.541359  7570 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:23:19.542523 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 57579.4us
I20260709 12:23:19.596593 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:19.543180  7398 raft_consensus.cc:2764] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
I20260709 12:23:19.478895  7564 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:23:19.558979 28337 rpcz_store.cc:269] 0709 12:23:17.682226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.682371 (+   145us) service_pool.cc:224] Handling call
0709 12:23:19.558223 (+1875852us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:19.558355 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":265079680}
W20260709 12:23:19.580235 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50229us
I20260709 12:23:19.577087 27658 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
W20260709 12:23:19.591164 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49890.2us
I20260709 12:23:19.616532 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.601158  7568 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.601336 27656 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: [40.21-40.21]
I20260709 12:23:19.601788  7535 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:19.595804  7380 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:19.602880  7287 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.602901  6823 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:19.652176 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.603606 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.605257 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:19.653259 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90232us
I20260709 12:23:19.602087  7501 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.605957 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:19.617741 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:19.651247 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 104032us
I20260709 12:23:19.605418 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.660451 27735 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:19.597975  7338 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.654533 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:19.665633 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 187655us
I20260709 12:23:19.612517  7506 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.666064 27588 raft_consensus.cc:2376] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36. Re-sending same reply.
I20260709 12:23:19.655411 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.656507 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.656509 27738 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:19.657780 27658 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 21. (term mismatch)
I20260709 12:23:19.654040 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.655840 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.598374  7554 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:19.596012  6951 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:19.672313 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:19.667995 27658 pending_rounds.cc:85] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 20
I20260709 12:23:19.673115 27658 pending_rounds.cc:107] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 28.21
I20260709 12:23:19.673329 27658 raft_consensus.cc:2889] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:19.674144 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.652202 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.668398 27729 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:19.665467 27734 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:19.640336  7572 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:19.671020 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.671353 27656 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:23:19.675143 27582 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:19.645857  7380 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:19.675004 27704 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.674502 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:19.680622 27656 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I20260709 12:23:19.683878 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:19.684517 27732 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:19.688271  7560 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:19.618882  7399 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.060s
I20260709 12:23:19.583657  7070 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.653043  7539 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:19.658756  7540 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.659008  7520 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.713676 27661 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:19.701876 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.712715 27738 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:19.734642 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.739149 27582 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:19.666955  7570 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:19.745659 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:19.676924  7270 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:19.699106  7571 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:19.756896 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:19.757539 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82612.2us
I20260709 12:23:19.677590  7338 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.700089  7581 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:19.773260 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.775930 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:19.740849  7554 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:19.747246  7583 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.694208  6951 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.676519  7484 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.100s
I20260709 12:23:19.764869  7015 raft_consensus.cc:2764] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:23:19.764156  7270 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.749428  7540 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:19.808634 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 106870us
I20260709 12:23:19.812754 27661 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:23:19.692728  7506 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.765322  7480 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:19.744302  7539 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.747521  7393 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:19.748908  7520 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:19.834959 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.895982 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.789892  6945 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:23:19.776842  7278 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:19.901698 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.896250 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:19.777668  7571 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.908558 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:19.838455  7572 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:19.899917  7270 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:19.944279 27586 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:19.947105 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:19.944496 27588 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:19.899189  6823 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:19.946436 27580 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:23:19.900038  7480 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.899071  7399 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.902415  6945 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:19.904249  7560 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.953358  7568 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:19.960729 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.908243  7571 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:19.815584  7583 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.813232  7470 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.141s
I20260709 12:23:19.860708  7587 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.110s
I20260709 12:23:19.946750  7592 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:19.989336 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:19.954041  7398 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.953272  7474 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.955626  7572 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.978694  7568 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:19.980422  7547 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:19.979192 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58252 (request call id 1776) took 1137 ms. Trace:
I20260709 12:23:19.902052  7554 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.969995  7480 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:19.951742  7484 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.027s
I20260709 12:23:19.970261  7445 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.975548  6945 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:19.996932 28329 rpcz_store.cc:276] 0709 12:23:18.841820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.843745 (+  1925us) service_pool.cc:224] Handling call
0709 12:23:18.941089 (+ 97344us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b58000d4a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f30e0a0 00007f0e9f30d984 00007f0e988886d4 00007f0e98888662
0709 12:23:19.978939 (+1037850us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400590954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:19.979108 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364559616}
I20260709 12:23:19.972061  7589 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:19.899211  7570 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.002557  7070 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.008073 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.016438 27582 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:20.019845 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.020589 27582 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:19.969789  6823 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.013206  7592 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:19.975716  7399 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:19.996701  7547 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:20.001124  7338 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.029847 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.036448 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.014670  7589 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:20.017073  7586 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:20.044595 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.045526 27738 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:20.046016  7070 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:20.046461  7070 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.046835  7070 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.047195  7219 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:20.048859  7219 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.049388  7270 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.051204  7219 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.051591 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5092) took 1608 ms. Trace:
I20260709 12:23:20.052032 27584 rpcz_store.cc:276] 0709 12:23:18.443312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.629843 (+186531us) service_pool.cc:224] Handling call
0709 12:23:18.651239 (+ 21396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.145542 (+494303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.051346 (+905804us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:20.051530 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130285440}
I20260709 12:23:20.015506  7594 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:19.986720  7560 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.041746 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:20.070803 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 49087.8us
I20260709 12:23:20.069875 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.056475 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5096) took 1494 ms. Trace:
I20260709 12:23:19.998909  7398 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.034010  6951 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.010854  7565 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.120278 27583 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:20.120970 27587 rpcz_store.cc:276] 0709 12:23:18.561912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.985920 (+424008us) service_pool.cc:224] Handling call
0709 12:23:19.002353 (+ 16433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.055974 (+1053621us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b54004194a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:23:20.056151 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.056176 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.056216 (+    40us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:20.056252 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.056307 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.056415 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9415040}
I20260709 12:23:20.125905 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.018672  7445 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.069146  7572 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:20.069585  7588 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:23:20.055927  7583 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.067967  7520 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.058045  7506 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.116479  7546 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.117136  7270 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:20.070302  7589 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:20.118707  7393 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.142609 27659 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:20.142242 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:20.055085  7070 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.171943 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:20.156955 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.160332 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.163856 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.172631 27660 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:20.168040 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.135253 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4026) took 1057 ms. Trace:
I20260709 12:23:20.073817  7287 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:20.173575 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:20.177486 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.178200 27658 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:20.178277 27587 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:20.173763 27655 rpcz_store.cc:276] 0709 12:23:19.078102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.597768 (+519666us) service_pool.cc:224] Handling call
0709 12:23:19.653486 (+ 55718us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.662970 (+  9484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.663004 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.669319 (+  6315us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.974645 (+305326us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:20.134552 (+159907us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.134849 (+   297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.134942 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.135009 (+    67us) raft_consensus.cc:1685] finished
0709 12:23:20.135028 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.135184 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2022,"prepare.queue_time_us":161326,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":631,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":17010,"thread_start_us":217563,"threads_started":1,"wal-append.queue_time_us":278906}
I20260709 12:23:20.180083 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:20.183965 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:20.186729 27657 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:20.189709 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:20.157166  7583 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:20.191538 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.192149 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.192588 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.147261  7484 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.047s
I20260709 12:23:20.194939  7474 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:20.195174  7560 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.139468  7547 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.196938  7565 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.138641  7470 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.040s
I20260709 12:23:20.143743  6945 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.162662  7539 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.207506  7501 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:20.209087  7398 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.210531 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:20.234967 28329 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:20.214358 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.214952 27587 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:20.219873 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.241285 27588 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:20.221297  7587 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:20.163427  7506 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:20.222613 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:20.221841 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 439425us
W20260709 12:23:20.221015 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 179332us
I20260709 12:23:20.250687 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:20.225585 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.127158  7568 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.219632  6709 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:20.264026 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:20.264967 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:20.265457 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:20.266567  7480 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.216439  7583 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.219475  7380 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.267921  7270 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:20.247789  7587 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.269361  7587 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:20.269266  7270 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:20.270663  7270 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:20.247447 27587 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 41 index: 21. (index mismatch)
I20260709 12:23:20.264843  7278 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:20.267532  7583 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:20.271796  7565 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.272447  7565 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.267112 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:20.276472 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.276744  7278 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:20.220145 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:20.280773  7592 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:20.225696 27582 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:23:20.283806 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:20.283886 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.284698  7594 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:20.284986  7070 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.285115  7594 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:20.285144 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.285799  7594 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.286347  7572 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:20.286872  7572 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:20.287523  7520 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:20.288108  7520 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:20.288607  7589 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.288944  7520 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.291213 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.293174  7539 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:20.293572  7539 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.293932  7539 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:23:20.294049  7543 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.881s	user 0.013s	sys 0.002s
I20260709 12:23:20.294531  7543 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.292160 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.292002  7572 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.296638  7543 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.285601  7070 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:20.167402  7546 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.298605 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.300137 28334 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:23:20.300377  7546 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.249244  7560 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:20.301309  7560 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.301662  7560 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:20.299649 27587 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 42 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:23:20.303097 27655 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.264849 28335 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:20.303988  7583 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.306219  7583 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.211158  7287 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.244657 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:20.283664  7399 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:20.308440 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63952.2us
I20260709 12:23:20.309166 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.321864  7287 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.328675 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:20.319504 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.329543  7522 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:20.329950 27738 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:20.328292 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:20.332093 27553 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38.
I20260709 12:23:20.332412 27553 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:20.333889  7594 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.275238  7565 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.309041 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.339380 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.341854 27655 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:20.342926 27730 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:20.344606  7320 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.345940 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.347174 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:20.222309  7571 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.349684 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.346537 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:20.352823 27587 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:20.348182 27738 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.350762 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.307690  7547 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:20.298539  6945 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:20.343921 27588 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:20.320693  7603 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:20.345830  7522 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38.
W20260709 12:23:20.360874 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 169548us
I20260709 12:23:20.350602  7501 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:20.321081  7506 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.370127 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.367434 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:20.404807  7601 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:20.405243  7601 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.405772  7601 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:20.413132 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45818.1us
I20260709 12:23:20.414842  7380 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.416574  6823 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.416322  7601 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.416955  7506 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.418607  7601 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.420457 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:20.368031  7583 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.423223  7381 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.431110  7589 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.432276  7539 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.403633  7522 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:20.433967 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:20.420451  7571 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:20.369168  7399 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.065s
I20260709 12:23:20.437341 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:20.437942 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:20.477756 28330 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:20.439980 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2189) took 1108 ms. Trace:
I20260709 12:23:20.442246 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4836) took 1087 ms. Trace:
I20260709 12:23:20.442359  7587 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.479699 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:20.480103 27659 rpcz_store.cc:276] 0709 12:23:19.354758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.742392 (+387634us) service_pool.cc:224] Handling call
0709 12:23:19.742710 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.171712 (+429002us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.441977 (+270265us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54001e50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:20.442188 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":919281024}
I20260709 12:23:20.481986  7587 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:20.443310 27583 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:20.482690 27659 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:20.451232 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.483749 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.484519 27583 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.478965 27585 rpcz_store.cc:276] 0709 12:23:19.331570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.331717 (+   147us) service_pool.cc:224] Handling call
0709 12:23:19.332003 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.335863 (+  3860us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.335907 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:19.335947 (+    40us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:23:19.335981 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.439753 (+1103772us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b54001892d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:20.439804 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.439923 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8427776}
W20260709 12:23:20.490608 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:20.466495 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46142.7us
I20260709 12:23:20.471225 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:20.473171 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:20.477137 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:20.497752 27583 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:20.498297 27663 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:20.448117 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.506542 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:20.507658  7587 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:20.508639  7587 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:20.509945  7587 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.515744 27583 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: [39.21-39.21]
W20260709 12:23:20.516168 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45032.3us
I20260709 12:23:20.516415  7134 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:20.516835  7134 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.517170  7134 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.517563 27583 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:20.519233  7287 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:20.520382 27583 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [34.22-36.23]
I20260709 12:23:20.520008  7287 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 39.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:20.522480  7543 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:20.522848  7543 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:20.523733  7543 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.524917  7543 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:20.527833  7134 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.530268  7134 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.533739  6951 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:20.535908  7134 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:20.537603 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:20.472506  7583 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.468132  7594 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:20.557396  7522 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.476302  7527 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:20.474323  7470 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:20.478842  7381 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.567690 27587 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
W20260709 12:23:20.569011 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90370.8us
W20260709 12:23:20.599736 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62363.8us
I20260709 12:23:20.638845 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:20.639133 28295 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:20.639810  7528 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:20.640395  7528 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:20.572348 28330 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:20.641238  7528 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.642216  6823 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:20.642783  6823 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:20.643632  6823 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.572446 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.647007 27732 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 28: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:20.648460 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.573413 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:20.651141 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.651924 27738 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 37 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:23:20.652961 27738 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.20-41.20]
I20260709 12:23:20.653908 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:20.576422 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:20.654743  7482 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:20.654927 28336 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:20.655268  7482 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:20.577247 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:20.579527 27730 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:20.656445  7482 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.581487 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.658548 27736 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:20.581797 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:20.607642  7116 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.009s	sys 0.044s
I20260709 12:23:20.451515  7501 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.658667 27738 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:20.662686 27732 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:20.599807  7134 raft_consensus.cc:2764] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election decision vote started in defunct term 40: won
I20260709 12:23:20.584864 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:20.667987 27733 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:20.586118 27729 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:20.586772 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.634855  7527 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.630064  7574 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:20.557025  7539 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.630681 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:20.633332 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:20.657042 27730 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:23:20.581161 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:20.664969 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87816.8us
I20260709 12:23:20.661252 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:20.664322  7116 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.009s	sys 0.045s
I20260709 12:23:20.679016 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.676476 27738 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:20.660153  7568 raft_consensus.cc:2764] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:23:20.682125 28328 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:20.583847 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:20.588449 27663 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:20.703716 27736 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:23:20.730540 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:20.669853  7571 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:20.748409  7443 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.747995 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 5107) took 1130 ms. Trace:
I20260709 12:23:20.762100 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4835) took 1423 ms. Trace:
I20260709 12:23:20.751801 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:20.720026 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.844409 27587 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:23:20.844728 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 93023.5us
I20260709 12:23:20.726414 27735 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:20.713272 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:20.769532 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.846035 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:20.847045  7380 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:20.847921  7380 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:20.848322  7380 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:20.848762  7380 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:20.713514 27731 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:20.748245 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:20.842676 27586 rpcz_store.cc:276] 0709 12:23:19.617405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.617515 (+   110us) service_pool.cc:224] Handling call
0709 12:23:20.747821 (+1130306us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400188dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:20.747939 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":586344320}
I20260709 12:23:20.851924 28336 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:20.743229  7540 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:20.790763 27733 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
W20260709 12:23:20.791370 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 77259.1us
I20260709 12:23:20.725108 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:20.679347  7527 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:20.728014 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:20.763137  7585 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.922s	user 0.002s	sys 0.001s
W20260709 12:23:20.760950  7590 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.982s	user 0.003s	sys 0.000s
I20260709 12:23:20.835457 27588 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:20.888197 27580 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:23:20.838276  7573 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.783993  7480 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:20.729996 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.842849 27661 rpcz_store.cc:276] 0709 12:23:19.339066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.750087 (+411021us) service_pool.cc:224] Handling call
0709 12:23:19.750445 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.895419 (+144974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.761852 (+866433us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b5400291154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:20.762041 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135450368}
I20260709 12:23:20.914247 27588 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 28.
W20260709 12:23:20.869920 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 208728us
I20260709 12:23:20.858003 27584 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:20.882488 28336 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
W20260709 12:23:20.883775 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 299964us
I20260709 12:23:20.667410  7546 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.119s
I20260709 12:23:20.888486 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:20.886572  7547 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.522s	user 0.003s	sys 0.006s
I20260709 12:23:20.854909  7443 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:20.887965  7399 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:20.923353 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:20.946372 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57955.1us
I20260709 12:23:20.888298  7520 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:20.704418  7574 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:20.963323 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.980026 27630 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:21.006255 27630 leader_election.cc:403] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:20.959671  7480 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:20.960211  7520 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.953784  7399 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:21.010697 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:21.084029 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 73483.4us
W20260709 12:23:21.028975 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.087504 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:20.966773  7546 consensus_queue.cc:1243] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 42.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:23:21.080322  7510 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:23:21.093617 27731 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
W20260709 12:23:21.094859 27662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.011s	sys 0.038s
W20260709 12:23:21.098201 27662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.011s	sys 0.038s
W20260709 12:23:21.017167  7595 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:23:20.962713  7613 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:20.920607  7573 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:20.962924  7594 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.129s
I20260709 12:23:21.093148  7399 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:20.967275  7587 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.134s
I20260709 12:23:21.082203  7583 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:21.080999  7520 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:21.091171  7398 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.096427  7480 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:21.098879  6951 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:21.122004  7614 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
W20260709 12:23:21.148366 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.107632  7522 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:21.099159  7549 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:23:21.151544 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:21.123288  7501 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:21.123646  7618 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:23:21.111004  6945 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:21.162176 27628 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:21.124506  7615 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.237s	user 0.000s	sys 0.000s
W20260709 12:23:21.125231  7609 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.263s	user 0.000s	sys 0.000s
W20260709 12:23:21.125669  7608 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.447s	user 0.000s	sys 0.001s
W20260709 12:23:20.964526  7560 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.662s	user 0.009s	sys 0.003s
I20260709 12:23:21.135224 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:21.138959 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:20.968484  7603 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.145s
W20260709 12:23:21.140786 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 294786us
I20260709 12:23:21.143784 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:21.151708 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:21.122730  7607 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.267s	user 0.000s	sys 0.000s
I20260709 12:23:21.155676 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:21.111577  7381 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:21.114053  7613 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:21.134703  7581 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.294s	user 0.037s	sys 0.080s
I20260709 12:23:21.161175 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.220954 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.141134  6951 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:21.182399 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.182796 27734 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:23:21.184654 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:21.224079 27735 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:21.187647 28331 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:21.190604 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.225492 27729 raft_consensus.cc:2368] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:21.225860 27738 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:23:21.193619 27730 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:23:21.217723 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:21.168169  7537 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:21.182202  7287 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:21.235095 27737 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 42 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:23:21.227620 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:21.148968  7398 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:21.224673  7581 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.386s	user 0.038s	sys 0.081s
I20260709 12:23:21.265996 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:21.235975  7618 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:23:21.274787 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.232146  6951 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:21.283784 27580 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
W20260709 12:23:21.278487 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51247.9us
I20260709 12:23:21.191689  7560 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.286623 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4850) took 1068 ms. Trace:
I20260709 12:23:21.285898 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:21.290598  7625 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:21.291949  7560 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.293682  7380 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.296469  7380 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:21.299253  7619 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.196s	user 0.000s	sys 0.000s
W20260709 12:23:21.300973  7624 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:23:21.301941  7626 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
I20260709 12:23:21.302052  7574 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.266060  7581 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:21.283504  7608 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.090s	user 0.002s	sys 0.000s
I20260709 12:23:21.314175  7581 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.314839 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4844) took 1166 ms. Trace:
I20260709 12:23:21.315475 27658 rpcz_store.cc:276] 0709 12:23:20.148912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.207935 (+ 59023us) service_pool.cc:224] Handling call
0709 12:23:20.208408 (+   473us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:20.208973 (+   565us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:23:20.209345 (+   372us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.356165 (+146820us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:20.359833 (+  3668us) log.cc:844] Serialized 52 byte log entry
0709 12:23:20.359981 (+   148us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:20.360021 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.360084 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.314550 (+954466us) raft_consensus.cc:1685] finished
0709 12:23:21.314581 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.314770 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":687,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":1175,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":3757,"thread_start_us":146292,"threads_started":1,"wal-append.queue_time_us":382449}
I20260709 12:23:21.307055  7443 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.266202  7546 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:21.318501  7546 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.321331  7546 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:21.283810  7539 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:21.322345  7539 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.322814  7539 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:21.307680 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:21.312096  7628 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:23:21.326591 28331 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:21.274636  7338 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.328008 27729 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:21.328253 28328 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
W20260709 12:23:21.275985 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 188567us
W20260709 12:23:21.327518 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103116us
I20260709 12:23:21.330008 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.330358 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:21.359668 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:21.349094 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 209120us
I20260709 12:23:21.256189  7537 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.353960  7573 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:21.352165 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 212573us
I20260709 12:23:21.361794 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:21.361040 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:21.367564  7603 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:21.368371  7603 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:21.369422  7603 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.370056  7399 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.371260 27584 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:21.371498  7320 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:21.372521  7510 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:21.372921  7613 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.347821  7540 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:21.367543  7627 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:23:21.387655 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.351830  7338 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:21.389389  7617 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.464s	user 0.001s	sys 0.000s
I20260709 12:23:21.390528  7338 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.392447  7398 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.287664  7480 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.384040  7520 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.352015  7583 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:23:21.381910  7506 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:21.412386 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:21.427966 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66911.2us
I20260709 12:23:21.429064 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:21.417091 27729 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:23:21.417496 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.444041 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:21.444536 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.450187 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.451762 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:21.453794 27737 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:21.455382 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.460304 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:21.411717  7537 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:21.464799 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:21.468727 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.472806 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:21.381145  7540 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.474766 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:21.476073 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:21.287024 27659 rpcz_store.cc:276] 0709 12:23:20.218635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.489975 (+271340us) service_pool.cc:224] Handling call
0709 12:23:20.551363 (+ 61388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.552246 (+   883us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:23:20.612790 (+ 60544us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.122467 (+509677us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b18004b8f80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:23:21.284540 (+162073us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.284612 (+    72us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:21.284696 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.284780 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.286567 (+  1787us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":30219,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":831,"raft.queue_time_us":1951,"spinlock_wait_cycles":45513472,"thread_start_us":161774,"threads_started":1}
I20260709 12:23:21.444079  7573 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:23:21.479790 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111408us
I20260709 12:23:21.480108 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:21.453482  7338 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:21.485385 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.483459 28328 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:21.453959  7506 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:21.464860  6823 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:21.463474  7399 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:21.488052 27581 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:23:21.489876 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:21.477921  7482 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.001s	sys 0.011s
I20260709 12:23:21.490931 27582 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
W20260709 12:23:21.502669 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 333207us
W20260709 12:23:21.479521  7527 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.595s	user 0.008s	sys 0.003s
I20260709 12:23:21.473631  7386 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:21.494443 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.457918  7360 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.031s
I20260709 12:23:21.504572 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:21.485044  7594 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:21.515763 27581 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:23:21.512823 28328 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:21.485277  6945 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:23:21.458050  7554 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:23:21.484946  7015 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:21.515722 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:21.470939  7443 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:23:21.520850 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:21.508265  7482 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.002s	sys 0.011s
W20260709 12:23:21.524269 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.517177 27734 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:23:21.488232  7641 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:21.524890 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:21.531006 27581 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
W20260709 12:23:21.545897 28292 leader_election.cc:336] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.757s (SENT)
I20260709 12:23:21.546356 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.547250 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.547783 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.546352 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.548404 27663 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:21.549324 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:21.550961 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:21.544788  7559 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:21.546228 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.572461 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:21.573690 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.545378  7574 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:21.575616 27658 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 32: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:21.576627 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:21.577319 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.577484 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.577935 27664 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 41 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 21.
I20260709 12:23:21.578011 27661 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:21.528784  7558 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:21.504065  7587 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:23:21.583281 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.584103 27664 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 30.
I20260709 12:23:21.584908 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.579046 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.533691  7510 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:21.518231  7527 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.522006  7338 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.032s
I20260709 12:23:21.518277  7640 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.083s
I20260709 12:23:21.585490 27664 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:21.575008  7574 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:21.587039 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88066us
I20260709 12:23:21.535734  6945 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:21.552122  7320 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:21.590344 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.572778  7549 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:21.614622 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.615705 27658 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:21.590966 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.591470 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:21.586892 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.576190  7641 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:21.616925 27655 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:21.608032 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.576452  7379 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:21.575294  7482 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:21.576947  7134 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:21.595479  7572 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.031s
I20260709 12:23:21.621949 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.627060 27657 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: [42.24-42.24]
W20260709 12:23:21.630285 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50526.8us
I20260709 12:23:21.648277 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:21.601142  7360 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:21.654086 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51475.8us
I20260709 12:23:21.655903 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.630102 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:21.659778 27658 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:21.662746 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60442.3us
I20260709 12:23:21.663951 27664 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:23:21.676890 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:21.678350 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 55498.2us
W20260709 12:23:21.679221 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64400.5us
I20260709 12:23:21.619797  7641 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.680011 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.680774 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:21.613083  7638 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:21.619832  7549 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:21.588439  7558 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.687186 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.648212  7594 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:21.625174  7320 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:21.630420  7636 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:21.699659  7636 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:21.701290  7632 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.418s	user 0.004s	sys 0.000s
I20260709 12:23:21.702068  7636 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:21.702369  7645 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:23:21.702567 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4849) took 1550 ms. Trace:
I20260709 12:23:21.703658 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5131) took 1040 ms. Trace:
I20260709 12:23:21.702966 27662 rpcz_store.cc:276] 0709 12:23:20.152517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.486666 (+334149us) service_pool.cc:224] Handling call
0709 12:23:20.487002 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:20.521969 (+ 34967us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.522037 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:20.529860 (+  7823us) log.cc:844] Serialized 26 byte log entry
0709 12:23:21.098661 (+568801us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:21.098790 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.098952 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.551202 (+452250us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:21.702258 (+151056us) raft_consensus.cc:1685] finished
0709 12:23:21.702284 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.702506 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28342,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":7180160,"thread_start_us":567973,"threads_started":1,"wal-append.queue_time_us":639981}
I20260709 12:23:21.704295 27588 rpcz_store.cc:276] 0709 12:23:20.663425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.915405 (+251980us) service_pool.cc:224] Handling call
0709 12:23:20.915809 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:20.925182 (+  9373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.925245 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:21.157413 (+232168us) log.cc:844] Serialized 26 byte log entry
0709 12:23:21.415840 (+258427us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:21.415904 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.415999 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.703428 (+287429us) raft_consensus.cc:1685] finished
0709 12:23:21.703454 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.703603 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4473,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":22570,"thread_start_us":196494,"threads_started":1,"wal-append.queue_time_us":306506}
I20260709 12:23:21.705255 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.706185 27662 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:21.691810 27660 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:21.628924  7637 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:21.697790  7662 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:23:21.698796  7527 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.732611 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.587724  7506 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.806183 27662 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:21.688136  7549 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:21.686374  7520 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:21.815946 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 150007us
I20260709 12:23:21.675966  7360 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.682992  7641 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:21.813252 27661 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:21.706511  7393 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:21.822335 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.827867 27627 leader_election.cc:400] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:21.828837 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:21.814072  7559 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:23:21.830721 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:21.839274 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 174698us
W20260709 12:23:21.835600 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 310591us
I20260709 12:23:21.838739 27662 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:21.856335 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.838579 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:21.861085 27661 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:21.862771 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.863833 27662 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:21.685648  6945 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.921222 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:21.921209 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:21.922260 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58805.6us
I20260709 12:23:21.926595 27662 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:21.925909 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72255.9us
I20260709 12:23:21.828058  7393 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.943835 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:21.947005 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:21.821691  7638 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.118s
I20260709 12:23:21.959103 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:21.960619  7651 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.382s	user 0.001s	sys 0.001s
I20260709 12:23:21.925871  7639 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.035s
W20260709 12:23:21.978605  7644 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.409s	user 0.000s	sys 0.001s
I20260709 12:23:21.978718  7393 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:21.979099  7667 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:21.980428  7660 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:21.981182  7667 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:21.981061  7660 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.981766  7667 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.982259  7667 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:21.939401  7594 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:21.983845  7670 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:23:21.987543  7672 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.054s
I20260709 12:23:21.957388  7581 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.923557  7480 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 36.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.033s
W20260709 12:23:21.975123  7539 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.000s
W20260709 12:23:21.975723  7643 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
I20260709 12:23:21.949921  7399 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:21.972496 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.042873 28328 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:21.956632  7338 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:21.962612  7638 consensus_queue.cc:1243] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 42.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
W20260709 12:23:22.007746 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47748.9us
I20260709 12:23:22.013111 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:21.952042  7559 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.039s
I20260709 12:23:21.925860  7320 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:21.963366  7662 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 12:23:22.056401 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:22.059146  7527 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.060428 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:22.063433 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.064322 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.066279 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.072647 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.073712 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.074194 27662 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:22.082662 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:22.084231 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.051095  7660 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:22.084667 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 255868us
I20260709 12:23:22.085122 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.040345  7581 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:22.087530  7665 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.148s	user 0.002s	sys 0.000s
I20260709 12:23:22.088809  7581 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:22.088871  7677 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:22.089285 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:22.090448 27734 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:22.094403 27738 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:22.094488 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.099196 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:22.100168 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:22.101116 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.106349 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5007) took 1552 ms. Trace:
I20260709 12:23:22.046990  7539 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.106896 27586 rpcz_store.cc:276] 0709 12:23:20.554462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.852582 (+298120us) service_pool.cc:224] Handling call
0709 12:23:21.158447 (+305865us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.158828 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.158859 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:21.171235 (+ 12376us) log.cc:844] Serialized 26 byte log entry
0709 12:23:21.398380 (+227145us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:21.398439 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.398532 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:22.106084 (+707552us) raft_consensus.cc:1685] finished
0709 12:23:22.106110 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.106278 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4710,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":14416,"thread_start_us":173251,"threads_started":1,"wal-append.queue_time_us":312664}
I20260709 12:23:22.075134  7662 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.115041 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:22.113830 27585 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:22.053334  7399 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.120247 27662 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:22.121876 27737 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 41 index: 21. (index mismatch)
I20260709 12:23:22.118386  7677 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.121560  7662 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:22.127458  7646 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.383s	user 0.000s	sys 0.001s
I20260709 12:23:22.080643  7527 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.129642 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.171738 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.177955 27734 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:22.137130 27737 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:22.196755 27738 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:22.210469 27738 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:22.211138 27734 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:22.134792 27658 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:22.211740 27734 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:22.212441 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.211333 27738 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.213207 27658 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:22.212759 27734 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.213883 27738 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:22.214051 27734 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:22.145208  7676 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:22.215909  7676 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.147231 27583 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:23:22.214150 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.220412 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.221524 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.146862  7631 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:22.145208  7650 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.383s	user 0.001s	sys 0.000s
I20260709 12:23:22.210008 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.222453 27658 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:22.228912 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.094887  7558 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.229882 27659 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 41 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 21.
I20260709 12:23:22.230119 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:22.197858  7644 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:23:22.237332 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.237455 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.239127 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:22.241657 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.231905 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:22.243470 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.224363  7631 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.131045  7360 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.251842  7520 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:22.257645  7682 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:23:22.257948  7520 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:22.258445  7682 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:23:22.258816  7682 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:22.229043  7386 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.259393  7682 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:22.259774  7386 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.262010  7386 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:22.251999  7538 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:22.258864  7520 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.276561 27734 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:23:22.277652 27663 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:22.250968 27659 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:22.233289  7569 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:22.296619 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50836.5us
W20260709 12:23:22.296227 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52015.4us
I20260709 12:23:22.300853 27663 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:22.318904 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.304785 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:22.320405 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 81319.9us
W20260709 12:23:22.312968 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:22.321688 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.322471 27662 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:23:22.322609 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.323369 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:22.324250 27662 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:22.331018 27662 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:22.332821 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.334707 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.336351 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.340161 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.341643 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:22.342967 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.307657 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.320763  7631 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:23:22.348304 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.256932  7558 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:22.344223 27659 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:22.348459  7675 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:22.345221 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:22.278322  7679 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:22.350104 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.350142 27628 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.198122  7549 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.348923  7639 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:22.357300  7681 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:22.357972  7681 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.354444  7674 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.006s
I20260709 12:23:22.351590  7660 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:23:22.354333  7685 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:23:22.363462 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:22.364269 27659 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:22.369167 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.370564 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.342702  7287 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.409761 27655 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:22.338541  7538 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:23:22.345443  7543 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.066s
I20260709 12:23:22.366014  7558 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:22.424350  7693 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:23:22.424743  7676 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.428783  7667 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:22.432472  7687 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:23:22.433702  7681 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.394110  7639 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:22.422075  7682 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.442284  7682 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:22.443938 27662 raft_consensus.cc:2393] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:22.445429  6951 consensus_meta.cc:287] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.157s	user 0.013s	sys 0.000s
I20260709 12:23:22.445868  6951 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.451148 27660 raft_consensus.cc:2393] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:22.366420  7656 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:22.367067  7662 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.344300  7569 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.442910  7682 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:22.431003  7675 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:22.479666  7549 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.480588  7631 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:22.481298 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5132) took 1817 ms. Trace:
I20260709 12:23:22.482196  7569 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.495529 27584 rpcz_store.cc:276] 0709 12:23:20.663972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.918404 (+254432us) service_pool.cc:224] Handling call
0709 12:23:20.918757 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.371659 (+452902us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.481042 (+1109383us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54000bccd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:22.481225 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1040208768}
W20260709 12:23:22.472504  7641 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.638s	user 0.005s	sys 0.006s
I20260709 12:23:22.463346  7558 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:22.414103  7287 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:22.483255  6823 consensus_meta.cc:287] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.987s	user 0.008s	sys 0.000s
I20260709 12:23:22.477098  7539 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.483850  7540 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.501278  7399 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.483986  7667 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.511420  6823 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:23:22.518702 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165119us
I20260709 12:23:22.502349  7675 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.522593 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.509445  7641 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.546413 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:22.548480 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.521853  7697 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:22.550953  7697 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.553710  7697 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.520843  6951 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.560734 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:22.566712  7399 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:22.566833 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.567912  7540 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:22.571528 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.572187 27734 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:22.572576 27737 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:23:22.567201  7399 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.576238 28331 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:22.572556  7231 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:22.583822 27738 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:23:22.589133 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.589728 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:22.590039  7381 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 37 current term: 37 is higher.
I20260709 12:23:22.592010  7445 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:22.593976 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4244) took 1120 ms. Trace:
I20260709 12:23:22.594357 27729 rpcz_store.cc:276] 0709 12:23:21.473761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.474282 (+   521us) service_pool.cc:224] Handling call
0709 12:23:22.593803 (+1119521us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54001d38d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:22.593923 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1336667520}
I20260709 12:23:22.575670  7399 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:22.579682  7569 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.599858  7569 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.601487 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.602795 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.603057 28329 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:22.603546 28330 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 33: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:22.604698 28334 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:23:22.602061  7569 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.589929 28336 raft_consensus.cc:2393] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 32: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:22.572386  7520 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:22.565348  7703 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:22.609923 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.610214 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.610606 28336 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:22.611501  7703 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:22.613857  7703 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:22.615015  7287 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:22.525084  7558 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.26, Last appended by leader: 26, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.576478  7674 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:22.617293  7674 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.576903  7697 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:22.636533  7231 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:23:22.629420 28336 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:22.628885 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.628096 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:22.632169 28330 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:23:22.619313  7641 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.643462 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.646903 27584 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:22.649039 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.649494 27581 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:22.611814  7520 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:22.689095 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.717123 28330 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:22.709192 27582 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: [34.22-36.23]
I20260709 12:23:22.708318 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.718768 28336 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
W20260709 12:23:22.714638 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:22.719776 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.721065 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:22.722289 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:22.723102 27582 raft_consensus.cc:2410] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 37 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:23:22.723342  7506 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:22.724126 27584 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: [39.21-39.21]
I20260709 12:23:22.723858  7506 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.642910  7674 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.737787 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.738658 27729 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:22.739852 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.744266 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:22.744647 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 209939us
I20260709 12:23:22.705834  7381 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.750538 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.709975  7520 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:22.756186 28337 raft_consensus.cc:2368] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:23:22.757346 28330 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:22.749987  7116 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.758455 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.759477 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.637066  7697 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.764519 28329 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:22.653789  7660 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.099s
W20260709 12:23:22.760254  7665 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:23:22.780880 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.752311  7639 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.829844 27729 pending_rounds.cc:85] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 25
I20260709 12:23:22.831400 27729 pending_rounds.cc:107] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 37.26
I20260709 12:23:22.831856 27729 raft_consensus.cc:2889] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:22.833694 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.836495 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.837396 28329 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:22.772469  7225 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:22.856014  7571 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:22.777585  7116 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.847820 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:22.849431  7706 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:23:22.830868  7540 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:22.766266  7381 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.877929  7574 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 41.20, Last appended by leader: 20, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.871244 27584 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:22.879660 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.849972 28330 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:22.769696  7386 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.092s
I20260709 12:23:22.835382  7674 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:22.870502 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.872288 27586 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:23:22.866657  7705 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:22.892056 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5143) took 1407 ms. Trace:
I20260709 12:23:22.933163 27585 rpcz_store.cc:276] 0709 12:23:21.484709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.484883 (+   174us) service_pool.cc:224] Handling call
0709 12:23:22.108623 (+623740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.116150 (+  7527us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.891813 (+775663us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54001879d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:22.891975 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1433801984}
I20260709 12:23:22.933856 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.934808 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:22.935727 28330 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
W20260709 12:23:22.897650  7623 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 1.937s	user 0.003s	sys 0.000s
I20260709 12:23:22.899302 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5142) took 1417 ms. Trace:
I20260709 12:23:22.941893 27580 rpcz_store.cc:276] 0709 12:23:21.481740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.488862 (+  7122us) service_pool.cc:224] Handling call
0709 12:23:21.489153 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.491239 (+  2086us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.491276 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.491339 (+    63us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:21.491374 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.898978 (+1407604us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5400087fd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:22.899043 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.899220 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4353280}
W20260709 12:23:22.901425 27663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2165) took 1353 ms (client timeout 1716 ms). Trace:
W20260709 12:23:22.943468 27663 rpcz_store.cc:269] 0709 12:23:21.548197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.549120 (+   923us) service_pool.cc:224] Handling call
0709 12:23:22.901205 (+1352085us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b540005fd54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:22.901352 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1673674624}
I20260709 12:23:22.944435 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:22.944882 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.945851 27663 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:22.938841 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.946727 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.903182 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4852) took 2609 ms. Trace:
I20260709 12:23:22.947288 27733 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:22.947485 27657 rpcz_store.cc:276] 0709 12:23:20.294190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.732676 (+438486us) service_pool.cc:224] Handling call
0709 12:23:20.733156 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.627602 (+894446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.902858 (+1275256us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54000e60d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:22.903100 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2054950272}
I20260709 12:23:22.948274 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.906973 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5112) took 3252 ms. Trace:
I20260709 12:23:22.949035 27733 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:22.950065 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:22.908166  7137 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:22.909996  7381 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:22.913950  7697 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:22.885414  7116 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:22.916240  7470 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:22.918663  7386 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:22.920805 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.887979  7674 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:22.925828  7554 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:22.928429 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:22.930955 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2163) took 1383 ms. Trace:
I20260709 12:23:22.896482  7399 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.952970 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:22.951273 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:22.953665 27702 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:22.954720 27628 leader_election.cc:400] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:23:22.956457 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.839680  7639 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:22.949379 27589 rpcz_store.cc:276] 0709 12:23:19.654606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.872807 (+218201us) service_pool.cc:224] Handling call
0709 12:23:19.873146 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.980570 (+107424us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:23:20.124584 (+144014us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.359163 (+234579us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.359211 (+    48us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:20.360017 (+   806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.051262 (+1691245us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b1800362980. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:23:22.906680 (+855418us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:22.906737 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.906883 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61549,"prepare.queue_time_us":240775,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1608,"raft.queue_time_us":1339,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7541,"spinlock_wait_cycles":278103168,"thread_start_us":369986,"threads_started":2,"wal-append.queue_time_us":588614,"wal-append.run_cpu_time_us":2114,"wal-append.run_wall_time_us":1427433}
I20260709 12:23:22.959947 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:22.964668 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 196905us
I20260709 12:23:23.008134 27628 leader_election.cc:403] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:23.008651 27663 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:23.009164 28337 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:23.010857 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:22.957552  7338 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.028s
W20260709 12:23:23.010975 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56450.7us
I20260709 12:23:23.012172 27589 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:23.012745 27733 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:23.013013 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.012190 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:23.012243 27655 rpcz_store.cc:276] 0709 12:23:21.547700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.649385 (+101685us) service_pool.cc:224] Handling call
0709 12:23:21.683507 (+ 34122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.410747 (+727240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.930726 (+519979us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b5400292054. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:22.930895 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1669610240}
W20260709 12:23:23.017277 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63687.7us
I20260709 12:23:22.984680 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.958442  7137 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:22.963147 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.006s	sys 0.049s
W20260709 12:23:23.024087 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.006s	sys 0.049s
I20260709 12:23:23.019444 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.020053 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.012171 27704 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.014443 28335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1809) took 2411 ms. Trace:
W20260709 12:23:23.025689 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 685605us
W20260709 12:23:23.022411 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62575.3us
I20260709 12:23:23.020854  7558 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:23:23.028591 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:23.031587 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:23.032807 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.033428 27589 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:23:23.035538  7116 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:23.035914  7638 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:23.038688  7638 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:23.038604 27589 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:23:23.039714  7638 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.036101  6951 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:23.041713 27586 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:23.036317 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.018923 27657 raft_consensus.cc:2376] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37. Re-sending same reply.
I20260709 12:23:23.028927 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:23.042537 27662 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:23.025281 28335 rpcz_store.cc:276] 0709 12:23:20.579676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.579820 (+   144us) service_pool.cc:224] Handling call
0709 12:23:20.580113 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:20.580529 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.580568 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:20.581029 (+   461us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.671374 (+ 90345us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:20.671499 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.671740 (+   241us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:22.896953 (+2225213us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:22.983112 (+ 86159us) raft_consensus.cc:1685] finished
0709 12:23:22.983144 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.990591 (+  7447us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":127,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":137,"spinlock_wait_cycles":10298752,"thread_start_us":89578,"threads_started":1,"wal-append.queue_time_us":300859}
I20260709 12:23:23.041041  7399 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.036190  7549 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:22.959517  7543 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:22.979846  7470 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:23.041649  6951 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:23.045047 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 255633us
I20260709 12:23:23.049938 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.038612  7134 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:23.036433 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.044807 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.052652 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.053226 27655 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:23.053525 27662 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:23:23.046731 27628 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:22.959276  7445 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.077s
I20260709 12:23:23.009651  7554 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:23.046159  7116 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:23.054695 27628 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:23:23.056702 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:22.960795  7640 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.091s
I20260709 12:23:23.055258  7134 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:22.952807  7705 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.059450 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.046635  7543 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:23.041074  7638 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:23.062717 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.078258 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.066159 27663 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:23.046627  7338 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:23.066996 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:22.983937  7381 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.074s
I20260709 12:23:23.045662  7636 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.019s
I20260709 12:23:23.073572 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.051613  6951 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:23.074332 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.075798 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.060664  7641 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:23.088313 27657 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:23.056282  7634 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:23:23.070724  7700 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:23.077100  7549 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:23.094055 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.076591  7543 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:23.045476  7386 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.086310  7338 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.110469 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:23.099254  7231 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:23.101934  7568 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:23.072487  7697 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.033s
I20260709 12:23:23.112411  7639 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:23.112861 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.109732  7386 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.099112  7700 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:23.114001  7711 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:23:23.114779  7703 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.500s	user 0.007s	sys 0.005s
I20260709 12:23:23.131551  7703 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.134156  7703 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.102326  7641 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:23.107316  7554 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.105502 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.135521  7641 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.108523  7613 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:23.136659  7613 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:23.137072  7613 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:23.106912 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:23.143683 27663 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:23.097007  7719 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:23.144188 27582 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:23.100494  7630 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:23:23.145900  7630 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:23.136291 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:23.146884  7630 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.147186  7470 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.147708 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:23.148242  7470 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:23.124970 27658 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:23.097162  7070 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:23.097450  7559 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:23.128017  7700 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:23.107537 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.150868 27657 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:23.118834  7338 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:23.148283  7630 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:23:23.111088  7705 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.148583 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:23.141192  7572 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:23.154073 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.121265  7547 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:23.155831 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.160667 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.162000 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.162597 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:23.162827 27658 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:23.096815  7134 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.090377  7137 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.167023 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:23.152132  7231 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:23.156428  7482 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:23.167373 27663 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: [39.27-39.27]
I20260709 12:23:23.177721 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4855) took 1062 ms. Trace:
I20260709 12:23:23.164806  7705 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:23.164762  7399 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:23.179855 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4859) took 1063 ms. Trace:
I20260709 12:23:23.164711  7703 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:23.155306  7559 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.178488 27663 rpcz_store.cc:276] 0709 12:23:22.115491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.166813 (+1051322us) service_pool.cc:224] Handling call
0709 12:23:23.167139 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.177396 (+ 10257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.177644 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.166000  7398 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:23.184777 27658 rpcz_store.cc:276] 0709 12:23:22.116711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.167273 (+1050562us) service_pool.cc:224] Handling call
0709 12:23:23.179268 (+ 11995us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.179523 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.179555 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.179600 (+    45us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:23.179645 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.179702 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.179801 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.172006  7231 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:23.156072  7070 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.190667  7231 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:23.193486  7716 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:23.188303  7398 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:23.196378  7398 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.197439  7398 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:23.200831 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.201722 27734 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
W20260709 12:23:23.205688  7648 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.623s	user 0.005s	sys 0.005s
I20260709 12:23:23.206540  7637 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:23.206944  7637 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:23.207768  7637 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.208772  7637 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:23.209081 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4829) took 3914 ms. Trace:
I20260709 12:23:23.209471 27656 rpcz_store.cc:276] 0709 12:23:19.294846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.681363 (+386517us) service_pool.cc:224] Handling call
0709 12:23:19.681648 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.120914 (+3439266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.120960 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.121018 (+    58us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:23.121060 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.208840 (+ 87780us) spinlock_profiling.cc:243] Waited 3.44 s on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:23.208886 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.209024 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2022,"raft.queue_time_us":2132,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":1655,"spinlock_wait_cycles":7903945472}
I20260709 12:23:23.211759 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:23.214160 27587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 5130) took 2551 ms (client timeout 1921 ms). Trace:
W20260709 12:23:23.214566 27587 rpcz_store.cc:269] 0709 12:23:20.662841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.845162 (+182321us) service_pool.cc:224] Handling call
0709 12:23:23.213924 (+2368762us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54000898d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:23.214055 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":690064128}
I20260709 12:23:23.217814  7137 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:23.219978  6951 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.222934 27731 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:23:23.234028 27731 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:23.234427 27656 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: [39.21-39.21]
I20260709 12:23:23.223933  7613 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.225265  7654 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.225724  7639 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.229756  7674 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.230432 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4245) took 1746 ms. Trace:
W20260709 12:23:23.249843 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.231859 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 4053) took 1406 ms. Trace:
I20260709 12:23:23.232664 27659 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:23.250799 27659 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:23.250736 27660 rpcz_store.cc:276] 0709 12:23:21.825251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.825932 (+   681us) service_pool.cc:224] Handling call
0709 12:23:23.231657 (+1405725us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b54000e79d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:23.231800 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1365553024}
I20260709 12:23:23.252750 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4865) took 1130 ms. Trace:
I20260709 12:23:23.253118 27656 rpcz_store.cc:276] 0709 12:23:22.122386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.210883 (+1088497us) service_pool.cc:224] Handling call
0709 12:23:23.211271 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.234790 (+ 23519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.252689 (+ 17899us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.237859 27731 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:23.186685  7703 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:23.181999  7705 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:23.165246  7547 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.249709 27732 rpcz_store.cc:276] 0709 12:23:21.484167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.484303 (+   136us) service_pool.cc:224] Handling call
0709 12:23:21.484600 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.706634 (+222034us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.706665 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.706715 (+    50us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:21.706753 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.230194 (+1523441us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540017ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:23.230239 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.230367 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510096640}
W20260709 12:23:23.240850 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 228750us
I20260709 12:23:23.177156  7482 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:23.192322  7134 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:23.236609 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.256156 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.223001  6951 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.264027 27658 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:23:23.251852 27659 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.262622  7705 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:23.258373  7703 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:23.288407 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4866) took 1140 ms. Trace:
W20260709 12:23:23.192212 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 173827us
I20260709 12:23:23.313285 27659 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:23.308316 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4864) took 1188 ms. Trace:
I20260709 12:23:23.316259 27660 rpcz_store.cc:276] 0709 12:23:22.148213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.251736 (+1103523us) service_pool.cc:224] Handling call
0709 12:23:23.287919 (+ 36183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.288111 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.288136 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.288175 (+    39us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:23.288204 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.288253 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.288347 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.318974 27658 rpcz_store.cc:276] 0709 12:23:22.120149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.239500 (+1119351us) service_pool.cc:224] Handling call
0709 12:23:23.263618 (+ 24118us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.308017 (+ 44399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.308253 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.321296 27656 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:23:23.322230 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4073) took 1099 ms. Trace:
I20260709 12:23:23.322701 27656 rpcz_store.cc:276] 0709 12:23:22.222611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.254010 (+1031399us) service_pool.cc:224] Handling call
0709 12:23:23.318040 (+ 64030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.321920 (+  3880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.322180 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:23.327899 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 249691us
I20260709 12:23:23.328783 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.328130 27731 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:23.298339  7713 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:23:23.317224 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:23.315114  7520 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260709 12:23:23.312956  7558 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 31. Current term is 32.
I20260709 12:23:23.267062  7482 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:23.286095  7559 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.335412 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:23.335487 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98943.7us
I20260709 12:23:23.287459  7070 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.305228  7613 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.341327 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2170) took 1002 ms. Trace:
I20260709 12:23:23.375905 27659 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 24. (term mismatch)
I20260709 12:23:23.386332 27659 pending_rounds.cc:85] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 23
I20260709 12:23:23.386894 27659 pending_rounds.cc:107] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 41.24
I20260709 12:23:23.310672  7338 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.291227  7700 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.376366 27658 rpcz_store.cc:276] 0709 12:23:22.339435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.325382 (+985947us) service_pool.cc:224] Handling call
0709 12:23:23.340854 (+ 15472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.341020 (+   166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.341043 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.341085 (+    42us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:23:23.341120 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.341174 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.341277 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.263125  7639 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.290723  7543 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.387720 27659 raft_consensus.cc:2889] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:23.411998 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4061) took 1439 ms. Trace:
I20260709 12:23:23.412369 27659 rpcz_store.cc:276] 0709 12:23:21.972122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.390886 (+418764us) service_pool.cc:224] Handling call
0709 12:23:22.391276 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.232568 (+841292us) spinlock_profiling.cc:243] Waited 77.9 ms on lock 0x7b54001df720. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:23:23.411727 (+179159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.411936 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179157888}
I20260709 12:23:23.340528  7725 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.335795  7386 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:23:23.333775  7445 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:23.397519  7381 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:23.399937  7231 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:23.432214 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.341176  7520 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.433243 27585 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:23.433396  7231 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.403194 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:23.404562 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69254.6us
I20260709 12:23:23.435649  7520 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.436224 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.436380  7520 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:23.405216 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.439640 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:23.341009  7399 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:23.342941  7697 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 43.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.072s
I20260709 12:23:23.441012  7399 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:23.441164 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:23.441509  7399 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:23.417413 27662 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:23.429041 27658 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:23:23.443632  7381 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:23.257907  7654 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.444077  7381 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:23.446877  7381 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:23.445771  7654 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:23.446341 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.450709 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.460965 27663 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:23.464718 28335 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
W20260709 12:23:23.490694 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:23.425550  7565 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 42 current term: 42 is higher.
I20260709 12:23:23.498802 28337 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:23:23.512898 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.514272 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:23.516690 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77129.3us
I20260709 12:23:23.426270  7725 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:23.517396 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.763s (SENT) [suppressed 24 similar messages]
I20260709 12:23:23.504070  7727 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:23.521714 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.497833  7634 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.024s
I20260709 12:23:23.524725 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:23.533871 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:23.534703 27582 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:23.528095 27630 connection.cc:256] RPC call timeout handler was delayed by -1.1201028499999666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260709 12:23:23.536193 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.519186  7654 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.020s
I20260709 12:23:23.547940 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.523711 27587 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:23.521131  7445 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:23.554278  7445 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.559689 27734 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 21. (term mismatch)
I20260709 12:23:23.560102 27734 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 20
I20260709 12:23:23.560329 27734 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 39.21
I20260709 12:23:23.560556 27734 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:23.562465  7543 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.530604  7638 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.563403  7638 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.565601  7638 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.566403 28334 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:23.539695  7703 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.554296  7639 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:23.569270  7703 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.570783 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:23.571368 27583 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:23:23.572851  7574 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:23.575028  7506 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:23.576478 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 4049) took 1889 ms (client timeout 2205 ms). Trace:
I20260709 12:23:23.523157  7727 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:23.577122 27664 rpcz_store.cc:269] 0709 12:23:21.686698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.686863 (+   165us) service_pool.cc:224] Handling call
0709 12:23:23.576303 (+1889440us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400041d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:23.576412 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345868032}
I20260709 12:23:23.577807  7727 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:23.578647 27664 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:23:23.553772 27587 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 27. Current term is 28. Ops: []
I20260709 12:23:23.580052  7677 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.580520  7506 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:23.581041  7506 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.581652 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4860) took 1464 ms. Trace:
I20260709 12:23:23.548774 27581 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:23.583210 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.576406 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.584076  7705 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.584512 27588 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:23.584587 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.582386 27656 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:23.584995  7700 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.585274 27738 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:23.583235  7506 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.586843 27656 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:23.587872  7725 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:23.581861 27664 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42. Ops: []
W20260709 12:23:23.541476 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.595866 27584 raft_consensus.cc:2468] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:23.597215 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:23.582094 27663 rpcz_store.cc:276] 0709 12:23:22.116981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.233949 (+1116968us) service_pool.cc:224] Handling call
0709 12:23:23.258777 (+ 24828us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.461678 (+202901us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.581431 (+119753us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b5400229f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:23.581597 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464553856}
I20260709 12:23:23.590229  7520 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:23.623644 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.629410 27738 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:23.636718 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.638674 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.640647 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.641338 27582 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:23.643391 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.551342  7547 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.646616 27738 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:23.646472 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:23.651429 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 133949us
I20260709 12:23:23.588063  7705 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.658628 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.664770 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:23.668452 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:23.644052  7703 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:23.670014 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.673049 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4868) took 1443 ms. Trace:
I20260709 12:23:23.673738 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.678879 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:23.655900  7338 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.673470 27660 rpcz_store.cc:276] 0709 12:23:22.229259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.320373 (+1091114us) service_pool.cc:224] Handling call
0709 12:23:23.337775 (+ 17402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.338263 (+   488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.338305 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.338896 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.544791 (+205895us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:23.645163 (+100372us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.672775 (+ 27612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.672851 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.672895 (+    44us) raft_consensus.cc:1685] finished
0709 12:23:23.672906 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.673003 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27260,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":35163,"thread_start_us":233802,"threads_started":2,"wal-append.queue_time_us":161435}
I20260709 12:23:23.668493  7697 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.710902  7637 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.001s
W20260709 12:23:23.724087  7740 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:23:23.702556  7700 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:23:23.679677  7703 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.683297  7727 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.778424  7742 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.813227 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.762056  7697 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.788872  7700 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:23.802832  7742 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.831043  7703 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.830878  7399 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.832716  7697 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:23.832926  7727 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:23:23.833045  7399 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.833591  7700 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.859138  7501 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:23.859879  7501 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.861649  7742 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:23.862238  7501 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.835914  7719 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:23.863267  7381 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.863746  7719 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.865907  7719 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:23.906302 28330 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:23.871268  7482 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:23.912101  7482 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:23.919190 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 216856us
I20260709 12:23:23.919346 28331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58252 (request call id 1788) took 1332 ms. Trace:
I20260709 12:23:23.920161 28331 rpcz_store.cc:276] 0709 12:23:22.587114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.587258 (+   144us) service_pool.cc:224] Handling call
0709 12:23:22.587638 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.587894 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:23:22.763080 (+175186us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.918477 (+1155397us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b1800249760. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e99ef04bc 00007f0e9723e48b 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a
0709 12:23:23.918823 (+   346us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.918913 (+    90us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:23.918993 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.919102 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.919246 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":371735,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":2080,"spinlock_wait_cycles":23127936,"thread_start_us":409210,"threads_started":2,"wal-append.queue_time_us":338052}
I20260709 12:23:23.936012  7656 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:23.936455  7656 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:23.937374  7656 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:23.938443  7656 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:23.943233  7629 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:23.946468 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:23.923527  7702 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:23.948056 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:23.949515 28335 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:23:23.956879 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:23.956794 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:23.959132 27553 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:23.971657 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.971163 27553 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:23.975879  7705 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:23.976565  7705 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:23.977058  7705 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:23.977578 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4779) took 1029 ms. Trace:
I20260709 12:23:23.978020 27733 rpcz_store.cc:276] 0709 12:23:22.948036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.019750 (+ 71714us) service_pool.cc:224] Handling call
0709 12:23:23.977388 (+957638us) spinlock_profiling.cc:243] Waited 85.1 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:23.977515 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195734400}
I20260709 12:23:23.972668  7482 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:23.980760  7381 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:23.968405  7725 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.014s
I20260709 12:23:23.983552  7725 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:23.984073  7725 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:23.984745 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.987596 27553 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.991218 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:23.993427 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.995676 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:23.996721 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:23.997473 27733 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:24.008440 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.952344  7742 consensus_queue.cc:1059] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:23:24.010330 27733 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:24.015769 27582 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 31. Current term is 32. Ops: []
I20260709 12:23:24.016971 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.023152 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5033) took 1275 ms. Trace:
I20260709 12:23:24.023613 27586 rpcz_store.cc:276] 0709 12:23:22.747935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.890293 (+142358us) service_pool.cc:224] Handling call
0709 12:23:22.925902 (+ 35609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.044117 (+118215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.022914 (+978797us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540004f954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:24.023091 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":265863040}
W20260709 12:23:24.025535 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.030987 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:23.985181  7725 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.045599  7727 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:24.009393  7482 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:23:24.011519  7762 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:23:24.047640  7482 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:24.049104 27735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4772) took 1962 ms. Trace:
I20260709 12:23:24.049000  7482 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.050747  7742 raft_consensus.cc:3055] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:24.051191  7742 raft_consensus.cc:740] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:24.040586  7677 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:23:24.061640  7763 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:23:24.062088  7719 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:23:24.065965 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.066975 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.011368  7231 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.046s
I20260709 12:23:24.053762  7725 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:24.049572 27735 rpcz_store.cc:276] 0709 12:23:22.086831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.269538 (+182707us) service_pool.cc:224] Handling call
0709 12:23:23.827219 (+1557681us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:23.827656 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.828061 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.828088 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.828576 (+   488us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.918390 (+ 89814us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:23.918512 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.918683 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:24.048817 (+130134us) raft_consensus.cc:1685] finished
0709 12:23:24.048836 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.049048 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1801,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":73,"spinlock_wait_cycles":50169088,"thread_start_us":82209,"threads_started":1,"wal-append.queue_time_us":111817}
I20260709 12:23:24.079501  7742 consensus_queue.cc:260] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.083848 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:24.085448 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.085800 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:24.086036 27588 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:24.086323 27587 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:24.088596 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.089818 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.090673 27731 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:24.090585 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.091781 27582 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:24.094272 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.094890 27583 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:24.096285 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.096417 27735 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:23:24.100359 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:24.082798  7728 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.742s	user 0.002s	sys 0.003s
I20260709 12:23:24.108949 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:24.109504 27586 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:24.111035 27630 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:24.114327  7703 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:24.117543 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 34, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 34 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:24.119979 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 30 to 32, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:24.122138 27487 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 35 to 37. New cstate: current_term: 37 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:24.124822  7749 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 12:23:23.934396 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:24.129731  7673 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:24.080873  7554 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:24.133334 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:24.138135 28330 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:24.142524 28328 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:24.143496  7656 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.005s
I20260709 12:23:24.145943 27630 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:24.149142 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:24.149197 27630 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:24.152488 27630 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:24.135336  7443 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:24.154655  7443 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.156015 27586 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:24.145182  7673 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:24.155166  7443 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:23:24.166826 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.172344  7594 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:24.172156  7629 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:23:24.184984 27630 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:24.185599  7573 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:24.194366  7654 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:24.149914  7630 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:24.177129  7287 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:24.175230  7637 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.054s
W20260709 12:23:24.208061  7765 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:23:24.209239 27585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2203) took 1276 ms. Trace:
I20260709 12:23:24.264873 27585 rpcz_store.cc:276] 0709 12:23:22.932592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.934345 (+  1753us) service_pool.cc:224] Handling call
0709 12:23:22.934666 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.433652 (+498986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.208965 (+775313us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54001892d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:24.209176 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1146185472}
I20260709 12:23:24.219419 28330 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:24.267169 27581 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
W20260709 12:23:24.225718 27661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2169) took 2666 ms (client timeout 1763 ms). Trace:
I20260709 12:23:24.226334  7629 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:24.229274 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4773) took 2142 ms. Trace:
I20260709 12:23:24.271160 27736 rpcz_store.cc:276] 0709 12:23:22.087138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.270494 (+183356us) service_pool.cc:224] Handling call
0709 12:23:23.114894 (+844400us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:23.115094 (+   200us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.570940 (+455846us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.570972 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.571499 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.779012 (+207513us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:23.779069 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.078294 (+299225us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54002e77d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:24.078366 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:24.123976 (+ 45610us) spinlock_profiling.cc:243] Waited 7.92 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:24.229060 (+105084us) raft_consensus.cc:1685] finished
0709 12:23:24.229082 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.229216 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4653,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":1113435008,"thread_start_us":207131,"threads_started":1,"wal-append.queue_time_us":301184}
I20260709 12:23:24.156009  7381 consensus_queue.cc:1059] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:23:24.156147  7679 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:24.151067  7574 raft_consensus.cc:2804] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:24.203910 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:24.206677  7760 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:23:24.216727  7594 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.149796  7666 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.105s
I20260709 12:23:24.291772  7666 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:24.292176  7666 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:24.293040  7666 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.294327  7666 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:24.254673  7654 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:24.296021  7654 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.297076  7654 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:24.297204  7679 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:24.298169  7679 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.225826  7690 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:24.301688  7690 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.304020  7690 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:24.265862 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:24.152156  7647 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.106s
I20260709 12:23:24.305753 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:24.227669 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.306664 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.308681  7338 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:24.253867  7672 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:24.309495  7672 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.309885  7672 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:24.310609  7594 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:24.267867 27661 rpcz_store.cc:269] 0709 12:23:21.549109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.578735 (+ 29626us) service_pool.cc:224] Handling call
0709 12:23:24.215736 (+2637001us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:24.215860 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452353792}
I20260709 12:23:24.155821  7677 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.102s
I20260709 12:23:24.258701  7670 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:23:24.269057  7287 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:24.314142  7677 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:23:24.314550  7677 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:24.315006  7287 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.315400  7677 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.316339 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:24.316754  7677 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:24.287600  7568 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 40.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.030s
I20260709 12:23:24.317570 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:24.276813  7574 raft_consensus.cc:697] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:24.318359  7568 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:24.318760  7568 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:24.319841 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:24.280004  7629 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.256836  7630 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:24.259438  7745 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.287632 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83788.5us
I20260709 12:23:24.258893  7506 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:24.288908  7393 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:24.261193  7634 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:24.328985 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.321732 28328 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:24.321651 28334 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 37 index: 27. (index mismatch)
I20260709 12:23:24.327698 27733 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:23:24.333503 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.323081  7703 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:24.335460  7629 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:24.274892  7673 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.337067  7673 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:24.339425  7647 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:24.332849  7634 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:24.340914  7634 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.323582  7520 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:24.342208  7634 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:24.342259  7520 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.342870  7520 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:24.343354  7506 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.345736  7506 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:24.346891 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:24.347813  7675 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:24.347074 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.365193 27661 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:24.349073  7690 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:24.350901  7393 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:24.347654  7568 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.323261  7670 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.322028  7381 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.027s
I20260709 12:23:24.329423  7630 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:24.351024  7398 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:24.349015  7745 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.318845  7574 consensus_queue.cc:237] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.347944  7705 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.323740  7583 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.022s
W20260709 12:23:24.358556 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.360414 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.370841 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.493192 27656 raft_consensus.cc:2376] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35. Re-sending same reply.
I20260709 12:23:24.370832 27664 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 43. Current term is 44. Ops: []
W20260709 12:23:24.424947 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.429303 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.338773  7443 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.499950 27735 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 37 index: 27. (index mismatch)
I20260709 12:23:24.358837  7667 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.478135 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.508340 27663 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:24.511972 27662 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:24.471091  7700 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:24.350900  7719 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.119s
I20260709 12:23:24.498070 27733 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 44. Current term is 45. Ops: []
W20260709 12:23:24.498247 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.498870 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:24.495087 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:24.483412 27661 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
I20260709 12:23:24.485242 27584 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:23:24.526604 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.517836 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.519950 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:24.528069 27737 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:24.528573 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.522958 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:24.529151 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:24.503340  7381 consensus_queue.cc:1243] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 40.28 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 27
I20260709 12:23:24.504849  7398 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 40.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:24.499919  7116 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.538103 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.531924 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:24.530966 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:24.544780 27661 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:24.533201 27735 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.20-41.20]
I20260709 12:23:24.531770 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.531265 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.534979 27655 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:24.484772  7670 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.503278  7565 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.034s
I20260709 12:23:24.539557 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:24.554342 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 194860us
I20260709 12:23:24.557171 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:24.557554 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 207047us
W20260709 12:23:24.558418 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.534874  7656 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:24.560447 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86082.2us
I20260709 12:23:24.516832  7667 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.599762 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:24.564396  7768 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:23:24.569507  7443 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:24.570304  7594 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.540140  7393 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.543946  7527 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:24.573102  7705 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:24.573426 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:24.636106 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62758us
W20260709 12:23:24.574440 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.576613  7656 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.580442  7670 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:24.581422 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.582419  7568 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:24.584080 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1946) took 1113 ms. Trace:
W20260709 12:23:24.619503  7725 consensus_meta.cc:287] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 12:23:24.660094 27732 rpcz_store.cc:276] 0709 12:23:23.471120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.871865 (+400745us) service_pool.cc:224] Handling call
0709 12:23:23.979429 (+107564us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.979634 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.979661 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.979705 (+    44us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:23.979737 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.979802 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.583915 (+604113us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54001d2ed0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a3478ad 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:24.584023 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":246209920}
I20260709 12:23:24.584455 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.585127 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.588114 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4266) took 1952 ms. Trace:
I20260709 12:23:24.590216  7338 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.593171 27661 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:24.593103 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.595098 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.596618 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.663823 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.662278 27729 rpcz_store.cc:276] 0709 12:23:22.635705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.768579 (+132874us) service_pool.cc:224] Handling call
0709 12:23:22.829595 (+ 61016us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.832588 (+  2993us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:23:22.832646 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.316410 (+1483764us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:23:24.316820 (+   410us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.531823 (+215003us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:24.531858 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.531925 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:24.587794 (+ 55869us) raft_consensus.cc:1685] finished
0709 12:23:24.587820 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.588050 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2807040,"thread_start_us":214680,"threads_started":1,"wal-append.queue_time_us":269127}
I20260709 12:23:24.623725 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.624382 27589 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:24.628197 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:24.628940 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 133956us
I20260709 12:23:24.528828  7700 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.677640 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:24.553836  7554 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:24.640969 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:24.664819 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.622277 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:24.662932 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:24.666241 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.671388 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:24.665939 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:24.543884  7116 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.661618 27663 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:24.682463 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:24.689425 27589 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:24.669456 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.690469 27664 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:24.691372 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.691944 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:24.693408 27664 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:23:24.685863 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.694288 27661 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:24.685253 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.694754 27662 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
W20260709 12:23:24.696959 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 56080.8us
I20260709 12:23:24.711529 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:24.700155 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.659688  7443 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:24.684092 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.679106  7705 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.723021 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.725118 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:24.734388 27663 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 43 because replica has last-logged OpId of term: 41 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 19.
I20260709 12:23:24.727866 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.735294 27661 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:24.740341 27664 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:23:24.749239 27589 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:24.756525 27662 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
W20260709 12:23:24.765581 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 265512us
I20260709 12:23:24.702821  7703 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:24.722518  7679 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:24.709677  7554 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:24.771239 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4771) took 2675 ms. Trace:
W20260709 12:23:24.773380 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 134815us
I20260709 12:23:24.679804  7338 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.769829 27656 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:24.732076  7719 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:24.727535  7443 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:24.710779  7787 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.085s
I20260709 12:23:24.783190 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:24.811156 28337 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:24.776953 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:24.776131  7786 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.785887 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:24.779207 27660 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: [42.24-42.24]
I20260709 12:23:24.792243  7795 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:24.793777 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.816743 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:24.817529 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:24.818171 27732 raft_consensus.cc:3060] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:24.696201  7670 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.080s
W20260709 12:23:24.786859 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 90597.9us
I20260709 12:23:24.783511  7700 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:24.782370 27730 rpcz_store.cc:276] 0709 12:23:22.086512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.268406 (+181894us) service_pool.cc:224] Handling call
0709 12:23:24.203866 (+1935460us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b5800220298. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:24.204086 (+   220us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.204498 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.204531 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.205035 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.532711 (+327676us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:24.549919 (+ 17208us) log.cc:844] Serialized 14 byte log entry
0709 12:23:24.726698 (+176779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.726800 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:24.726861 (+    61us) raft_consensus.cc:1685] finished
0709 12:23:24.761811 (+ 34950us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.762011 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22812,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":33,"raft.queue_time_us":118716,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":14524,"spinlock_wait_cycles":52425472,"thread_start_us":493963,"threads_started":2,"wal-append.queue_time_us":374483}
I20260709 12:23:24.786389 28334 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:23:24.794148  7788 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.041s
I20260709 12:23:24.801329  7443 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:24.827138 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:24.827382  7666 consensus_meta.cc:287] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.000s
W20260709 12:23:24.782563 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 105007us
I20260709 12:23:24.820312 28295 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:24.821453  7776 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:23:24.784756  7745 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:24.815410  7795 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:24.759271  7705 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.832118 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:24.847952 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:24.840237 28337 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:24.841050 27732 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
W20260709 12:23:24.842800 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.858800 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.864707 27581 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:24.833884  7647 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:23:24.875660 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48595.1us
I20260709 12:23:24.854063  7700 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:24.847271  7795 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:24.888293 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51739.6us
I20260709 12:23:24.891089 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:24.891958 27585 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 38
W20260709 12:23:24.879011 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 102184us
W20260709 12:23:24.900499 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.855111  7788 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 43.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.017s
I20260709 12:23:24.900049 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.909102 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:24.900753  7784 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:24.913803 27662 raft_consensus.cc:1217] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Deduplicated request from leader. Original: 36.19->[38.20-41.22]   Dedup: 38.20->[39.21-41.22]
I20260709 12:23:24.920039 27662 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [38.20-41.22]
I20260709 12:23:24.914983 27581 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:24.864144  7745 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:24.871614  7667 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:24.926007 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:24.926905 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.877549  7554 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:24.900190  7800 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:24.820258  7786 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.911052  7791 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:23:24.930532 27737 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
W20260709 12:23:24.980394 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160608us
I20260709 12:23:24.981236 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:24.982266 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61915.8us
I20260709 12:23:24.932360  7594 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:24.964936 27585 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:24.910244  7785 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.913929  7797 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:25.016022 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:25.038491 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 161402us
W20260709 12:23:24.983356  7798 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:24.981112  7381 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:24.913022  7801 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.132s
I20260709 12:23:24.982487  7791 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:24.855648  7703 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.026372 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:24.986006  7393 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:25.056643 27585 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:25.041761 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:25.060205 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:25.039537 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:24.879076  7804 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.095s
I20260709 12:23:25.014962  7783 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:24.949816  7745 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:24.973150  7554 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:24.981647  7700 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.071s
I20260709 12:23:25.018097  7116 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.027032  7594 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:25.053270  7338 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.054873  7527 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:25.037985  7797 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:25.021392  7806 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:25.080628 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.062083  7641 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:25.092368  7806 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.093029  7641 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:25.085911 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:25.088120 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48770.1us
I20260709 12:23:25.067622  7800 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:23:25.096019 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.095942  7800 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:25.097333  7800 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.070421  7393 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:23:25.099467 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 222797us
W20260709 12:23:25.082628  7634 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.740s	user 0.010s	sys 0.002s
I20260709 12:23:25.142189 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:25.182626 27627 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:25.081321  7527 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.040956  7785 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:25.179836  7629 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.844s	user 0.010s	sys 0.001s
W20260709 12:23:25.203058 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 181384us
I20260709 12:23:25.100325  7816 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:25.082818  7594 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.074072  7783 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.183849  7789 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:25.194759  7634 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.212105  7672 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.902s	user 0.004s	sys 0.008s
I20260709 12:23:25.188344  7806 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.193645  7818 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:25.212440  7813 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:25.237049  7594 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.237679  7785 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.181306  7791 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.243754  7802 raft_consensus.cc:2804] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:25.242945 27487 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 31 to 34, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 34 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:25.244112  7672 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.179649  7443 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.236181  7786 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.222752  7824 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:25.276679 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.233557  7803 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:25.224905  7705 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.248154  7818 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:25.224211  7817 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.058s
I20260709 12:23:25.273294  7813 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.299703  7827 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.301841 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.309335 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.292441 27584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5039) took 1210 ms. Trace:
I20260709 12:23:25.309901 27584 rpcz_store.cc:276] 0709 12:23:24.082289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.084398 (+  2109us) service_pool.cc:224] Handling call
0709 12:23:24.084717 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.526287 (+441570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.292071 (+765784us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400187ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:25.292373 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920411008}
I20260709 12:23:25.312054  7745 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:25.312990  7673 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.975s	user 0.012s	sys 0.000s
I20260709 12:23:25.256733  7393 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.224395  7634 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.245358  7797 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.323778  7594 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:25.324754  7594 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.284814  7803 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.222275  7789 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.330341 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:25.333709 27737 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:25.336310  7594 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.338653 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:25.339298  7819 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.140s	user 0.002s	sys 0.000s
I20260709 12:23:25.341032  7672 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.341368  7736 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:25.341755  7736 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:25.342814  7527 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.342675  7736 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.329073 27584 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:25.343685  7736 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:25.343957 27584 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:25.345393  7654 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 1.048s	user 0.013s	sys 0.001s
I20260709 12:23:25.345216 27584 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:25.281553  7338 consensus_queue.cc:1175] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" responder_term: 44 status { last_received { term: 44 index: 26 } last_committed_idx: 26 last_received_current_leader { term: 44 index: 26 } }
W20260709 12:23:25.348846  7677 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.032s	user 0.008s	sys 0.006s
I20260709 12:23:25.296157  7818 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:25.334316  7803 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:25.354905 27731 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:25.356693 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:25.358134 27737 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:25.373004 27729 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 34 index: 22. (term mismatch)
I20260709 12:23:25.373642 27729 pending_rounds.cc:85] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 21
I20260709 12:23:25.373973 27729 pending_rounds.cc:107] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 33.22
I20260709 12:23:25.374286 27729 raft_consensus.cc:2889] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:25.380213  7630 consensus_meta.cc:287] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.885s	user 0.009s	sys 0.003s
I20260709 12:23:25.380393  7789 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.380904  7630 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:25.384613  7520 consensus_meta.cc:287] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 1.041s	user 0.011s	sys 0.000s
I20260709 12:23:25.385154  7520 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:25.387108  7568 consensus_meta.cc:287] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.712s	user 0.011s	sys 0.000s
I20260709 12:23:25.388937  7813 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.392334 27587 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: [38.24-38.24]
I20260709 12:23:25.286886  7824 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.396494  7824 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.326239  7827 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.294740  7795 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.404464  7827 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.415601 27738 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:25.427920  7630 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.428862  7520 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.429332 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4777) took 2481 ms. Trace:
I20260709 12:23:25.429744 27734 rpcz_store.cc:276] 0709 12:23:22.947484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.214248 (+266764us) service_pool.cc:224] Handling call
0709 12:23:23.214571 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.560955 (+346384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.429077 (+1868122us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b540017f7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:25.429269 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":792832384}
I20260709 12:23:25.431058  7816 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:25.432724 27586 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:25.433293  7795 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.434535 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:25.435200 27588 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:25.436866 27730 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
W20260709 12:23:25.451116 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.460145 27737 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:25.435550  7803 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.470652  7803 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.472267  7791 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:25.455528  7818 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.486449 27584 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:25.488389  7818 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.489370 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:25.506238 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:25.507861 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:25.511591 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:25.518617 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:25.519431 28337 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
W20260709 12:23:25.540849 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.552999 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.559067 27584 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:23:25.561517 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5051) took 1276 ms. Trace:
I20260709 12:23:25.562084 27580 rpcz_store.cc:276] 0709 12:23:24.285486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.286102 (+   616us) service_pool.cc:224] Handling call
0709 12:23:24.286355 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.346345 (+ 59990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.346381 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.346425 (+    44us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:24.346456 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.561251 (+1214795us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b540004f454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:25.561318 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.561444 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137526144}
W20260709 12:23:25.565976 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.555894  7386 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:25.570415  7386 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:25.572134  7792 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.533s	user 0.002s	sys 0.005s
I20260709 12:23:25.572889  7386 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.576555 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:25.569134  7537 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:25.583089 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2179) took 1255 ms. Trace:
I20260709 12:23:25.583668 27658 rpcz_store.cc:276] 0709 12:23:24.327942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.359717 (+ 31775us) service_pool.cc:224] Handling call
0709 12:23:24.373428 (+ 13711us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.373796 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.373840 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.495173 (+121333us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.799217 (+304044us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:24.799275 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.799409 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.571528 (+772119us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:25.579002 (+  7474us) raft_consensus.cc:1685] finished
0709 12:23:25.579034 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.583027 (+  3993us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76682,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":15219,"spinlock_wait_cycles":458954112,"thread_start_us":257637,"threads_started":1,"wal-append.queue_time_us":368960}
I20260709 12:23:25.568238  7528 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:25.582034 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:25.586268  7537 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:25.569370  7818 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.021s
I20260709 12:23:25.590833  7537 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.591874  7537 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:25.592541  7549 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:25.592962  7549 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:25.593351  7549 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:25.589792  7528 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.598763  7528 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.589745  7780 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.602515 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.578089  7811 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.022s	sys 0.036s
I20260709 12:23:25.603724  7780 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.614816  7573 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
W20260709 12:23:25.622202 28292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.005s	sys 0.021s
W20260709 12:23:25.616851  7811 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.023s	sys 0.036s
W20260709 12:23:25.616935 27552 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.002s	sys 0.024s
W20260709 12:23:25.625424 28292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.005s	sys 0.021s
W20260709 12:23:25.641945 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 580536us
I20260709 12:23:25.624135  7573 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:25.640079  7780 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:25.615775  7700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.016s	sys 0.008s
W20260709 12:23:25.644935 27552 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.002s	sys 0.024s
W20260709 12:23:25.648733 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 754538us
I20260709 12:23:25.656266  7780 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:25.658002 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 559365us
W20260709 12:23:25.649921  7700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.017s	sys 0.008s
W20260709 12:23:25.660231 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.662719 27660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.030s	sys 0.016s
I20260709 12:23:25.663757 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:25.657552  7802 raft_consensus.cc:697] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:25.646132  7788 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 25, Time since last communication: 0.017s
W20260709 12:23:25.668967 27660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.030s	sys 0.016s
I20260709 12:23:25.668900 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:25.669850 27661 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:25.671272 27655 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:25.670358 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:25.678057 27658 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:23:25.667649 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.679947 27662 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:25.660544  7780 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.682986 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:25.691215 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:25.689630 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:25.699609 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:25.706086 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.677661  7835 consensus_queue.cc:1059] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.021s
I20260709 12:23:25.721571  7820 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:25.721642  7835 consensus_queue.cc:1243] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 38.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:23:25.722108 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:25.723479 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:25.682860  7554 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.688177  7782 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:25.703307  7811 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:23:25.696594  7787 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:25.704371  7675 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:25.733615 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:25.737579 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:25.740955 27630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.005s	sys 0.027s
W20260709 12:23:25.742096 27630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.005s	sys 0.027s
W20260709 12:23:25.743294 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 539312us
I20260709 12:23:25.665932  7573 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.747265 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:25.763067 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:25.762948 27661 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:25.765674 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:25.736455  7655 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:25.772233 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:25.739847  7820 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:25.774272 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2212) took 1160 ms. Trace:
I20260709 12:23:25.774284 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:25.777463 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:25.779230 27587 rpcz_store.cc:276] 0709 12:23:24.614021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.614628 (+   607us) service_pool.cc:224] Handling call
0709 12:23:24.614936 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.392789 (+777853us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.774024 (+381235us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b5400089dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:25.774208 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1787146624}
I20260709 12:23:25.759003  7386 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:23:25.750682  7482 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:25.749399  7735 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:25.749397  7833 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:25.786506 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.785173 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:25.771767  7528 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:25.763358  7811 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:25.774294  7573 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:25.779083  7782 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:25.740345  7787 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.780526  7675 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:23:25.773960  7581 consensus_queue.cc:1175] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 42 status { last_received { term: 42 index: 23 } last_committed_idx: 23 last_received_current_leader { term: 42 index: 23 } }
I20260709 12:23:25.812835 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:25.788040  7820 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:25.788043  7482 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:25.796536  7791 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:23:25.818902 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.801697  7840 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:25.801154  7655 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:25.815043  7528 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.824101  7386 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:25.842494 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 152986us
I20260709 12:23:25.843127  7528 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:25.834817  7791 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.837471  7840 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:25.832933  7795 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:25.833201  7702 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:25.843652  7386 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.827142  7818 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.036s
I20260709 12:23:25.760823  7802 consensus_queue.cc:237] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 33, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.844157  7655 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.858934  7780 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:25.870313  7791 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:25.873023  7840 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:25.881757  7702 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:25.884979  7386 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:25.891914  7741 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 2.064s	user 0.013s	sys 0.007s
I20260709 12:23:25.907635  7549 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.864837  7482 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.912446  7549 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.825073  7833 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:25.923159  7844 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.073s	user 0.002s	sys 0.000s
I20260709 12:23:25.880995  7820 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:25.936853  7843 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:25.939657  7791 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.941107  7833 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:25.942679  7791 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:25.948324  7845 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:23:25.958662  7702 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:25.964798  7840 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:25.969084  7840 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:26.026625 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 249088us
I20260709 12:23:26.032317 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.034083 27580 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 38 index: 27. (index mismatch)
I20260709 12:23:26.035396 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.036263 27581 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 43
I20260709 12:23:26.037302 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:26.044070 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.038908 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.045378 27582 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:26.046826 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.047785 27582 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:26.049958 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.050866 27585 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 28.
I20260709 12:23:26.052743 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.057049 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.058405 27589 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:26.063683 27627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.021s	sys 0.024s
W20260709 12:23:26.064170 27627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.021s	sys 0.024s
W20260709 12:23:26.066987 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 884401us
I20260709 12:23:26.044423 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:25.800740 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.076288 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.079912 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4086) took 3318 ms. Trace:
I20260709 12:23:26.080426 27659 rpcz_store.cc:276] 0709 12:23:22.761311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.413278 (+651967us) service_pool.cc:224] Handling call
0709 12:23:23.413589 (+   311us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.413924 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.413952 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.423189 (+  9237us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.677337 (+254148us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:23.882405 (+205068us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.883248 (+   843us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.883359 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.823631 (+1940272us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:26.079189 (+255558us) raft_consensus.cc:1685] finished
0709 12:23:26.079344 (+   155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.079677 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":118705,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":569,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":33034,"spinlock_wait_cycles":2303872,"thread_start_us":278576,"threads_started":2,"wal-append.queue_time_us":317798}
I20260709 12:23:26.084105 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4113) took 1213 ms. Trace:
I20260709 12:23:26.086009 27664 raft_consensus.cc:1240] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:26.084568 27660 rpcz_store.cc:276] 0709 12:23:24.870354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.879640 (+  9286us) service_pool.cc:224] Handling call
0709 12:23:24.923726 (+ 44086us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.018392 (+ 94666us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.018438 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.037341 (+ 18903us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.669649 (+632308us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:25.691160 (+ 21511us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.692164 (+  1004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.692309 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.083422 (+391113us) raft_consensus.cc:1685] finished
0709 12:23:26.083472 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.083931 (+   459us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94226,"prepare.queue_time_us":26619,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":1738,"raft.queue_time_us":122819,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":12740,"spinlock_wait_cycles":18730880,"thread_start_us":631869,"threads_started":1,"wal-append.queue_time_us":770061}
I20260709 12:23:25.833900 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:26.093721 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 259809us
I20260709 12:23:26.085243 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:25.839483 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.939090  7655 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.150373  7583 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:26.165576  7655 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.926792  7443 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.167359  7569 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.168237  7569 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.171674  7549 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:26.173986  7840 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.148753  7522 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:26.174885  7840 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.176133  7522 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:26.176416  7837 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.176662  7522 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:26.177497  7837 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.178594  7840 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.166730  7655 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.178969 27581 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:26.174939  7444 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.195963  7444 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.198397  7655 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:25.928886  7482 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:26.179997  7818 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.930356  7656 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.200639  7569 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.232662  7583 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.013s
I20260709 12:23:26.260905  7444 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.249385  7573 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:26.258056  7675 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:23:26.277858 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.274245 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58060.3us
I20260709 12:23:26.282025  7603 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:26.258471  7549 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.027s
W20260709 12:23:26.286580 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4793) took 1789 ms (client timeout 2118 ms). Trace:
W20260709 12:23:26.288048 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.299278 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.311096  7554 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.312351  7554 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:26.287106 27738 rpcz_store.cc:269] 0709 12:23:24.496488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.531722 (+ 35234us) service_pool.cc:224] Handling call
0709 12:23:26.286141 (+1754419us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54000a50d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:26.286340 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1842604288}
I20260709 12:23:26.282764 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.311188  7837 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.301429 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.319363 27588 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:26.322979  7858 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:26.324023  7858 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.330740 27587 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:26.275542  7818 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.335882  7818 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.339890 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.340687 27580 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:23:26.341399  7837 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.344018  7742 raft_consensus.cc:3060] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:26.344269  7837 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.346830  7837 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:26.352161 27472 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4aa1523b9af84c3db99ba394e2b9aa9a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 33 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:26.332854  7858 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.376259  7445 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:26.377416  7445 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.378273  7783 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.379201  7783 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:26.380527  7802 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.382928  7838 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.384253  7443 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.384636 27660 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:23:26.392650 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.384827  7787 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.385274  7445 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.385918 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.386195 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.387746 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:26.387995  7554 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.390868 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.392791 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:26.396493 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.399989  7848 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.508s	user 0.002s	sys 0.001s
W20260709 12:23:26.400691 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 315743us
I20260709 12:23:26.361192  7573 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.404899  7443 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.405189  7838 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.406395  7802 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.408295 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.412945 27587 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:26.416993 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.429986 27585 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
W20260709 12:23:26.430676 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44975us
I20260709 12:23:26.429493 27730 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:26.438179 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.438980 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.439906 28335 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:26.440150  7807 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.440826 27588 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:26.441103  7807 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.436053  7787 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:23:26.444147  7787 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.444820  7787 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:26.436903 27659 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:26.407996  7522 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.457084 27661 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:26.458020 27587 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: [38.24-38.24]
W20260709 12:23:26.459203 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.438246  7670 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.204s	user 0.008s	sys 0.007s
I20260709 12:23:26.463773  7703 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.463860 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.464506  7637 raft_consensus.cc:3055] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:26.464843  7703 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.467667 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.468650 27657 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:26.464934  7637 raft_consensus.cc:740] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:26.463898 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.450423  7573 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.472754 27730 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:23:26.474363  7573 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.476100  7802 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.481436  7637 consensus_queue.cc:260] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.486912  7522 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.488737 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.447281  7783 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.491464  7783 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.491765  7831 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:26.492385  7831 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:26.492689 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.493818 27580 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:26.493623  7831 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 41.20, Last appended by leader: 20, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.494517 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.494763  7398 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:26.488899  7838 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.494059 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:26.498415  7634 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.495234 27589 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:26.495064 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:26.469969  7670 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.276s	user 0.009s	sys 0.008s
I20260709 12:23:26.504195 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.504594  7702 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.504662  7443 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.532641 28335 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:26.506155  7820 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:26.507875  7807 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.508399  7797 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:26.510205 28328 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:26.483274  7637 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:26.514629 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:26.517647 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.517551 27630 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.519585  7748 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.520025  7783 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:26.561088  7783 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:26.518872  7793 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:26.562714  7793 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.565618  7793 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.520833  7801 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.566589  7801 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.567126 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.568234 27734 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:26.568667  7788 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:23:26.569181  7801 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.521517  7581 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.523739  7860 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:26.524995 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.570227  7581 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.571116 27735 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:26.526252  7703 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:26.571765 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54415.2us
I20260709 12:23:26.569846 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.573207  7443 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:26.573823 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.573954  7443 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:26.576490  7827 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:26.576592 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.576938 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.535885 28333 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:26.577822  7581 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.578513 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:26.530224 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:26.582172 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.577602  7443 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.584190 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:26.585391 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.541095 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.543926 27736 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:26.544303 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.547272 27588 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:26.514344 28329 raft_consensus.cc:1240] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:26.559835 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.560204  7748 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.522208  7736 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.004s
I20260709 12:23:26.577039  7827 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.538527 28331 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:26.586403 27736 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:26.587276 27580 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 31.
I20260709 12:23:26.606443 27584 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:26.606679 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.606980  7581 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:26.607640  7581 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.608084 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:26.608911 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:26.610052 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:26.610216  7581 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:26.610806 27660 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:23:26.608217 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.611658 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.612082 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.613240  7828 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.060s
I20260709 12:23:26.617187 27736 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.618369 27736 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:26.618853  7702 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:26.619457  7702 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:26.620528  7702 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.621423  7748 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.622838 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.623133 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.623677 27587 raft_consensus.cc:2393] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:26.605101  7827 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:26.626546 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4882) took 2099 ms. Trace:
I20260709 12:23:26.620945 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5189) took 1597 ms. Trace:
I20260709 12:23:26.627821 27583 rpcz_store.cc:276] 0709 12:23:25.023018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.023129 (+   111us) service_pool.cc:224] Handling call
0709 12:23:26.619966 (+1596837us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800002a98. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97223b71 00007f0e97223a22 00007f0ea013c8d9 00007f0e9f296ee0 00007f0e9f3600ab 00007f0e9f360131 00007f0e9f321fc0 00007f0e9f30c929 00007f0e9f30c3d2 00007f0e98883bd4
0709 12:23:26.620280 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.620526 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.620552 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.620592 (+    40us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:26.620622 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.620671 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.620767 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2801024}
I20260709 12:23:26.626998 27664 rpcz_store.cc:276] 0709 12:23:24.527026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.742170 (+215144us) service_pool.cc:224] Handling call
0709 12:23:24.742550 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.086496 (+1343946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.626233 (+539737us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540005f854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:26.626485 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3088270720}
W20260709 12:23:26.614785  7528 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.751s	user 0.004s	sys 0.016s
I20260709 12:23:26.637027 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.642177 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.649585 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.655901 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.676815 27661 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:26.680087 27729 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:23:26.684566 28335 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:23:26.663564 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:26.693120 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:26.699815 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.675629 27704 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:26.673931  7820 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:26.709265 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:26.704754 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54385.1us
I20260709 12:23:26.712502 27735 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 22. (term mismatch)
W20260709 12:23:26.709321 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 53006.1us
I20260709 12:23:26.715231 27731 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:26.715566 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.652108  7528 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.712909 27735 pending_rounds.cc:85] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 21
I20260709 12:23:26.716879 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.722818  7673 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.723593  7673 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.724793 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:26.725978  7398 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:26.726452  7398 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:26.726919  7398 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:26.731407 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.732191 27729 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 32.
I20260709 12:23:26.735416 27584 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:23:26.735775  7398 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.738193  7398 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.738116 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:26.721102 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76942.3us
I20260709 12:23:26.726733 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:26.746512 28331 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:26.714516  7338 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:26.717886 27735 pending_rounds.cc:107] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 41.22
I20260709 12:23:26.751749 27735 raft_consensus.cc:2889] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:26.756350 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.756769  7820 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.758572  7679 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:26.752976  7338 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.759214  7679 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.759982 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.760789 27664 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:26.761662  7679 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.761731  7559 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:26.762295  7559 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.717691  7736 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:26.761663 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:26.777189  7736 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:26.780043 27736 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:23:26.780786 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 1951) took 2454 ms. Trace:
I20260709 12:23:26.781097 27736 rpcz_store.cc:276] 0709 12:23:24.326304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.326472 (+   168us) service_pool.cc:224] Handling call
0709 12:23:24.326824 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.543648 (+2216824us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54000477a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:23:26.780445 (+236797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.780713 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":479248256}
I20260709 12:23:26.782363 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.786226 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:26.786913 27734 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:26.791675 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.767551  7559 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:26.794857 27661 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
I20260709 12:23:26.769269  7528 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.771677 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.809041 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:26.809734  7788 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:23:26.810149  7482 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.004s
I20260709 12:23:26.810688 28334 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:26.772585 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:26.767968 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.759166  7820 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:26.776474  7866 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:23:26.735646  7748 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:26.777320 27664 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:26.762853 27732 raft_consensus.cc:2410] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 38 because replica has last-logged OpId of term: 36 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 19.
I20260709 12:23:26.767968 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:26.822515 27731 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:26.797333 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.778556  7736 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.797327  7444 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:26.803593 27656 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 36: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:26.801040 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.759204  7338 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:26.811041 28334 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:23:26.818782 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.819228 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 99764.4us
I20260709 12:23:26.820133 27660 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
W20260709 12:23:26.802858  7717 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 40 current term: 40 is higher.
I20260709 12:23:26.831452 27734 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:26.831841 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.833186 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.810173  7788 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:26.821899  7748 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.844695 27661 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
W20260709 12:23:26.845903 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.824141  7386 consensus_meta.cc:287] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.867s	user 0.011s	sys 0.003s
W20260709 12:23:26.835525 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 109030us
I20260709 12:23:26.848938 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:26.853658  7794 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.647s	user 0.006s	sys 0.003s
W20260709 12:23:26.855259  7794 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.653s	user 0.006s	sys 0.003s
I20260709 12:23:26.834219 27660 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:26.859179 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.862303 27660 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:26.863206 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:26.868535 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.887904 27582 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:26.888087 27584 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:26.869277  7386 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.869666 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.840318  7787 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.875434 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.892639 27737 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:26.893668 27588 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:23:26.894253 27664 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:26.875252  7231 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:26.881511 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:26.882145  7735 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:26.928437 27731 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:26.882853 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.930142 27661 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:26.883728 28334 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.890252 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.890575  7667 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.890287 27735 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:23:26.845175  7748 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:26.866842 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.895146 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.896291  7852 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.898344 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:26.836347  7826 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:23:26.904588 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:26.905443 28330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58246 (request call id 1888) took 1820 ms. Trace:
I20260709 12:23:26.941440 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:26.907266 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.912206  7787 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.912825 27657 raft_consensus.cc:2393] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 29: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:26.913085  7549 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:26.913924  7790 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.916528  7679 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.915055 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:26.936048 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54719.1us
I20260709 12:23:26.932199 27663 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:26.940263 27737 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:26.941537 27730 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:26.945269 28334 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:26.942296 28330 rpcz_store.cc:276] 0709 12:23:25.085431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.085581 (+   150us) service_pool.cc:224] Handling call
0709 12:23:26.905072 (+1819491us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54005d93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:26.905248 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4159254272}
I20260709 12:23:26.952148 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.952765  7679 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.955147  7809 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:26.955492 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.956789 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:26.955211  7790 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.962795  7790 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.963581 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:26.964543 27655 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:26.965359  7231 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:26.965584  7852 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:26.966511  7231 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:26.966993  7231 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:26.934729  7443 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.968056  7443 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:26.968631  7852 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.969537  7833 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:26.970082  7833 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:26.970543  7833 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:26.970745  7443 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:26.965392  7697 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:26.973896 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.974035 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.974853 27585 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:26.975486 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.975968 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:26.976662 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.978713 28331 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:26.966101 27655 raft_consensus.cc:1217] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Deduplicated request from leader. Original: 38.26->[39.27-39.27]   Dedup: 39.27->[]
I20260709 12:23:26.980305 27655 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40. Ops: [39.27-39.27]
I20260709 12:23:26.982146 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.982152 27657 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:26.984681 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.985544 27663 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:26.987933 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:26.990362 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:26.930956  7482 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.052s
I20260709 12:23:27.012253 27731 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 45: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:27.003463 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.003710  7679 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.005070 27657 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [41.23-41.23]
I20260709 12:23:27.006326  7881 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:26.978960  7794 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:23:27.007216 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.017374 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.007653 27660 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:26.944105  7797 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.064s
I20260709 12:23:27.008284 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:27.010021 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:27.010155 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.002758 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:26.989398 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.012861 27589 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:23:27.014657 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47970.1us
I20260709 12:23:27.016296 27583 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:27.007675  7510 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.014144  7581 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.018256 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
W20260709 12:23:27.032967 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 92908.6us
I20260709 12:23:27.034847 27663 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42. Ops: [41.23-41.23]
I20260709 12:23:27.024761  7510 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.036445  7338 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.038295  7510 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.029529  7573 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:27.039763  7573 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.040225  7573 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:27.042716  7231 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.044462  7231 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:27.048113  7386 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.048381  7393 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:27.048895  7393 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.049324  7393 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:27.051640 28334 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 38 index: 27. (index mismatch)
W20260709 12:23:27.052376  7719 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 38 current term: 38 is higher.
I20260709 12:23:27.053123 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:27.054618 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:27.055565  7794 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:27.029588  7581 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.056807 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.057646 27731 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:27.058014  7581 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.058610  7581 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.058714 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.059520 27737 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:27.060981 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.066274 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.060233 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1889) took 1838 ms. Trace:
I20260709 12:23:26.955909  7809 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.067602 28334 rpcz_store.cc:276] 0709 12:23:25.224838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.237469 (+ 12631us) service_pool.cc:224] Handling call
0709 12:23:25.237779 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.810395 (+1572616us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b5400611820. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:23:27.052056 (+241661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.060134 (+  8078us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":997766400}
I20260709 12:23:27.016039  7881 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.038987  7338 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.060075  7634 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:23:27.055073  7833 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.083657 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.084429  7833 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.085105 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.085757 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.086802 27587 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 44 because replica has last-logged OpId of term: 42 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:23:27.091930 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.092690 27738 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 43.
I20260709 12:23:27.056852  7794 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.096179 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:27.096525 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.097328  7634 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:27.097846  7634 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:27.060596  7801 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:27.098392  7634 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:27.098788 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.099476  7881 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:27.100368  7637 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.100989  7637 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.102723 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:26.950371  7667 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.059263  7581 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:27.081984  7875 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.137s	user 0.004s	sys 0.000s
I20260709 12:23:27.060878  7793 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:27.103229  7719 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.983868  7735 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.106729  7727 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.005s
I20260709 12:23:27.070775  7482 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:27.103312  7637 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:26.981277  7559 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.120441 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.118206  7700 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.007s
I20260709 12:23:27.081329  7827 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.134904 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:27.124027 27702 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.136652 27658 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
W20260709 12:23:27.155395 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56715.2us
I20260709 12:23:27.119741  7697 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:27.160463 27661 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:27.120122  7482 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:27.213220  7482 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:27.123692  7637 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:27.165369 27664 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:27.168263  7717 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.168942  7641 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:27.249624  7809 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.250051  7641 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.250519  7641 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:27.170852  7581 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.252624 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.171150  7735 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.174100  7393 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.253744 27737 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 43 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:23:27.176767  7667 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.107391  7444 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.046s
I20260709 12:23:27.254909 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.183274 27487 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 34 to 42. New cstate: current_term: 42 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.188077 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.188433 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.188886  7690 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:27.191082 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.192866 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.193015  7827 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.194722 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.159755  7793 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:27.107689  7797 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.046s
I20260709 12:23:27.196533 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.197548  7783 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.198351 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.200009 27580 raft_consensus.cc:1240] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:27.200358  7881 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.206849  7559 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.207418 27662 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: []
I20260709 12:23:27.121807  7807 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.091s
I20260709 12:23:27.121109 27589 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 29.
I20260709 12:23:27.317874 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.221228 27584 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:27.236800 27582 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:27.243402 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.222620 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.243165  7637 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.249301 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:27.322454 27658 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
W20260709 12:23:27.322674 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 100166us
I20260709 12:23:27.249112  7717 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.179706 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 7274fd8a2115405c9b78c61214bff736 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:51028
I20260709 12:23:27.149550  7801 consensus_queue.cc:1059] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.106s
I20260709 12:23:27.165093  7820 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.091s
I20260709 12:23:27.255923 27660 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: [39.25-39.25]
I20260709 12:23:27.155066  7833 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.101s
I20260709 12:23:27.157791  7700 consensus_queue.cc:1243] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 36.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:23:27.173895 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.336302 27659 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:27.337776 27659 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:27.333945  7735 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:27.338411  7735 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.338750  7735 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:27.257750 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.339751 27735 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:27.340646  7634 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.341971  7381 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.341729  7573 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.342545  7381 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.258903  7748 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.343595 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.344899  7748 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.345265 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.264060 27737 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:27.347061 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.347713 27583 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:27.348747 27730 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:27.353318 27735 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:27.354070 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.353515  7788 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.354876 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:27.294546 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.355736 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:27.356709 27731 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:27.357014 27655 raft_consensus.cc:1240] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:27.357707 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:27.357802 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.358382  7783 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:27.358882 27731 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 44 because replica has last-logged OpId of term: 42 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
W20260709 12:23:27.269073 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.359120 27735 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:27.361068  7393 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.265601  7690 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:27.363379  7690 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.364825  7690 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:27.329015 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.368208  7851 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:27.369263 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:27.370625 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.271564  7793 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:27.372453  7793 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:27.373468 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.374245 27657 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:27.376196 27628 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:27.377516  7827 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:27.378078  7827 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.378826  7784 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:27.380246  7801 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:27.380640  7827 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.380797  7801 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:27.381138 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:27.332140 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.387703 27584 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:27.330528 27487 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 32 to 35. New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.322011  7637 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:23:27.267544 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109685us
I20260709 12:23:27.391150 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.392107 27584 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:27.392189 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.392988 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.355561 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:27.273087 27732 raft_consensus.cc:2393] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 45: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:27.357980 27478 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 238) took 77422 ms (client timeout 15000 ms). Trace:
I20260709 12:23:27.382177  7801 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.397245  7801 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:27.397406  7444 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:27.398027  7444 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:27.394392 27478 rpcz_store.cc:269] 0709 12:22:09.912121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.912459 (+   338us) service_pool.cc:224] Handling call
0709 12:23:27.322264 (+77409805us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:17.807763 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:17.808534 (+   771us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:17.808602 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:23:17.808635 (+    33us) tablet.cc:662] Decoding operations
0709 12:23:17.810628 (+  1993us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:17.810746 (+   118us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:17.810784 (+    38us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:23:17.827268 (+ 16484us) tablet.cc:701] Row locks acquired
0709 12:23:17.827340 (+    72us) write_op.cc:260] PREPARE: finished
0709 12:23:17.827517 (+   177us) write_op.cc:270] Start()
0709 12:23:17.827783 (+   266us) write_op.cc:276] Timestamp: P: 1783599797827452 usec, L: 0
0709 12:23:17.827831 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 12:23:17.828669 (+   838us) log.cc:844] Serialized 7835 byte log entry
0709 12:23:18.464786 (+636117us) op_driver.cc:464] REPLICATION: finished
0709 12:23:18.929634 (+464848us) write_op.cc:301] APPLY: starting
0709 12:23:18.929743 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:19.291623 (+361880us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:19.291794 (+   171us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:19.388793 (+ 96999us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:19.389159 (+   366us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=0,mrs_lookups=0
0709 12:23:19.389223 (+    64us) write_op.cc:312] APPLY: finished
0709 12:23:19.389745 (+   522us) log.cc:844] Serialized 180 byte log entry
0709 12:23:19.416814 (+ 27069us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:19.417003 (+   189us) write_op.cc:454] Released schema lock
0709 12:23:19.417084 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:19.515501 (+ 98417us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:19.516142 (+   641us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":84298,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":349545,"apply.run_cpu_time_us":39391,"apply.run_wall_time_us":538366,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87727,"num_ops":20,"prepare.queue_time_us":263520,"prepare.run_cpu_time_us":54505,"prepare.run_wall_time_us":138743,"raft.queue_time_us":273002,"raft.run_cpu_time_us":36412,"raft.run_wall_time_us":423026,"replication_time_us":578322,"spinlock_wait_cycles":3328,"thread_start_us":632318,"threads_started":5,"wal-append.queue_time_us":266374,"wal-append.run_cpu_time_us":9745,"wal-append.run_wall_time_us":2270546}]]}
I20260709 12:23:27.393245 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.394209 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:27.401532 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.402493 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.156728  7719 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.369065  7851 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:27.327772 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:27.298997 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.363379 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.361193 27735 raft_consensus.cc:1275] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:23:27.347074 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.416818 27580 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 43 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
W20260709 12:23:27.416849 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.355506 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.418850 27738 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 43 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:23:27.412788 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.332841 27589 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:27.436823 27584 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:27.437662 27735 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:27.353576  7673 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.421180  7828 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:27.441485  7828 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.441992  7828 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:27.442279  7381 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:27.443975 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 88532.3us
I20260709 12:23:27.447503  7719 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.448499 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.449951 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.451422  7742 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:27.451965  7742 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:27.452469  7851 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.452900  7742 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.453953  7851 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:27.456346 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.461473  7717 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.463052  7881 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:27.467610 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 63692.2us
I20260709 12:23:27.467765  7573 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.468503 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:27.421952  7748 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:27.452492 27487 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 31 to 37. New cstate: current_term: 37 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.471274  7748 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.470587  7863 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:23:27.470546 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:27.473899  7827 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:27.474298 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:27.435114  7581 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:27.475620 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:27.471782  7748 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:27.475731  7581 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:27.474402  7827 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:27.475886 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:27.471580 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 75356.7us
I20260709 12:23:27.478401 27701 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.486564 27583 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:27.486057 28328 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:27.493522 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:27.494438 28331 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:27.463425  7559 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:23:27.496727 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 76211.1us
W20260709 12:23:27.487291 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.499922 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:27.505089 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.495648 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:27.485450  7581 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:27.505831 27738 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:27.511082 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.505300 27630 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:27.511686 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.507375 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.491022  7788 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:27.479348  7807 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.518851 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:27.532716 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 470116us
I20260709 12:23:27.521181 27589 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: []
I20260709 12:23:27.534886 27580 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
W20260709 12:23:27.535790 27586 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.005s	sys 0.017s
W20260709 12:23:27.536259 27586 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.005s	sys 0.017s
I20260709 12:23:27.537782 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:27.487664  7679 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:27.520567 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:27.515749  7381 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:27.531723  7790 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:23:27.537148  7794 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:27.502045  7393 consensus_queue.cc:1059] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.039s
I20260709 12:23:27.487490  7667 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:23:27.511055  7634 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.547297 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:27.539150 27734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.042s	sys 0.046s
I20260709 12:23:27.536355  7783 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:27.521425  7559 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:27.549170  7673 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.009s
I20260709 12:23:27.534863  7641 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.549383  7794 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.534463  7717 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:27.517761  7852 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.549489  7381 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:27.541635  7736 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:27.566388 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 89960.8us
I20260709 12:23:27.564801  7784 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:27.578477 27487 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 38, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 38 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:27.537583  7807 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.559388  7790 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:27.616741 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.631125 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:27.567775 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 444918us
I20260709 12:23:27.577235 28334 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
W20260709 12:23:27.632032 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46497us
I20260709 12:23:27.578783 27730 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:23:27.657686 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.584117 27582 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:23:27.631122 28331 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:27.571579 28328 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
W20260709 12:23:27.585664 27734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.043s	sys 0.046s
I20260709 12:23:27.504408  7444 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.547394  7735 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.642030 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 30 to 35. New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.566758  7802 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:27.569445  7783 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.666641 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.655851 27630 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:27.674561 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129693us
I20260709 12:23:27.577046  7794 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:27.630976  7881 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:23:27.665788  7848 log.cc:927] Time spent T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:23:27.568118 27733 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.012s	sys 0.010s
W20260709 12:23:27.688162 27733 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.012s	sys 0.010s
I20260709 12:23:27.645017 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.679900 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.634397  7736 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.579536  7717 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.576803  7641 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.683838 27738 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:27.688299 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.690903 27657 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:23:27.691588 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:27.690568 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 22 to 27. New cstate: current_term: 27 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.703998 28333 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 45: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:27.665788  7784 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:27.707726 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:27.709237 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78192.2us
I20260709 12:23:27.579685  7793 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.718204 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:27.719377 27585 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 35. Current term is 36. Ops: [35.21-35.21]
I20260709 12:23:27.719278 27732 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:27.720834 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.685500  7783 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:27.641187  7637 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.737465 27586 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39. Ops: [38.24-38.24]
I20260709 12:23:27.733258 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.738417 27655 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 44.
I20260709 12:23:27.732434 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 40 to 43. New cstate: current_term: 43 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.721627 27657 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
I20260709 12:23:27.729280 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:27.694854  7863 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:27.694094  7736 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:27.714641 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:27.710202  7670 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:27.712313  7832 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:27.705369  7717 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:27.569270  7827 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.749879 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:27.786566 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.789052 27585 raft_consensus.cc:2393] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:27.789907 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:27.787833 27487 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 25 to 31, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 31 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:27.761827  7831 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:27.762854 27734 raft_consensus.cc:1240] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:23:27.764096 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:27.765070 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.787629 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:27.791106 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82646.8us
I20260709 12:23:27.779949  7832 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.707706  7679 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.093s
W20260709 12:23:27.819453  7883 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:23:27.784556 27732 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:27.723549  7897 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.767239  7863 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:27.790036 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:27.759431 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67091us
I20260709 12:23:27.790804 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.693696  7852 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:27.801517 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.770241 27735 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:27.828759 27487 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 33 to 36, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 36 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.795756  7338 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:23:27.809309 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.811012  7833 raft_consensus.cc:3055] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:27.810709  7667 consensus_queue.cc:1059] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:27.811385  7629 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:27.798902  7831 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:27.812582  7549 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:27.816043  7697 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:27.824934  7735 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.826064 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:27.826655  7637 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:27.788744  7670 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:27.858176 27734 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:27.791505 27657 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:27.788182  7797 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.042s
I20260709 12:23:27.812492  7793 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:27.820623  7832 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:27.769263  7559 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:27.855546  7890 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:23:27.853820 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:27.799237  7381 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.835973  7863 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:27.886912 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:27.892060 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.835904  7828 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.838320  7833 raft_consensus.cc:740] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:27.861300 27487 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 32. New cstate: current_term: 32 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:27.790441 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.852285  7831 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:27.831740  7581 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.799635  7790 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.896327 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:27.835173  7897 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.897766 27581 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:23:27.901494 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:27.865582  7670 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.920639 27660 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:27.921164 27655 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:27.751314 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:27.857429  7629 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:27.928726 27487 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 25 to 30. New cstate: current_term: 30 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.936323 27734 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
I20260709 12:23:27.873669  7748 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:27.849424  7784 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:27.921002  7482 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.707s	user 0.013s	sys 0.000s
I20260709 12:23:27.919142  7807 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:27.970582 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:27.950038 27581 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:23:27.960060 27661 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
W20260709 12:23:27.962628 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 153951us
I20260709 12:23:27.975521 27582 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:27.969882 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:27.979633 27660 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:27.953405  7703 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.956066  7667 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:27.981240 27487 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 33 to 36. New cstate: current_term: 36 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.974133  7581 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:27.989238 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 238018us
I20260709 12:23:27.976711  7482 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:28.000587  7905 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:23:28.001863  7482 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.002465 27584 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:27.989033  7703 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.009676  7703 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.012645  7811 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:27.977221  7833 consensus_queue.cc:260] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.014201  7703 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.014286  7833 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:27.988857  7892 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.205s	user 0.001s	sys 0.001s
I20260709 12:23:28.015035  7703 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.016786 27656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2197) took 1400 ms. Trace:
I20260709 12:23:28.017239  7703 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.018334  7801 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:28.019068  7801 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:28.022574  7903 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:23:28.027714  7851 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.573s	user 0.010s	sys 0.003s
I20260709 12:23:28.013096  7783 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.030160 28333 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 21. (term mismatch)
I20260709 12:23:28.030633 28333 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 20
I20260709 12:23:28.030833 28334 raft_consensus.cc:2368] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:28.030876 28333 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 35.21
I20260709 12:23:28.031584 28333 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:27.984862  7807 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:28.037681  7444 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:28.042160  7887 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
W20260709 12:23:28.043893 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 98042.7us
I20260709 12:23:28.044819 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:28.049413 27657 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:28.055168 27664 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: [38.24-38.24]
I20260709 12:23:28.055433 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:28.060699  7748 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.017141 27656 rpcz_store.cc:276] 0709 12:23:26.615896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.967626 (+351730us) service_pool.cc:224] Handling call
0709 12:23:26.968067 (+   441us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:27.016468 (+ 48401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.016520 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:27.064228 (+ 47708us) log.cc:844] Serialized 52 byte log entry
0709 12:23:27.515040 (+450812us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:27.515076 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.515136 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.016536 (+501400us) raft_consensus.cc:1685] finished
0709 12:23:28.016558 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.016725 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44020,"raft.queue_time_us":49234,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7654,"thread_start_us":450364,"threads_started":1,"wal-append.queue_time_us":615190}
I20260709 12:23:28.057125 27487 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 29 to 36, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 36 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:28.066296 27660 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
W20260709 12:23:28.068434  7888 log.cc:927] Time spent T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:23:28.068915 27656 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 42 index: 22. (index mismatch)
I20260709 12:23:28.069989  7673 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.074244  7788 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.542s	user 0.007s	sys 0.001s
W20260709 12:23:28.077126 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 290607us
W20260709 12:23:28.098791 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 108751us
I20260709 12:23:28.080883  7637 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:23:28.081979  7902 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260709 12:23:28.082895 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.083808  7581 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.085646 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:28.086126  7573 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:28.111717  7573 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.112895 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 30 to 36, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 36 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:28.086953 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.116319  7573 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.074123  7801 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.119912  7801 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:28.087733 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:28.121085 28330 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:28.124727  7717 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.125531 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:28.125946 27734 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:28.127455  7742 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:28.129711  7717 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.132092  7832 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.091514 27733 raft_consensus.cc:1240] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:28.096037 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4814) took 1329 ms. Trace:
I20260709 12:23:28.216377 27736 rpcz_store.cc:276] 0709 12:23:26.761510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.783157 (+ 21647us) service_pool.cc:224] Handling call
0709 12:23:26.794043 (+ 10886us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.821949 (+ 27906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.821981 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:26.843941 (+ 21960us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.422806 (+578865us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:27.500888 (+ 78082us) log.cc:844] Serialized 14 byte log entry
0709 12:23:27.505266 (+  4378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.505334 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.090976 (+585642us) raft_consensus.cc:1685] finished
0709 12:23:28.090997 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.091124 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10405,"prepare.queue_time_us":77946,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":5211,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":2944,"thread_start_us":472690,"threads_started":1,"wal-append.queue_time_us":899655}
I20260709 12:23:28.099552 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:28.110543  7581 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.104931 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:28.110419  7673 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.133018  7700 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:23:28.223042  7700 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:23:28.223738  7904 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.225s	user 0.002s	sys 0.000s
I20260709 12:23:28.224704  7673 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.224242  7700 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.134099 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 7274fd8a2115405c9b78c61214bff736 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:23:28.137301 27702 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:28.138799  7828 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.139504  7797 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:28.139047 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52192us
I20260709 12:23:28.140094  7897 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.140142  7697 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:28.141440 27628 leader_election.cc:400] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
W20260709 12:23:28.142736 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.232226 27628 leader_election.cc:403] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:28.143090  7811 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:28.143837  7807 consensus_queue.cc:1059] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:28.149246 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4916) took 1104 ms. Trace:
I20260709 12:23:28.103874  7820 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
W20260709 12:23:28.150785  7910 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:23:28.154752  7907 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260709 12:23:28.157390  7898 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.348s	user 0.000s	sys 0.001s
W20260709 12:23:28.165599  7886 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.297s	user 0.000s	sys 0.001s
I20260709 12:23:28.114997 27487 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 21 to 27, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 27 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:28.167142 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122415us
I20260709 12:23:28.168231  7629 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:28.169238  7913 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:28.169929  7831 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.090590 27660 raft_consensus.cc:1275] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:23:28.261425  7559 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:23:28.176316  7703 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.177707  7735 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:28.178845  7641 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:28.182104 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.183197  7554 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:28.183586 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.184340 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.185602 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.188987  7852 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.192759  7783 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.226166 27702 leader_election.cc:403] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Cancelling election due to peer responding with higher term
W20260709 12:23:28.303398 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 166280us
W20260709 12:23:28.230487 27471 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96529.4us
I20260709 12:23:28.250846 27738 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: [42.22-42.22]
W20260709 12:23:28.257809 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 116561us
I20260709 12:23:28.230819 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:28.261355 28332 raft_consensus.cc:1240] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:28.261750 27659 rpcz_store.cc:276] 0709 12:23:27.045075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.338613 (+293538us) service_pool.cc:224] Handling call
0709 12:23:27.338945 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.339308 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.347416 (+  8108us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.348096 (+   680us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.833315 (+485219us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:27.845909 (+ 12594us) log.cc:844] Serialized 14 byte log entry
0709 12:23:27.864871 (+ 18962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.865946 (+  1075us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.149001 (+283055us) raft_consensus.cc:1685] finished
0709 12:23:28.149024 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.149169 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14088,"prepare.queue_time_us":12329,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":917,"raft.queue_time_us":18694,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":23874,"thread_start_us":484840,"threads_started":1,"wal-append.queue_time_us":576083}
I20260709 12:23:28.236553  7808 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:28.280597 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2202) took 1382 ms. Trace:
I20260709 12:23:28.285542 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:28.287892 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 188403us
I20260709 12:23:28.282922 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:28.284756  7896 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.478s	user 0.004s	sys 0.000s
I20260709 12:23:28.281253  7700 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:28.302127 27581 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.304883 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:28.297538 27487 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 32, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:28.273571  7736 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.257295  7797 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.319669 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.296826  7554 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:28.348001 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:28.295217  7788 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.043s
I20260709 12:23:28.350009 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.307060  7381 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election lost for term 44. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:28.318958  7820 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.375422 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4130) took 1479 ms. Trace:
I20260709 12:23:28.369860 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:28.376860 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5234) took 1254 ms. Trace:
I20260709 12:23:28.320734  7828 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:28.312065 27658 rpcz_store.cc:276] 0709 12:23:26.897792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.334964 (+437172us) service_pool.cc:224] Handling call
0709 12:23:27.375726 (+ 40762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.376199 (+   473us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:23:27.376540 (+   341us) log.cc:844] Serialized 14 byte log entry
0709 12:23:27.829427 (+452887us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.863778 (+ 34351us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.863956 (+   178us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:28.223024 (+359068us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:28.274240 (+ 51216us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.275072 (+   832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.275178 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.280330 (+  5152us) raft_consensus.cc:1685] finished
0709 12:23:28.280355 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.280533 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6862,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":2551,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":5728,"spinlock_wait_cycles":76342144,"thread_start_us":844824,"threads_started":2,"wal-append.queue_time_us":574081}
I20260709 12:23:28.355916 27587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5228) took 1372 ms. Trace:
W20260709 12:23:28.346555  7827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.018s	sys 0.012s
I20260709 12:23:28.352134  7573 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:28.374145  7919 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:23:28.319860  7690 raft_consensus.cc:3055] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:28.361626 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:28.362216 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49651.9us
I20260709 12:23:28.407001  7690 raft_consensus.cc:740] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:28.407833 27627 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:28.364179 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:28.354465 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 123697us
I20260709 12:23:28.408289  7690 consensus_queue.cc:260] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.409749  7690 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:28.307224  7703 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.411219 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:28.315351  7808 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:28.347375  7807 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.412990  7703 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:28.412824  7807 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.371551  7697 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:28.415192 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:28.415705 27588 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 38 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
I20260709 12:23:28.416939  7807 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:28.378798 27583 rpcz_store.cc:276] 0709 12:23:27.122780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.539704 (+416924us) service_pool.cc:224] Handling call
0709 12:23:27.540100 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:27.543673 (+  3573us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:23:27.573080 (+ 29407us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b480030c3d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:27.866092 (+293012us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.226167 (+360075us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.226481 (+   314us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:28.256778 (+ 30297us) log.cc:844] Serialized 52 byte log entry
0709 12:23:28.256941 (+   163us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:28.256975 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.257049 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.281549 (+ 24500us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b18003e32c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:28.293555 (+ 12006us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b18003e32c0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:28.376424 (+ 82869us) raft_consensus.cc:1685] finished
0709 12:23:28.376458 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.376795 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2666,"prepare.queue_time_us":238446,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":99,"raft.queue_time_us":4657,"raft.run_cpu_time_us":2917,"raft.run_wall_time_us":43660,"spinlock_wait_cycles":103699968,"thread_start_us":607489,"threads_started":2,"wal-append.queue_time_us":417476}
W20260709 12:23:28.385422 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65856.6us
I20260709 12:23:28.419818 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.380039 27587 rpcz_store.cc:276] 0709 12:23:26.983674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.087420 (+103746us) service_pool.cc:224] Handling call
0709 12:23:27.087794 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.116480 (+ 28686us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.116530 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.150983 (+ 34453us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.646816 (+495833us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:27.646890 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.646985 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.156605 (+509620us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:28.348938 (+192333us) raft_consensus.cc:1685] finished
0709 12:23:28.348959 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.355843 (+  6884us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28208,"raft.queue_time_us":28989,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4274,"spinlock_wait_cycles":24931584,"thread_start_us":463540,"threads_started":1,"wal-append.queue_time_us":585579}
W20260709 12:23:28.421482  7863 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.512s	user 0.003s	sys 0.007s
I20260709 12:23:28.397279  7828 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:28.412372 27704 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:28.421921  7863 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.358325 27487 catalog_manager.cc:5697] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 28 to 34, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 34 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:28.422938  7828 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.424860 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:28.425585  7673 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:28.384690  7820 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.426375  7554 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.426904 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:28.427817  7673 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:28.427855  7554 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:28.428380  7673 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.428833  7673 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:28.428773 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:28.341367  7634 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:28.394188  7581 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:28.369691  7918 log.cc:927] Time spent T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:28.434832  7863 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.447448  7832 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.361075 27659 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 31. Current term is 32. Ops: [31.22-31.22]
I20260709 12:23:28.377972 27663 rpcz_store.cc:276] 0709 12:23:26.896108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.326482 (+430374us) service_pool.cc:224] Handling call
0709 12:23:27.379795 (+ 53313us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:27.411687 (+ 31892us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.411729 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:27.436015 (+ 24286us) log.cc:844] Serialized 52 byte log entry
0709 12:23:27.789341 (+353326us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:27.789384 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.789462 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.153787 (+364325us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:28.375107 (+221320us) raft_consensus.cc:1685] finished
0709 12:23:28.375128 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.375362 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71756,"raft.queue_time_us":31635,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":5619,"spinlock_wait_cycles":5253888,"thread_start_us":337382,"threads_started":1,"wal-append.queue_time_us":514812}
I20260709 12:23:28.402583 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:28.352303  7811 consensus_queue.cc:1059] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.052s
I20260709 12:23:28.341516  7338 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:23:28.454622  7338 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:28.454998  7338 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:28.455175 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.456207 27580 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.455842  7338 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.457154  7338 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:28.398671  7794 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.000s
I20260709 12:23:28.412277  7808 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:28.341172  7231 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:28.417711  7736 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:28.418424 27588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 5082) took 1264 ms (client timeout 1610 ms). Trace:
I20260709 12:23:28.423017  7381 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:28.422057  7801 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:28.470450  7381 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.470921  7381 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:28.436120  7703 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:28.437780 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:28.472429  7797 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.435133  7788 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.472939 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:28.473292  7788 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.348559  7667 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:28.474255 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:28.474745  7667 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.475481  7634 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:28.381662  7897 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:23:28.460580  7697 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:28.439137  7637 consensus_queue.cc:1059] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:23:28.439118 27587 raft_consensus.cc:1240] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:28.476845  7667 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:28.441838 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.477238  7637 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:28.442905 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:28.478627  7629 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:23:28.479044  7629 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:28.479476 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.479923  7629 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.480774 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.481003 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.481261  7629 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:23:28.481523 27587 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:28.468729 27588 rpcz_store.cc:269] 0709 12:23:27.154400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.270138 (+115738us) service_pool.cc:224] Handling call
0709 12:23:28.418232 (+1148094us) spinlock_profiling.cc:243] Waited 1e+03 ms on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:28.418344 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2299534720}
I20260709 12:23:28.483512  7549 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:28.483923  7549 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:28.484221  7510 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:28.484362  7549 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:28.483538  7922 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:23:28.438817  7921 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.072s	user 0.000s	sys 0.002s
I20260709 12:23:28.388583  7482 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:28.487509  7482 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:28.445896 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:28.488268 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:28.488430  7482 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.489636  7482 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:28.478504  7735 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.491402  7735 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.445266  7831 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.493506  7811 raft_consensus.cc:3055] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:28.493949  7811 raft_consensus.cc:740] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:28.477345  7697 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:28.469133 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 105031us
I20260709 12:23:28.496030 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:28.339712  7670 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.094s
I20260709 12:23:28.428617  7820 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:28.397626  7573 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:28.476164  7634 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:28.477933 27583 raft_consensus.cc:2393] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 32: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.468747  7231 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:28.443889  7790 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:23:28.389524  7827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.019s	sys 0.012s
I20260709 12:23:28.482981 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:28.458907  7794 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:28.536736 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 74631.8us
I20260709 12:23:28.556250 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.556311 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:28.537380 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.559417 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:28.560125 28334 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:28.561497 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.565398 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.495160  7735 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.571579 27580 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:28.568501 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:28.570535 27588 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:28.570680 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:28.579978 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.580776 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.537533  7573 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:28.494833  7811 consensus_queue.cc:260] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 33.25, Last appended by leader: 25, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.586336 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:28.586634 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 165463us
I20260709 12:23:28.506210  7833 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:28.589167 27586 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.589917 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.591105 27734 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.592972  7735 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:28.467725  7444 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.100s
I20260709 12:23:28.546689  7231 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:28.567562  7788 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.596843 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:28.544447  7801 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.054s
I20260709 12:23:28.551837  7790 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:28.601228 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.627415 27581 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:28.606774 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.588312  7897 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:28.599578 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:28.619293 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.582362  7783 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:28.586421  7637 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.628029 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:28.594110 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:28.655292 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:28.657996 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58474.1us
I20260709 12:23:28.605465  7667 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:28.648101 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.666621 27738 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:28.666531 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.615056  7703 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:28.679478 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.679605 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.614727  7833 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:28.650749  7790 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:23:28.663542 27701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.610s (SENT) [suppressed 7 similar messages]
W20260709 12:23:28.700695 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72755.2us
I20260709 12:23:28.703395 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.709067 27580 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.681352 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.713141 27583 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:28.673861  7811 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:28.714303 28295 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.674537  7852 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:28.727502  7852 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.727972  7852 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:28.724808  7828 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:28.732429 27583 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:28.733850 27583 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:23:28.731211  7828 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.742059  7828 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.668486  7444 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:28.668511  7393 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:28.679280  7581 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:28.705971  7717 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.669935  7667 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:28.713466  7783 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:28.723268  7735 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:28.753547 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:28.753434 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 270488us
I20260709 12:23:28.756006 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:28.750864 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 193249us
I20260709 12:23:28.674213  7637 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.748595  7393 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.748975  7930 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:28.771829 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.747015  7444 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:28.758473  7667 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:28.766942 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118979us
I20260709 12:23:28.718320  7703 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.765812  7863 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:28.754030  7690 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:23:28.776634  7393 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:28.789659 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.761256  7717 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.781678  7559 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 38. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
W20260709 12:23:28.841636 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46704.2us
W20260709 12:23:28.844492 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54906us
I20260709 12:23:28.764161  7783 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.791388  7863 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:28.872607 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58216.5us
W20260709 12:23:28.847604  7929 log.cc:927] Time spent T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:23:28.849537  7931 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:28.853120  7703 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:28.855101  7794 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:28.898150  7231 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:28.855897  7933 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:23:28.856813  7637 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.913499 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:28.795684  7510 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:28.789106  7808 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:28.864877  7935 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:23:28.865660  7941 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:28.845974  7444 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.889588  7790 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:28.938426 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 231008us
I20260709 12:23:28.879901  7863 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:28.895432  7783 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.883993 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:28.901402 27660 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:28.844558  7833 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.857954  7852 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.949680  7717 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.950309  7717 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:28.951567  7852 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.952917  7231 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:28.870879  7381 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.953776 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.954625 27589 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:23:28.955735  7381 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:28.958086  7940 consensus_queue.cc:1048] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:23:28.959398  7338 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:28.952483  7717 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:28.960052  7338 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.960736  7717 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:28.963768 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44900.5us
I20260709 12:23:28.967057 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:28.967823 27580 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:28.968677  7802 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.969025  7554 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.000s
I20260709 12:23:28.969300  7802 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.970499  7944 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:28.969033 27580 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 41 index: 20. Preceding OpId from leader: term: 43 index: 21. (index mismatch)
I20260709 12:23:28.967936 27587 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:23:28.867257  7549 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.916858 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:28.980439 27663 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
W20260709 12:23:28.980460 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 69389.5us
I20260709 12:23:28.917706 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:28.981359 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:28.919912 27735 raft_consensus.cc:2393] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 45: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:28.921490 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:28.869897  7897 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:28.925346 27736 raft_consensus.cc:1275] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 20. (term mismatch)
W20260709 12:23:28.946913  7673 consensus_meta.cc:287] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.000s
I20260709 12:23:28.925953 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:28.926599 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:28.991995 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:28.925695 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:28.879441  7832 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.860028  7398 consensus_queue.cc:1048] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:23:28.937187 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:28.989586 27736 pending_rounds.cc:85] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 19
I20260709 12:23:28.914722  7629 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:29.001659  7629 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:28.946550 28331 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:28.948431 27661 raft_consensus.cc:1240] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:23:29.003841  7338 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:29.004832  7930 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:29.006064  7735 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:28.948892 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:28.915303 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.978468 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:28.984886  7941 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:28.984869 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:28.962450  7717 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:28.987210 27589 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:28.925693 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.008740 27655 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
W20260709 12:23:28.996455 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.996214 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:28.999765  7832 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.009536 27658 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
W20260709 12:23:29.013852 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98641.9us
I20260709 12:23:29.026484 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:29.027534  7482 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:29.027992 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:28.994359 27736 pending_rounds.cc:107] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 36.20
I20260709 12:23:29.030300 27736 raft_consensus.cc:2889] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:29.031639  7231 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:29.031834 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.032893 27736 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:29.015352 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:28.952930  7808 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.036509  7690 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 38.26, Next index: 27, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:29.036957  7629 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:29.037778  7690 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:29.038213  7690 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:29.001368  7549 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.039739  7444 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:29.033968  7381 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:29.040786 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:28.971094  7944 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.043776 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.045185 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.045370  7944 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.045747 27655 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:29.047565  7802 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.048650  7717 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:29.049418  7863 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.040807  7381 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.051837  7381 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:29.052223  7667 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.053297 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.054082 27663 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:28.971880  7637 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.052s
W20260709 12:23:29.056977 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.039840 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.040222 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.062943 27657 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:28.991279  7897 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:29.066574 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.066852 27732 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:29.066574 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:29.078605 27657 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:29.014305  7930 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:29.085359 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.089680 27734 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 19. Preceding OpId from leader: term: 43 index: 21. (index mismatch)
I20260709 12:23:29.034180  7801 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.034255  7941 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:29.099793 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.101658 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:29.103899 27736 raft_consensus.cc:606] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.104437 27736 raft_consensus.cc:614] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:29.104580 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.102579  7820 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:29.112051  7820 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:29.113155  7820 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.119359 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:29.120699  7637 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:29.121158  7637 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:29.122107  7637 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.070904  7703 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:29.010392  7673 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.125653 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:29.152971 27730 raft_consensus.cc:606] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.153386 27730 raft_consensus.cc:614] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:29.154347 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.154608  7673 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:29.130548 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.132588 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:29.136507 27736 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:29.155972  7393 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.156960 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.158718  7398 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.158906  7393 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:29.136700 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.178061 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.178738  7703 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.179720 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.179601  7703 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.180570 27656 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:29.181736 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.139672 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.182343  7703 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.128573  7790 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.144065 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:29.136364  7930 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.048s
I20260709 12:23:29.185354  7930 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:29.185431 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.185956  7930 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:29.186326 27731 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:29.141120  7807 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.004s
I20260709 12:23:29.128660  7941 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:29.191555  7942 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:29.192060  7942 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:29.192406  7932 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.326s	user 0.003s	sys 0.000s
I20260709 12:23:29.193295  7942 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.194713  7942 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:29.145221  7832 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:29.141470 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.140695 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.140838 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.130376 27704 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:29.124187  7811 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.027s
I20260709 12:23:29.039151  7690 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.123301  7637 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:29.142334  7801 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.159459 27663 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:29.159391  7398 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.160353 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.208185 27658 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:29.161242 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.162346 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.163103 27737 raft_consensus.cc:2368] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:29.163035 27589 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:29.164196 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.211866  7811 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:29.212287  7703 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.212857  7703 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.126502 28331 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:29.138223 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.209560 27586 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:29.217756 27735 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:29.166813 27585 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:23:29.167052 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.196188 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:29.183890 27732 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:29.197696 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.183023 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.203521 27661 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:29.203639 27664 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
W20260709 12:23:29.206691 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 76406.9us
I20260709 12:23:29.219354 27584 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:29.221899 27582 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:29.209071 27655 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:29.222218 27656 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:29.198613  7863 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.253196 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.254015 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.264533 27730 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:29.261204 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:29.267225 27704 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:29.269869 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.272684 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.282068 27586 raft_consensus.cc:2410] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 44 because replica has last-logged OpId of term: 41 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 19.
I20260709 12:23:29.290023 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:29.247648  7573 consensus_meta.cc:287] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.626s	user 0.005s	sys 0.006s
I20260709 12:23:29.291083 27661 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:29.292469 27664 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:29.304325 27662 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:29.308063 27657 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:29.248049  7673 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:29.212291  7811 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:29.369362 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 69611.9us
I20260709 12:23:29.259490  7482 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:29.370597 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.393157 27658 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:29.371361 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.393994 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:29.292636 27581 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:29.396625 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.397339 27662 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:29.187134  7930 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.402163 27664 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:29.290712  7629 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:29.384423 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:29.274457  7831 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.394487 27657 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:29.414585  7629 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.420317  7629 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:29.255002  7581 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.143s
I20260709 12:23:29.411265 27661 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:29.300438  7735 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.290917  7947 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:29.444078 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.444772 27657 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
I20260709 12:23:29.290529  7832 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.124s
I20260709 12:23:29.371631 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:29.431453 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.381381  7667 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.416078  7559 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:23:29.395893  7833 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:29.376622  7573 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.479853 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.480726 27663 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:29.452831 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:29.482373  7393 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:29.483054  7393 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:29.484311  7393 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.438329  7703 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.451771 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.387313  7554 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.056s
I20260709 12:23:29.443881  7947 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.508101 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.498317 27580 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:29.515705 27581 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:29.516644 27585 raft_consensus.cc:2368] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:29.466425 27586 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:23:29.521727 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.437915  7831 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.530923  7831 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.491617  7930 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:29.519376  7667 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:29.540500  7667 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.542900  7667 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:29.478760  7444 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.525754  7955 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
W20260709 12:23:29.527859  7954 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:23:29.546124 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.546829 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.547822 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:29.548753 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 164423us
I20260709 12:23:29.552295 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.502445  7820 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:29.458889  7735 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.558081 27657 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:23:29.561085 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:29.557163 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:29.573639 27587 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:29.603869 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.605129 27589 raft_consensus.cc:2410] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:23:29.515026  7947 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:29.492558  7833 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.612049 27585 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
W20260709 12:23:29.618342 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.471073  7811 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:29.619966 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.621886 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.622517 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:29.624910 27657 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 32.
I20260709 12:23:29.561962  7703 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:29.555649  7673 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:29.614534  7897 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.002s
I20260709 12:23:29.563356  7820 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.630429 27630 leader_election.cc:400] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:29.634276 27630 leader_election.cc:403] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
I20260709 12:23:29.631026 27657 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:23:29.645057 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.544999  7381 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.654681 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:29.563699  7444 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:29.668699 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49845.3us
W20260709 12:23:29.688835 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 131763us
I20260709 12:23:29.638638  7820 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:29.691303 27734 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:29.690439 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:29.653388  7717 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.604s	user 0.010s	sys 0.002s
I20260709 12:23:29.616746  7941 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:29.702345 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:29.706698  7956 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.494s	user 0.002s	sys 0.000s
W20260709 12:23:29.710760  7959 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
W20260709 12:23:29.713325 27660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2214) took 1368 ms (client timeout 1791 ms). Trace:
W20260709 12:23:29.713892 27660 rpcz_store.cc:269] 0709 12:23:28.345013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.345167 (+   154us) service_pool.cc:224] Handling call
0709 12:23:29.713025 (+1367858us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54000e56d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:29.713179 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1090277120}
I20260709 12:23:29.700635  7667 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.028s
W20260709 12:23:29.732453  7964 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:23:29.742159  7444 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.744771  7965 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:29.745374  7965 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:29.745874  7965 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:29.759819 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61234us
I20260709 12:23:29.761238 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:29.775380 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.778065 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.682778  7811 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:29.743053  7444 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:29.679649  7554 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37.
I20260709 12:23:29.648105  7897 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:29.797008 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.787254 27701 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:29.806099 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.681648  7703 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.105s
I20260709 12:23:29.789552  7444 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.827476 27582 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:29.701174  7559 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.115s
I20260709 12:23:29.830957 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.790544  7717 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.835469  7231 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.803s	user 0.009s	sys 0.004s
I20260709 12:23:29.833524  7444 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:29.853256  7703 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:29.827280  7554 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:29.855544  7629 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.855844  7554 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.856916  7971 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:29.857800  7971 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:29.844236  7573 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:29.836745  7832 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.859905  7231 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:29.858407  7966 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:29.814026  7673 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.920970 27735 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
W20260709 12:23:29.914505  7637 consensus_meta.cc:287] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.698s	user 0.008s	sys 0.004s
I20260709 12:23:29.912637 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:29.858016  7629 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:29.853788  7968 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.071s
I20260709 12:23:29.840677  7717 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.857887  7554 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.855549  7703 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.859480  7735 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:29.914880  7975 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.923669  7969 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.914880 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 273895us
W20260709 12:23:29.933331 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 231047us
I20260709 12:23:29.918421  7971 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:29.935164 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:29.937323  7703 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:29.933387  7966 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:29.928920  7930 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:29.916870  7832 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:29.947263 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.942514 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.945700 27704 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:29.923403  7573 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.018s
I20260709 12:23:29.964591 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.941814  7833 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:29.971374 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:29.974885 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:29.980427 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 149602us
I20260709 12:23:29.983294 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:29.990363 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:29.991343 27588 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:29.994545 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:29.995507 27738 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:29.995666 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.961864  7930 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:29.998498 27732 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 19. (term mismatch)
I20260709 12:23:29.996436 27735 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:30.000788 27732 pending_rounds.cc:85] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:23:30.001174 27732 pending_rounds.cc:107] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260709 12:23:30.001432 27732 raft_consensus.cc:2889] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:30.001503 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:29.976250  7231 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:29.982334  7629 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.984860  7968 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.006359 27735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:29.988574  7974 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:29.999750  7790 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.008194 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:29.945534  7969 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.013870  7790 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.015445  7969 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:29.977753  7973 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:30.016484 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:30.018183 27701 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:30.013652 27735 raft_consensus.cc:606] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:30.019279 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:30.020504 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:29.943770  7975 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.024485 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:30.024066  7972 log.cc:927] Time spent T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.101s	user 0.001s	sys 0.001s
I20260709 12:23:30.028813 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.029386 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:30.031937 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.032577 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:30.049475 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.052136 27735 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:30.055800  7559 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.069630  7559 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.016934  7629 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.073360  7629 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.056643  7790 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.055812 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:30.057416  7863 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.059551 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.074446  7863 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.074925  7790 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:30.075380  7790 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:30.075750  7790 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:30.067608  7977 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 12:23:30.078622 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 59458.9us
I20260709 12:23:30.079365 27733 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:30.080143 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.027415  7971 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:30.021816  7973 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:30.051398  7970 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:23:30.020879  7979 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:30.026769  7984 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.084668 27735 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:30.076735  7863 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.087102 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:30.091786  7974 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:30.149482  7974 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:30.153760  7942 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.959s	user 0.010s	sys 0.006s
I20260709 12:23:30.092016  7971 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:30.156064  7971 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.095613 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.159605 27735 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:30.160847 27735 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:30.110132  7975 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.111269  7973 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.021713  7968 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.114075 27738 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:30.118307  7947 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.120523  7832 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.124116 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.082737  7573 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:30.128196  7953 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:30.128422  7717 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.131906  7979 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.135533 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:30.135872  7559 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:30.136710  7958 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.705s	user 0.004s	sys 0.000s
I20260709 12:23:30.122002  7984 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:30.138597 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52058.1us
I20260709 12:23:30.138958 27729 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:30.139228  7794 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:30.190340 27737 raft_consensus.cc:2368] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:30.140712  7444 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.141093 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:30.192451 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51453.3us
I20260709 12:23:30.196295  7965 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.176709  7573 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.203056  7573 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.209061  7942 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:30.209897  7942 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.210322  7942 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.198998  7965 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.144591 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.114661  7820 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.085881  7637 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:23:30.165660 27586 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:30.170089 27581 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:30.177497 27735 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:30.185817 27736 raft_consensus.cc:606] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:30.189543 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:30.144238  7703 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.195304 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:30.189045  7863 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:30.219458 27736 raft_consensus.cc:614] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:30.220981 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:30.226769 27627 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:30.223670 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:30.216744  7559 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.188462  7979 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.212785  7717 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:23:30.218889  7637 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.225152  7634 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:23:30.272680 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.225095  7863 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:30.288417 27588 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:30.239197 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:30.305917 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.225535  7953 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.306221 27584 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:30.300774 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.234146  7965 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:30.240710  7899 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:23:30.283396 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 46786.5us
I20260709 12:23:30.237736  7831 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:30.248874  7832 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:30.273078  7975 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:30.276752  7811 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.297505  7637 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:30.297923  7969 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.326974 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.330318 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:30.271505  7573 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.043s
I20260709 12:23:30.273675  7827 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.042s
I20260709 12:23:30.303128  7634 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.313120  7965 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.289093  7807 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.039s
W20260709 12:23:30.351071 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 122157us
I20260709 12:23:30.334056 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.340082 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:30.317133  7899 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.298035  7559 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.351967 27627 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:30.359105 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.360755 27587 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 33 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:23:30.361618  7559 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.359879  7444 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.361861  7989 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:30.362253  7989 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:30.362354  7820 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.315438  7953 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.332016  7811 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.365700 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.365993 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.366737 27662 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:30.366811 27656 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:30.350239  7634 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:30.370014 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.370733 27657 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:30.371677  7667 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:30.371768  7827 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:30.372349  7667 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.375257  7667 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.377862 27656 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:30.351624  7965 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:30.380329 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.382555 27656 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:30.381397 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.380599 27627 leader_election.cc:400] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:30.352546  7899 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:30.351490  7987 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:30.384797 27657 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.361430  7703 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.385458 27627 leader_election.cc:403] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:23:30.367626  7444 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.363173  7989 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.349813  7975 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.336573  7969 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:30.399384  7986 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:23:30.402375  7953 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.396111  7987 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:30.404672  7987 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.408269 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.409162 27585 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:23:30.363873  7790 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.364583  7997 consensus_queue.cc:1048] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:23:30.386762  7947 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.409538  7984 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.416877 27584 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:30.419648 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:30.415865  7991 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:30.427584 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.436967 27580 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:23:30.444605  7969 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.416575  7827 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:30.446015  7827 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.422607  7989 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:30.408880  7444 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:30.463915 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67270.6us
I20260709 12:23:30.416066  7953 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.037s
I20260709 12:23:30.419550  7998 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.051s
I20260709 12:23:30.433036  7634 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.477823  7790 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.478858  7634 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.486346  7979 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.497330 27582 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:30.499805 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4334) took 1354 ms. Trace:
I20260709 12:23:30.500020  7482 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.500259 27730 rpcz_store.cc:276] 0709 12:23:29.145384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.163936 (+ 18552us) service_pool.cc:224] Handling call
0709 12:23:30.499646 (+1335710us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b54001d1ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:30.499750 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121380736}
I20260709 12:23:30.500587  7482 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.501210 28334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1915) took 1383 ms. Trace:
I20260709 12:23:30.501689 28334 rpcz_store.cc:276] 0709 12:23:29.117859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.117999 (+   140us) service_pool.cc:224] Handling call
0709 12:23:29.118299 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.118536 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:23:29.442850 (+324314us) log.cc:844] Serialized 14 byte log entry
0709 12:23:29.678493 (+235643us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.500569 (+822076us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b18003cd840. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:23:30.500860 (+   291us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:30.500916 (+    56us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:30.500961 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.501043 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.501159 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6421,"prepare.queue_time_us":326076,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":6710,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9965,"spinlock_wait_cycles":22942336,"thread_start_us":559003,"threads_started":2,"wal-append.queue_time_us":423626}
I20260709 12:23:30.505589  7947 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:30.507108 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.508078 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:30.508790 27582 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:30.509490 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.524035  7975 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.524559 27730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:30.525305 27730 raft_consensus.cc:606] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:30.525727 27730 raft_consensus.cc:614] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:30.518913  7393 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:30.524923  7828 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.543607 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.518318  7947 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.544288 28334 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:30.546931 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.544135  7852 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.553800  7947 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.541430  7393 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.559800  7444 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:30.563154  7967 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.855s	user 0.004s	sys 0.005s
I20260709 12:23:30.570631 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4333) took 1427 ms. Trace:
I20260709 12:23:30.566363  7965 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.545358  7828 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.611567  7554 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.612412  7554 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.615868  7554 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.616046  7381 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.617036  7482 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.617715 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 5273) took 1058 ms. Trace:
I20260709 12:23:30.663837 27586 rpcz_store.cc:276] 0709 12:23:29.559824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.560740 (+   916us) service_pool.cc:224] Handling call
0709 12:23:30.577946 (+1017206us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54001fe0d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:30.617635 (+ 39689us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1278927744}
I20260709 12:23:30.619266  7801 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.620731  7941 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.621470  7852 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.667675  7852 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.668383  7690 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.624044  7942 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.695804  7942 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.620719  7965 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.682593  7852 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.700102  7852 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:30.703539 27733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 4833) took 1801 ms (client timeout 1687 ms). Trace:
I20260709 12:23:30.626727  7828 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.581141 27734 rpcz_store.cc:276] 0709 12:23:29.142724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.156723 (+ 13999us) service_pool.cc:224] Handling call
0709 12:23:30.570404 (+1413681us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54001d42d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:30.570565 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1226653440}
I20260709 12:23:30.627727 28332 raft_consensus.cc:2410] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 33 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:23:30.628623  7947 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:30.631809 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.634801 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:30.704114 27733 rpcz_store.cc:269] 0709 12:23:28.902338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.902706 (+   368us) service_pool.cc:224] Handling call
0709 12:23:30.695293 (+1792587us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54002a1a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:30.703282 (+  7989us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648775424}
I20260709 12:23:30.712957 28334 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:30.631875  7393 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:30.635991 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.637090 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:30.641402  7444 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.642510 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.643950 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:30.644217 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.745800 28336 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:30.647948  7637 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.622797 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 5271) took 1077 ms. Trace:
W20260709 12:23:30.670270  7963 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.979s	user 0.000s	sys 0.003s
I20260709 12:23:30.670827  7581 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:30.671444 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.672462  7863 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.681452  7899 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.685007  7398 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.691798 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.758792 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.624512  7991 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:30.682538  7930 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.745213 28331 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:30.644786 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.763082 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.748136 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:30.763484 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.763827 28337 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:30.746484 27581 rpcz_store.cc:276] 0709 12:23:29.545465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.545812 (+   347us) service_pool.cc:224] Handling call
0709 12:23:30.577425 (+1031613us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54000bc7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:30.622737 (+ 45312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1376737024}
I20260709 12:23:30.764166 27738 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:30.756476 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:30.757293 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.765594 28328 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:30.759562 27735 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:30.761241 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:30.761518 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.752838 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:30.767284 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 95952.7us
I20260709 12:23:30.768512 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.765666 27734 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:30.770951 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.771754 27657 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:30.772433 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.773033 27656 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.774073 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.774863 27660 raft_consensus.cc:2393] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 32: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:30.777715 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.783649 27737 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:30.786984 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:30.787551 27733 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.787914 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:30.783839 27729 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 44: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:30.788937 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:30.756053  7581 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:30.794551 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.794723 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.797842 27664 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:30.795156 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.798708 27658 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:30.799835 27585 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:30.800284 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.800455 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.802963 27581 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:23:30.803146 27657 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:30.770402  7863 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:30.805331 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.802048 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.807024 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.808911 27586 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 31.
W20260709 12:23:30.809835 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 344914us
I20260709 12:23:30.810832 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.811601 27586 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:30.809276 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.813946 27585 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:23:30.814304 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.756549  7444 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.813784 27733 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:30.815694 27586 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:30.815727 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.817734 27661 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.816869 27582 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:30.792845  7965 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:30.815085 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 393090us
I20260709 12:23:30.756912  7947 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.793725  7398 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.783540  8010 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:30.820798 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5108) took 1662 ms. Trace:
I20260709 12:23:30.819111 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.822660 27588 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:30.822345 27583 rpcz_store.cc:276] 0709 12:23:29.158422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.163428 (+  5006us) service_pool.cc:224] Handling call
0709 12:23:29.163762 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.164092 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.164120 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:29.164651 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 12:23:29.445355 (+280704us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:29.445417 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.445519 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.669417 (+1223898us) spinlock_profiling.cc:243] Waited 62.3 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:30.816586 (+147169us) raft_consensus.cc:1685] finished
0709 12:23:30.816608 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.820707 (+  4099us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":199,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":67,"spinlock_wait_cycles":143276928,"thread_start_us":280234,"threads_started":1,"wal-append.queue_time_us":447252}
I20260709 12:23:30.821894 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:30.807960  7930 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:30.806908  7231 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:30.828073 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:30.828929 27733 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:30.829833 27730 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I20260709 12:23:30.830718 27737 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: []
I20260709 12:23:30.832626 27734 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 32. Current term is 33. Ops: []
I20260709 12:23:30.814389  7482 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:30.819059  7975 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.846264 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:30.844182 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:30.818187  7852 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:30.849363 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:30.845628 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.850461 27735 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:30.846575 27580 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:30.816454  7983 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:30.812047  7942 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:23:30.845082 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:30.852638 27587 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:30.820719  8010 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:30.848206 27702 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:30.854995 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:30.855700 27580 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40.
I20260709 12:23:30.841153  7987 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.856312  7987 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.859417 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:30.861346 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:30.862466 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:30.864033  7637 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.827443  7398 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.865880  7700 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:30.866596  7398 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.832206  7231 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:30.867205  7231 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:30.868441 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.868880  7947 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.870536  7444 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.876020 27584 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
W20260709 12:23:30.875927 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 90684.9us
I20260709 12:23:30.878257 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.879935 27657 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:23:30.826217  7393 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:30.880671 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.896303 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.897347 27736 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:30.883006 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.898785 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.842988  7482 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.925940 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:30.902230 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 80400.1us
I20260709 12:23:30.911413 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
W20260709 12:23:30.911602 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 87384.6us
I20260709 12:23:30.899680 27730 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:30.940012 27660 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I20260709 12:23:30.947723 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:30.951018 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.828193  7965 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:30.951406 27659 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:30.952105 27663 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:30.953452  7965 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:30.954470 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:30.955598 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.957618 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:30.958567 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.959738 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.936084  7482 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:30.938305  7984 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:30.961097 27657 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:23:30.961238  7984 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:30.962144  7984 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.909976 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:30.936559 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:30.972810 27656 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:30.969133 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:30.973991 27660 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 32.
I20260709 12:23:30.973069 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:30.975586 27735 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:30.970443 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:30.939062 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 55080.5us
I20260709 12:23:30.936084 27664 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:30.897688  7637 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.873703  8010 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:30.881651  7983 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:30.897136  7832 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:30.899623  7969 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:30.971897 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:30.933852  7398 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:30.885460  7852 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:31.005072 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.852959  7975 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:30.947898  7991 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:30.984637  7942 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.025504 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:31.041411  8005 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:31.042066  8005 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.044350  8005 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:31.045703 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 88174.5us
I20260709 12:23:31.048100  8001 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:31.048699  8001 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:31.049192  8001 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:31.005861  7735 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.044s
W20260709 12:23:31.058458 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 121968us
I20260709 12:23:31.059543 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.061208 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.062534 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.062879  7784 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:31.063551  7982 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:31.063714  7784 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:31.005591  7897 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.064447  8011 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:31.065063  8011 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.065601  8011 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:31.067840  7811 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.023871 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:31.068923 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45232.3us
I20260709 12:23:31.069739 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:31.023128  7899 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.037206 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:30.968103  7833 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.061s
I20260709 12:23:31.064378  7982 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:31.076399  7982 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:31.078720 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4149) took 1908 ms. Trace:
I20260709 12:23:30.854451  7581 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.079934  7581 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:31.079123 27655 rpcz_store.cc:276] 0709 12:23:29.169948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.245761 (+ 75813us) service_pool.cc:224] Handling call
0709 12:23:29.255793 (+ 10032us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:29.262100 (+  6307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.262150 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:30.690188 (+1428038us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b2c00010da8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:23:30.690917 (+   729us) log.cc:844] Serialized 52 byte log entry
0709 12:23:30.905711 (+214794us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:30.951912 (+ 46201us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.004514 (+ 52602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.004591 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:31.076152 (+ 71561us) spinlock_profiling.cc:243] Waited 4.64 ms on lock 0x7b1800415b40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:31.078478 (+  2326us) raft_consensus.cc:1685] finished
0709 12:23:31.078501 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.078655 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96521,"prepare.queue_time_us":46149,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":59818,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":37724,"spinlock_wait_cycles":46103808,"thread_start_us":214309,"threads_started":1,"wal-append.queue_time_us":336656}
I20260709 12:23:31.083701 27732 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:31.085036  7987 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.000674  7637 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.071136 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:31.091213 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.091339 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.091212 27735 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:31.092320 28337 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:31.093297 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:31.096275 27658 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:31.001441  7832 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.001739  7969 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:31.105641 28336 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:31.107484 28334 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:31.098487 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.108723 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:30.930263  7393 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.112910 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:31.112417 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:31.113989 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.118938  7899 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:31.119812  7393 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.035070  7398 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:31.121299  7398 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 41.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.122853 27664 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:31.110067  7735 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:31.123723  7735 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.124099  7735 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:31.129787 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:31.130267 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.130450 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.130945 28337 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
W20260709 12:23:31.133459 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.135306 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.141465 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.142302 28336 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:31.105823  7832 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:31.111387  7969 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:31.132603  8020 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:31.168283 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.128332  7947 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.136559  7381 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:23:31.175997 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.105289  7667 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.042s
I20260709 12:23:31.023038  8014 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.144s
I20260709 12:23:31.109961  7998 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.041s
I20260709 12:23:31.145196  7991 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:23:31.145011  7997 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:31.151712  7784 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.106559  7942 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.212442 28337 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: [37.24-37.24]
I20260709 12:23:31.168829  8020 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:31.222947 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 151888us
W20260709 12:23:31.224773 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 116767us
I20260709 12:23:31.204288  8005 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:31.225524 27552 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:31.233621 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.248787 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.218519  7998 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.248960 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:31.257745 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.227628  8020 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:31.259127 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.259909 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:31.260211  8019 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:31.261040 27658 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:23:31.261125  7975 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.260219  7930 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:31.270462  7930 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:31.217805  7784 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:31.263909  7690 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.000s
I20260709 12:23:31.227182  7968 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:31.273413  7784 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.274248  7930 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.277833  7989 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.161870  7983 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.279806 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:31.278499  7989 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.267577 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:31.249686  7897 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:31.282918  7897 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:31.283995  7897 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.204751  7947 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.255287  7833 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:31.210053  7231 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:31.295397 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 160217us
I20260709 12:23:31.284291  7482 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:31.314633 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 176619us
I20260709 12:23:31.314841 27662 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:31.315981 27627 leader_election.cc:400] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:31.317318 27627 leader_election.cc:403] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:23:31.332870 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:31.342314 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.343272 27729 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:23:31.345854 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.347580 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:31.357859  7637 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:31.357928 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.358799 27733 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:31.360039 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.361822 27731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4845) took 1445 ms. Trace:
I20260709 12:23:31.362217 27731 rpcz_store.cc:276] 0709 12:23:29.916627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.917207 (+   580us) service_pool.cc:224] Handling call
0709 12:23:29.917594 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.007232 (+ 89638us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:23:30.141324 (+134092us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b18002b9480. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:23:30.336281 (+194957us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.469043 (+132762us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.473810 (+  4767us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.473911 (+   101us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.479746 (+  5835us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.479848 (+   102us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:30.481710 (+  1862us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.492986 (+ 11276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.505285 (+ 12299us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b18002b9480. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:30.574135 (+ 68850us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b18002b9480. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:30.680653 (+106518us) spinlock_profiling.cc:243] Waited 68.1 ms on lock 0x7b54002e81d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:30.680734 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:31.245440 (+564706us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b18002b9480. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:31.361598 (+116158us) raft_consensus.cc:1685] finished
0709 12:23:31.361622 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.361764 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61505,"prepare.queue_time_us":550943,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":113,"raft.queue_time_us":27639,"raft.run_cpu_time_us":489,"raft.run_wall_time_us":7729,"spinlock_wait_cycles":196509824,"thread_start_us":266750,"threads_started":2,"wal-append.queue_time_us":123323}
I20260709 12:23:31.364859  7482 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.365330  7942 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.364898 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.370450 27731 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:31.361241 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.370666 27736 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
W20260709 12:23:31.372067 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 113787us
I20260709 12:23:31.388494 27733 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:31.389248 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:31.374251 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.390540 27664 raft_consensus.cc:2368] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39.
I20260709 12:23:31.391654 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.392453 27664 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:31.376226 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.376482 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.377810 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:31.379256 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.395888 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.380472 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.297345  7667 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.071s
I20260709 12:23:31.399500 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.291853  7998 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.401310 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.406454 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:31.409998 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.410316 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.410974 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.415608 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:31.413239 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.329568  8005 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.050s
I20260709 12:23:31.368656  7997 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:31.358793  7947 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.442032 27658 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:31.416742 27663 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
I20260709 12:23:31.420205 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.420861 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.386431  8001 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.427644 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.429181 27660 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: [38.22-38.22]
I20260709 12:23:31.430104  8030 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:23:31.431609  8025 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:23:31.358968  7735 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.462481 27663 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 35. Current term is 36. Ops: []
W20260709 12:23:31.472769 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52665.5us
I20260709 12:23:31.455178 27656 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:23:31.475520 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 300054us
W20260709 12:23:31.487280 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.476102 27552 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:31.524493 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48433.4us
W20260709 12:23:31.529422 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 304014us
I20260709 12:23:31.399870  8031 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:31.476208 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:31.534173 27664 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:31.416064  7942 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:31.532186 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:31.534426 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 58291.6us
I20260709 12:23:31.409821  7930 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.122s
I20260709 12:23:31.522593  7690 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:31.545173 27704 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:31.439440  7983 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:31.473255  7979 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.550168 28337 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 36: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:31.543274 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.439239  7989 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.541775 27730 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:31.457271  8030 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:31.456766  7482 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:31.473280  7947 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:31.547602 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.522792  7975 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.553028 27704 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:31.555888 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.554821  7968 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:31.559476 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:31.532092  7811 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:31.560537 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.557545  7482 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.564592  7930 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:31.564488  7975 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.565104  7930 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:31.566128  7930 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.561882 27730 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:31.541050  7942 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.570611  7942 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:31.570935  7482 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.573341 27737 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:31.574268  7811 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:31.575419  7811 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.440383  7969 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.579455  7969 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.580435 27589 raft_consensus.cc:1240] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:31.549382  7983 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.581436  7983 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:31.534629 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:31.539826  7381 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:31.544715  7637 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:31.544269  7965 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:31.596089 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61534.7us
I20260709 12:23:31.588748 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:31.594405  7852 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.559s	user 0.010s	sys 0.000s
I20260709 12:23:31.558966  8030 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:31.594645  7930 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:31.609614 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.556378  8031 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:31.595407  7690 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:31.554876  7444 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.050s
I20260709 12:23:31.598124  8043 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:31.646268  7690 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.656227  7690 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:31.647706  7381 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.655196  8043 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.663688  8043 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:31.598162 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 257668us
I20260709 12:23:31.599402  7637 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.600301  7794 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:31.558226  7979 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.647682  7831 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:31.651505  8039 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:23:31.585282  7947 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.630447  7852 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:31.653144  8031 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:31.667301  7581 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.002s
I20260709 12:23:31.667037 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.670775 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.604259  7965 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.707697 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.708593 27656 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:31.613322  7897 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:31.670022 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.708910 27735 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:23:31.709271 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.668237  7231 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:31.711422 27655 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:31.674858 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.706104 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:31.677613  7637 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:31.715418 28334 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:31.681392  7833 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:31.675573  7794 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:31.718891 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109411us
I20260709 12:23:31.725294 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:31.746042 27663 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:31.719138 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:31.677382  7831 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:31.721081 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.701507  7998 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:31.745127 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:31.757067 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:31.752488 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:31.732723 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65783.8us
W20260709 12:23:31.721611 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 148717us
I20260709 12:23:31.759419 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.761756 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:31.762507 27581 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:31.749687 27656 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:31.711238  7897 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.765758 27656 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: []
W20260709 12:23:31.722414  8014 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.009s	sys 0.022s
I20260709 12:23:31.768213  7897 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:31.768004  7998 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:31.677050  8038 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.083s
I20260709 12:23:31.793970 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:31.783834 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:31.806648 27588 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:31.807152 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 4158) took 1437 ms. Trace:
I20260709 12:23:31.807585 27662 rpcz_store.cc:276] 0709 12:23:30.369643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.369772 (+   129us) service_pool.cc:224] Handling call
0709 12:23:31.806967 (+1437195us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b5400345154. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:31.807088 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2035684224}
I20260709 12:23:31.808220 27655 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:23:31.749290  7989 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:23:31.820164 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.751232  7794 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:23:31.784240  8010 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.760s	user 0.008s	sys 0.004s
I20260709 12:23:31.824985 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:31.749560  8041 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:31.804322  8033 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:31.862157 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:31.863689 27662 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:23:31.806280  7381 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:31.840572  7989 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.817204  7998 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:31.884132 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 111206us
I20260709 12:23:31.887377  8041 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:31.897748  8041 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:31.899863  8045 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:23:31.888509 27630 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:31.846935  8014 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:23:31.901587 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.902086 27630 leader_election.cc:403] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:23:31.844187  8049 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:23:31.864120  8055 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:23:31.865129  7969 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.911653 27630 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40.
I20260709 12:23:31.914871 27630 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20260709 12:23:31.873931  8032 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:23:31.955036 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43571.2us
I20260709 12:23:31.895277  7989 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:31.959203  8059 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:31.889138  7984 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:31.955552  8011 consensus_meta.cc:287] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.854s	user 0.017s	sys 0.002s
I20260709 12:23:31.853866  8010 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:31.968864 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72733.2us
I20260709 12:23:31.915163  7997 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
W20260709 12:23:31.979769  8048 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
W20260709 12:23:31.981277  8040 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
I20260709 12:23:31.981945  7998 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.889014  7381 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.987900 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:31.986912 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:31.998421 27661 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:31.905998  8047 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.096s
I20260709 12:23:31.867777  7831 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.004299 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.004419 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:31.966921  7667 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40.
I20260709 12:23:31.978440  8014 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:31.957944  7969 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.020268  8054 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.023s
I20260709 12:23:32.024709  8060 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:32.025816 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.026798 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.026978 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.028067 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.028949 28328 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:32.025287  8060 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:32.008462  7832 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.842s	user 0.008s	sys 0.005s
I20260709 12:23:32.001331  7398 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:31.978017  7984 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:31.987426  8011 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:31.914310  7833 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.012609  7852 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:32.014150  8041 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:32.052789  8060 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.015043  7811 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.058457 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5126) took 1552 ms. Trace:
I20260709 12:23:31.969995  8059 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.043176  7965 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.061710 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:32.057219  8041 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:32.081667  8041 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:32.082192  8059 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.082775  8059 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:23:32.061295 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48661.6us
I20260709 12:23:32.035372  7947 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:32.046618  7982 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:32.082919 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.063472 27582 rpcz_store.cc:276] 0709 12:23:30.506304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.519248 (+ 12944us) service_pool.cc:224] Handling call
0709 12:23:30.519511 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.818414 (+298903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.058185 (+1239771us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54001879d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:32.058380 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":683459968}
I20260709 12:23:32.089061 28332 raft_consensus.cc:2368] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:23:32.011137  8033 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.049s
I20260709 12:23:32.050163  7398 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.047888  7832 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.083634  8011 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:32.112532  8068 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:32.106593  8060 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.123638 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.104040  7997 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:32.079756  7811 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.087132  7690 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.143296 27735 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:32.150091 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 46.
W20260709 12:23:32.151665 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.152458 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.156908 27738 raft_consensus.cc:2393] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:32.135648  7997 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.107434  8014 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:32.160566  8053 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:23:32.169762 28329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1931) took 1291 ms. Trace:
I20260709 12:23:32.167610 27738 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:32.174624 28329 rpcz_store.cc:276] 0709 12:23:30.878461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.878575 (+   114us) service_pool.cc:224] Handling call
0709 12:23:30.878872 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.127855 (+248983us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.127888 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.127943 (+    55us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:31.127979 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.169432 (+1041453us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b540073e954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:32.169504 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.169677 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":572042496}
I20260709 12:23:32.161249  7997 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:32.167480  8010 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.132263  8068 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:32.190796 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.160815  8043 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.202826  7690 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.203408  7832 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.179272  7953 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.015s
W20260709 12:23:32.204739  8035 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.640s	user 0.003s	sys 0.000s
I20260709 12:23:32.204941  8074 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.205533  8074 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.210671 28332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1864) took 1120 ms. Trace:
I20260709 12:23:32.210283  8031 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:32.211531  8031 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:32.211923 27732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4352) took 1454 ms (client timeout 1831 ms). Trace:
W20260709 12:23:32.212352 27732 rpcz_store.cc:269] 0709 12:23:30.757524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.757766 (+   242us) service_pool.cc:224] Handling call
0709 12:23:32.211760 (+1453994us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54001d38d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:32.211861 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":735681280}
I20260709 12:23:32.213693  7984 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.212977  8031 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:32.189764  7930 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.554s	user 0.004s	sys 0.007s
I20260709 12:23:32.185302  7381 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:32.211120 28332 rpcz_store.cc:276] 0709 12:23:31.090783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.090921 (+   138us) service_pool.cc:224] Handling call
0709 12:23:32.210472 (+1119551us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b5400520dd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:32.210611 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2254904704}
I20260709 12:23:32.215193  7811 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.188570  7969 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.187989  8073 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.213102  8068 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.199414  7897 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:32.227064 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.219554  8031 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:32.217290  8069 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.232962  7942 consensus_meta.cc:287] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.662s	user 0.007s	sys 0.006s
I20260709 12:23:32.234998  8075 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:32.229446  8073 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:32.232348  7983 consensus_meta.cc:287] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.650s	user 0.006s	sys 0.006s
I20260709 12:23:32.248096 27588 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:23:32.247717  7965 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.248989 27585 raft_consensus.cc:1240] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:32.249703 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:32.283564 27580 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:32.283681  7989 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.284682  7983 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.285751  7989 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.251174 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:32.286576 27587 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 29.
I20260709 12:23:32.287191  7983 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:32.243081  8030 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.577s	user 0.007s	sys 0.000s
I20260709 12:23:32.290162  8030 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.251758 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:32.291920 27584 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:32.254076 27553 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.264928  7794 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.294034 27553 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.294566  7794 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.247268  7637 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.295029  8073 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.295607  8073 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:32.296625  7637 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:32.267599 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 179340us
I20260709 12:23:32.251386 27583 raft_consensus.cc:1240] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38. Ops: [37.19-37.19]
I20260709 12:23:32.298481  8030 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.299172  7983 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.300204  8081 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:32.300071  7983 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.300942  8081 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.295523  7989 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:32.303723  7983 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.305267  8081 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:32.305636 27661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2228) took 1367 ms (client timeout 1677 ms). Trace:
W20260709 12:23:32.306066 27661 rpcz_store.cc:269] 0709 12:23:30.938582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.939685 (+  1103us) service_pool.cc:224] Handling call
0709 12:23:32.305455 (+1365770us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000e83d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:32.305578 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2396657024}
I20260709 12:23:32.296541 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.308254 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:32.308413 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.309584 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.310041 27738 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
W20260709 12:23:32.311580 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.276122  7398 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.313894 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.317584  7983 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:32.315111  8030 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:32.320091  7398 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:32.281556  8041 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.281618  7942 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.313831  7965 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.280445  8069 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.353485  8069 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.354030  8069 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.354480  8069 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.281456  8075 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.371667 27586 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:32.373536  8043 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.376965 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.377311  7942 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.377873 27731 raft_consensus.cc:3060] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:32.378165  7897 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:32.381625  8020 consensus_meta.cc:287] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 1.087s	user 0.005s	sys 0.006s
I20260709 12:23:32.384615 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.385370 27738 raft_consensus.cc:2393] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:32.383487  8041 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.386080 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:32.394903  7965 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.397634  7997 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.399717 27627 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39.
I20260709 12:23:32.400588 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:32.402722  7790 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:32.403262  7790 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.403635  7790 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:32.405570  7735 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.419889 27731 raft_consensus.cc:2468] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:32.391603  8020 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.445645  8020 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.420162  7997 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.446514 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4876) took 1089 ms. Trace:
I20260709 12:23:32.420303 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:32.421241  8069 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.424911 27585 raft_consensus.cc:1240] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40. Ops: [39.25-39.25]
I20260709 12:23:32.425999 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:32.448469 27584 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:23:32.427784 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.431018 27583 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:23:32.433266  8009 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.452806 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.453461 27655 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:32.453025  8009 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.453959 27664 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:23:32.455054 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.456269 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.457242 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.458142  7831 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.458957  7393 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:23:32.459764 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.460637 27655 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:32.461392  7393 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.461809  8069 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:32.463807 28292 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.465408 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.466023 28332 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.466410  8074 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:32.467231  7700 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.467666 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.467852  7700 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.467815  8033 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:32.469275 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.470029  7700 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:32.437595 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.470474  7637 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:32.470867  7637 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.471220  7637 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:32.471316 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.472432 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.472868 27589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2275) took 1378 ms. Trace:
I20260709 12:23:32.472977 27661 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:32.473263 27589 rpcz_store.cc:276] 0709 12:23:31.094927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.095078 (+   151us) service_pool.cc:224] Handling call
0709 12:23:31.095403 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.580929 (+485526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.472664 (+891735us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b540011c954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:32.472811 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1114968832}
W20260709 12:23:32.420365 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.475443 27736 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 36: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:32.476748  8009 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.402776 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.477610  7828 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:32.452225 27660 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: [45.27-45.27]
I20260709 12:23:32.477989  7828 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:32.478392  7828 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:32.477632 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.479831 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.462121 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.481242  8029 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:23:32.481624  8029 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.481950 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.446895 27737 rpcz_store.cc:276] 0709 12:23:31.356661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.574992 (+218331us) service_pool.cc:224] Handling call
0709 12:23:31.575601 (+   609us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.895634 (+320033us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.895666 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.917859 (+ 22193us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.989040 (+ 71181us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:32.163662 (+174622us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.164136 (+   474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.445143 (+281007us) spinlock_profiling.cc:243] Waited 5.93 ms on lock 0x7b18002bb100. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e972292a6 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419
0709 12:23:32.446212 (+  1069us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540017e8d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:32.446263 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.446302 (+    39us) raft_consensus.cc:1685] finished
0709 12:23:32.446313 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.446443 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":160078,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":48221,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":37191,"spinlock_wait_cycles":318362752,"thread_start_us":378958,"threads_started":3,"wal-append.queue_time_us":75941}
I20260709 12:23:32.482630 28336 raft_consensus.cc:2368] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35.
W20260709 12:23:32.487622 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 84912.6us
I20260709 12:23:32.488385 28295 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.489223 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:32.489796 27585 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:32.491453 27589 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:32.491840  8011 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:32.492568  7794 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.493062  7794 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.494740 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.495440 27731 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:32.467368  8014 raft_consensus.cc:2804] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:32.498550 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:32.435597  8073 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:23:32.507483 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 100156us
I20260709 12:23:32.497156 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:32.458244 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.508095 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.451864 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:32.508642 27655 raft_consensus.cc:3060] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:32.508308 27664 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:32.498910 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.501130 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.505129 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.497412 27584 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:32.511438 27661 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:32.510406 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.510574 27583 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:32.517045  8043 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:23:32.509522  8014 raft_consensus.cc:697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.518680  8014 consensus_queue.cc:237] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 33.25, Last appended by leader: 25, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.522038  7811 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:32.522854  8011 consensus_queue.cc:1059] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:23:32.523550  8075 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:32.523788  8011 consensus_queue.cc:1059] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:32.524530 28337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1872) took 1250 ms. Trace:
I20260709 12:23:32.524828  8011 raft_consensus.cc:3055] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:32.525008  7852 raft_consensus.cc:3055] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:32.525316  8011 raft_consensus.cc:740] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.522464 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:32.526326  8011 consensus_queue.cc:260] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.526949 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:32.527892 27589 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:32.527700  8043 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:23:32.524904 28337 rpcz_store.cc:276] 0709 12:23:31.274272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.279504 (+  5232us) service_pool.cc:224] Handling call
0709 12:23:32.524348 (+1244844us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54005eb854. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:32.524465 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":609504640}
I20260709 12:23:32.525523  7852 raft_consensus.cc:740] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.531368  7852 consensus_queue.cc:260] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.532088 27735 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: [45.27-45.27]
I20260709 12:23:32.532349  7852 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:32.532963  7979 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:32.534479  8033 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:32.535055  8033 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.527771 27583 raft_consensus.cc:2393] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 37: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:32.535502  7700 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:32.535971  8033 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.539772 27658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4943) took 1164 ms. Trace:
I20260709 12:23:32.539883 27655 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:32.540354  7794 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:32.540264 27658 rpcz_store.cc:276] 0709 12:23:31.375073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.375217 (+   144us) service_pool.cc:224] Handling call
0709 12:23:32.539543 (+1164326us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b54000e4cd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:32.539698 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58765696}
I20260709 12:23:32.515278 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:32.549959 27583 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:23:32.550940 27737 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:32.551383 27737 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:32.553033 28332 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:32.553414 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.552348 27737 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.576136 27737 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:32.577332 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.578486 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.581136 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:32.582044 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 65795.7us
W20260709 12:23:32.582511 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.583546 27735 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:23:32.585019 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:32.586424 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 80487.3us
I20260709 12:23:32.587481 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.589404 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.591074 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.593119 27735 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:32.594879 27735 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:32.596670 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.597275 27736 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:32.599145 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.600360 27732 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:32.608309 27731 raft_consensus.cc:2468] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:32.609851 28292 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.508366  7833 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.093s
I20260709 12:23:32.601305  7667 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.009s
I20260709 12:23:32.610908  7797 raft_consensus.cc:2804] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:32.611449  7797 raft_consensus.cc:697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:32.612301  7797 consensus_queue.cc:237] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.616093 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.617640 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.581060  7444 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:32.619663 27731 raft_consensus.cc:2393] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:32.620656 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.596295  8011 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:32.621742 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.525334  7790 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.622371 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.527629  7997 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.079s
W20260709 12:23:32.625437 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 98145.3us
I20260709 12:23:32.637079 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.638401 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.627673 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.630527  7991 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:23:32.631124  8090 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
W20260709 12:23:32.631026 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 43683.3us
I20260709 12:23:32.641644 27580 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:32.631991  8076 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:32.625932  7444 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.610255  7998 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:32.627265 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.482543  8029 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.639073 27585 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:32.640437 27732 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:23:32.640753 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.642318 27628 leader_election.cc:400] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35.
I20260709 12:23:32.650827 27628 leader_election.cc:403] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:23:32.639112 27736 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:32.633761 27586 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:32.645423 27735 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:32.605507  7700 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:23:32.649195 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.642856 27630 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:32.652019 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.662132 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.653267 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.664105 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:32.665144 27732 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:32.657006 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:32.661223 27736 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:32.667826 27736 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 38 index: 22. (term mismatch)
I20260709 12:23:32.662940 27731 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.668237 27736 pending_rounds.cc:85] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 21
I20260709 12:23:32.669520 27736 pending_rounds.cc:107] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 37.22
I20260709 12:23:32.669971 27736 raft_consensus.cc:2889] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:32.666067 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:32.641366  7991 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:32.647637  8078 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.681087 27735 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:23:32.650075  7998 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:32.697587  7998 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:32.698487  7811 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:32.641695  8073 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:23:32.703745  7941 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:32.704608  7941 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:32.705725  7941 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.708802 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.710430  8073 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:32.710852  8073 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.711199  8073 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:32.714145  8051 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.853s	user 0.002s	sys 0.002s
I20260709 12:23:32.648798  7444 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:32.715538  7811 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:32.717412 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.718561  8030 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:32.719028 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.719265  8082 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:32.721256  8082 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:32.721715  8082 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.722232  8082 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:32.725418 27553 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.726514  8074 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:32.726814 27553 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.727561  8029 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:32.727720  8001 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:32.728251 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.719069  8030 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:32.730530  7637 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.678797  8031 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 34. Current term is 35.
I20260709 12:23:32.680429 27736 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:23:32.733052 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.790012 28334 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:32.765596 27737 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:32.742830 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:32.735100 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.793483 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:32.794044 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.794633 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.794776 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.795352 27661 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:32.795773 27655 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:32.796002 27731 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:32.796643 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.791884 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.801797 28337 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:23:32.804483 28333 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:32.734086 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 70468us
I20260709 12:23:32.735824 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:32.774894 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.775475 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.773697 27630 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:32.809391 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:32.662436  7231 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:32.783304 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.786945 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:32.812484 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.813190 27655 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:32.814086 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.815157 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:32.816123 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.816330 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:32.828619 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 101898us
I20260709 12:23:32.647024  7828 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.831264 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.780185  8075 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:32.840751 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.810606 28330 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:32.844389 27729 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:32.845734 27655 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.784162  8030 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:32.850358 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.851503 27553 leader_election.cc:400] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:23:32.850642 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:32.784446  7863 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.823925 27701 connection.cc:256] RPC call timeout handler was delayed by -1.2035669179999786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20260709 12:23:32.858943 27701 connection.cc:256] RPC call timeout handler was delayed by -1.1144669179998345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20260709 12:23:32.856875 28334 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:32.858369 27553 leader_election.cc:403] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:32.859462 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0703669179997632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20260709 12:23:32.783596  7794 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:32.863775 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.840801  7833 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:32.803471  8068 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.067s
I20260709 12:23:32.819973  7231 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.891839 28328 raft_consensus.cc:1240] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
W20260709 12:23:32.895673 27701 connection.cc:256] RPC call timeout handler was delayed by -1.0064669180001147s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
I20260709 12:23:32.920709 27701 leader_election.cc:400] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39.
I20260709 12:23:32.921075 27701 leader_election.cc:403] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:23:32.802294  7979 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:32.899668 27655 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:32.899573  8059 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:32.901585  7444 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.907773 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.914803  7828 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:32.681633  8049 consensus_queue.cc:1175] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 32 status { last_received { term: 32 index: 20 } last_committed_idx: 20 last_received_current_leader { term: 32 index: 20 } }
I20260709 12:23:32.922605 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:32.817525  7968 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.074s
W20260709 12:23:32.928277 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.860903  7984 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.861236  8031 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:32.847527  7989 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:32.897965 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:32.926548  7231 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:32.965332 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 67521.4us
W20260709 12:23:32.931416 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 80514.4us
W20260709 12:23:32.894194  8040 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:32.869865  7398 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:32.934234 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.870283  7863 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.855414  7482 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.075s
I20260709 12:23:32.947288 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4871) took 1592 ms. Trace:
I20260709 12:23:32.948637 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.947606  7581 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:32.937750  8074 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 32. Current term is 33.
I20260709 12:23:32.860836  7991 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:32.995031 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.000037 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:33.001535 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:32.993732  7581 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.004328 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:32.991524 27730 rpcz_store.cc:276] 0709 12:23:31.354821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.568599 (+213778us) service_pool.cc:224] Handling call
0709 12:23:31.568922 (+   323us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:31.569636 (+   714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.569666 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:31.570368 (+   702us) log.cc:844] Serialized 52 byte log entry
0709 12:23:31.722509 (+152141us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:31.768138 (+ 45629us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.794063 (+ 25925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.794175 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.461099 (+666924us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b180045d5a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:32.519648 (+ 58549us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b48001a01d8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:32.713362 (+193714us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:32.931015 (+217653us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b180045d5a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:32.947049 (+ 16034us) raft_consensus.cc:1685] finished
0709 12:23:32.947073 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.947222 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25418,"prepare.queue_time_us":48436,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":7681,"raft.queue_time_us":42949,"raft.run_cpu_time_us":2366,"raft.run_wall_time_us":975655,"spinlock_wait_cycles":82896128,"thread_start_us":151718,"threads_started":1,"wal-append.queue_time_us":271221}
I20260709 12:23:32.931901  7794 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:32.934986  7979 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.011554 27730 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
W20260709 12:23:33.013511 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.950816  7984 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.016127 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:33.016693 27730 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:23:33.020195 27730 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:33.012734  7991 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:23:33.027113 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.004488  7398 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:33.007494  7581 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:33.006284  8069 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:23:33.034673  8095 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260709 12:23:33.014422  8093 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:23:33.006945  8073 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.023818  8059 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:33.045130 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:33.029958  7637 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:33.055832 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 264017us
I20260709 12:23:33.058252 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:33.049918  8026 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:23:33.063993  7444 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.065577  7863 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.071975 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:33.047679  8059 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.077653 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:33.032642  7998 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.053268  7968 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39.
I20260709 12:23:33.079672 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:33.084625 27663 raft_consensus.cc:2376] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37. Re-sending same reply.
I20260709 12:23:33.057868  8052 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.085716 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:33.086380 27663 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:33.083940 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:33.087209 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:33.102484 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 181888us
I20260709 12:23:33.080607  8059 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:33.060916  7975 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.106731 27658 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:23:33.109709 27663 raft_consensus.cc:1240] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:33.076637  7637 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:23:33.145802  7637 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:33.146340  7637 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:33.110700 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.147595  7393 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.112851 27660 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:23:33.148867  7393 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:33.106706  7398 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.114538  7968 consensus_queue.cc:1059] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:23:33.152863  7968 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:33.153297  8106 raft_consensus.cc:3055] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:33.153720  8106 raft_consensus.cc:740] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:33.154132 27587 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
W20260709 12:23:33.116210  8096 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:23:33.154505  8106 consensus_queue.cc:260] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.120772  7998 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.155769  8106 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:33.157202 27656 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:23:33.158682 27656 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: [45.27-45.27]
I20260709 12:23:33.122602  8073 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.123292  8104 consensus_queue.cc:1059] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.011s
W20260709 12:23:33.125087  8100 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:23:33.127352 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4173) took 2170 ms. Trace:
I20260709 12:23:33.107045  8052 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.129829  8078 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:33.138659 27655 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:33.140793  7794 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.141992  7979 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:33.115672  8103 log.cc:927] Time spent T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
W20260709 12:23:33.179765 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 194267us
I20260709 12:23:33.237839 27553 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.171473 27657 rpcz_store.cc:276] 0709 12:23:30.956959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.961786 (+  4827us) service_pool.cc:224] Handling call
0709 12:23:30.962106 (+   320us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:31.051129 (+ 89023us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:23:31.051503 (+   374us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.300451 (+248948us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:31.336588 (+ 36137us) log.cc:844] Serialized 52 byte log entry
0709 12:23:31.336742 (+   154us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:31.336994 (+   252us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.378777 (+ 41783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.378861 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.803867 (+1425006us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b180036b380. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:33.127047 (+323180us) raft_consensus.cc:1685] finished
0709 12:23:33.127079 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.127252 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":137861,"prepare.queue_time_us":5183,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":25818,"raft.queue_time_us":138445,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":30582,"spinlock_wait_cycles":12833920,"thread_start_us":246102,"threads_started":1,"wal-append.queue_time_us":323532}
I20260709 12:23:33.240345 27553 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.179898 27660 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:23:33.180687 27658 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
W20260709 12:23:33.183477 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72612us
I20260709 12:23:33.242090 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.175122 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.243588 27660 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:23:33.241153 27657 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
W20260709 12:23:33.250361 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75363.9us
I20260709 12:23:33.251693 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:33.252264 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.253801 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.258404 27553 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:33.263267 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 489644us
I20260709 12:23:33.264990 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.245185  8073 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:33.247521  7984 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.266335 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.260839  8061 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:33.272547  8074 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:33.254885  8105 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:33.284657 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5309) took 1004 ms. Trace:
I20260709 12:23:33.263113  7444 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:33.242980  8030 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.299309 27588 rpcz_store.cc:276] 0709 12:23:32.280597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.282375 (+  1778us) service_pool.cc:224] Handling call
0709 12:23:32.283730 (+  1355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.372142 (+ 88412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.372192 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.372254 (+    62us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:32.372312 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.284398 (+912086us) spinlock_profiling.cc:243] Waited 88.1 ms on lock 0x7b540011c454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:33.284452 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.284597 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":202665856}
I20260709 12:23:33.290980 27552 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.305243 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.306846  7444 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:33.267675  7975 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.027s
I20260709 12:23:33.270035  7482 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:33.288623  7968 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:33.304963  8105 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.281165 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.273262  8113 raft_consensus.cc:3055] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:33.274289 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:33.336437 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:33.336838 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 55787.1us
I20260709 12:23:33.337147 27731 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:33.338096 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:33.338654 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.340682 27701 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.311463 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.315632  8114 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:33.293304  8061 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.317317 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4877) took 1468 ms. Trace:
W20260709 12:23:33.317451  8082 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.595s	user 0.015s	sys 0.000s
I20260709 12:23:33.291319  8115 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:33.296886  8073 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:33.318585  7833 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:33.320963  8030 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.321578 27585 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:33.402743  8082 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.347440 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:33.404297 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56997.8us
I20260709 12:23:33.405105 27701 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.406546  7794 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:33.406342  8059 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.335880  8113 raft_consensus.cc:740] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:33.407300  7794 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:33.408195  8113 consensus_queue.cc:260] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.409595  8113 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:33.404721  7581 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.348286 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:33.417600 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:33.318619  8052 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.412464 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5308) took 1282 ms. Trace:
I20260709 12:23:33.422106 27582 rpcz_store.cc:276] 0709 12:23:32.129818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.129977 (+   159us) service_pool.cc:224] Handling call
0709 12:23:32.130370 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.156334 (+ 25964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.156383 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.156444 (+    61us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:32.156505 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.412231 (+1255726us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b540011ba54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:33.412284 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.412406 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59088640}
I20260709 12:23:33.406368 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:33.412885  8105 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:33.413664 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:33.426681 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.378270  8110 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:33.437345 27734 rpcz_store.cc:276] 0709 12:23:31.849386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.849596 (+   210us) service_pool.cc:224] Handling call
0709 12:23:31.850026 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.904645 (+ 54619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.904676 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.905175 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.716948 (+811773us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b180012b200. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:32.898568 (+181620us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:23:32.902930 (+  4362us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.903242 (+   312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.903375 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.317006 (+413631us) raft_consensus.cc:1685] finished
0709 12:23:33.317106 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.317249 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11265,"prepare.queue_time_us":4107,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":202367,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":42129,"spinlock_wait_cycles":3555072,"thread_start_us":235492,"threads_started":2,"wal-append.queue_time_us":1028703}
W20260709 12:23:33.359062 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.440729 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.441104 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.441480 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:33.442576 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:33.442838 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:33.443367 27580 raft_consensus.cc:2368] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:33.443724 27581 raft_consensus.cc:2393] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:33.443718 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:33.444259 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:33.444541 27589 raft_consensus.cc:2368] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41.
I20260709 12:23:33.444434 27701 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.445252 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:33.446022 27586 raft_consensus.cc:2393] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 38: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:33.409819  8082 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.454553 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.348816  8104 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:33.349285  7984 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:33.343124 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.453881 27735 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:33.456429 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:33.378415  8061 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:33.464888 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:33.462145 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.463861 27588 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
W20260709 12:23:33.465263 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 122223us
I20260709 12:23:33.466794  7794 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:33.467486 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.343951  7968 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:33.473031 27704 leader_election.cc:400] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:33.473824 27704 leader_election.cc:403] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:23:33.437798  8105 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:23:33.445746  8052 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:33.418228  7231 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.477428 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:33.477139 27704 leader_election.cc:400] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41.
I20260709 12:23:33.481091 27704 leader_election.cc:403] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:33.463230  7634 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:23:33.482249 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:33.463284  8115 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:33.468439  8114 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:33.468626  8106 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:33.467953 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.498165 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:33.505543 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:33.480559  7984 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:33.499840  7581 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:33.476455  8105 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:33.512669  8105 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:33.479511  8110 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 34. Current term is 35.
I20260709 12:23:33.510548 27552 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:33.511125 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.516671 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:33.517354 27729 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 44.
I20260709 12:23:33.489506  8104 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 41
I20260709 12:23:33.529397 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:33.535883 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58246 (request call id 1936) took 1054 ms. Trace:
I20260709 12:23:33.536355 28328 rpcz_store.cc:276] 0709 12:23:32.480991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.483635 (+  2644us) service_pool.cc:224] Handling call
0709 12:23:32.483895 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.926628 (+442733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.535571 (+608943us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54003f5a54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:33.535795 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":937651584}
I20260709 12:23:33.540992  8059 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.542143  8115 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.545238  8075 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:33.545766  8075 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:33.557590 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:33.559684 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:33.559780  7984 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:33.560436 28333 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:33.560714  7581 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:33.563834 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:33.570874 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:33.571681 28333 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:33.572671 28328 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:33.515435  7968 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:33.573760 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 44.
I20260709 12:23:33.569159 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:33.584627 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.586058 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62667.6us
W20260709 12:23:33.589272 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60005us
I20260709 12:23:33.532725  8052 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:33.599445 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:33.589767  7827 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:33.573518  8031 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:33.573314  7444 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:33.605720 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 341615us
I20260709 12:23:33.599892  8114 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:33.608260 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:33.619019 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:33.620836 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:33.603688  7998 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.600580  7794 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.020s
I20260709 12:23:33.627821  7581 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:33.618566  8114 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.615758  7979 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:33.615051  8110 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.622820  8031 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.652472  8105 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.640180  7581 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.653802  8114 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:33.641971  8032 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.654701  8041 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.657568  7444 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.660081 27659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2233) took 1221 ms. Trace:
I20260709 12:23:33.662184  8061 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.664131 27735 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: [45.27-45.27]
I20260709 12:23:33.679402 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:33.721952 27662 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:33.680935 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:33.690400 27733 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 35 index: 26. (index mismatch)
W20260709 12:23:33.691071 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 48637.3us
I20260709 12:23:33.692718 27663 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:33.702049 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:33.703850 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:33.726948 27656 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:33.705677 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.612036  7827 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.711944 27732 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: []
I20260709 12:23:33.715466  8105 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:33.715819 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 102418us
I20260709 12:23:33.716842 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:33.738468 27662 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:33.718417 27659 rpcz_store.cc:276] 0709 12:23:32.438545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.438681 (+   136us) service_pool.cc:224] Handling call
0709 12:23:32.438983 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.439207 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:23:32.607711 (+168504us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.101756 (+494045us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.659571 (+557815us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.659666 (+    95us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:23:33.659753 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.659850 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.660008 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1258609,"prepare.queue_time_us":245682,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":7042,"raft.queue_time_us":573490,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":38953,"spinlock_wait_cycles":70190848,"thread_start_us":631400,"threads_started":3,"wal-append.queue_time_us":323014}
I20260709 12:23:33.748358 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:33.628129  8109 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.086s
W20260709 12:23:33.756407 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 149398us
W20260709 12:23:33.757782 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 52190.7us
I20260709 12:23:33.708232  7581 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:33.759549 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52916.8us
I20260709 12:23:33.760293 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:33.636134  7998 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:33.772789 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.774523 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.769920 27656 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:33.770948 27589 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [33.21-33.21]
W20260709 12:23:33.777010 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72418.7us
I20260709 12:23:33.778242 28292 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.643973  7637 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:33.787293 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 62177.3us
I20260709 12:23:33.790948  7827 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.791293  7559 consensus_queue.cc:1059] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:23:33.791908  7790 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.792380  7559 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:33.792814  7559 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.792946  8116 raft_consensus.cc:3055] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:33.793244  7559 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:33.793367  8116 raft_consensus.cc:740] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:33.794315  8116 consensus_queue.cc:260] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.800908  7637 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:33.782469  8029 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:33.754392  8041 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.802116  7559 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.804039  8041 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.749449  8032 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.806383  8032 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.768927  8105 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:33.783200  8052 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:23:33.776475  7667 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:33.817306  8004 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:33.709807  8110 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.818024  7831 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:33.783421  7231 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.818075  8078 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:33.807942  7968 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:33.817330  8028 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.006s
I20260709 12:23:33.823807  7667 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:33.836759 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 55286.5us
I20260709 12:23:33.819936  8116 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:23:33.850780  8119 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.544s	user 0.006s	sys 0.000s
I20260709 12:23:33.852195  8135 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:33.815536  8033 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:23:33.855400  7998 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.856674  8061 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.857625  8005 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.858131  8004 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:33.861397  8083 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.861506  8110 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.875892 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:33.894738 28328 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:33.880666 27656 raft_consensus.cc:1275] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:23:33.882458 27662 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:23:33.886906  8029 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.838738  8052 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:33.889705 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 134) took 28031 ms (client timeout 15000 ms). Trace:
I20260709 12:23:33.862027  8010 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.882943  7953 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:33.939085 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51209.4us
I20260709 12:23:33.885001  7982 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.888257 27657 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40. Ops: [39.27-39.27]
I20260709 12:23:33.890064  8134 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.890924  8028 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.942718  8114 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.954618 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:33.953831 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:33.892666  8004 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:33.947875 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 173419us
I20260709 12:23:33.963521 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4397) took 1189 ms. Trace:
I20260709 12:23:33.970510 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:33.970293 27736 rpcz_store.cc:276] 0709 12:23:32.774159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.865337 (+ 91178us) service_pool.cc:224] Handling call
0709 12:23:32.865740 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.090583 (+224843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.090605 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.090938 (+   333us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.291858 (+200920us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:23:33.291896 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.291967 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.850249 (+558282us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:33.952008 (+101759us) raft_consensus.cc:1685] finished
0709 12:23:33.952034 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.963451 (+ 11417us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":164658,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":7145,"spinlock_wait_cycles":9986432,"thread_start_us":425015,"threads_started":2,"wal-append.queue_time_us":215841}
W20260709 12:23:33.948642 27487 rpcz_store.cc:269] 0709 12:23:05.906845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.224046 (+5317201us) service_pool.cc:224] Handling call
0709 12:23:33.886210 (+22662164us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:28.739202 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:28.778590 (+ 39388us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:28.778644 (+    54us) write_op.cc:435] Acquired schema lock
0709 12:23:28.778665 (+    21us) tablet.cc:662] Decoding operations
0709 12:23:28.896085 (+117420us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:28.896190 (+   105us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:28.896220 (+    30us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:23:28.897963 (+  1743us) tablet.cc:701] Row locks acquired
0709 12:23:28.898026 (+    63us) write_op.cc:260] PREPARE: finished
0709 12:23:28.898217 (+   191us) write_op.cc:270] Start()
0709 12:23:28.898380 (+   163us) write_op.cc:276] Timestamp: P: 1783599808898139 usec, L: 0
0709 12:23:28.898420 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:23:29.034690 (+136270us) log.cc:844] Serialized 8254 byte log entry
0709 12:23:30.095788 (+1061098us) op_driver.cc:464] REPLICATION: finished
0709 12:23:30.415377 (+319589us) write_op.cc:301] APPLY: starting
0709 12:23:30.421410 (+  6033us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:30.626430 (+205020us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:30.626577 (+   147us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:30.862044 (+235467us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:30.863294 (+  1250us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=0,mrs_lookups=0
0709 12:23:30.863373 (+    79us) write_op.cc:312] APPLY: finished
0709 12:23:30.863858 (+   485us) log.cc:844] Serialized 188 byte log entry
0709 12:23:30.864750 (+   892us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:30.864968 (+   218us) write_op.cc:454] Released schema lock
0709 12:23:30.865054 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:30.958173 (+ 93119us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:30.958203 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4830363,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":245002,"apply.run_cpu_time_us":35676,"apply.run_wall_time_us":562254,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91907,"num_ops":21,"prepare.queue_time_us":248611,"prepare.run_cpu_time_us":30926,"prepare.run_wall_time_us":681263,"raft.queue_time_us":271063,"raft.run_cpu_time_us":44943,"raft.run_wall_time_us":286232,"replication_time_us":1181544,"thread_start_us":784864,"threads_started":4,"wal-append.queue_time_us":410888,"wal-append.run_cpu_time_us":67366,"wal-append.run_wall_time_us":2347550}]]}
I20260709 12:23:33.950686  7559 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.958310  7953 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.976716  8139 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:23:33.986943 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.972673 27701 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:34.001850 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4184) took 1191 ms. Trace:
I20260709 12:23:34.004644  8113 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:34.005117  8113 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:34.005934  8113 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.006930  8113 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:33.973524  7554 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:34.009023  7554 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.009404  7554 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:34.017046 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:34.017517  7827 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.017693 28328 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:34.018046  7827 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.938455  8083 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.022149  8083 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:33.976320  7790 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:34.023428  8099 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.129s	user 0.000s	sys 0.004s
I20260709 12:23:33.968461  8028 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.033937  8028 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:33.956796  8010 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.036399  8010 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.037058  8052 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.26, Last appended by leader: 26, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:33.966446  7982 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:34.044371  8098 log.cc:927] Time spent T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 1.076s	user 0.006s	sys 0.000s
I20260709 12:23:34.045892  7982 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:33.894630  8005 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.023061  7790 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.004108  8004 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.002318 27661 rpcz_store.cc:276] 0709 12:23:32.810769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.811051 (+   282us) service_pool.cc:224] Handling call
0709 12:23:32.811524 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.642164 (+830640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.642215 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.664286 (+ 22071us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.711157 (+ 46871us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:33.711213 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.001406 (+290193us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001f0254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:34.001465 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.001528 (+    63us) raft_consensus.cc:1685] finished
0709 12:23:34.001545 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.001783 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":669421,"raft.queue_time_us":648091,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":89,"spinlock_wait_cycles":450392960,"thread_start_us":46422,"threads_started":1,"wal-append.queue_time_us":154809}
I20260709 12:23:34.020282  8069 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.050084 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:34.078433 28333 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:34.051106 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:34.051797 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.080775 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.052878 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:34.075184 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 102595us
I20260709 12:23:34.082223 27580 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:34.086400 27585 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:34.094064 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:33.953809  8029 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.085039 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:34.095054 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 284172us
I20260709 12:23:34.077917  8109 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.966621  8134 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.052453  7953 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.146809 28330 raft_consensus.cc:2410] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 47 because replica has last-logged OpId of term: 45 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 26.
I20260709 12:23:34.146982 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.147198 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:34.134207  8109 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:33.974256  7581 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.142719 27736 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:34.145649 27731 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
W20260709 12:23:34.148851 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63920.7us
I20260709 12:23:34.153463 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.150796 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:34.133008  8075 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.000s
I20260709 12:23:34.094760  8004 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:34.157157 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 202391us
I20260709 12:23:34.073350  8135 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.073s
I20260709 12:23:34.165946 27587 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
W20260709 12:23:34.102785 27664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 4977) took 1308 ms (client timeout 1638 ms). Trace:
W20260709 12:23:34.169396 27664 rpcz_store.cc:269] 0709 12:23:32.793904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.797782 (+  3878us) service_pool.cc:224] Handling call
0709 12:23:34.102598 (+1304816us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000e7ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:34.102728 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2467625856}
I20260709 12:23:34.168974 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:34.171371 27589 raft_consensus.cc:1275] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 35 index: 26. (index mismatch)
I20260709 12:23:34.174312 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:34.091450  8009 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:34.089831  8110 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:34.150058  7979 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:34.175915 27738 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:34.088568  8069 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:34.182395 27580 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:34.150041  7827 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.185875 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.190785 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.187776 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:34.167645  7482 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.030s	sys 0.037s
I20260709 12:23:34.190917 27588 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:34.192270 27735 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:34.158701  7953 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.156101  7398 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.161832  7833 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:34.198657  7231 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:34.204583 27735 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: [37.24-37.24]
I20260709 12:23:34.175451  8074 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:34.206264  7953 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.215605  8074 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.217833  7953 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.170509  8139 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.053s
I20260709 12:23:34.224198  8139 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:34.224326  8029 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:34.224653  8139 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:34.182070  8009 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.180549  7811 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:34.217914  7581 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:34.213973  8136 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
W20260709 12:23:34.214726  8141 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:23:34.177448  7393 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.032s
I20260709 12:23:34.176815  8047 consensus_queue.cc:1048] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.032s
I20260709 12:23:34.193919  8110 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:34.161401  8075 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.230597 28328 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:34.234681 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:34.207898  8104 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.003s
I20260709 12:23:34.239094  7231 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:34.257290 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.257462  7231 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:34.194522  7482 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.031s	sys 0.038s
I20260709 12:23:34.241106 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.229902  8009 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:23:34.295457 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.192669  7788 consensus_queue.cc:1059] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 45.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.052s
I20260709 12:23:34.303992 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:34.231662  7965 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:34.306618 27735 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:23:34.237421  7667 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.308729 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.246073  8134 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.247051  7984 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:34.314568 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.249392  8005 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.209754  7398 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.318632  7965 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.273866  8104 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 40. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41.
I20260709 12:23:34.272459  8114 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.336905 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:34.227772  7833 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.305680  8029 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.309535 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:34.238111  7811 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.250753  8144 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.092s
I20260709 12:23:34.390338 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:34.321666  8033 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.065s
W20260709 12:23:34.409708 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.408071 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:34.410176 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:34.388545  8120 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:34.379242  7790 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.418489 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:34.422475 28328 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:34.423779 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.423936 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.424866 27659 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:34.342192  7984 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.225628  8139 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.428655 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:34.433125 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.434033 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.435096 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:34.406549  8118 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:23:34.444294 27660 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:34.445534 27663 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 46.
I20260709 12:23:34.447762 27657 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:34.459630 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:34.460386 28328 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:34.468561 27585 raft_consensus.cc:1240] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:34.435192  8083 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.401058  8029 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.435452  7667 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.477417 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.476706 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:34.482339 27659 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:34.491750 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:34.492419 27585 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:34.492940 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.484558 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.470360  8114 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:34.463459  8123 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:34.499058 27584 raft_consensus.cc:1240] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: [43.25-43.25]
I20260709 12:23:34.505769 27628 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.517053 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.476913  8139 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:34.527184 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.507944 28328 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:34.517930 27583 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:34.412904  8110 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.532490 27585 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:34.512437  8010 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:34.488996  7482 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:23:34.479773  8083 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.526171  8075 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.605571 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:34.606146 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 189513us
I20260709 12:23:34.495967  8156 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.057s
I20260709 12:23:34.564934 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.609064 27585 raft_consensus.cc:2393] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:34.491789  7667 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.568693 27583 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:23:34.523360  8114 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:34.556638  8145 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
I20260709 12:23:34.556926  7953 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:34.552479  8010 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:34.561515  8146 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.244s	user 0.001s	sys 0.000s
I20260709 12:23:34.565270  7811 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.637071 28292 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.643687 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.646386 27585 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:34.646353 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:34.634163  8147 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
I20260709 12:23:34.648101 28292 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.610836  7833 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.653137  7833 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:34.653738  7833 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.654059  7833 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:34.654208  8028 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:23:34.654673  8028 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:34.655652  8028 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 30, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.660076 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.661046 27581 raft_consensus.cc:1240] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:34.661168 27704 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:34.688848 27586 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:34.662547  7984 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.631597  7398 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.669703 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.670624 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.693336 27581 raft_consensus.cc:1240] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:34.674443  8114 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.651486  7393 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:34.651547  7788 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:23:34.636757  7953 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.676440 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:34.676918  7982 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:34.677449  7554 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.668s	user 0.009s	sys 0.001s
I20260709 12:23:34.677619  8074 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.651795  7482 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:34.679529  7735 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:34.645185  8010 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:34.681039  8148 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:23:34.681643  8154 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
W20260709 12:23:34.682111 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 47238.6us
I20260709 12:23:34.682595  8123 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:34.683100  7667 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.650754  8075 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:23:34.683596  7998 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:34.683841  8149 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 12:23:34.685096  8083 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:34.685953  8157 log.cc:927] Time spent T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:34.662031  8029 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.691870 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:34.681074  8004 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.001s
I20260709 12:23:34.696871  7393 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.651947  7965 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.722589 27586 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:34.730620  7998 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.731464 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:34.732477  8159 consensus_queue.cc:1059] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:34.733237  8166 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.733496  8159 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:23:34.734005  8159 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:34.733742  8166 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.713249  7482 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:34.734913  8159 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.736512  7482 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.738200 27664 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 38 index: 24. (term mismatch)
I20260709 12:23:34.738605 27664 pending_rounds.cc:85] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 23
I20260709 12:23:34.738844 27664 pending_rounds.cc:107] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 37.24
W20260709 12:23:34.739434  8137 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.781s	user 0.005s	sys 0.000s
I20260709 12:23:34.712234  7982 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.713304  7735 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:34.713311  7554 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.699146  7953 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:34.699137  8028 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:34.697654  7788 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.785254  7788 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:34.786406  8109 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.726869  8075 consensus_queue.cc:1243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 40.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:34.729148  8156 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:34.735527  8161 raft_consensus.cc:3055] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:34.792917  8161 raft_consensus.cc:740] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:34.793931  8161 consensus_queue.cc:260] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:34.739142 27664 raft_consensus.cc:2889] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.742055  7393 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:34.750193  8047 raft_consensus.cc:3055] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:23:34.696208 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 190514us
I20260709 12:23:34.770121 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.802191 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.771404 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.771572 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.772449 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.773273 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.804847 27660 raft_consensus.cc:2393] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:34.804736 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.805996 27659 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 33. Current term is 34. Ops: [33.21-33.21]
W20260709 12:23:34.801783 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 70404.3us
I20260709 12:23:34.807147 27657 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:34.808662 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:34.809492 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:34.811213 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:34.811936 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:34.811630 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 137731us
I20260709 12:23:34.818315 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.826372 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:34.826817 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.827890 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.837704 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:34.838733 27659 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:34.796851  8161 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:34.855552  7554 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:34.837121  7997 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:34.857471  7997 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:34.857853  7997 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:34.859409  7735 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.859949  7998 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:34.802101  8047 raft_consensus.cc:740] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:34.840732 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.862443 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:34.863035 28337 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:34.842731 27663 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40. Ops: [39.27-39.27]
I20260709 12:23:34.840943 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
W20260709 12:23:34.824735  8068 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.000s	sys 0.013s
I20260709 12:23:34.867849 27661 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:34.839494  7444 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.007s	sys 0.010s
W20260709 12:23:34.852038 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 176168us
I20260709 12:23:34.860518  7735 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:34.850455  8115 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:34.877817 27589 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:34.883011  8123 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:34.883845  7965 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.884820 27553 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:34.887853  7667 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:34.888329  8169 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.888650 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51062.2us
W20260709 12:23:34.888970  7989 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.012s	sys 0.004s
I20260709 12:23:34.889825 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:34.889436 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:34.891453 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:35.012519 28294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.011945  8169 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:34.891518 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:35.014472 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:35.015556 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.892052 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:34.901811  8143 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.571s	user 0.001s	sys 0.000s
W20260709 12:23:34.907109  8153 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260709 12:23:35.016499 27659 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:35.015815  7982 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.017407 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:34.928364  7482 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:23:35.034986 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.938651 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:34.939536  8009 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.625s	user 0.013s	sys 0.005s
W20260709 12:23:34.937443 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66908.8us
W20260709 12:23:34.870916  8082 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.019s	sys 0.029s
I20260709 12:23:34.852723  8075 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:23:35.042127  8082 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.019s	sys 0.030s
I20260709 12:23:35.042474  8075 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:34.841758  8156 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.036s
W20260709 12:23:35.043422 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 733887us
W20260709 12:23:34.871357  8068 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.000s	sys 0.014s
W20260709 12:23:34.871557  7444 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.007s	sys 0.010s
I20260709 12:23:34.951757 27733 raft_consensus.cc:2410] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 34 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
I20260709 12:23:34.867398  8144 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:34.956854 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 72130.1us
I20260709 12:23:35.051605 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:35.051507  8171 consensus_queue.cc:1048] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.100s
I20260709 12:23:35.052439 27729 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:34.952874  8073 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:35.008992 27663 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 45
I20260709 12:23:34.909134 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4408) took 1411 ms. Trace:
I20260709 12:23:35.022558 27655 raft_consensus.cc:1217] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Deduplicated request from leader. Original: 36.19->[38.20-43.23]   Dedup: 38.20->[39.21-43.23]
I20260709 12:23:35.024211  8059 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
W20260709 12:23:35.026584 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 137100us
I20260709 12:23:35.031860 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:35.010046  7989 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.013s	sys 0.004s
I20260709 12:23:34.957731  7965 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.047487  8113 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:35.060035 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.062738 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:35.095399 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:35.009541 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:34.955118  8115 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:35.056823 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.074378 27734 rpcz_store.cc:276] 0709 12:23:33.497556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.505211 (+  7655us) service_pool.cc:224] Handling call
0709 12:23:34.908894 (+1403683us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b54002a1f54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:34.909055 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2025295616}
I20260709 12:23:35.077771 27655 pending_rounds.cc:85] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 20
I20260709 12:23:35.105532 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:34.959545  8123 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:35.080921 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:35.035701  8131 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:23:35.105151 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95705us
I20260709 12:23:35.107911 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.108420 27655 pending_rounds.cc:107] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 40.21
I20260709 12:23:35.116451 27655 raft_consensus.cc:2889] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:35.110929 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.118891 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.124137 27737 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:23:35.127296 27734 raft_consensus.cc:2393] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 33: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:35.078346 28337 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:35.079151  7634 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.115165 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.125492 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:35.088043  8166 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:35.096558  8106 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:23:35.133306 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:35.090490  7794 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:35.137300 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.154487 27734 raft_consensus.cc:2410] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 40 because replica has last-logged OpId of term: 39 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 21.
I20260709 12:23:35.142851 27729 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 45.
I20260709 12:23:35.154966 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:35.112439 27479 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 27 to 32. New cstate: current_term: 32 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.088373  8073 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.139498 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:35.150774 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.110088  7979 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:35.073256  8009 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.152669 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:23:35.174783 28337 raft_consensus.cc:606] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.175087 27630 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:35.130954 27663 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 45.
I20260709 12:23:35.114114  7559 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:35.047528  8144 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.132107  8059 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:35.077005  8169 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.173004 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:35.175283 28337 raft_consensus.cc:614] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:35.159106 27729 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 45. Current term is 46. Ops: []
I20260709 12:23:35.176498 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:35.176498 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:35.149788  8069 consensus_meta.cc:287] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.946s	user 0.014s	sys 0.007s
I20260709 12:23:35.038398  8047 consensus_queue.cc:260] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.160050  7444 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:35.194485 28329 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:23:35.195091 27737 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:23:35.226532 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.144381  8131 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:35.187505 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 526375us
I20260709 12:23:35.200179 27479 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 30 to 32, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 32 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:35.230896 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.212360 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:35.216339 28333 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:35.204007 27661 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:23:35.222411 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:35.216202 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.234390 27729 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:35.235823 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:35.220788  8033 consensus_queue.cc:1059] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:35.198925  7811 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:23:35.221676  8184 log.cc:927] Time spent T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:23:35.151728  8083 consensus_queue.cc:1059] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 40.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.023s
I20260709 12:23:35.196067  8106 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:35.137122  7634 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.236900 27729 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:35.110617  8075 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.192175  8166 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:35.227769 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:35.160215  8109 consensus_queue.cc:1059] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.072s
I20260709 12:23:35.200218  8016 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:35.199668  8059 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:35.253450  8174 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.212s	user 0.001s	sys 0.003s
I20260709 12:23:35.253657  7965 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:35.254027 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 4909) took 1076 ms. Trace:
I20260709 12:23:35.254506  8162 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.165s
W20260709 12:23:35.254731  7833 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.600s	user 0.009s	sys 0.002s
I20260709 12:23:35.254336 27479 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:35.256080  8175 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:35.200897  8005 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:23:35.256809  8172 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:23:35.256969  7393 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.165241  7794 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.239437  8069 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.248989  7975 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:35.212262  7984 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:35.213162  7554 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
W20260709 12:23:35.264345 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:35.265821  8170 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.228s	user 0.001s	sys 0.001s
I20260709 12:23:35.238094  8159 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:23:35.267747 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:35.360687 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93026.8us
W20260709 12:23:35.271732 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.272979  7634 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:35.238886  8123 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.250342  8110 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.363973  8110 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.229794  8115 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.274216  8131 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:35.275655  8010 consensus_meta.cc:287] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.540s	user 0.002s	sys 0.015s
I20260709 12:23:35.276144  8047 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:35.273473 27730 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:35.278613 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.280923 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:23:35.278666 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.242774  8139 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.284785  8073 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.191733  7979 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.283120 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.291932 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.378691 27732 raft_consensus.cc:2368] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 44. Current term is 45.
I20260709 12:23:35.372907 27729 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
W20260709 12:23:35.379441 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 96423.5us
I20260709 12:23:35.373203 28337 raft_consensus.cc:606] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.380225 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.291273 27736 rpcz_store.cc:276] 0709 12:23:34.177411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.177563 (+   152us) service_pool.cc:224] Handling call
0709 12:23:34.177958 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.178398 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.178427 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.915945 (+737518us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:23:34.916500 (+   555us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.934578 (+ 18078us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:34.935483 (+   905us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.094054 (+158571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.094130 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.253755 (+159625us) raft_consensus.cc:1685] finished
0709 12:23:35.253836 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.253971 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":218748,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":162223,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":16036,"spinlock_wait_cycles":10124672,"thread_start_us":17551,"threads_started":1,"wal-append.queue_time_us":156301}
I20260709 12:23:35.373481 27733 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:35.326802  7393 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:35.315292  8005 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.381820  8005 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:35.370792  8010 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.383643  7794 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.369393  8144 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:35.366227  8178 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:23:35.385104  7700 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:23:35.385288  8010 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.371573  8069 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:35.371266  7788 consensus_meta.cc:287] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.000s
I20260709 12:23:35.388038  7788 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.389062  8009 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.390307  7788 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:35.391721 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 190621us
I20260709 12:23:35.326715  8106 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.329182 27734 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:35.286135  8059 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:35.336112 27479 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 32 to 33, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 33 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.362950  8179 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:35.380434 28337 raft_consensus.cc:614] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:35.316890  7833 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:35.372028  8161 consensus_meta.cc:287] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.003s
I20260709 12:23:35.400300 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:35.296154 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.395747  7581 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:35.401377  7634 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:35.418080 27479 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 24 to 27, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 27 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.431581  8109 raft_consensus.cc:3055] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:35.408731  7998 consensus_queue.cc:1059] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.024s
W20260709 12:23:35.432921  8195 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:23:35.438597 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:35.359833  7984 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.391489  8139 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.472605 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:35.480405 27731 raft_consensus.cc:606] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.481040 27731 raft_consensus.cc:614] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:35.480352 27479 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 20 to 21, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 21 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.488547 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.401582  8116 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:35.496646 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:35.499348 28337 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:35.497887 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:35.469763  8170 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:35.498020 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:35.554411 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 56519.7us
W20260709 12:23:35.517204 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 290774us
I20260709 12:23:35.501191 28334 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:35.438728  8166 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:35.446096  7979 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:35.552742 28337 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:35.543284 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:35.540954 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 46563.1us
I20260709 12:23:35.557552 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:35.575090 28329 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 46. Current term is 47. Ops: []
I20260709 12:23:35.487031  7794 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:35.439255  8016 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:35.439409  8109 raft_consensus.cc:740] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:35.570685 27479 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 27 to 30, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 30 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.578941 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:35.559171 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.586776 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:35.501415  7998 consensus_queue.cc:1243] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 44.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:23:35.590768 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:35.617954  8105 consensus_queue.cc:1059] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.009s
I20260709 12:23:35.615929  8113 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.013s
I20260709 12:23:35.631759 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:35.561429  8052 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.560701  7637 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:23:35.594578  7735 consensus_meta.cc:287] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.644s	user 0.002s	sys 0.009s
I20260709 12:23:35.569685  7444 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:35.661950 27731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:35.682879 27479 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 20 to 21, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 21 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.589339  7811 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:35.540401  8116 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.706305 27553 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:35.706543 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:35.707360 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:35.708784 27585 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:35.708920 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:35.704447 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:35.671233  7975 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:35.712037 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:35.733687  8199 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260709 12:23:35.744916 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:35.739500  7833 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:35.710451  8166 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.736056  8191 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:23:35.749356 27584 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: []
W20260709 12:23:35.735415  8183 log.cc:927] Time spent T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:23:35.746151  7997 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.888s	user 0.009s	sys 0.000s
I20260709 12:23:35.786696  7965 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.692129  8110 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:35.669041  8052 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.715932  8109 consensus_queue.cc:260] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.808198  8073 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:35.733065  8123 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:23:35.752183  8005 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.780519 27479 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.781675 27731 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:35.835143 27731 raft_consensus.cc:614] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:35.704087  7637 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:35.788625 27582 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:35.704638  7794 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:35.791524 27589 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
W20260709 12:23:35.804194 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 97964.7us
I20260709 12:23:35.804414 27583 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:35.805104 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61704.5us
I20260709 12:23:35.870425 28334 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
W20260709 12:23:35.783344  8201 log.cc:927] Time spent T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
W20260709 12:23:35.780604  8204 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.081s	user 0.010s	sys 0.000s
I20260709 12:23:35.779479  8156 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.886600  8156 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:35.812460 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 233601us
I20260709 12:23:35.809352  8016 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.786782  8033 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:35.818185  8113 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:35.861442 27479 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 33 to 34, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 34 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.808316  7811 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.827059  7735 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.814036  8105 consensus_queue.cc:1243] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 45.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:23:35.876228 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2258) took 1050 ms. Trace:
I20260709 12:23:35.871944 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:35.883785  7788 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:23:35.884454  8169 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:23:35.885051  8207 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.065s	user 0.000s	sys 0.003s
W20260709 12:23:35.885430  8208 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:23:35.885723  7989 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:35.783356  8202 log.cc:927] Time spent T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:23:35.821785  7997 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.889417 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:35.748001  7984 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:35.863246  8109 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:35.823892  7965 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.916070 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:35.981456 28333 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:23:35.874831  7637 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:35.935665 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 583947us
I20260709 12:23:35.865218  7975 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.879954  8139 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.875629  8073 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:35.900228  8144 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.948210 27582 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:23:35.890395  8191 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:35.918542  8212 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:35.979070  8028 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:36.031594  8028 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:23:36.032048  8218 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:23:35.994542  7788 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.032827  7788 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:35.890919 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:35.862887  7794 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.960440  8030 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:35.960973  8180 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.039850  8180 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.045459  7794 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.045898  8052 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:36.046321 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.965363 27583 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:35.968202 27660 rpcz_store.cc:276] 0709 12:23:34.826160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.829094 (+  2934us) service_pool.cc:224] Handling call
0709 12:23:34.840126 (+ 11032us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.841795 (+  1669us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:23:34.889309 (+ 47514us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b18002bca80. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e97258455 00007f0e9dd403c7 00007f0e9a3529a7 00007f0e9a3620bf 00007f0e9a362038
0709 12:23:34.898375 (+  9066us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.212240 (+313865us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.392373 (+180133us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:35.392879 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.393051 (+   172us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:35.393085 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.393154 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.876024 (+482870us) raft_consensus.cc:1685] finished
0709 12:23:35.876047 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.876172 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":453750,"prepare.queue_time_us":125708,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":72,"raft.queue_time_us":217127,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":29597,"spinlock_wait_cycles":14934400,"thread_start_us":267429,"threads_started":2,"wal-append.queue_time_us":363365}
I20260709 12:23:35.935900  7482 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.048334  7482 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.049418  7997 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.899997  8219 consensus_queue.cc:1059] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.064s
I20260709 12:23:35.891649  8059 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.051316  8219 consensus_queue.cc:1243] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 39.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:23:35.977066  7735 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.053092  8059 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.053833  7735 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.042749  8180 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.055033  7968 raft_consensus.cc:3055] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:23:36.061643  7968 raft_consensus.cc:740] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:36.062898  7968 consensus_queue.cc:260] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.27, Last appended by leader: 27, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.064545  7968 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:35.889010  8004 raft_consensus.cc:3055] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:36.065563  8004 raft_consensus.cc:740] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:36.066421  8004 consensus_queue.cc:260] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.067205 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.067777  8004 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:36.068308 28329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:36.069972 28337 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
W20260709 12:23:35.984802 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 112922us
I20260709 12:23:35.983312 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:36.071835 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 88661.4us
W20260709 12:23:35.998687 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 109384us
I20260709 12:23:36.072813 28295 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:35.907648  8047 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.537s	user 0.005s	sys 0.005s
I20260709 12:23:36.073763  8047 raft_consensus.cc:2804] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:36.074337  8047 raft_consensus.cc:697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:36.074569 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:36.075533 28334 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:23:36.075469  8061 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:36.076486  7965 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.083076 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.084735 27730 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 46.
I20260709 12:23:36.006644 27479 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:35.984449  7989 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.021543  8215 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:36.019275  8073 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.002760  7984 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:35.999047  7393 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:36.022240  7811 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.094442  8010 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:36.094971  8010 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:36.022744  8213 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:23:36.024001  8005 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.095995  8010 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.097211  8010 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:36.024724  8211 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:23:36.027458  8144 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.101478  8144 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:36.028299  8205 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:23:36.102571  8073 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.102888 28329 raft_consensus.cc:2393] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:36.028301  8191 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:35.964932  8033 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:35.997841  7953 consensus_meta.cc:287] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.193s	user 0.012s	sys 0.003s
I20260709 12:23:36.105242  8191 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.104898  7953 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.106305  8033 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.107354  7953 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.107770  8215 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:36.108829  8215 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.110035  8215 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:36.037020  8030 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.110708  7811 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.051016  8123 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.111193  7811 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.111665  8123 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.112214  8030 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.112857  8030 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.113272  7811 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:35.984449 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:35.967727  8113 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.003530 27658 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:36.077095  8061 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:36.082072 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.115000 27589 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:36.083563 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.089052 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.086972 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:36.116549 27479 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 25 to 30, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 30 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:36.061134  8206 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:23:36.092846 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:23:36.093140  8082 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:36.098870  7982 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.107185 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:36.034879 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144003us
I20260709 12:23:36.129467 28330 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:36.113633  8123 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.137846 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.122972 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.080394  7979 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:36.138769 27737 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:36.123638 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.139716 27584 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:36.123698 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.106115  8173 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:23:36.148749 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.032737  8028 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.051012  7482 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.133512 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.002303  7637 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.160405  8113 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.161438  7637 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.036660  8116 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:36.136427 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 742405us
I20260709 12:23:36.131206 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:36.137765 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.121181 28337 raft_consensus.cc:606] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:36.164624 28337 raft_consensus.cc:614] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:36.155561 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:36.167198 28295 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.145167  7735 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.149829 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.128017 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:36.113333  8030 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.151721 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:36.138569  8139 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.181089  7989 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.181342  8139 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.183506 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:36.185477 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:36.186319 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:36.148399  8219 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.188822 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4227) took 1257 ms. Trace:
I20260709 12:23:36.189198  8219 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:36.189814 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 205451us
I20260709 12:23:36.190579 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.192770 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.189210 27663 rpcz_store.cc:276] 0709 12:23:34.931036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.216097 (+285061us) service_pool.cc:224] Handling call
0709 12:23:35.216424 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.398039 (+181615us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:23:35.408722 (+ 10683us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.056763 (+648041us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b18002834e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b3f3 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:36.108311 (+ 51548us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.108498 (+   187us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.108969 (+   471us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.109109 (+   140us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:36.109282 (+   173us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.109399 (+   117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.109562 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.188598 (+ 79036us) raft_consensus.cc:1685] finished
0709 12:23:36.188620 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.188756 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1384847,"prepare.queue_time_us":13565,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":92,"raft.queue_time_us":856420,"raft.run_cpu_time_us":450,"raft.run_wall_time_us":32810,"spinlock_wait_cycles":11943552,"thread_start_us":180835,"threads_started":1,"wal-append.queue_time_us":414167}
I20260709 12:23:36.195114  8232 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:36.153086 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.086536  8052 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:36.137126  7393 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:23:36.165527 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:36.140491 28331 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:36.149852  7979 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.195277 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:36.204505  8191 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.204852 27663 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:36.206539 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.207193 27581 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:36.197000  7790 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:36.197176  8052 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.208496  8052 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:36.210489  7581 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.197683 28337 raft_consensus.cc:2368] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42.
I20260709 12:23:36.199604 27655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4225) took 1271 ms. Trace:
I20260709 12:23:36.200693 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.175997 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.212384 27655 rpcz_store.cc:276] 0709 12:23:34.929929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.021926 (+ 91997us) service_pool.cc:224] Handling call
0709 12:23:35.022328 (+   402us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:23:35.147374 (+125046us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:23:35.193344 (+ 45970us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.324412 (+131068us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b18003c6be0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee823a 00007f0e9a352a2a
0709 12:23:35.382880 (+ 58468us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b18003c6be0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9e21c58f 00007f0e9725b571 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd
0709 12:23:36.105983 (+723103us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:36.106718 (+   735us) log.cc:844] Serialized 78 byte log entry
0709 12:23:36.106899 (+   181us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:36.107125 (+   226us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.107470 (+   345us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.107742 (+   272us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.107874 (+   132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.107944 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.199389 (+ 91445us) raft_consensus.cc:1685] finished
0709 12:23:36.199412 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.199554 (+   142us) inbound_call.cc:177] Queueing success response
0709 12:23:36.206012 (+  6458us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
Metrics: {"mutex_wait_us":612440,"prepare.queue_time_us":60846,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":102,"raft.queue_time_us":590369,"raft.run_cpu_time_us":1229,"raft.run_wall_time_us":131933,"spinlock_wait_cycles":143334144,"thread_start_us":249354,"threads_started":1,"wal-append.queue_time_us":622857}
I20260709 12:23:36.200403 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.216045 27583 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:36.203460 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:36.217357 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.178489 27660 raft_consensus.cc:3060] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:36.103142  7984 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.155053  7982 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.197322  7393 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.196532 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.177825 27656 raft_consensus.cc:3060] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:36.226204 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.236124 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4231) took 1138 ms. Trace:
I20260709 12:23:36.150521  8173 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:36.211129  7581 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.258844 27584 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:36.258893 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:36.259737 27580 raft_consensus.cc:2368] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:36.135352  8156 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.236636 27663 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:36.085896  8047 consensus_queue.cc:237] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.236622 27662 rpcz_store.cc:276] 0709 12:23:35.097732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.275820 (+178088us) service_pool.cc:224] Handling call
0709 12:23:35.276185 (+   365us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:35.416298 (+140113us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:35.416332 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:35.433082 (+ 16750us) log.cc:844] Serialized 52 byte log entry
0709 12:23:35.594094 (+161012us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:35.659101 (+ 65007us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.123785 (+464684us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.235806 (+112021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.235878 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.235917 (+    39us) raft_consensus.cc:1685] finished
0709 12:23:36.235928 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.236071 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":619311,"prepare.queue_time_us":66181,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":80,"raft.queue_time_us":696262,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":259,"thread_start_us":272314,"threads_started":2,"wal-append.queue_time_us":278090}
I20260709 12:23:36.020079 27586 raft_consensus.cc:3060] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:36.252556 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.253952 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.272888 27587 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:36.256184 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.275110 27660 raft_consensus.cc:2468] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:36.276171 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:36.178082  7788 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.257390 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.261061 27659 raft_consensus.cc:2393] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 39: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:36.292604 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.293166 27659 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
I20260709 12:23:36.293929 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.294538 27659 raft_consensus.cc:2393] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:36.261138 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:36.262595 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:36.295827 27581 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:36.198822  7979 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:36.181106  8030 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.177712  8113 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.264698 27734 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 19. (term mismatch)
I20260709 12:23:36.302394 27734 pending_rounds.cc:85] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 18
I20260709 12:23:36.302749 27734 pending_rounds.cc:107] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 37.19
I20260709 12:23:36.303053 27734 raft_consensus.cc:2889] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:36.265422 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.305380 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.306872 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.227720  8144 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:23:36.228086  7637 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:36.170468 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:36.267237 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.225411  7634 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.327495 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:36.229775 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62806.2us
I20260709 12:23:36.270591 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.330246 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:36.331075 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:36.331146 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 160779us
W20260709 12:23:36.331719 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.332147 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.332676 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:36.332940 27630 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:36.333220 27630 leader_election.cc:403] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Cancelling election due to peer responding with higher term
I20260709 12:23:36.333431 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.248916 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:36.228896  8082 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.201992 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.287660  8217 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:36.290026  7982 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.290087  8139 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.237725  7393 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:36.294831 27660 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:36.322741  8144 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.229597  8191 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.243808  8180 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:36.203565  7735 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.249439  8105 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.203614  8219 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.308583  8009 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.322495  7637 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.395555  7637 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:36.334939  7634 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.336339  8214 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.399915  8214 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.370265 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.402197 28332 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:23:36.404091 28330 raft_consensus.cc:1240] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 38. Current term is 39. Ops: [38.24-38.24]
I20260709 12:23:36.375169  8144 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:36.342262  7827 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.065s
I20260709 12:23:36.384318 28335 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 33.
I20260709 12:23:36.337641  7788 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:36.327849  8113 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.364439  7984 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.342746  8082 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.394608  8009 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.368793  7581 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:36.508610 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.268355 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:36.516607 27584 raft_consensus.cc:2410] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 38 because replica has last-logged OpId of term: 37 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 21.
W20260709 12:23:36.522915 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 320795us
I20260709 12:23:36.609162 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.523669 27628 leader_election.cc:400] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
W20260709 12:23:36.611434 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 88509us
I20260709 12:23:36.612512 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:36.343264 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.612913  8113 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.307217 27656 raft_consensus.cc:2468] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 47.
I20260709 12:23:36.615062 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.616863 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.617852 27655 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
W20260709 12:23:36.525698 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 192659us
I20260709 12:23:36.526507 27583 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:23:36.527258 27658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36354 (request call id 2260) took 1700 ms (client timeout 1973 ms). Trace:
I20260709 12:23:36.620465 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:36.527365 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50014 (request call id 4987) took 1719 ms. Trace:
W20260709 12:23:36.620548 27658 rpcz_store.cc:269] 0709 12:23:34.826848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.827663 (+   815us) service_pool.cc:224] Handling call
0709 12:23:36.526715 (+1699052us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:36.526964 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2701729152}
I20260709 12:23:36.334055 27662 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:36.621857  7975 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.622071  8217 raft_consensus.cc:493] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:36.323493 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.623256  8217 raft_consensus.cc:515] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.529343 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4229) took 1597 ms. Trace:
I20260709 12:23:36.531219 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4228) took 1599 ms. Trace:
I20260709 12:23:36.357091 27729 raft_consensus.cc:3055] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:36.626516 27729 raft_consensus.cc:740] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:36.624773 27661 rpcz_store.cc:276] 0709 12:23:34.931617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.272475 (+340858us) service_pool.cc:224] Handling call
0709 12:23:35.272848 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.099763 (+826915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:36.099843 (+    80us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.100933 (+  1090us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.200470 (+ 99537us) spinlock_profiling.cc:243] Waited 9.9 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:36.337334 (+136864us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:36.349051 (+ 11717us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.528425 (+179374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.528570 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.528668 (+    98us) raft_consensus.cc:1685] finished
0709 12:23:36.528847 (+   179us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.529121 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1005039,"prepare.queue_time_us":11230,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":831264,"raft.run_cpu_time_us":2028,"raft.run_wall_time_us":96159,"spinlock_wait_cycles":22775552,"thread_start_us":235666,"threads_started":1,"wal-append.queue_time_us":352191}
I20260709 12:23:36.625980 27664 rpcz_store.cc:276] 0709 12:23:34.931315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.261539 (+330224us) service_pool.cc:224] Handling call
0709 12:23:35.261910 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.262166 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:23:35.262525 (+   359us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.083793 (+821268us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b180006a780. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:23:36.406698 (+322905us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:36.530367 (+123669us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:36.530509 (+   142us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:36.530645 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.530735 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.531013 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":122573,"prepare.queue_time_us":9148,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":723010,"raft.run_cpu_time_us":1528,"raft.run_wall_time_us":304730,"spinlock_wait_cycles":110830208,"thread_start_us":562267,"threads_started":2,"wal-append.queue_time_us":236051}
I20260709 12:23:36.628367 27729 consensus_queue.cc:260] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.543038 27702 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:36.544319 27704 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:36.358395 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.357506 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:36.637197 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 94634.5us
I20260709 12:23:36.638474 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:36.639605 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95599us
I20260709 12:23:36.334461 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.641008 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:36.643015 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.644876 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.646533 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.621750 27657 rpcz_store.cc:276] 0709 12:23:34.808076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.808185 (+   109us) service_pool.cc:224] Handling call
0709 12:23:34.808424 (+   239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.808615 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:23:34.975440 (+166825us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.204799 (+1229359us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:36.525988 (+321189us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:36.526117 (+   129us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:36.526313 (+   196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.526660 (+   347us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.526982 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":868820,"prepare.queue_time_us":121037,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":888183,"raft.run_cpu_time_us":879,"raft.run_wall_time_us":101485,"spinlock_wait_cycles":25778432,"thread_start_us":812494,"threads_started":3,"wal-append.queue_time_us":316915,"wal-append.run_cpu_time_us":1347,"wal-append.run_wall_time_us":1001541}
I20260709 12:23:36.336992 27734 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 21. (term mismatch)
I20260709 12:23:36.650480 27734 pending_rounds.cc:85] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 20
I20260709 12:23:36.650909 27734 pending_rounds.cc:107] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 33.21
I20260709 12:23:36.356459 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.552744 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.337574 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:36.386025  7393 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.625963  7735 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.398895  7979 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.392305  8105 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.373608  8173 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.419214  7788 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:36.466620  8214 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:36.458724  8228 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:36.594236  7634 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.598402 27581 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:36.601918  8082 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:36.609853  8004 consensus_meta.cc:287] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.541s	user 0.014s	sys 0.000s
I20260709 12:23:36.624112  8191 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.625777 27663 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:36.295161 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:36.638705  8009 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.649462  8110 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:36.703106  8110 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.703708  8110 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:36.357858 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.700503 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.708449 27587 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:36.510865 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:36.712515 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 202092us
I20260709 12:23:36.712671  7482 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:36.630093 27662 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 44 index: 23. (term mismatch)
I20260709 12:23:36.713891 27662 pending_rounds.cc:85] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 22
I20260709 12:23:36.714246 27662 pending_rounds.cc:107] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 43.23
I20260709 12:23:36.714527 27662 raft_consensus.cc:2889] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:36.610826  8028 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:36.660266 28336 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 46. Current term is 47.
I20260709 12:23:36.663858  7788 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:36.655642  8185 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:23:36.655015 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 317361us
I20260709 12:23:36.653874  8219 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.015s
I20260709 12:23:36.664453 27733 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:36.653043 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:36.653985 27736 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 44 index: 23. (index mismatch)
I20260709 12:23:36.666060 27656 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: []
W20260709 12:23:36.651261 28333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.014s	sys 0.031s
I20260709 12:23:36.622913  7975 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.651304 27734 raft_consensus.cc:2889] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:36.661443  8113 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:36.662590  8105 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.658859  7393 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:36.683931 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 71639.3us
I20260709 12:23:36.654754  7581 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:36.729768  8231 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 40. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:36.656867  8221 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:36.696234 27586 raft_consensus.cc:2468] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:36.681639  8216 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:36.723367  8185 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.724776  8233 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:36.724959  8116 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:23:36.734037 28333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.015s	sys 0.032s
I20260709 12:23:36.731258 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:36.730648  8224 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:36.730340  7581 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.725531  8113 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.734683  7559 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:36.735970  8185 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:36.737007  8233 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:36.746786  8113 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:23:36.754309 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 101541us
I20260709 12:23:36.763594  8166 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:36.764324  8115 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:36.764657  7989 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:23:36.764868  8115 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.764988 27586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56296 (request call id 2310) took 1052 ms. Trace:
I20260709 12:23:36.765717 27730 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:36.761312 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:36.762377 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 142258us
I20260709 12:23:36.768471 27586 rpcz_store.cc:276] 0709 12:23:35.712915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.744601 (+ 31686us) service_pool.cc:224] Handling call
0709 12:23:36.764747 (+1020146us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540011c954. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:36.764907 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462691584}
I20260709 12:23:36.758387  8233 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:36.749003  8191 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:23:36.760696  7979 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.761533  7559 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:36.770329  8115 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:36.785657 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.785984 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 177124us
I20260709 12:23:36.793540 27628 leader_election.cc:400] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42.
I20260709 12:23:36.805131 27628 leader_election.cc:403] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:23:36.793543 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.783258  8106 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:36.807703  8217 leader_election.cc:290] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:36.765578  8221 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.803077 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.818733 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:36.824602  7559 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.832183 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.834137  7554 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:23:36.836366  8216 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42.
I20260709 12:23:36.836971  8221 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:36.836529  7581 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:36.835150 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 66696.4us
I20260709 12:23:36.845368  8105 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.853359  8105 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:36.858628 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:36.860073 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.861055 27584 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:36.862572 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.863595 27729 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:36.863965 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.871783 28295 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:36.872875 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 83343.3us
I20260709 12:23:36.874225  8047 raft_consensus.cc:2804] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:36.874888  8047 raft_consensus.cc:697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:36.875728 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.877136 27583 raft_consensus.cc:2393] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:36.876009  8047 consensus_queue.cc:237] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.879643  8166 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:36.891286  8166 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.883831 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.884294 27628 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:36.895452 27628 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 46. Current term is 47.
I20260709 12:23:36.895746 27628 leader_election.cc:403] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:36.887522 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.973s (SENT) [suppressed 7 similar messages]
I20260709 12:23:36.897716  8214 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:36.898434  8214 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:36.899749  8214 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:36.792618  8219 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
W20260709 12:23:36.846961 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67061us
I20260709 12:23:36.890386 27584 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:36.835124  7975 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:36.811740  8106 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:36.887689 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44884.5us
I20260709 12:23:36.923305 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:36.924242 27736 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:23:36.923070 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80577.3us
W20260709 12:23:36.943806 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.913879  8106 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:36.947080 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.926201  8116 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 47
I20260709 12:23:36.970019 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:36.919135  8217 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:23:36.931818  8191 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:36.933107  8221 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.955642  8243 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:23:36.987774 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:37.008100 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 206551us
W20260709 12:23:37.011695 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 136369us
I20260709 12:23:37.022961 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:36.956769  8033 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.045s
I20260709 12:23:37.030164 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:37.034409 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:37.041476  7634 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:37.040947  7788 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.042174  7634 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.043166  7968 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:37.072371  8191 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.072801  7968 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.074214  8173 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:37.043388  7979 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:37.075088  7979 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.077440  7979 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.077654  7968 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.018519  8105 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:37.079937  8246 log.cc:927] Time spent T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.052s	user 0.000s	sys 0.007s
I20260709 12:23:37.044000  8222 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:37.044791  8052 consensus_meta.cc:287] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.836s	user 0.005s	sys 0.015s
I20260709 12:23:37.053516 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:37.054163  7554 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:37.018314  8233 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.045720 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:37.064262  8144 consensus_meta.cc:287] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.659s	user 0.008s	sys 0.009s
W20260709 12:23:37.069882  7637 consensus_meta.cc:287] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.674s	user 0.014s	sys 0.007s
W20260709 12:23:37.070930  8245 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:23:37.071223  7634 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.071288  8221 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.072041 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.088780 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.102752 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.095050 27736 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 47.
I20260709 12:23:37.109495 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.121549 27581 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
W20260709 12:23:37.124780 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61595.1us
W20260709 12:23:37.132616 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87415.7us
I20260709 12:23:37.092689  7975 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:37.151937 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.080292  8185 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.140635 27630 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:37.131688  8144 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.125841  8113 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.160251  8082 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:37.160915 27582 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:37.201272 27729 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 24. (term mismatch)
I20260709 12:23:37.163791  8221 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.172477  7788 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:37.173354 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119988us
I20260709 12:23:37.177018  8233 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.177968  7482 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.108750  8217 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:37.185738 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:37.196596 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.190083  8033 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.144441  8231 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:37.083820  8222 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.198385  8105 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:37.202771 27729 pending_rounds.cc:85] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 23
I20260709 12:23:37.203662 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:37.203784 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 106518us
I20260709 12:23:37.206161  7554 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:37.248605 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:37.248972  7788 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:23:37.249523 28330 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:37.183635  8166 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.210325  7989 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:37.204476  8082 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.204377 27589 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:37.238669 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.247769 28328 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 34: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:37.249571  7788 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:37.140043  7637 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.210325 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:37.258278 27729 pending_rounds.cc:107] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 38.24
W20260709 12:23:37.233136 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47519.9us
I20260709 12:23:37.259298 27729 raft_consensus.cc:2889] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:37.208455  8191 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.123850  8052 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.268380 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:37.256009  7554 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:37.261971  8082 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:37.251724  8105 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:37.282838 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.274489  8221 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:37.256639  8222 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.312704 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.315536  8123 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.315920 27737 raft_consensus.cc:2393] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:37.316202  8123 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.319893  8052 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.261266  8106 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.319725  7554 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.321424 27735 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 44: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:37.322397  8106 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.324203  7637 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.324641 28328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58264 (request call id 1901) took 1203 ms. Trace:
I20260709 12:23:37.283288  8033 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.325168 28328 rpcz_store.cc:276] 0709 12:23:36.121021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.121161 (+   140us) service_pool.cc:224] Handling call
0709 12:23:37.324324 (+1203163us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400962654. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:37.324546 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2383956864}
W20260709 12:23:37.326735  8194 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 39 current term: 39 is higher.
W20260709 12:23:37.340195  8115 consensus_meta.cc:287] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.543s	user 0.008s	sys 0.004s
I20260709 12:23:37.340780  8115 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:37.344707  8109 consensus_meta.cc:287] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 1.361s	user 0.009s	sys 0.016s
I20260709 12:23:37.345682  7953 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.346325  7953 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.348760  8131 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.349416  8131 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.293956  7975 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.061s
I20260709 12:23:37.358228  8113 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.306883  8221 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:37.363188  8221 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:37.366379  8116 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader pre-election lost for term 46. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 46. Current term is 47.
I20260709 12:23:37.366439  7637 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:37.367604  8233 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:37.367476  7637 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.368485  8233 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.370453  8033 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.373894 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.374920 27581 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:37.375833 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:37.376353  7637 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.377374 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:37.378353 28328 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:37.371213  8233 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.387205  8166 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:23:37.388217 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 178075us
I20260709 12:23:37.390251  8144 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.392132 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:37.393410 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:37.394397 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.298367  8105 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:37.395298  8115 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.383026 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.403680  7953 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:37.408952 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:37.292757  8191 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.428697  8082 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.430757  8123 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.432590 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:37.432687  8185 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.435386 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4427) took 1237 ms. Trace:
I20260709 12:23:37.435948 27729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4441) took 1082 ms. Trace:
I20260709 12:23:37.384495  7788 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 30, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.284013  7482 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:37.437866  8110 consensus_meta.cc:287] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.733s	user 0.016s	sys 0.001s
I20260709 12:23:37.481842 27736 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:37.464787 28328 raft_consensus.cc:2468] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
W20260709 12:23:37.468245 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 85319.6us
I20260709 12:23:37.473793 27589 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:37.475464 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.477028 27729 rpcz_store.cc:276] 0709 12:23:36.353706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.356567 (+  2861us) service_pool.cc:224] Handling call
0709 12:23:36.356924 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:36.767167 (+410243us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b5800689fd0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:23:37.302910 (+535743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.435871 (+132961us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39024000}
I20260709 12:23:37.480151 27581 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:37.476981 27737 rpcz_store.cc:276] 0709 12:23:36.198384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.218719 (+ 20335us) service_pool.cc:224] Handling call
0709 12:23:37.435068 (+1216349us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000a5ad4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:37.435269 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2418215680}
I20260709 12:23:37.485723 28330 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:37.471146  7968 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:23:37.500804 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.502794 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:37.512094  7797 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:37.513273 27588 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:23:37.505755  7968 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:37.514922  7968 raft_consensus.cc:3060] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:37.521638 27702 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.525728 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:37.531840 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:37.535293  8194 raft_consensus.cc:2804] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:23:37.489529  8110 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.538405  8115 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:37.553884 27580 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 42 index: 27. (index mismatch)
W20260709 12:23:37.554010 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75699.1us
I20260709 12:23:37.539170  8131 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:37.552776 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.564563 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.566306 27656 raft_consensus.cc:1240] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:37.480791  8191 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.561164  8115 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.575161 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:37.579715 27587 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:37.583426 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:37.562799  8194 raft_consensus.cc:697] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:37.603971 28292 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:37.588408  8115 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:37.601359  7634 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.628469 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 77222.6us
W20260709 12:23:37.629374 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 45022.9us
I20260709 12:23:37.633057  8106 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:37.594353  8144 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.042s
I20260709 12:23:37.645915 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.650159 27658 raft_consensus.cc:1275] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:23:37.651404 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.635305  8082 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:37.659503 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 405968us
I20260709 12:23:37.659670 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:37.643595  7637 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:37.673314 27581 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:37.679147 27627 leader_election.cc:400] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 44. Current term is 45.
I20260709 12:23:37.679600 27627 leader_election.cc:403] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:37.670922 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.680962 27585 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:37.676962 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:37.662268  8047 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:37.696187 27589 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:37.686008 27704 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.701107 27704 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:37.696870  8166 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.701939  8219 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:23:37.697085  8257 log.cc:927] Time spent T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:23:37.704732 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:37.706378 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:37.657260  7634 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:37.717382 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.702102  8166 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:37.717823 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.706689  8047 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:37.685536  7482 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:37.716066 27655 raft_consensus.cc:3060] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:37.724359 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:37.704022  8214 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.014s	sys 0.014s
I20260709 12:23:37.677049  8106 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.718384 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:37.699993  7393 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:37.729120 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65159.9us
I20260709 12:23:37.704476  8173 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election lost for term 44. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 44. Current term is 45.
W20260709 12:23:37.730691  8214 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.014s	sys 0.014s
I20260709 12:23:37.731424  8052 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:37.745474 27589 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:23:37.744156 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:37.704191  7637 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.759057 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:37.757576  7393 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.759920 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:37.758262  8180 raft_consensus.cc:2804] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:23:37.771735 27588 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:37.760727  8109 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.779042 28330 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:37.762708  8052 raft_consensus.cc:697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:37.770547  8231 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.842073 28337 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 40 index: 27. (term mismatch)
I20260709 12:23:37.842562 28337 pending_rounds.cc:85] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting all ops after (but not including) 26
I20260709 12:23:37.842831 28337 pending_rounds.cc:107] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting uncommitted NO_OP operation due to leader change: 39.27
I20260709 12:23:37.843112 28337 raft_consensus.cc:2889] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:37.844128 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70870.7us
W20260709 12:23:37.824255  8254 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:23:37.734392  8194 consensus_queue.cc:237] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:37.849475 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 93358.3us
I20260709 12:23:37.816694  8180 raft_consensus.cc:697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:37.818495  7559 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:37.849097 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 129568us
W20260709 12:23:37.844973 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72656.8us
W20260709 12:23:37.852350 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 199405us
I20260709 12:23:37.853910 28295 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:37.840757  8191 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:37.861543 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 160511us
I20260709 12:23:37.823770  8109 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.854389  7559 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.842831  7968 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.886852 27663 raft_consensus.cc:1275] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 42 index: 27. (index mismatch)
I20260709 12:23:37.846349  8231 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.857436  8219 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:37.843268  8047 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.908078  8271 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:37.948588 27661 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:23:37.949158  7968 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:37.949091  8271 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.949844 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.950539 27661 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:37.950966 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:37.908834  8260 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:23:37.909423  7975 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:37.867049  8144 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:37.920528  8004 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.924834 27585 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 43.
I20260709 12:23:37.926851 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5865d310fddd49789e9de0a6a7bb260a" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:37.885659  7559 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:37.928041 27736 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:37.929613 27630 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:37.933169 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.939358 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:37.948585 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66409.4us
I20260709 12:23:37.971279 27729 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:37.941076 27656 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 47.
W20260709 12:23:37.940935 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 54510.4us
I20260709 12:23:37.972096 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.882334  8222 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:37.947690  8258 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.188s	user 0.000s	sys 0.002s
I20260709 12:23:37.948233 27738 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:23:37.953359 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.975669 27661 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 45 because replica has last-logged OpId of term: 43 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 24.
W20260709 12:23:37.944996  8256 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
I20260709 12:23:37.930167  8113 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.956664  8052 consensus_queue.cc:237] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.979259 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:37.979485 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:37.980334  8222 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:37.973318 28295 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:37.981637  8033 consensus_queue.cc:1048] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:37.983233  8115 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:37.984740 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:37.984992 27582 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 29. Current term is 30. Ops: []
I20260709 12:23:37.985735 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:38.010386 27735 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:23:37.985924  7637 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:37.986250 27731 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:23:38.009546 27661 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:37.986946 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.011601 27657 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:23:37.987811  8109 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:37.988837 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 59305.8us
I20260709 12:23:37.998014 27736 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:23:37.951426  8148 consensus_queue.cc:1059] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.109s
I20260709 12:23:38.006940 27730 raft_consensus.cc:2410] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 45 because replica has last-logged OpId of term: 43 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 24.
I20260709 12:23:37.967870  8144 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.989441  8004 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.008023 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:38.020107 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:38.020764 27729 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:37.994151  8216 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.001657  8106 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.009191  8271 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.026577 27630 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:38.019181 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.079257 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.033910 27658 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:23:38.021585  8144 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:38.001886  8113 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.008675  8180 consensus_queue.cc:237] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.023947  7637 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.047554 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:38.045246 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51797.1us
I20260709 12:23:38.059664 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.087819 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:38.088889 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45607.9us
I20260709 12:23:38.090991 27736 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
I20260709 12:23:38.038014  8116 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:38.044620  8259 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:23:38.059185  8262 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:38.095198 28333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58264 (request call id 1904) took 1273 ms. Trace:
I20260709 12:23:38.098150 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.029719  8216 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.107165 27737 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:38.101768 28333 rpcz_store.cc:276] 0709 12:23:36.814561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.828188 (+ 13627us) service_pool.cc:224] Handling call
0709 12:23:36.912156 (+ 83968us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:37.013598 (+101442us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.013645 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.044328 (+ 30683us) log.cc:844] Serialized 26 byte log entry
0709 12:23:37.469437 (+425109us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:37.469478 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.469551 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.088016 (+618465us) raft_consensus.cc:1685] finished
0709 12:23:38.088036 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.088165 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18701,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":11287,"thread_start_us":424672,"threads_started":1,"wal-append.queue_time_us":627982}
I20260709 12:23:38.109614  7393 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:38.110334  8267 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:23:38.110621  8276 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:38.107189 27553 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:38.117075 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 130251us
I20260709 12:23:38.117946 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.097206  8033 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:38.095883  8116 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:38.100390  7554 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.126016  8115 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.131084 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:38.132490 28334 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:38.033345  8219 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 43.21, Last appended by leader: 21, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.094856  8233 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:23:38.136842  8233 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:38.137228  8233 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.138095  8233 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.138639  8216 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.139456  8231 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.140336  8004 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.140909  8278 raft_consensus.cc:2804] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:38.142050  8113 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.142163  7634 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.106662  8109 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.025s
I20260709 12:23:38.145893 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.146826  7975 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:38.148437 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.150478  8222 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:38.151721  8105 consensus_meta.cc:287] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.671s	user 0.005s	sys 0.007s
W20260709 12:23:38.152364  8268 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:23:38.152181  8105 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.153227 27732 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:38.153553  8275 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:38.155970 27735 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:38.158178  8105 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.119784  8123 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:38.155077 27655 raft_consensus.cc:2468] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 43.
I20260709 12:23:38.163352 27729 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:38.124022  8116 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:38.153240  8221 consensus_meta.cc:287] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.789s	user 0.011s	sys 0.001s
W20260709 12:23:38.189474 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.188356 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.192514 27731 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:23:38.199060 27661 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 47 index: 31. (index mismatch)
I20260709 12:23:38.124382  7637 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.188983  8123 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:38.173194  8282 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:23:38.165274  7975 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:38.212349 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.159739  7554 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:38.200791 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 95827.1us
I20260709 12:23:38.205742 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.213054 28334 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:38.214794 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:38.215543 27553 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:38.219635  8217 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:38.220211  8217 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:38.220543  8123 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.221163  8217 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.223870  7989 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:38.226994 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.183797  8148 consensus_queue.cc:1048] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.028s
I20260709 12:23:38.231158  7554 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.194067  8221 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.234977 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:38.249420 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 144031us
I20260709 12:23:38.251262 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:38.249300  8278 raft_consensus.cc:697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:23:38.248140  8166 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.516s	user 0.011s	sys 0.001s
I20260709 12:23:38.251029  7989 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:38.299528 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:38.304265  8030 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.271118  8214 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:38.324824 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.327827 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.329587 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.267431  8291 raft_consensus.cc:2804] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:38.330521 27580 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:38.337348 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.267791  8166 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.299968  7953 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.038s
I20260709 12:23:38.325212  8214 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.347064  8274 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.348483 28333 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
W20260709 12:23:38.349457 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 79455.7us
W20260709 12:23:38.347272  8289 log.cc:927] Time spent T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.133s	user 0.001s	sys 0.001s
W20260709 12:23:38.354254 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 139530us
I20260709 12:23:38.335865  8115 raft_consensus.cc:2764] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election decision vote started in defunct term 42: won
I20260709 12:23:38.339253  8291 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:38.356362  8214 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:38.365556 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.367609  8110 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.367857  8278 consensus_queue.cc:237] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.304908  8030 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.370024  8274 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.360983 27630 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:38.385352 27659 raft_consensus.cc:2368] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40.
I20260709 12:23:38.374285 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.385998 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:38.456055 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 63472.9us
W20260709 12:23:38.462252 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 75840.6us
I20260709 12:23:38.355877  7637 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:38.477245 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 103009us
I20260709 12:23:38.372499  8274 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.373373  8291 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:38.485442 27586 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:23:38.486616 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.371238  8030 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.454936  8110 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.498272  8110 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.500449  8110 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.503674 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.487919  8297 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.504997  8297 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.505367  7554 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:38.506414  7554 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:38.506752  7554 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.491586 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.509790  8144 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.513346  8166 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.514092  8047 raft_consensus.cc:3055] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:38.514510  8047 raft_consensus.cc:740] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:38.515776  8116 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.455003  7559 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.504407 28330 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:38.510895  8221 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.516918  8214 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.517722  8233 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:38.522063 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:38.492936 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.491993 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:23:38.495440  8298 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:23:38.460242  8275 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:38.463308  7975 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:38.540285 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:38.541306 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.550810  8144 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:38.551748  8082 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:38.552620  8082 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.574882 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:38.544842  8030 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:38.557504  7559 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.544842  8221 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:38.540148  8275 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.579793 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.580615 27736 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:38.561170 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:38.531777  8047 consensus_queue.cc:260] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.569748 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:38.571223 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.572264  8214 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.579808 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:38.558794  8291 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.585335 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:38.572757 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 86624.1us
I20260709 12:23:38.573694 27656 raft_consensus.cc:1240] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:38.582525 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:38.586381 27732 raft_consensus.cc:2368] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 46. Current term is 47.
I20260709 12:23:38.586923  8276 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:38.589350  8109 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.589779  8224 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:38.615108  8109 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.615302  8224 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.617225  8109 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.617980 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.589968  7559 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.619086 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:38.619196  7559 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.590246  7953 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.624063 27655 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:38.623190 28332 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:38.629711  8276 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.631517 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:38.631911  8276 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.627203 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.633571 28332 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:38.634374 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.627259 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:38.635265  8030 raft_consensus.cc:2764] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election decision vote started in defunct term 39: won
I20260709 12:23:38.626724 28333 raft_consensus.cc:1240] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: [40.27-40.27]
I20260709 12:23:38.636059  8030 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:38.637941 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.597805 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.598857 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.599504  7554 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.603018 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.603355 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:38.605386  8082 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.607823 27657 raft_consensus.cc:1240] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 46. Current term is 47. Ops: []
I20260709 12:23:38.607671 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.608361  8219 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:38.585793  8221 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:38.610404 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.649637 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:38.650959 27586 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:23:38.653412  7797 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260709 12:23:38.653643 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.653975 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.654670  8144 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:38.656268 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.656718  7554 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:38.610920  8116 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.613111  8291 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.584609  8275 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:38.642138 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.642499 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.640372 27663 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:38.647073  8219 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.662490  8219 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:38.663722 27658 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:23:38.643803 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:38.667096 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.621412  7953 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.675608 27660 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 39 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 23.
I20260709 12:23:38.683084 27733 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:38.684489 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:38.685338 28333 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:38.686726  8105 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:38.686982 28337 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:38.681800 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.647228  7393 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.702103 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.703509 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.704209 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:38.704686 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.705890 28292 leader_election.cc:400] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40.
I20260709 12:23:38.707191 27660 raft_consensus.cc:2410] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 39 because replica has last-logged OpId of term: 38 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 19.
I20260709 12:23:38.710250 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:38.679056  8224 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.696004  8144 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:38.712268 27586 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:38.705394  7393 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.712821 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.701191  8082 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.014s
I20260709 12:23:38.707042  7559 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.683175  8028 consensus_queue.cc:1059] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.026s
I20260709 12:23:38.724378  8028 raft_consensus.cc:3055] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:38.724792  8028 raft_consensus.cc:740] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:38.725651  8028 consensus_queue.cc:260] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.726936  8028 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:38.730338  7953 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.731043 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:38.711943  8180 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.020s
I20260709 12:23:38.732200  8180 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:38.732451  7393 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:38.732615 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.733259 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:38.733521  8297 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.689357  8116 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.020s
I20260709 12:23:38.732574  8180 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:38.748294  8180 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.749464  8180 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:38.752944  8291 consensus_queue.cc:1059] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:38.753902  8291 raft_consensus.cc:3055] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:38.754349  8291 raft_consensus.cc:740] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.758728 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.759086  8291 consensus_queue.cc:260] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.739583 27660 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
W20260709 12:23:38.756731  8307 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.186s	user 0.002s	sys 0.000s
I20260709 12:23:38.756989  8297 consensus_queue.cc:1059] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:38.758010 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.798049 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.735469 27732 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:38.760962  8173 consensus_queue.cc:1059] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20260709 12:23:38.761854  8305 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.206s	user 0.002s	sys 0.000s
I20260709 12:23:38.819659 27733 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:38.820163 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.820765 27733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:38.821293 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:38.821588 27733 raft_consensus.cc:606] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.821913 27733 raft_consensus.cc:614] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:38.822355 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.822928 27662 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:38.823472  8283 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:23:38.823858 27586 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:38.824352 27732 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:38.764142 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.825578 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.825759 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.831254 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.732569 27588 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:38.792223 27587 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:38.824851 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:38.801488 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.801869  8047 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:38.806066 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.807789 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:38.845454 27736 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 47 index: 31. (index mismatch)
I20260709 12:23:38.847159 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:38.847021  7989 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:38.807950 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.809799 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.809973  8297 raft_consensus.cc:3055] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:38.848052 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.810578 27737 raft_consensus.cc:1240] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:38.810493 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.811069  7975 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:38.747117  8219 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.812052 27659 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:23:38.848453 27580 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:38.812674 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.812827 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.813555 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.815501 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.850050 27729 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:38.850162 27655 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:38.816183 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.817245 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.819720 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.836126 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:38.794888 27585 raft_consensus.cc:2468] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:38.836642 27731 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:38.838326 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:38.838465 27630 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:38.839002  8173 raft_consensus.cc:3055] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:38.840809 27732 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:23:38.860363 27660 raft_consensus.cc:2410] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 45 because replica has last-logged OpId of term: 42 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:23:38.859762 27737 raft_consensus.cc:1275] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:23:38.861152 27664 raft_consensus.cc:2368] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:38.851843 27658 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:38.863770 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.854635 27734 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:38.853231 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:38.875659 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.878100 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:23:38.878159 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.879109 27655 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:38.881664 27735 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:38.883420 27730 raft_consensus.cc:1240] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:38.885629 27583 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:38.879140 27704 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:38.860176  8123 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:38.859850  8297 raft_consensus.cc:740] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.867743  8191 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:38.868741  8291 raft_consensus.cc:3060] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:38.860371  8173 raft_consensus.cc:740] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.878855 28337 raft_consensus.cc:606] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:38.891426 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:38.871812  8276 raft_consensus.cc:2804] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:38.847683  8082 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.037s
I20260709 12:23:38.901441 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.883467  8217 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:38.912235  8276 raft_consensus.cc:697] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.913391  8276 consensus_queue.cc:237] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:38.918875 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.892613  8105 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:38.920701  8105 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:38.921684  8105 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.923763  8217 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.924343  8217 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:38.927496  8222 raft_consensus.cc:2804] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:38.927827 27587 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 43 index: 21. Preceding OpId from leader: term: 44 index: 22. (index mismatch)
I20260709 12:23:38.928318  8222 raft_consensus.cc:697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:38.929698  8222 consensus_queue.cc:237] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.935441  8173 consensus_queue.cc:260] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.884775  8116 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 43.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.029s
I20260709 12:23:38.937009  8262 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.937515  8262 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.868149  8287 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.035s
I20260709 12:23:38.937515  8116 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:38.941821  8116 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:38.942643  8116 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 43.21, Last appended by leader: 21, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:38.943892  8116 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:38.944741  8219 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:38.946447  8297 consensus_queue.cc:260] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:38.947861  8297 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:38.952387 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:38.953284 28335 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:38.956344  8148 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:38.956491 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65210.6us
I20260709 12:23:38.956836  8148 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:38.957206  8148 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:38.896803  8283 consensus_queue.cc:1048] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 25, Time since last communication: 0.057s
I20260709 12:23:38.959589  8283 consensus_queue.cc:1243] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 46.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:23:38.965284 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.966122 27663 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:38.966661 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.966969 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:38.967890 27655 raft_consensus.cc:2410] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 46 because replica has last-logged OpId of term: 43 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 24.
I20260709 12:23:38.968888 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.970440 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.971100 27659 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:38.972173 27660 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:23:38.973459 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:38.974061 27657 raft_consensus.cc:3060] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:38.869275  8115 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.036s
I20260709 12:23:38.972100 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:38.978564 27730 raft_consensus.cc:1217] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Deduplicated request from leader. Original: 42.25->[44.26-47.28]   Dedup: 44.26->[45.27-47.28]
I20260709 12:23:38.979135 27730 pending_rounds.cc:85] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 26
I20260709 12:23:38.979411 27730 pending_rounds.cc:107] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 46.27
I20260709 12:23:38.979718 27730 raft_consensus.cc:2889] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:38.983485  8047 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:23:38.986380 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:37660
I20260709 12:23:38.987217 27738 raft_consensus.cc:606] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.987670 27738 raft_consensus.cc:614] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:38.989107  8194 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:38.989606 28337 raft_consensus.cc:1275] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:23:38.991477 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:23:38.992010 28337 raft_consensus.cc:606] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:38.967892 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:38.992987 27658 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:38.996774  7968 consensus_queue.cc:1048] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:38.992409 28337 raft_consensus.cc:614] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:39.001607 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.003809 28337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
I20260709 12:23:39.004523 28337 raft_consensus.cc:606] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:39.007393 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:39.009495 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:39.010795 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:39.017807 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:39.022012 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.025626 27664 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:38.953310  8275 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:39.031602 27662 raft_consensus.cc:2368] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42.
I20260709 12:23:39.044129  8275 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:38.940953  8030 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:39.045743 27582 raft_consensus.cc:1240] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 33. Current term is 34. Ops: []
W20260709 12:23:39.047170 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.028136 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.052248 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:39.053963 27629 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.054713 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.056192 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.057361 27655 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:39.070175 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.080801 27657 raft_consensus.cc:2468] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:39.123383 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.124792 27655 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 39 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 23.
W20260709 12:23:39.130381 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 283280us
I20260709 12:23:39.135248 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.136430 27657 raft_consensus.cc:2368] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38.
I20260709 12:23:39.131731 28335 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:39.144371 27587 raft_consensus.cc:1275] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:23:39.147574 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:39.162475  8131 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:39.168303  8131 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.168819  8131 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:39.173578  8261 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:39.173429  8115 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.003s
I20260709 12:23:39.176373 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.180154 27552 leader_election.cc:304] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:39.182050 27662 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 37 index: 22. (term mismatch)
I20260709 12:23:39.191020 27662 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 21
I20260709 12:23:39.191298 27662 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 38.22
I20260709 12:23:39.191573 27662 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:39.191766 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:39.182834 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.183707  7975 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.182065 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.185577 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:39.198307 27732 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 45.
I20260709 12:23:39.194634 27731 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:39.199913 27662 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: [37.21-37.21]
I20260709 12:23:39.242305 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:39.248042 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.162042 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.156836 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:39.249590 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 92879.8us
I20260709 12:23:39.197072  8082 raft_consensus.cc:2749] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:39.204802  8123 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:39.265549  8123 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:39.266047  8123 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:39.271813  8320 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:39.273555  8320 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:23:39.261698 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.276587 27657 raft_consensus.cc:2376] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40. Re-sending same reply.
W20260709 12:23:39.282701  8325 log.cc:927] Time spent T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.151s	user 0.001s	sys 0.002s
I20260709 12:23:39.255353 27655 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:39.286820  8180 consensus_queue.cc:1048] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:23:39.287801  8148 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.289529 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:39.252117  7975 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.290422  8148 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:39.292994 27661 raft_consensus.cc:2410] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 34 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 20.
I20260709 12:23:39.253315  8105 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:39.294360  8105 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:39.295701  8105 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.296208  8217 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:39.298358  8278 raft_consensus.cc:993] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:39.313416 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:39.314292 28335 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
I20260709 12:23:39.284341 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.320626 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:39.335680 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74020.6us
I20260709 12:23:39.358177  8131 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:39.251423  8261 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.088s
I20260709 12:23:39.360219 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.361081 27657 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:39.293022  7975 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:39.375074  8131 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:39.376761  8123 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:39.378929  8123 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:39.339672  8279 raft_consensus.cc:1081] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:39.384435 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:39.352012  8327 log.cc:927] Time spent T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.220s	user 0.000s	sys 0.002s
I20260709 12:23:39.387770 28335 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:39.390378 27588 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 42: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:39.316884  8217 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:39.343402  8317 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.097s	user 0.002s	sys 0.004s
W20260709 12:23:39.403973 27581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.013s	sys 0.028s
W20260709 12:23:39.404498 27581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.013s	sys 0.028s
I20260709 12:23:39.408674 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.409407 27656 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:39.410533 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.411819 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.412891 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.408214 27583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:56296
I20260709 12:23:39.416198 27583 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:39.416630 27583 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:39.419665 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.421236 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.426759 27656 raft_consensus.cc:2393] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:39.429160 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.430316 27657 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:39.404686  8115 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.018s
I20260709 12:23:39.434681 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.435817 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.448913 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.459581 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:39.461270 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 267545us
I20260709 12:23:39.464403 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:39.483973 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.485406 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.486512 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.487634 27657 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
I20260709 12:23:39.488488 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.520531 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.720455 27656 raft_consensus.cc:2368] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42.
I20260709 12:23:39.668354 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.652125 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:39.722157 27732 raft_consensus.cc:3060] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 45
I20260709 12:23:39.668581 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:39.728906 27663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.027s	sys 0.059s
I20260709 12:23:39.516582 27583 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.580313 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:39.734704 27730 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:39.567509 27580 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:39.721467 27658 raft_consensus.cc:2376] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44. Re-sending same reply.
W20260709 12:23:39.738332 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58673.3us
I20260709 12:23:39.541976 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.741246 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
W20260709 12:23:39.541594 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.592939 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:39.579864 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:39.745076 27729 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:39.746089 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:39.543561 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:39.519996  8307 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:23:39.625741  8222 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.008s	sys 0.031s
I20260709 12:23:39.746765 27658 raft_consensus.cc:2376] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40. Re-sending same reply.
I20260709 12:23:39.759874 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:39.562909  8326 log.cc:927] Time spent T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:23:39.521483  8131 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:39.757356 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.760545 27655 raft_consensus.cc:2376] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44. Re-sending same reply.
I20260709 12:23:39.618788  8219 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:39.762666 27583 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:39.738246  7559 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:39.763031 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:39.760375 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 216885us
I20260709 12:23:39.766454 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.748847 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.817023 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.819667 27655 raft_consensus.cc:2393] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:39.820051 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.823035 27657 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:39.845928 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.762369  8332 log.cc:927] Time spent T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:23:39.884091 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.847537  7559 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.847821  8219 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.889364 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:39.894060 27658 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 43 index: 21. Preceding OpId from leader: term: 44 index: 22. (index mismatch)
I20260709 12:23:39.898602 27655 raft_consensus.cc:1275] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:23:39.896322  7559 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:39.933902 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 108514us
I20260709 12:23:39.815691  8123 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:39.961628 27732 raft_consensus.cc:2468] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
W20260709 12:23:39.759924  8222 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.009s	sys 0.031s
I20260709 12:23:39.952114 27585 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
W20260709 12:23:39.952272 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63035.1us
I20260709 12:23:39.762879  7554 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.134s
W20260709 12:23:39.954880 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 336405us
W20260709 12:23:39.949255  8105 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.038s	sys 0.102s
I20260709 12:23:39.952785  7559 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.972141 27729 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:39.993430 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:39.994125 27732 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:39.995416 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:39.997085 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:39.998955 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:39.970876  8105 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.038s	sys 0.102s
I20260709 12:23:40.026906 27732 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:40.026048 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:40.032167 27737 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:40.033775 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.034134 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.034657 27738 raft_consensus.cc:3060] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.037153 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.039743 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:40.040536 27735 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 44 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:23:39.977835  8194 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.972642  8123 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:40.044734  8123 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.048767  8262 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:40.052312  8329 log.cc:927] Time spent T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:40.053594 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5216) took 1026 ms. Trace:
I20260709 12:23:40.054049 27582 rpcz_store.cc:276] 0709 12:23:39.027664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.048735 (+ 21071us) service_pool.cc:224] Handling call
0709 12:23:39.049181 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:39.049628 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:39.049673 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:39.050314 (+   641us) log.cc:844] Serialized 26 byte log entry
0709 12:23:39.340752 (+290438us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:39.340793 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.340864 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.053306 (+712442us) raft_consensus.cc:1685] finished
0709 12:23:40.053339 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.053524 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":381,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":289992,"threads_started":1,"wal-append.queue_time_us":704559}
I20260709 12:23:40.057763  8283 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.058228  8105 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:23:40.058341  8283 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:39.971302  8217 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.073s
W20260709 12:23:40.061553  8336 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:23:40.067683 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.072921 27730 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:40.043282  7554 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:40.075564  7554 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:40.076463  7554 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.077745  7554 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.068091  8194 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:40.081568  8330 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:23:40.082785 27660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2301) took 1058 ms. Trace:
I20260709 12:23:40.083278 27660 rpcz_store.cc:276] 0709 12:23:39.024028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.037261 (+ 13233us) service_pool.cc:224] Handling call
0709 12:23:39.037723 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:39.058199 (+ 20476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:39.058255 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:39.058940 (+   685us) log.cc:844] Serialized 26 byte log entry
0709 12:23:39.397510 (+338570us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:23:39.397570 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.397668 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.082457 (+684789us) raft_consensus.cc:1685] finished
0709 12:23:40.082490 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.082713 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":337674,"threads_started":1,"wal-append.queue_time_us":703893}
I20260709 12:23:40.062146  8219 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.003902  7559 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.092136  7559 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.099634 27738 raft_consensus.cc:2468] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:40.119004 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:40.119628 28334 raft_consensus.cc:3055] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:40.119956 28334 raft_consensus.cc:740] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:40.082155  8262 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:40.130822 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49176.3us
I20260709 12:23:40.126992 28334 consensus_queue.cc:260] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.132901 28334 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:23:40.130292  8331 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.176s	user 0.002s	sys 0.000s
I20260709 12:23:40.132134  8262 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.145421 27627 leader_election.cc:400] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 46. Current term is 47.
W20260709 12:23:40.157552 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 258266us
W20260709 12:23:40.164173 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 160915us
I20260709 12:23:40.166854 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.169473 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.172959 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.154640  7953 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 41.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
W20260709 12:23:40.198180 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52956.1us
I20260709 12:23:40.198329 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.198637  8262 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.199056 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.199497 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.200131  7953 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:40.200497  8297 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:40.200555  7953 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:40.201112 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.202286 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:40.203053  8278 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.201574  7953 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.21, Last appended by leader: 21, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.204419  8357 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.204919  8173 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:40.211741  8341 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:23:40.219524  8278 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.233486  8351 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:40.234516 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50014 (request call id 5021) took 1269 ms. Trace:
I20260709 12:23:40.259929 27661 rpcz_store.cc:276] 0709 12:23:38.965393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.971835 (+  6442us) service_pool.cc:224] Handling call
0709 12:23:40.234212 (+1262377us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54000c2454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:40.234436 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":721997056}
I20260709 12:23:40.242173  8308 raft_consensus.cc:993] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:40.243183  8082 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Stepping down as leader of term 40
W20260709 12:23:40.226310 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 266836us
I20260709 12:23:40.270287 28334 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
W20260709 12:23:40.304382 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 103321us
I20260709 12:23:40.305439 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:40.308583  7559 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:23:40.313445 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:40.229544  8350 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:23:40.334456 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.366746  8283 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:40.369465  8278 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:40.312377  7554 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:40.370466  7554 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:40.370867  7554 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.294745  8219 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:40.296056  8082 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:40.294121  8262 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:40.384521 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.385457 28335 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:40.303746  8308 raft_consensus.cc:1081] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:40.391099 28334 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:40.387063 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:40.395779 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 82902.1us
I20260709 12:23:40.400419 27553 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:40.403532 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.405154 28337 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:40.374455  8351 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:40.231945  8357 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:40.410145 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 446239us
I20260709 12:23:40.389367  8131 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:23:40.408660 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 206688us
I20260709 12:23:40.415302 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.399281  8217 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:40.426537 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:40.420706 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 247973us
I20260709 12:23:40.431432 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.435492 27627 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.438385 27701 leader_election.cc:400] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38.
I20260709 12:23:40.413528  8262 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.441143 27701 leader_election.cc:403] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:40.415409 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.416100  8131 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:40.453081 27585 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:40.434361  8343 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.443722  8353 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:40.440438  8116 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:23:40.453117  8335 log.cc:927] Time spent T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.418s	user 0.004s	sys 0.000s
I20260709 12:23:40.473311 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4274) took 1286 ms. Trace:
I20260709 12:23:40.424388  8352 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.043s
I20260709 12:23:40.464180 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.440960  8217 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.466902  8116 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:40.460001  8343 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.455441  8350 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:40.451479  8262 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.470711  8368 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:23:40.453922  8131 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:40.489985  8357 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:40.493278  8217 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.495585  8343 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:40.482270  8364 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:40.497399  7559 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:40.502012 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.502710 28334 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:40.433868  8082 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.503997  8082 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:40.507416  8322 consensus_queue.cc:1175] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" responder_term: 40 status { last_received { term: 40 index: 27 } last_committed_idx: 27 last_received_current_leader { term: 40 index: 27 } }
I20260709 12:23:40.474220  8353 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:40.510146  8353 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.510370 27553 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:40.516229  8217 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:40.516659  8217 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:40.517055  8217 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:40.477456  8219 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 37. Current term is 38.
W20260709 12:23:40.531932 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 96542.4us
I20260709 12:23:40.542639 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:40.480952 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 65643.6us
I20260709 12:23:40.473749 27662 rpcz_store.cc:276] 0709 12:23:39.186707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.291589 (+104882us) service_pool.cc:224] Handling call
0709 12:23:39.291915 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:39.321812 (+ 29897us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:23:39.321857 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:39.346753 (+ 24896us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.254702 (+907949us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b18003d08a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:40.405994 (+151292us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:40.406036 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.406102 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.471170 (+ 65068us) raft_consensus.cc:1685] finished
0709 12:23:40.471191 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.473246 (+  2055us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1172,"raft.queue_time_us":43095,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6186,"spinlock_wait_cycles":68665600,"thread_start_us":151068,"threads_started":1,"wal-append.queue_time_us":1094267}
I20260709 12:23:40.460004 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.506157  8131 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.555936 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:40.544265  8291 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.026s
I20260709 12:23:40.551824  8370 consensus_queue.cc:1059] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:23:40.581810  8349 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.011s
I20260709 12:23:40.582883  8349 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:40.583351  8349 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:40.584236  8349 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.585518  8349 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:40.573714  8358 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.552035  8219 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:40.586388  8358 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.593047 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:40.591818 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:40.593767 27738 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
W20260709 12:23:40.530807  8123 consensus_queue.cc:1175] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" responder_term: 41 status { last_received { term: 40 index: 27 } last_committed_idx: 27 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:23:40.607548 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 151608us
W20260709 12:23:40.570823 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 157208us
I20260709 12:23:40.577440 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:40.570950 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 190677us
I20260709 12:23:40.552985  7559 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:40.575373 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:40.662286 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.663596 27659 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:36354
I20260709 12:23:40.599985  8350 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:40.664357 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.637336  8217 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.627580 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:40.625063  7968 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:40.639245  8276 raft_consensus.cc:3055] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:40.675870 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:40.677369 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.662022  8374 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:40.682569  8131 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:40.684818  8194 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.684937  7554 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.685858  8351 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:40.687083 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.722628  8351 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.709626  8344 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:40.636816  8358 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:40.729429 28334 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 41 because replica has last-logged OpId of term: 39 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 24.
I20260709 12:23:40.687119  8116 raft_consensus.cc:3055] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:40.730455  8116 raft_consensus.cc:740] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:40.651619 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:40.700008 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.731515  8116 consensus_queue.cc:260] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.732259 27661 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:40.700731 27659 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:40.733237 27659 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:40.733314  8116 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:40.719580 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.702320 27663 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:40.646180 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:40.703588 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.704530 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 129431us
I20260709 12:23:40.706564  8378 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:40.701799  8216 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.714026  8262 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.713599  8374 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.727166 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.750806 27587 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:40.700723 27657 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
W20260709 12:23:40.736641 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90622.4us
W20260709 12:23:40.716955 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.703682 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76334.9us
I20260709 12:23:40.737190 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.737793 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.736249 28332 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:40.745798 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.722853  8109 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:40.753285 28334 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:40.728103  8351 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.706629  8276 raft_consensus.cc:740] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:40.758934 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.709313  8350 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:40.763258 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.735605  8344 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.763302  8352 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:40.768638  8370 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:40.775595  8370 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:40.776579  8370 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.777243  7953 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.778012  8370 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:40.746735  8378 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:40.779630 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:40.751071  8216 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.780369 27661 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:40.779671  8378 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:40.787798 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.790041 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:40.790315 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.790762 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:40.791189 27735 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 43 because replica has last-logged OpId of term: 42 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
I20260709 12:23:40.793421  8105 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:40.794009  8105 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.795176 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:40.796937 27701 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.798870  8191 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:40.771741  8109 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:40.799332  8191 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.799738  8191 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:40.772403  8344 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.799990  7554 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:40.750139  7968 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:40.759806  8173 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:40.803507 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.808874 27730 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:40.798347  8350 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.805230 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.809649 27738 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.815817 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:40.802299 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.763527  7559 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:40.820959 27737 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 37. Current term is 38. Ops: [37.21-37.21]
W20260709 12:23:40.824944 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 54663.4us
I20260709 12:23:40.805332 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:40.806370 27661 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:40.807476 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:40.807854 27659 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.826927 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.826910 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:40.828034 27661 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:40.828787 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:40.825629 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:40.833046 27655 raft_consensus.cc:2393] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
W20260709 12:23:40.840298 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 84455.4us
I20260709 12:23:40.807413  8351 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:40.843379 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:40.838984 28292 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:40.844300 27659 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 48 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.845216 27730 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:40.847029 27738 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:40.770139  8030 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.064s
I20260709 12:23:40.833083  8224 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:40.839032  8358 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:40.857795 28292 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:40.840054  8370 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:40.839589  8377 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:23:40.822474  8262 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 40.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.040s
I20260709 12:23:40.871723 27730 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
W20260709 12:23:40.871627 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 66354.6us
I20260709 12:23:40.848145  8109 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:40.886379 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 51530.8us
I20260709 12:23:40.876044 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:40.887122 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:40.881978 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.887816 27580 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:40.865824  8364 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:40.861142  7559 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:40.894810 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:40.863783  8358 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.901176 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.896936 28334 raft_consensus.cc:3055] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:23:40.901962 27581 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:40.902161 28334 raft_consensus.cc:740] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:40.903334 28334 consensus_queue.cc:260] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.904791 28334 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:40.852627  8123 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:23:40.895901  8392 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:40.916678 27660 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:40.889539 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.876856  8357 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:40.940502  8357 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.940955  8357 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:40.873433  8370 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:40.927556  8393 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.943876  8370 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:23:40.944231  8385 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:40.944150  8393 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.946054  8334 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:40.946455  8393 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:40.948051 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63638.8us
I20260709 12:23:40.949360 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.909427 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.950151  8378 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.950599  8349 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:40.951416 27663 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 35.
W20260709 12:23:40.951659  8389 log.cc:927] Time spent T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:23:40.921934  8388 log.cc:927] Time spent T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:23:40.886861  8109 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:40.891729  8364 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.909231  8358 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:40.932582 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:40.983537 27655 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 42 because replica has last-logged OpId of term: 40 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 26.
I20260709 12:23:40.934347 27661 raft_consensus.cc:1240] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 47. Current term is 48. Ops: []
I20260709 12:23:40.861529  8276 consensus_queue.cc:260] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 37.27, Last appended by leader: 27, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:40.948588  7968 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:40.874150  8377 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:40.888871 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.908232 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:40.963971 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.997816 28334 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 48.
I20260709 12:23:40.996804 27588 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 34.
I20260709 12:23:40.918900  8392 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:41.000479 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 82775.5us
I20260709 12:23:40.953258  8030 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:40.984292 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:40.929566  8217 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:40.997835 27583 raft_consensus.cc:2410] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 42 because replica has last-logged OpId of term: 40 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 26.
I20260709 12:23:40.998142 28332 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:40.950793 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:40.999217 27628 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:41.010042  8109 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:40.964476  8216 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.015283 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:40.982738  8364 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:41.026263 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 75568.9us
I20260709 12:23:41.030179 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.032025 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:40.984516  8105 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:41.034817 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 206111us
I20260709 12:23:40.997704  8377 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:41.003513  8320 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:23:41.021049  7554 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:23:41.013312  8392 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:41.016206  8334 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:41.025178  8352 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:41.022574  8374 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.034049  8341 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:41.036301  8109 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:23:41.060578 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 184598us
I20260709 12:23:41.035732 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.040174  8320 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:41.022181  8030 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.068312 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.069908 27627 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.056778  7554 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 48 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.060250  8109 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.061645  8352 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.093148  8030 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.080839  8109 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:41.106767 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:41.110682 28337 raft_consensus.cc:1275] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:41.116614 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:41.116791 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:41.122395 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:41.125422 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:41.127837  8343 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:41.128423  8343 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:41.128451 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58620.3us
I20260709 12:23:41.129948 27627 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.112834  8216 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:41.112790  8030 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:41.162770 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:41.161489 27627 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.164310 28334 raft_consensus.cc:2393] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:41.164888  7554 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.175384  8224 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:41.175800  8224 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.176632  8224 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.178097  8224 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:41.173959  8377 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.160781  8375 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:41.187249  8399 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:41.162978  8004 raft_consensus.cc:2804] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:41.168082  8291 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:41.169518  8367 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:41.195641  8004 raft_consensus.cc:697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.195497  8395 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.205464 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:41.207568 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.214170 27731 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:41.161360  8030 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.220968  8377 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:41.221817  8004 consensus_queue.cc:237] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.159412  8334 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.190799  7953 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.224454 27659 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:41.224689 27662 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:41.198849  8392 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.228178  8191 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.229295  8109 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.203168  8370 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.231972  8343 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.213231  8291 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.233208  8357 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.218169  8395 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.160902  8400 consensus_queue.cc:1048] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.087s
W20260709 12:23:41.239114 27630 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.163218  8320 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.255919 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:41.257354  8109 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.249691  7953 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.261961 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:41.266409 28335 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:41.246304  8375 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.267531 27734 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: [43.24-43.24]
I20260709 12:23:41.274430  8357 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:41.277024 28337 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:41.214329  8399 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.280819  7953 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:41.281497 28293 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.281505  8399 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.284248  8370 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.260560  8378 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.287393  8392 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.281837  8291 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.295799  8030 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.297670  8320 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:41.288458  8344 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.288918 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:41.290217  8375 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.304783  8375 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:41.305876 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.302407 28333 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:41.311494 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:41.312849  8405 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:41.291416  8297 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:41.315210 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.294291  8395 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.318524 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:41.319890 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:41.320715 27583 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:41.321671 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.324342 28335 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:41.327715 27587 raft_consensus.cc:2393] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 35: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:41.328742 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:41.329396  8340 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:41.329896  8340 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.330366  8340 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:41.332856 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.333480 27587 raft_consensus.cc:3060] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 42
I20260709 12:23:41.325627 27628 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:41.335736 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.336277 27582 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:41.336556  8297 raft_consensus.cc:2804] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:41.336988 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.337064 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.337100  8297 raft_consensus.cc:697] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.336300 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:41.338474  8297 consensus_queue.cc:237] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.27, Last appended by leader: 27, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.339013 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:41.339149 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:41.339469 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 290548us
I20260709 12:23:41.342379  8358 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.343184  8173 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.343762  8173 raft_consensus.cc:3060] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:41.344661  8358 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.348039 27587 raft_consensus.cc:2468] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:41.351824 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.353617 27730 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:41.329432 27585 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:41.357826 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.360632 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.362074 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.362783 27730 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:41.377385 28333 raft_consensus.cc:2468] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:41.363890 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.379897 27731 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:41.390265 27731 raft_consensus.cc:1240] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 37. Current term is 38. Ops: []
I20260709 12:23:41.391134 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.391788 27733 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:41.407377 27736 raft_consensus.cc:1275] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:23:41.410912 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.412453 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.413035 27730 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:41.413786 27731 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 37: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:41.414026  8362 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:41.415306  8191 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:41.415575 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 308871us
W20260709 12:23:41.414017 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 345757us
I20260709 12:23:41.411772 27628 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:41.421696 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.426810 28334 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:41.418974  8362 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:41.436858 27582 raft_consensus.cc:2468] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:41.437254 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.437915 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.445484 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.402182  8375 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.446152 28295 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:41.446172 27738 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
W20260709 12:23:41.458163 27703 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.329908  8364 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.444196 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 48.
I20260709 12:23:41.462668 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.467896 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.468035 27730 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 35.
I20260709 12:23:41.450353  8362 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.465144  8191 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:41.473563  8344 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.473868 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.455489  8291 raft_consensus.cc:2804] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:41.481652  8291 raft_consensus.cc:697] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.482649  8291 consensus_queue.cc:237] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.455946  8358 raft_consensus.cc:2804] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:41.463482  8004 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:41.490989 28332 raft_consensus.cc:2393] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 35: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:41.493379 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.494356  8380 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:41.494737  8380 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.470446  8216 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:41.486770  8395 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:41.457628  8375 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.487241  8362 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:41.494668 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.491415  8351 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:41.506469 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.501770  8380 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:41.499737  8358 raft_consensus.cc:697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:41.504733  8395 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.515196 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.513206  8392 consensus_queue.cc:1059] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 37.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.007s
I20260709 12:23:41.515069  7559 consensus_queue.cc:1048] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.010s
I20260709 12:23:41.526736  8364 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:41.527519 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.528187 27582 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:41.511682  8297 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:41.513309  8368 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:41.547906 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.549113 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.516990  8395 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:41.551614  8297 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:41.552094  8297 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.552508  8297 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:41.557263  8358 consensus_queue.cc:237] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.529544  8348 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:41.564477  8375 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.565264 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:41.566473  8370 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:41.576332  8340 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.578455  8340 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:41.581535 27582 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:41.582779  8395 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.584965  8395 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.590515 28295 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:41.590440 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.528242  8173 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.593182  8173 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.593777 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.594564 27582 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:41.596956 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:41.599836 28333 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: [37.22-39.23]
I20260709 12:23:41.600250 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:41.601317  8276 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:41.604146  8380 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.607053 27553 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:41.607457 27553 leader_election.cc:403] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:23:41.608044  8372 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:41.608632  8372 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.608749  8362 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.610936  8362 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.613479 27628 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:41.610946  8372 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.616392  8353 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:41.617137  8353 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:41.617479  8216 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:41.617905 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.617615  8353 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.624930  8353 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.628052 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.628727 27738 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:41.633004 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:41.633793 27662 raft_consensus.cc:2368] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:41.636684 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.636998 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.637228  8352 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:41.638237 27736 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:41.640321 27738 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:23:41.641050 28334 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:23:41.642721  8308 raft_consensus.cc:2804] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:41.643527  8308 raft_consensus.cc:697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:41.644615  8308 consensus_queue.cc:237] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.649237 27657 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:41.651046 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.655797 27734 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:41.657769 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.658560  8283 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:41.659896 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:41.660193 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 147205us
I20260709 12:23:41.660967 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.662676 27701 leader_election.cc:400] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42.
I20260709 12:23:41.662942 27701 leader_election.cc:403] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:41.663240 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:41.663864  7968 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:23:41.664255  7968 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:41.664637  7968 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:41.665948 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.666582 27733 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:41.672293 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:23:41.675522  8397 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.587s	user 0.000s	sys 0.005s
I20260709 12:23:41.676434 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:41.677224  8352 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:23:41.677604  8352 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.677937  8352 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:41.679277  8372 consensus_queue.cc:1059] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:41.680644  8372 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:41.682104  8372 raft_consensus.cc:3055] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:41.682463  8372 raft_consensus.cc:740] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:41.664624  8194 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 41. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42.
I20260709 12:23:41.686058 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:41.686880 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 137827us
I20260709 12:23:41.688195 27627 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.689036  7554 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:41.689560  7554 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:41.707505 27586 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 45. Current term is 46. Ops: []
I20260709 12:23:41.715837 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.716805 27660 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 34.
I20260709 12:23:41.722108 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.723058 27736 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:41.695840  7554 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:41.748962 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 52927.7us
I20260709 12:23:41.755044 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:41.755841 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:41.757184 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 51496.7us
I20260709 12:23:41.768337 28332 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 41: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:41.768559 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:41.770603 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62700.1us
I20260709 12:23:41.773207 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.774031 27729 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 42
I20260709 12:23:41.774111 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:41.775590 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.775635 27657 raft_consensus.cc:1240] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:41.789765 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.792562  8380 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:41.754701  8320 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:41.793440  8320 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:41.797574  7968 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.748265  8341 raft_consensus.cc:2764] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
W20260709 12:23:41.801930 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 340041us
I20260709 12:23:41.809376 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:41.756980  8375 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:41.816684 27733 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:41.818167 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.814771 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.829715 27732 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:41.829910 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.783113  8368 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:41.837397 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.838449 27729 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:41.845082 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.782330  8131 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:41.783288  8340 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:41.871716 27657 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:41.888958 27736 raft_consensus.cc:2468] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:41.852427 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.683311  8372 consensus_queue.cc:260] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.826455  8348 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.044s
I20260709 12:23:41.848222  8343 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:41.902463 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 126939us
I20260709 12:23:41.848933  8362 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:41.907603 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:41.921303 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.851004  8375 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.937786 27586 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:41.922289 27662 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 30.
I20260709 12:23:41.868582  8131 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.872407  8297 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.842792  8334 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.074s
I20260709 12:23:41.947774 27580 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 42 index: 28. (index mismatch)
I20260709 12:23:41.911005  7968 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:41.935599  8348 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.871223  8320 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:41.959210 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:41.963521 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.938925  8372 raft_consensus.cc:3060] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:41.974505 27657 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: []
W20260709 12:23:41.976826 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 151581us
I20260709 12:23:41.984314 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:41.987075 27627 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:41.988065 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:41.911232  8343 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:41.994405 28332 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:41.990661 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:41.995652 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.999284 28334 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 48 index: 32. (index mismatch)
I20260709 12:23:41.974284  8341 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:41.974575  8219 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:42.003806 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.035826 28332 raft_consensus.cc:1275] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 42 index: 28. (index mismatch)
I20260709 12:23:42.007762 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.034646 27663 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 36. Current term is 37. Ops: []
I20260709 12:23:41.956739  8340 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:23:41.958909  8351 consensus_queue.cc:1048] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:23:42.042042 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:42.049080 27662 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:42.007767  8380 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.043s
I20260709 12:23:42.059850 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:41.974164  8348 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.037660  8318 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:41.997455  8216 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:42.043294  8374 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:42.008818  8350 raft_consensus.cc:2804] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:41.992300  8334 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 41.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.082s
W20260709 12:23:42.061230 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 218913us
I20260709 12:23:42.084103 27661 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 24. (term mismatch)
I20260709 12:23:42.110509 27661 pending_rounds.cc:85] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 23
I20260709 12:23:42.110805 27661 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 37.24
I20260709 12:23:42.111037 27661 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.111723 27661 pending_rounds.cc:107] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 39.25
I20260709 12:23:42.112295 27661 raft_consensus.cc:2889] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.083550  8380 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:42.087775  8352 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.120821  8352 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.122385  8348 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.122918  8348 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:42.123538  8348 consensus_queue.cc:1243] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 4aa1523b9af84c3db99ba394e2b9aa9a log is divergent from this leader: its last log entry 39.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:23:42.078938  8194 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:42.124007  8348 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:23:42.086055  8378 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:42.125247  8378 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:42.089658  8350 raft_consensus.cc:697] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:42.126021  8378 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.126467  8350 consensus_queue.cc:237] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.129772 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:42.072753  8318 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:42.118860  8380 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260709 12:23:42.127312  8378 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:42.137097 28292 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:42.084424  8400 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.055s
I20260709 12:23:42.072679  8390 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.074s
I20260709 12:23:42.142921  8294 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:42.143530  8318 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:42.145202  8297 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:42.154336  8426 raft_consensus.cc:2804] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:23:42.176863  8419 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:23:42.178366  8291 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.031s	sys 0.014s
W20260709 12:23:42.178774  8291 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.031s	sys 0.014s
I20260709 12:23:42.181383 27659 raft_consensus.cc:1240] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 39. Current term is 40. Ops: [37.22-39.23]
W20260709 12:23:42.182898 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 164870us
I20260709 12:23:42.185279 27704 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:42.186879 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:42.183619 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:42.190460 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.200877 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:42.201661 28337 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:42.204038 27737 raft_consensus.cc:2393] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 38: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:42.167685  8375 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.169137  8294 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:42.174615  8131 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:42.183358  8343 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:42.186507  8426 raft_consensus.cc:697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:42.207469  8340 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:42.227200  8340 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:42.227033  8426 consensus_queue.cc:237] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.228186  8340 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.215404  8294 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.224005  8368 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:23:42.241135 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 182375us
W20260709 12:23:42.245806 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 242114us
I20260709 12:23:42.247925  8341 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:42.249348  8341 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.21, Last appended by leader: 21, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.253808 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:42.255651 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 138322us
I20260709 12:23:42.239316  8224 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:23:42.269373 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:42.270850 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.272341 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.275967 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:42.277191 27660 raft_consensus.cc:3060] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:42.283864 27738 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: [40.25-40.25]
I20260709 12:23:42.284739 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:42.285112 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:42.294585 27737 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 48 index: 32. (index mismatch)
I20260709 12:23:42.296002 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.290124  8430 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:42.293442  7559 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.300700 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.300819  7559 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.301510 27663 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:42.301316  8436 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.302417 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.302215  8436 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.295903  8320 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:23:42.303527  8430 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.304557  8436 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.305766  8430 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.308565 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.309676 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:42.310235 27659 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:42.312239 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.313609 27662 raft_consensus.cc:2393] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:42.314407 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.314898 27662 raft_consensus.cc:3060] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:42.302991 27663 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:23:42.316737  8343 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:42.318192  8436 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:42.319010  8436 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.271819  8224 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:42.323336 27663 raft_consensus.cc:2468] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 35.
I20260709 12:23:42.311028 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:42.326628 27659 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:42.328923 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.286258  8291 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:42.331393 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:42.332274 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:42.332598 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:42.332887 27738 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:42.335498 27701 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:42.340437 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.341173 28334 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:42.342568 28337 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:42.303035  7559 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:42.295843  8368 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:42.345391 27660 raft_consensus.cc:2468] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:42.342449  8377 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:23:42.342742  8343 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:23:42.359040 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 61813.1us
I20260709 12:23:42.343693 27662 raft_consensus.cc:2468] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
I20260709 12:23:42.395087 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.367024  8438 raft_consensus.cc:2804] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:23:42.360927  8318 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.366837 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:42.400445  8438 raft_consensus.cc:697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:42.364001  8436 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.362743  8431 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:42.345566  8291 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.359493  8224 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.402454  8438 consensus_queue.cc:237] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.403374 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:42.400975 27660 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:42.404794 27660 pending_rounds.cc:85] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 20
I20260709 12:23:42.405057 27660 pending_rounds.cc:107] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 33.21
I20260709 12:23:42.405324 27660 raft_consensus.cc:2889] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.342659  8433 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:42.402280  8431 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:42.408155 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64692.5us
I20260709 12:23:42.408818 27627 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:42.366668 27659 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 44 because replica has last-logged OpId of term: 43 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 23.
I20260709 12:23:42.426540 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:42.410885  8428 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:23:42.438992 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.403622  8224 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:42.446141 27662 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 42 index: 26. (index mismatch)
I20260709 12:23:42.449710  8308 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:42.452545  8318 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.454269  8431 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.407150  8291 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.418035  8380 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.044s
I20260709 12:23:42.409938  7968 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:42.468685 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 169689us
I20260709 12:23:42.407516  8343 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:42.469686 27552 leader_election.cc:400] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:42.469926 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:42.475157 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.479393 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:42.476622  8318 consensus_queue.cc:1059] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.004s
I20260709 12:23:42.480785  8318 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 41.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:23:42.481313  8318 raft_consensus.cc:3055] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:42.481678  8318 raft_consensus.cc:740] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.482499  8318 consensus_queue.cc:260] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.486873 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:42.504204 28337 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:23:42.467479  8308 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:42.481855  8439 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:23:42.474185  8343 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:42.483405  8440 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:23:42.526212 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:42.537925 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.502475  8131 raft_consensus.cc:2764] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
I20260709 12:23:42.541234 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.478078  7968 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:42.577299 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 100589us
I20260709 12:23:42.527863  8318 raft_consensus.cc:3060] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:42.542711  8440 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:42.588099 27657 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 41 because replica has last-logged OpId of term: 40 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 25.
W20260709 12:23:42.590293 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 64168.1us
I20260709 12:23:42.583705  8131 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.594955  8440 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:42.546043  8452 consensus_queue.cc:1059] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.055s
W20260709 12:23:42.600634  8372 consensus_meta.cc:287] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.001s
W20260709 12:23:42.618835 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.603696  8340 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:42.605688 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:42.607602  8449 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:23:42.608379  8441 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:23:42.610951  8353 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:42.613375  8322 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:23:42.601670  7968 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:42.658552  8340 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.658591 27588 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:23:42.528333  8308 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.660280  8322 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.660187  8353 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.655876  8131 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:42.707965 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 114925us
I20260709 12:23:42.689160  8440 raft_consensus.cc:3055] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:42.710570 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.687917  8452 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.015s
I20260709 12:23:42.709728  8440 raft_consensus.cc:740] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.720206  8440 consensus_queue.cc:260] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.721765  8440 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:42.713470  8434 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.724310  8340 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.724411  8434 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.730432  8308 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:42.727160  8322 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.741185  8334 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:23:42.744499  8334 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:42.744545 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.745116  8334 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:42.748070  8219 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:42.749477  8452 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:42.750047  8452 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.750416 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:42.752343  8219 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:42.752553  8131 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:42.738865  8343 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.756372 28337 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:42.742302 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.743466 28333 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:23:42.759924 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:42.761684 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44011us
I20260709 12:23:42.762434 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.727099  8434 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.751081  8452 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.775741 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:42.776125 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:42.778330 27585 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:42.757517 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:42.785892 27585 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 21. Preceding OpId from leader: term: 42 index: 22. (index mismatch)
I20260709 12:23:42.791620  8131 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:42.792230  8131 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:42.792706  8131 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:42.792834  8415 log.cc:927] Time spent T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 1.102s	user 0.005s	sys 0.002s
I20260709 12:23:42.781407  8372 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.013s
I20260709 12:23:42.795732 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5458) took 1455 ms. Trace:
I20260709 12:23:42.796129 27583 rpcz_store.cc:276] 0709 12:23:41.340263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.341572 (+  1309us) service_pool.cc:224] Handling call
0709 12:23:41.341897 (+   325us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:41.342300 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:41.342332 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:41.342933 (+   601us) log.cc:844] Serialized 52 byte log entry
0709 12:23:41.507841 (+164908us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:41.507885 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.507956 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:42.792203 (+1284247us) spinlock_profiling.cc:243] Waited 57.8 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:42.794639 (+  2436us) raft_consensus.cc:1685] finished
0709 12:23:42.794662 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.795636 (+   974us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"spinlock_wait_cycles":132998016,"thread_start_us":158775,"threads_started":1,"wal-append.queue_time_us":347462}
I20260709 12:23:42.801738  8343 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:42.810199 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.811625 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:42.812165 27733 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:42.822041 27733 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 36.
I20260709 12:23:42.787747  7968 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.035s
I20260709 12:23:42.836774 27738 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 23. (term mismatch)
I20260709 12:23:42.837172 27738 pending_rounds.cc:85] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 22
I20260709 12:23:42.837393 27738 pending_rounds.cc:107] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 42.23
I20260709 12:23:42.823028 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:42.823505 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:42.825505  8418 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.750s	user 0.001s	sys 0.001s
I20260709 12:23:42.837776 27738 raft_consensus.cc:2889] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.840866 27736 raft_consensus.cc:2393] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:42.841235 27734 raft_consensus.cc:1240] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:42.840437 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:42.842481 27738 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 38.
I20260709 12:23:42.837363 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:42.846305 27731 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:42.840004  8082 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:42.853796  8082 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.854735  8082 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.855877  8082 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:42.840902  8374 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.010s	sys 0.015s
W20260709 12:23:42.876952  8435 log.cc:927] Time spent T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.591s	user 0.002s	sys 0.000s
W20260709 12:23:42.884889 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.834584  8466 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.045s
I20260709 12:23:42.889192 27552 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:42.890066  8318 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:42.900152  8318 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.904541  8466 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:42.904996  8466 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.905890  8466 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.908071  8471 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:23:42.907925  8334 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.924891  8351 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.925268 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:42.925302 27583 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 30. Current term is 31. Ops: []
I20260709 12:23:42.925472  8351 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.907207  8466 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:42.928602  8351 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:42.935365 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:42.935457  8351 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:23:42.937672  8434 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:23:42.941267 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 532481us
I20260709 12:23:42.941567 27664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4290) took 1171 ms. Trace:
I20260709 12:23:42.942169 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:42.943621 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:42.908759  8471 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:42.917701  8353 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.861313 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:42.921118 27585 raft_consensus.cc:1240] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 39. Current term is 40. Ops: []
I20260709 12:23:42.946744 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:42.890839  8322 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:23:42.880934  8131 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.953923 27580 raft_consensus.cc:1275] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 42 index: 26. (index mismatch)
I20260709 12:23:42.956063  8224 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:42.956765  8224 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:42.909241  8372 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:42.950351 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:42.961309 27586 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:42.951942 27588 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:23:42.941972 27664 rpcz_store.cc:276] 0709 12:23:41.769884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.770293 (+   409us) service_pool.cc:224] Handling call
0709 12:23:41.805003 (+ 34710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.805203 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:23:42.077491 (+272288us) log.cc:844] Serialized 14 byte log entry
0709 12:23:42.408597 (+331106us) log.cc:844] Serialized 14 byte log entry
0709 12:23:42.852790 (+444193us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b18004643e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9725b522 00007f0e9725d7c1 00007f0e9725d779 00007f0e9725d709 00007f0e9725d6d1 00007f0e9725c9cd 0000000000388634 0000000000388479 00007f0e9723e83a
0709 12:23:42.941042 (+ 88252us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b18004643e0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e971d5c0a 00007f0ea0915219 00007f0e9725da40 00007f0e972584e4 00007f0e97258455 00007f0e9a340060 00007f0e9a352716 00007f0e9a363287 00007f0e9a363208
0709 12:23:42.941205 (+   163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:42.941255 (+    50us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:23:42.941315 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.941396 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.941511 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70598,"prepare.queue_time_us":50694,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":2690,"raft.queue_time_us":67696,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":19369,"spinlock_wait_cycles":50962048,"thread_start_us":511501,"threads_started":2,"wal-append.queue_time_us":285057}
I20260709 12:23:42.955580  8471 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:42.881866  8374 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.010s	sys 0.016s
W20260709 12:23:42.960606 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 425752us
I20260709 12:23:42.952592  8276 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.949043  8436 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:42.958458  8350 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:42.953018  8417 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:42.971046 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:42.972811 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:42.980302  8224 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:42.982277  8374 consensus_queue.cc:1059] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:42.952224  8082 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:42.961247  8465 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:42.985118  8436 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.961009  8434 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:23:42.990482 28336 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:42.988897  8276 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:42.995731  8436 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:43.001518  8276 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.003897  8343 consensus_queue.cc:1048] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.006s
I20260709 12:23:42.994796  8082 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.005277  8082 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:42.988481  8350 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.007699  8350 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.012439  8173 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.012974  8173 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.014974  8173 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.016064  8334 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.016175 27663 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:23:43.016898 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.019475  8318 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.032058 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.032815 27585 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:43.022096  8374 consensus_queue.cc:1243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 35.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:23:43.035159  8291 consensus_queue.cc:1048] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:23:42.990841  8475 consensus_queue.cc:1059] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 39.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.013s
I20260709 12:23:43.036171 27588 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:43.037135 27658 raft_consensus.cc:1240] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 34. Current term is 35. Ops: []
I20260709 12:23:43.038100 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.017635 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.039685 27660 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:43.040838 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.041159 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:43.041764 27479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 243) took 79064 ms (client timeout 15000 ms). Trace:
I20260709 12:23:43.042188  8475 raft_consensus.cc:3055] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:43.042912  8475 raft_consensus.cc:740] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:43.043905  8475 consensus_queue.cc:260] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.045132  8475 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:42.996593 28295 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
W20260709 12:23:43.046849 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50384.2us
I20260709 12:23:43.012477  8131 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:43.020221 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 66805.9us
I20260709 12:23:43.021652 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.028374  8362 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:43.051371 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.059638 27660 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:23:43.067979 27664 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:43.075769 27659 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 38.
I20260709 12:23:43.067171  8358 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:23:43.081460 28292 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.086716 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 132047us
I20260709 12:23:43.087644 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.087875 27660 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 21. Preceding OpId from leader: term: 42 index: 22. (index mismatch)
I20260709 12:23:43.089037 27659 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 42: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:43.082912  8358 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.093430  8358 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.096101  8438 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:43.096670  8438 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.098341  8417 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.098842  8417 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.100004 27664 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:23:43.107829 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.108145 27582 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:43.108523 27661 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:43.110056 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.090044  8173 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.114528 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033363
I20260709 12:23:43.174436  8417 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:43.202656 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 243137us
W20260709 12:23:43.203022 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 91298.6us
I20260709 12:23:43.059592  8082 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.071295  8364 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.045s
I20260709 12:23:43.193081  8219 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:43.194887  8351 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:43.214490  8351 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.216046 27664 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:23:43.218868  8377 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:43.221359  8219 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.227945  8219 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:43.196431  8476 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.084s	user 0.003s	sys 0.000s
I20260709 12:23:43.197616  8350 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.199009 27589 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:43.199401 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.240276 27587 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:43.116119  8291 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:43.189502 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:43.208632 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.241904 27586 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:43.242218 27659 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:43.231732 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.233538 27733 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:23:43.189559  8283 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:43.236352 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:43.244701 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.237558 27585 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:43.248906 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.253018 27582 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
W20260709 12:23:43.249521 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 60119.9us
I20260709 12:23:43.254655 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:43.256254 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.256750 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.257251 27738 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 44 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
W20260709 12:23:43.042166 27479 rpcz_store.cc:269] 0709 12:22:23.977817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.307570 (+10329753us) service_pool.cc:224] Handling call
0709 12:23:43.041529 (+68733959us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:36.770053 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:37.018774 (+248721us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:37.063466 (+ 44692us) write_op.cc:435] Acquired schema lock
0709 12:23:37.063517 (+    51us) tablet.cc:662] Decoding operations
0709 12:23:37.064998 (+  1481us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:37.065114 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:37.065160 (+    46us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:23:37.066561 (+  1401us) tablet.cc:701] Row locks acquired
0709 12:23:37.066621 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:23:37.095309 (+ 28688us) write_op.cc:270] Start()
0709 12:23:37.142823 (+ 47514us) write_op.cc:276] Timestamp: P: 1783599817066745 usec, L: 0
0709 12:23:37.157984 (+ 15161us) op_driver.cc:348] REPLICATION: starting
0709 12:23:37.215347 (+ 57363us) log.cc:844] Serialized 4770 byte log entry
0709 12:23:38.160796 (+945449us) op_driver.cc:464] REPLICATION: finished
0709 12:23:38.564115 (+403319us) write_op.cc:301] APPLY: starting
0709 12:23:38.564220 (+   105us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:38.812108 (+247888us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:38.812232 (+   124us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:38.823200 (+ 10968us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:38.900758 (+ 77558us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=0,mrs_lookups=0
0709 12:23:38.900830 (+    72us) write_op.cc:312] APPLY: finished
0709 12:23:38.942142 (+ 41312us) log.cc:844] Serialized 113 byte log entry
0709 12:23:38.942368 (+   226us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:38.942533 (+   165us) write_op.cc:454] Released schema lock
0709 12:23:38.942616 (+    83us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:38.943243 (+   627us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:38.943289 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":483610,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":380177,"apply.run_cpu_time_us":19498,"apply.run_wall_time_us":379067,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54287,"num_ops":12,"prepare.queue_time_us":250042,"prepare.run_cpu_time_us":157107,"prepare.run_wall_time_us":895863,"raft.queue_time_us":60557,"raft.run_cpu_time_us":94135,"raft.run_wall_time_us":422676,"replication_time_us":962961,"thread_start_us":972120,"threads_started":4,"wal-append.queue_time_us":522684,"wal-append.run_cpu_time_us":31800,"wal-append.run_wall_time_us":2208253}]]}
I20260709 12:23:43.259590 27628 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:43.261580 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:43.262809 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:43.254963 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:43.263952 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.264554 27657 raft_consensus.cc:2368] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:43.236291  8430 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.217011  8351 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:43.270027 27585 raft_consensus.cc:3055] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:23:43.270587 27585 raft_consensus.cc:740] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:43.267313 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 65265.9us
I20260709 12:23:43.267778 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.239472  8334 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:43.266675 27628 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:43.243999  8291 consensus_queue.cc:1243] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 42.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:23:43.274384  7554 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:23:43.271862 27585 consensus_queue.cc:260] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.275684 27582 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 46: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:43.209512  8173 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.268769  8219 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.291514 27733 raft_consensus.cc:1275] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:23:43.292598 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 38 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:43.292906  8173 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.278781  8395 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:43.264885  8224 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:43.273648  8370 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:43.295979  8370 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.296396  8370 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:23:43.296770 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.296916  8417 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:23:43.297729 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.298012  8417 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:23:43.298419  8417 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:43.276963 27738 raft_consensus.cc:1240] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 35. Current term is 36. Ops: []
I20260709 12:23:43.300781  8481 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:43.301340  8481 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.266675  8438 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:43.299273  8417 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.272873  8434 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:43.304817  8434 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.305155  8417 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:43.305236  8434 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.306525  8395 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:43.272565  7968 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:23:43.307911 27734 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
W20260709 12:23:43.309206 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.257903  8283 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.278590  8334 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.286203  8082 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.276130  8430 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.240157  8436 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.289475  8358 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.265100  8276 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:23:43.290477 27702 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:43.344158 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 53759.1us
I20260709 12:23:43.290699  8351 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:43.363919 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:43.304067 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.366722 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.307096  8395 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.268604  8380 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:43.278203 27585 raft_consensus.cc:3060] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:23:43.339079 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.288486  8350 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:43.340602 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.395569 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:43.396179 27731 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:23:43.369168 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 106426us
I20260709 12:23:43.369707 27627 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:43.388681 27730 raft_consensus.cc:2368] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39.
W20260709 12:23:43.405579 28292 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.25.202.2:33051 timed out after 1.742s (SENT) [suppressed 30 similar messages]
I20260709 12:23:43.356841  8351 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.390440 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.423209 27585 raft_consensus.cc:1275] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:23:43.423686 27734 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:43.366199  8481 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.393803 28333 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42. Ops: [41.28-41.28]
I20260709 12:23:43.356431  7968 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:43.389612 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:43.428172 27587 raft_consensus.cc:3060] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.393368  8431 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:43.425130  7554 consensus_queue.cc:1243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 36.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:23:43.392908  8443 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:43.450412  8283 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:43.452554  8334 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.386727  8378 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.044s
I20260709 12:23:43.386943  8343 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.061s
I20260709 12:23:43.422377  8432 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:43.425333  8351 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:43.391294  8364 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.040s
W20260709 12:23:43.465893 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 188705us
I20260709 12:23:43.467602 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:43.471380 27584 pending_rounds.cc:85] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 23
I20260709 12:23:43.471735 27584 pending_rounds.cc:107] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 42.24
I20260709 12:23:43.471997 27584 raft_consensus.cc:2889] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:43.473126  8358 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.436213  8297 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:23:43.474332  8440 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:43.475426  8224 consensus_queue.cc:1059] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:43.476981 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:43.479580  8395 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.484591  8430 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.485644 28334 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:23:43.486992  8362 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:23:43.487964  8362 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:43.488412  8362 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:43.489406  8362 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.490830  8362 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.490990  8452 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:43.492900  8480 consensus_queue.cc:1059] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:43.498425 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.500267 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:43.501504 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 102051us
I20260709 12:23:43.498873 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.512337 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.513577 27659 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:43.514835 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.519125 28333 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7"
dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:46136
W20260709 12:23:43.517622 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 120972us
I20260709 12:23:43.464674 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:43.528962 27628 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:43.533425 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:43.536319  8491 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:43.536847  8491 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.537297  8491 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:43.541031  8380 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:43.541975  8380 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.542531  8380 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.519909 28333 raft_consensus.cc:606] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:43.546753 27655 raft_consensus.cc:1275] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:23:43.548302 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.548951 27737 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:43.549266  8380 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.561035 27587 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:43.562608 27630 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:43.564664 27730 raft_consensus.cc:1275] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:43.565361 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.488143  8461 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260709 12:23:43.579739 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.580447 27663 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 46.
I20260709 12:23:43.581247  8481 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:43.582315  8351 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.583578 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.583972 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:43.584558 27730 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:43.587760 27736 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:43.588330 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:43.594379 27662 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:43.581338 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.605295 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.595530  8490 raft_consensus.cc:2804] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:43.511987  8430 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:43.634198 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:43.583207  8219 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:43.597455  8390 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:23:43.638979  8482 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
W20260709 12:23:43.692101 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 166658us
I20260709 12:23:43.695019  7554 consensus_queue.cc:1048] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:23:43.697048 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 291487us
I20260709 12:23:43.696684 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:43.639307  8490 raft_consensus.cc:697] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:43.631518  8417 consensus_queue.cc:1048] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.058s
I20260709 12:23:43.713598 28292 leader_election.cc:400] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39.
I20260709 12:23:43.714025 28292 leader_election.cc:403] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:23:43.690528  8395 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.718264  8350 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:23:43.725804  8475 raft_consensus.cc:993] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:43.724576  8351 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:43.693560  8173 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:43.638759  8430 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.690303  8291 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:43.740680 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:43.737973  8460 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:43.750464 27731 raft_consensus.cc:2468] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:43.738519  8436 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:43.718986  8432 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.770560  8486 log.cc:927] Time spent T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:43.751785  8173 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.733814  8350 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.784778 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:43.743252  8364 raft_consensus.cc:1081] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:43.819988 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:43.822870 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:43.749132  8291 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.752439  8351 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.786254  8432 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.832649  8432 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:43.824147  8436 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:23:43.833412  8436 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:43.833791  8436 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:43.775569  8283 consensus_queue.cc:1048] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.060s
W20260709 12:23:43.836045 27475 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.369s	user 0.011s	sys 0.015s
I20260709 12:23:43.836936  8219 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:23:43.839236  8434 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.533s	user 0.010s	sys 0.001s
I20260709 12:23:43.839767  8434 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.841388  8490 consensus_queue.cc:237] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.841943  8173 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.844411  8430 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.844724  8446 consensus_queue.cc:1048] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:43.736656  8395 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.841987  8434 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:43.829332  8291 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:43.849548 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.827208  8417 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:43.851967  8417 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:43.852187 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.852757 27664 raft_consensus.cc:3060] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:43.851047 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:43.853821 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 72481.7us
I20260709 12:23:43.854055 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.854628 27661 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:43.752105  8426 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.073s
I20260709 12:23:43.849575  8395 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.864035  8426 consensus_queue.cc:1243] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer 9c09719959a54e8eb66d2041dd9c3113 log is divergent from this leader: its last log entry 40.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:23:43.864297  8320 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.864888  8320 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.820530  8350 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:43.870715 27664 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:43.775300  8370 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.880614 27664 raft_consensus.cc:3055] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:23:43.881017 27664 raft_consensus.cc:740] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:43.894579 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:43.904160 27661 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:43.910472 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:43.912618 27661 raft_consensus.cc:2410] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 41 because replica has last-logged OpId of term: 40 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 25.
I20260709 12:23:43.878429  8460 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 38. Current term is 39.
I20260709 12:23:43.924598 27662 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: [42.22-42.22]
I20260709 12:23:43.852454  8417 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.924041  8494 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:43.853003  8351 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.918711 27664 consensus_queue.cc:260] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.933284 27664 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:23:43.905397  8364 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:43.928973  8494 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:43.968644  8487 log.cc:927] Time spent T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.051s	user 0.002s	sys 0.000s
I20260709 12:23:43.971515  8480 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.972103  8480 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.972607  8417 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:43.973740 27663 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:36354
I20260709 12:23:43.974272 27663 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:43.974622 27663 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:43.976079  8480 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:43.977885  8351 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:43.982154 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.924770  8491 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:43.987046 27663 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:43.992811 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:43.959303 27664 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 20. (term mismatch)
I20260709 12:23:43.999342 27664 pending_rounds.cc:85] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 19
I20260709 12:23:44.002735 27663 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:44.003051 27660 pending_rounds.cc:85] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 24
I20260709 12:23:44.003609 27660 pending_rounds.cc:107] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 40.25
I20260709 12:23:44.003914 27660 raft_consensus.cc:2889] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:43.999838 27664 pending_rounds.cc:107] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 35.20
I20260709 12:23:44.011871 27664 raft_consensus.cc:2889] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:44.013892  8362 consensus_meta.cc:287] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.516s	user 0.006s	sys 0.004s
I20260709 12:23:44.020771  8353 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:44.030305  8497 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:23:44.013542  8417 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.014029  8351 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:44.040127  8496 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.049741  8496 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:44.037135  7559 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 36 current term: 36 is higher.
I20260709 12:23:44.053923 27659 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:44.061333  8436 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.063723  8436 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:44.051738  8417 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.069315  8494 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:44.069598  8417 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:44.056771  8351 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.074733  8491 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:44.073194  8351 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:44.078146  8431 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:44.078650  8431 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:44.083696  8370 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:44.089927 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.091045 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.055833  8496 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:44.093760 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:44.098558  8452 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:44.099085  8452 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:44.099941  8452 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.046566  8490 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:44.101365  8490 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.104203  8490 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:44.106937 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:44.042207  8283 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:44.110323  8283 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.115820 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.118834 27588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39872 (request call id 5469) took 1148 ms. Trace:
I20260709 12:23:44.133807 27588 rpcz_store.cc:276] 0709 12:23:42.970330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.970714 (+   384us) service_pool.cc:224] Handling call
0709 12:23:44.118646 (+1147932us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b54002b93d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:44.118748 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5678464}
I20260709 12:23:44.120245 27657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49998 (request call id 4303) took 1111 ms. Trace:
I20260709 12:23:44.134629 27582 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:44.135006 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.135025 27657 rpcz_store.cc:276] 0709 12:23:43.008804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.040581 (+ 31777us) service_pool.cc:224] Handling call
0709 12:23:44.120021 (+1079440us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540022a454. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:44.120184 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503151488}
I20260709 12:23:44.135879 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:44.136483 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.137027 27657 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 44.
W20260709 12:23:44.143906 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.145870 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:44.149024 27704 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:44.149857  8377 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:23:44.151446 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.152494 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.153085 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:44.153637 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.158264 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.160127 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.056406 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.865s	user 0.044s	sys 0.020s Metrics: {"bytes_written":4754,"delete_count":0,"lbm_write_time_us":876,"lbm_writes_lt_1ms":5,"reinsert_count":0,"thread_start_us":173747,"threads_started":1,"update_count":188,"wal-append.queue_time_us":323562}
I20260709 12:23:44.128284 27701 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.130967  7968 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:44.131275  8494 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:44.131717  8351 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.153760 27731 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: [43.23-43.23]
I20260709 12:23:44.161404 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.172315 27552 leader_election.cc:400] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:44.172705 27552 leader_election.cc:403] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:44.162388 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.174988 27737 raft_consensus.cc:1275] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:44.163076 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.121382  8291 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.165223 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.165505 27701 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.174539 27476 maintenance_manager.cc:419] P 04e0ca717d6b41608e4d35acaee1cac8: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.207272
I20260709 12:23:44.181790 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.182691 27734 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:44.173656 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:44.170658 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.176133 27588 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:44.183669 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:44.176090 27655 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 46.
I20260709 12:23:44.181442 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.186362 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:44.191653 27589 raft_consensus.cc:2393] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 46: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:44.188573 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.195791 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:44.198987 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.177294  8496 raft_consensus.cc:2804] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:44.207928 27656 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: [42.25-42.25]
I20260709 12:23:44.217381 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.187623  7559 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:44.203742  8417 consensus_queue.cc:1059] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 41.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.021s
I20260709 12:23:44.195932  8390 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 41. Reason: Vote denied by peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) with higher term. Message: Invalid argument: T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:44.232508  8320 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:44.181977  8481 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:23:44.236733 27736 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 31.
I20260709 12:23:44.176769  7968 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.248999 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:44.228662  8431 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.235896 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 37 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:44.206254  8334 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:44.201730  8341 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:44.204262  8434 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:23:44.181305  8475 consensus_queue.cc:1059] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.041s
I20260709 12:23:44.206665  8362 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:44.261384 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.293162  7968 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:44.289645  8283 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:44.286904  8436 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.296247  8436 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.296772  8291 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:44.298323  8362 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.299088  8362 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:44.293224 27657 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:44.301538  8431 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:44.304988 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.233985 27627 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.285872 27738 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:23:44.287076 27731 raft_consensus.cc:1240] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: [43.24-43.24]
I20260709 12:23:44.247509  8496 raft_consensus.cc:697] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:44.239461 27480 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 21 to 32, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 32 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:44.207453  8350 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.282639  8334 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.305094 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.321286 27734 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:44.321889 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.322281 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.322520 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:44.323712 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.324467 27734 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:23:44.306607 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 72737.9us
I20260709 12:23:44.181368  8494 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.257654  7559 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.181234  8351 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.310726 27730 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:44.308964 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.324775 28330 raft_consensus.cc:3055] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:23:44.329866 27736 raft_consensus.cc:1275] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:44.342060 28330 raft_consensus.cc:740] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:44.331943 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.343596 27731 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:44.306847 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:44.351935 27662 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 38: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:44.298400  8436 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:44.304344  8434 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:44.315912  8431 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:23:44.354023 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.325347  8334 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:44.343151 28330 consensus_queue.cc:260] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.356904 27480 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 31 to 36, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 36 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:44.357630 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.392477 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:44.355190 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:44.311450  8341 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:44.370272 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.355397 28333 raft_consensus.cc:2468] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:44.396613 27657 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 44: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:44.398679 27730 raft_consensus.cc:2410] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for term 42 because replica has last-logged OpId of term: 41 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 27.
I20260709 12:23:44.400105 28330 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:44.406720 27702 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:44.414041  8417 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:44.340639  8515 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:44.418810 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.423622 28330 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:44.420083 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.367749  8431 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:44.426457 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.426623 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.427722 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.440513 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.441361 27588 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 37.
I20260709 12:23:44.435384 27628 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:44.444547 27627 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:44.444954  7554 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:23:44.444839 27480 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 30 to 36, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 36 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:44.462896 27588 pending_rounds.cc:85] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 21
W20260709 12:23:44.429543  8475 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 34 current term: 34 is higher.
I20260709 12:23:44.463266 27588 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 37.22
I20260709 12:23:44.463586 27588 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:44.463924 27588 pending_rounds.cc:107] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 39.23
I20260709 12:23:44.464200 27588 raft_consensus.cc:2889] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:44.466625 27480 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 27 to 32, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 32 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:44.468086  7554 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 37.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:44.441725  8431 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:44.395960  8436 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 40.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.048s
I20260709 12:23:44.471894 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.474951 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.475167 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.476050 27583 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 31: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:44.476046 27584 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:44.444703  8432 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:44.482059 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:44.485576 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 287489us
W20260709 12:23:44.486651 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 291542us
I20260709 12:23:44.427778  8515 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.467761  8370 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:44.463148  8395 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:44.508939 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:44.510365 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.518996 27583 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:44.514026 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.520227 27732 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:44.475570  8297 raft_consensus.cc:2804] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:23:44.508275  8466 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 35 current term: 35 is higher.
W20260709 12:23:44.553568 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 44742us
I20260709 12:23:44.557011 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:44.557624 27580 raft_consensus.cc:3060] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:23:44.441239  8496 consensus_queue.cc:237] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 37.27, Last appended by leader: 27, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:44.581380 27552 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.558959 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.513844  7968 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:44.570128 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.539932  8508 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:44.577230  8438 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:44.585391 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:44.584380 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.502995  8479 consensus_queue.cc:1048] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.067s
I20260709 12:23:44.589247 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 37.
I20260709 12:23:44.516095  8395 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.540115  8353 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.516673  8370 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:44.589428 27732 raft_consensus.cc:2468] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:44.549952  8510 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:44.612515 27663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.007s	sys 0.007s
W20260709 12:23:44.617415 27663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.007s	sys 0.007s
I20260709 12:23:44.619180 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.588791  7968 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:44.585857  8473 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.502669  8341 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.609375 27580 raft_consensus.cc:2468] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 34.
I20260709 12:23:44.483789  8434 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.625115 27480 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 26 to 31, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 31 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:44.604720  8395 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:44.644438 27734 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:44.615545  8370 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:44.663458 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:44.666785 27589 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:44.649796 27731 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:23:44.599561  8297 raft_consensus.cc:697] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:44.644116  8473 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.615006  8390 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:44.628001  8276 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:44.615387  8353 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.670254 27480 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 32 to 39, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 39 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:44.690282 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.644162  8350 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:44.695232 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:44.698367 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:44.650439  8494 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:44.665082  8351 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:44.629881  8510 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.684190  8424 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:44.800487 27589 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:44.691666  7559 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:44.766084  8509 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:23:44.804945 27480 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:44.817528 27480 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 33 to 37. New cstate: current_term: 37 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:44.700065  8446 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:23:44.770687  8521 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.308s	user 0.001s	sys 0.000s
I20260709 12:23:44.767526  8494 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:44.819850 27480 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 27 to 31, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 31 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:44.691538  8452 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.113s
I20260709 12:23:44.699548  8276 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.826224 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:44.696547  8390 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:44.824673 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 201993us
I20260709 12:23:44.835966 27627 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.836519 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.831352 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:44.823583  8520 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.343s	user 0.000s	sys 0.001s
I20260709 12:23:44.849553 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:44.851787 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:44.852399 27731 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:44.852653 27480 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 21 to 25, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 25 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:44.855283 27627 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.857779 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:44.861049 27480 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 21 to 26. New cstate: current_term: 26 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:44.831238  8424 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:44.865157 27480 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 23 to 30, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 30 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:44.757475  7968 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.851397  8430 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:44.872681 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:44.878058  8461 consensus_queue.cc:1048] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:44.870424  8173 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:44.865420  8224 raft_consensus.cc:2804] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:44.844681  8494 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.888957 27583 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:23:44.831404  8297 consensus_queue.cc:237] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.860189  8515 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:44.881721  8524 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.284s	user 0.001s	sys 0.000s
W20260709 12:23:44.904471 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 57393.2us
W20260709 12:23:44.896471 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 120776us
I20260709 12:23:44.914887 27589 raft_consensus.cc:1275] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:23:44.909310 27552 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42.
I20260709 12:23:44.910683 27731 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:44.883052  8004 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.009s
I20260709 12:23:44.916184 27552 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.920395 27661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2344) took 1025 ms. Trace:
I20260709 12:23:44.920835 27661 rpcz_store.cc:276] 0709 12:23:43.894772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.941478 (+ 46706us) service_pool.cc:224] Handling call
0709 12:23:43.941753 (+   275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.942080 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.942107 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.942724 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.177471 (+234747us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:23:44.177507 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.177576 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.920172 (+742596us) raft_consensus.cc:1685] finished
0709 12:23:44.920192 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.920334 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":83,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":6502,"thread_start_us":169750,"threads_started":1,"wal-append.queue_time_us":425306}
I20260709 12:23:44.890280  8508 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:44.918470 27480 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 30 to 35, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 35 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:44.872095  8529 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:44.935662 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:44.874569  8424 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:44.903119  8320 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.899570  8473 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:44.881757  8540 consensus_queue.cc:1175] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" responder_term: 40 status { last_received { term: 40 index: 24 } last_committed_idx: 24 last_received_current_leader { term: 40 index: 24 } }
I20260709 12:23:44.917905  8173 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:44.957841 27581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39872 (request call id 5480) took 1849 ms. Trace:
I20260709 12:23:44.957839  8173 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:44.918201  8224 raft_consensus.cc:697] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:44.958509 27581 rpcz_store.cc:276] 0709 12:23:43.108092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.108200 (+   108us) service_pool.cc:224] Handling call
0709 12:23:43.108466 (+   266us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.108825 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.108854 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.109273 (+   419us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.081931 (+972658us) spinlock_profiling.cc:243] Waited 922 us on lock 0x7b18004daf40. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0ea0123ab1 00007f0ea0130281 00007f0e9723de07 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b 00007f0e99ee7d23 00007f0e99ee80d1 00007f0e9a303a0f
0709 12:23:44.396428 (+314497us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:23:44.396468 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.396532 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.957453 (+560921us) raft_consensus.cc:1685] finished
0709 12:23:44.957475 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.957782 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":78,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":4651,"spinlock_wait_cycles":2121088,"thread_start_us":314308,"threads_started":1,"wal-append.queue_time_us":1431014}
I20260709 12:23:44.960197  8224 consensus_queue.cc:237] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:44.961862 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 45789.1us
I20260709 12:23:44.944655 27480 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:44.911640  8283 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.920579  8506 raft_consensus.cc:3055] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:44.954443 27583 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 38.
I20260709 12:23:44.915568 27704 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:44.976653 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:44.976783 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61317.1us
I20260709 12:23:44.979997 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:44.971549 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 112863us
I20260709 12:23:44.983295 27480 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 21 to 23, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 23 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:44.995194 27661 raft_consensus.cc:1240] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: [42.22-42.22]
I20260709 12:23:45.000914 27581 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Deduplicated request from leader. Original: 34.19->[36.20-38.21]   Dedup: 36.20->[38.21-38.21]
I20260709 12:23:45.001703 27581 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: [36.20-38.21]
I20260709 12:23:44.994837 27480 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 29 to 35, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 35 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:44.997615 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:44.964344  8488 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.006218 27480 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 30 to 32, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 32 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:44.965652  8353 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:44.997721  8461 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:44.984016  8515 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:45.018844 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:45.003868  8526 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:23:45.010888 27581 raft_consensus.cc:1240] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 33. Current term is 34. Ops: []
I20260709 12:23:44.984460  8506 raft_consensus.cc:740] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:44.991222  8546 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:44.963580  8320 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:45.010278  8533 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:23:45.013798  8488 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.029695  8390 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.019095  8555 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:45.021653  8452 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
W20260709 12:23:45.018141  8532 log.cc:927] Time spent T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.205s	user 0.001s	sys 0.001s
I20260709 12:23:44.988898  8283 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.008253  8473 raft_consensus.cc:493] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:45.066299  8473 raft_consensus.cc:515] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.068770  8473 leader_election.cc:290] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.007634  8424 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.040927 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.052909 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.072288 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.054840 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.021574  8510 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:45.057690 28295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.029742  8539 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
W20260709 12:23:45.064886  8553 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:23:45.065641  8283 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:23:45.052907  8276 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.103849 28295 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:45.104570 27630 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:45.108457 27630 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:45.115991 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 118446us
I20260709 12:23:45.081952  8390 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:45.102530  8559 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:23:45.135854 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.137001 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:45.061973  8452 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.137729 27585 raft_consensus.cc:2376] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40. Re-sending same reply.
I20260709 12:23:45.139787 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.143566 27586 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:23:45.041791  8546 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.150643 28295 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 
I20260709 12:23:45.128996  8430 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:23:45.125927  8556 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:45.123588  8358 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:45.194929  8488 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.086899  8506 consensus_queue.cc:260] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.22, Last appended by leader: 22, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:45.186861  8431 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.702s	user 0.010s	sys 0.001s
I20260709 12:23:45.197350  8431 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.129246  8390 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
W20260709 12:23:45.190625  8334 consensus_meta.cc:287] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.775s	user 0.006s	sys 0.002s
I20260709 12:23:45.171905  8364 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:45.204186 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 215296us
I20260709 12:23:45.138870  8494 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:45.192838  8362 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.893s	user 0.004s	sys 0.008s
I20260709 12:23:45.194739  8556 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:45.202296 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 81897.9us
W20260709 12:23:45.215607 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 79409.8us
W20260709 12:23:45.247185 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.247900 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.208619  8364 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:45.252056 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48041.1us
I20260709 12:23:45.252899 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:45.257472 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.258101 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.258533 27586 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:45.203789  8559 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:45.259224 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.217449  8556 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:45.217665  8452 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.260008 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.216634 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:45.208184  8461 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
I20260709 12:23:45.276690 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:45.281383 27581 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:45.208957  8334 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.266063  8390 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.220266  8362 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:45.278683  8537 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 12:23:45.259402  8364 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:45.274078  8481 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.288777 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72235.7us
I20260709 12:23:45.314308  8390 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:45.331568  8523 log.cc:927] Time spent T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:23:45.332877 27663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36354 (request call id 2348) took 1392 ms. Trace:
I20260709 12:23:45.333154  8531 raft_consensus.cc:2764] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:23:45.333285 27663 rpcz_store.cc:276] 0709 12:23:43.940499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.009076 (+ 68577us) service_pool.cc:224] Handling call
0709 12:23:44.016204 (+  7128us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.016871 (+   667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.016902 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.017352 (+   450us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.624215 (+606863us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:44.624273 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.624362 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.332630 (+708268us) raft_consensus.cc:1685] finished
0709 12:23:45.332653 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.332810 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":566224,"threads_started":1,"wal-append.queue_time_us":821353}
I20260709 12:23:45.336194 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:45.337194  8568 log.cc:927] Time spent T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.185s	user 0.000s	sys 0.004s
I20260709 12:23:45.337355  8573 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:23:45.338061  8573 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:45.315107  8572 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:45.284992  8424 consensus_queue.cc:1059] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.033s
I20260709 12:23:45.340157  8424 raft_consensus.cc:3055] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:45.340628  8424 raft_consensus.cc:740] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:45.341179  8559 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.342097 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.341598  8424 consensus_queue.cc:260] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:45.288600  8527 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
W20260709 12:23:45.318023 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.289404  8395 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.617s	user 0.009s	sys 0.002s
W20260709 12:23:45.319799  8552 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
W20260709 12:23:45.288326  8530 log.cc:927] Time spent T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
W20260709 12:23:45.320159 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.288280  8550 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:23:45.323449  8545 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260709 12:23:45.313726  8491 raft_consensus.cc:3055] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:45.349475 27738 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:45.375123 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.358084 27730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 2092) took 1038 ms. Trace:
I20260709 12:23:45.358199 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.357882 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.350782 27737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34476 (request call id 5020) took 1066 ms. Trace:
I20260709 12:23:45.362450 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:45.363662 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.367012 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:45.374758 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.359810  8506 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:45.388914  8320 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.360013  8481 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.460283  8481 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.361672  8362 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:45.395512  8573 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.343281  8424 raft_consensus.cc:3060] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:45.396938 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.367887  8291 consensus_queue.cc:1048] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.100s
I20260709 12:23:45.289445  8454 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:45.469594  8454 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.471094 28337 raft_consensus.cc:2410] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 37 because replica has last-logged OpId of term: 36 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 19.
I20260709 12:23:45.472666  8454 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.435485 27584 raft_consensus.cc:2368] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45.
I20260709 12:23:45.432809 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.478189 27582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5270) took 1327 ms. Trace:
I20260709 12:23:45.478884 27582 rpcz_store.cc:276] 0709 12:23:44.150301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.171669 (+ 21368us) service_pool.cc:224] Handling call
0709 12:23:44.172032 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.172400 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.172441 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.172902 (+   461us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.487524 (+314622us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:23:44.487606 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.487707 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.475743 (+988036us) spinlock_profiling.cc:243] Waited 7.78 ms on lock 0x7b240013ac18. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e970e19c4 00007f0e970dfacb 00007f0e9729adc9 00007f0e9729aa55 00007f0e97299fb0 00007f0e99f10488 00007f0e99f10be8 00007f0e99ee8398 00007f0e99ee376a 00007f0e99ee2bb6
0709 12:23:45.477792 (+  2049us) raft_consensus.cc:1685] finished
0709 12:23:45.477826 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.478110 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10101,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6246,"spinlock_wait_cycles":17895680,"thread_start_us":314240,"threads_started":1,"wal-append.queue_time_us":518454}
I20260709 12:23:45.327112  8431 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.441709 27589 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:45.453256 27659 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:45.440364 27730 rpcz_store.cc:276] 0709 12:23:44.319852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.551491 (+231639us) service_pool.cc:224] Handling call
0709 12:23:44.588392 (+ 36901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.071038 (+482646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.071070 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:45.071112 (+    42us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:23:45.071142 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.357825 (+286683us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400316d54. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:45.357878 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.358008 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1109514624}
I20260709 12:23:45.418795 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.438989 27737 rpcz_store.cc:276] 0709 12:23:44.284252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.285119 (+   867us) service_pool.cc:224] Handling call
0709 12:23:44.285586 (+   467us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.302528 (+ 16942us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.302573 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.303359 (+   786us) log.cc:844] Serialized 52 byte log entry
0709 12:23:44.701321 (+397962us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:23:44.701369 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.701431 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.350584 (+649153us) raft_consensus.cc:1685] finished
0709 12:23:45.350605 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.350725 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16418,"raft.queue_time_us":18012,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"spinlock_wait_cycles":22912,"thread_start_us":396646,"threads_started":1,"wal-append.queue_time_us":511568}
W20260709 12:23:45.457204  8569 log.cc:927] Time spent T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 12:23:45.457476  8436 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.500698  8436 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.365501 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:45.313361  8297 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:45.484462 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.504503  8334 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.509605 27589 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:45.512205 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:45.513931 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:45.456000  8370 consensus_meta.cc:287] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.776s	user 0.006s	sys 0.006s
W20260709 12:23:45.517675 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 152230us
I20260709 12:23:45.527448 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:45.525648 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:45.404008  8491 raft_consensus.cc:740] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:45.533350 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:45.533394 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:45.539287  7554 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:45.539839  7554 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:45.542582 27662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49998 (request call id 4334) took 1020 ms. Trace:
I20260709 12:23:45.543015 27662 rpcz_store.cc:276] 0709 12:23:44.522290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.524709 (+  2419us) service_pool.cc:224] Handling call
0709 12:23:44.527804 (+  3095us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.618312 (+ 90508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.618347 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.618812 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.813140 (+194328us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:44.813266 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.813415 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.192347 (+378932us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b4800033e58. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9f271aa1 00007f0e9f268b37 00007f0e9f2689ca 00007f0e9f36026f 00007f0e9f35ff37 00007f0e9f37455c 00007f0e9f374508 00007f0e9f374488 00007f0e9f374443 00007f0e9f3736ec 00007f0e9e29da96
0709 12:23:45.534929 (+342582us) raft_consensus.cc:1685] finished
0709 12:23:45.534954 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.542506 (+  7552us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90040,"raft.queue_time_us":91370,"raft.run_cpu_time_us":3145,"raft.run_wall_time_us":567817,"spinlock_wait_cycles":15548800,"thread_start_us":193800,"threads_started":1,"wal-append.queue_time_us":257721}
W20260709 12:23:45.545636 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 267273us
W20260709 12:23:45.534466  8552 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:45.517041  7559 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:45.552284 27730 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:45.552345 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.550498 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:45.553058 27737 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 43. Current term is 44. Ops: [43.23-43.23]
I20260709 12:23:45.554186 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.568456 27735 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:45.568804 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.569284 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:45.569416 27732 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:45.569810 27735 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 45
I20260709 12:23:45.539116 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:45.547777  8496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.013s	sys 0.019s
I20260709 12:23:45.530886  8572 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:45.566336 27729 raft_consensus.cc:1240] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: [42.25-42.25]
I20260709 12:23:45.531682  8297 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:45.408288  8395 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.536098  8515 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:45.534300  8488 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:45.553490  7554 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:45.571185 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60167.6us
I20260709 12:23:45.576850 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:45.588392 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.589069 27731 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:45.545904  8362 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:23:45.582661 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:45.572947  8496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.014s	sys 0.019s
I20260709 12:23:45.588605 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:45.535019  8370 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.587428  7559 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.615613  8364 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.616916 28330 raft_consensus.cc:2368] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41.
I20260709 12:23:45.580524  8334 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:45.618198  8334 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.620692  8334 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.628206 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.631916 27702 leader_election.cc:400] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41.
I20260709 12:23:45.632344 27702 leader_election.cc:403] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:45.552680  8546 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.609273 27736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47076 (request call id 2094) took 1276 ms. Trace:
W20260709 12:23:45.633888 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.636045 28332 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:45.637665 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.634182 27736 rpcz_store.cc:276] 0709 12:23:44.333329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.552660 (+219331us) service_pool.cc:224] Handling call
0709 12:23:44.560355 (+  7695us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.613736 (+ 53381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.613789 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.637113 (+ 23324us) log.cc:844] Serialized 52 byte log entry
0709 12:23:44.899229 (+262116us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:23:44.900508 (+  1279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.586748 (+686240us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b540044ded4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:45.586814 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.586877 (+    63us) raft_consensus.cc:1685] finished
0709 12:23:45.586898 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.609181 (+ 22283us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16297,"raft.queue_time_us":17100,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":12366,"spinlock_wait_cycles":83967232,"thread_start_us":150390,"threads_started":1,"wal-append.queue_time_us":274736}
I20260709 12:23:45.641117 27735 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 45.
I20260709 12:23:45.546327  8575 consensus_queue.cc:1059] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.099s
I20260709 12:23:45.637768 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:45.595964  8550 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:45.660696 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.661664 27737 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:23:45.632153  8561 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:45.673460 27582 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:45.676324 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.644719  8291 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:45.675223 27704 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:45.655220  8575 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 37.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:45.636164  8480 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.041s
I20260709 12:23:45.661569  8550 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.665982  8508 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 41
I20260709 12:23:45.682953  8400 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:45.725066 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 74816.3us
I20260709 12:23:45.678170  8561 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:45.726238 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:45.696915 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 157854us
I20260709 12:23:45.700332  8362 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:45.699671  8550 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:45.724438  8400 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.722939  8498 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:45.750537 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:45.750954 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.753760 27585 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:23:45.729223  8561 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:45.740866  8480 consensus_queue.cc:1243] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer ce69a3f9261a47a7a22f02f9c1f07964 log is divergent from this leader: its last log entry 42.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:23:45.762617 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.764742 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.765913 27660 raft_consensus.cc:1240] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 45. Current term is 46. Ops: []
I20260709 12:23:45.765965 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:45.767035  8586 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:45.767795  8586 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.769899  8586 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.774336  8173 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.775882  8578 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:23:45.758049  8400 raft_consensus.cc:3060] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:45.774859  8173 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:45.780150 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.781250  8364 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
I20260709 12:23:45.754748  8353 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:45.782649  8353 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:45.797116 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.799029 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:45.800339 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 154447us
I20260709 12:23:45.801213 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.792781 27657 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 46.
I20260709 12:23:45.803344 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.803467 27658 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:45.806293 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4811ee81866e469abca31f212c08ace2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.806994 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.807710 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:45.807478 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.808281 27664 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:45.811014 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:45.818992 28337 raft_consensus.cc:1275] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 37 index: 27. Preceding OpId from leader: term: 39 index: 28. (index mismatch)
I20260709 12:23:45.828171 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:45.830711 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:45.848839 27664 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 45.
I20260709 12:23:45.809130  8586 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:45.839843  8524 log.cc:927] Time spent T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:23:45.825778  8362 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:45.841444  8446 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:45.763806  8498 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:45.869879  8430 consensus_meta.cc:287] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.687s	user 0.012s	sys 0.000s
I20260709 12:23:45.885635  8587 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:45.886260  8587 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.891607  8494 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:45.893566  8494 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:45.900872 27701 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.907483  8173 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.908453  8587 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.907200  8362 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.917625 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:45.921298  8446 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:45.921496  8515 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:45.922227  8446 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:45.865720  8353 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.923352 27732 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 40.
W20260709 12:23:45.923414  8580 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:23:45.924830  8446 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:45.898597  8430 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:23:45.925410  8494 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:45.882501  8495 raft_consensus.cc:493] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:45.928570 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:45.868100  8586 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:45.945008  8495 raft_consensus.cc:515] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:45.946506  8556 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.655s	user 0.010s	sys 0.000s
W20260709 12:23:45.948046  8585 log.cc:927] Time spent T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:23:45.951547 27738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34476 (request call id 5026) took 1643 ms (client timeout 2103 ms). Trace:
W20260709 12:23:45.952037 27738 rpcz_store.cc:269] 0709 12:23:44.308588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.308712 (+   124us) service_pool.cc:224] Handling call
0709 12:23:45.951313 (+1642601us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54000a41d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:45.951483 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2317164928}
I20260709 12:23:45.926339  8438 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:45.877206  8572 consensus_queue.cc:1048] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.040s
I20260709 12:23:45.954224  8438 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:46.036849  8599 log.cc:927] Time spent T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260709 12:23:46.037200  8592 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.124s	user 0.002s	sys 0.001s
I20260709 12:23:46.046552  8498 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:45.935539  8515 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.047369  7968 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:46.047941  8515 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.052428  8395 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.062886  8400 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.078019 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:46.097968 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 278070us
W20260709 12:23:46.099496 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 250112us
I20260709 12:23:46.103732 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:45.947242  8556 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.093402  8320 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:46.106694  7554 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:46.099771  8495 leader_election.cc:290] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.113202  7554 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:23:46.104738  8508 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 40. Reason: Vote denied by peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321) with higher term. Message: Invalid argument: T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 40. Current term is 41.
I20260709 12:23:46.115499 27663 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:46.127805 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:46.114642  8424 consensus_meta.cc:287] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.649s	user 0.010s	sys 0.001s
I20260709 12:23:46.134084 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:46.185241 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:46.128831  8400 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:46.205672 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 75359.9us
I20260709 12:23:46.206362  7554 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:46.112610  8561 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.113490  8320 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.178821  8572 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.215873 28337 raft_consensus.cc:1275] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:23:46.217561 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.219381 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.220281 27734 raft_consensus.cc:2468] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:23:46.195188  8586 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.210656  7554 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.203586  8424 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.204380  7559 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.193869  8498 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.245108 27663 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:23:46.227283  8605 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:46.271427  7554 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.234082  8572 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.255487  8424 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.260545  8498 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:46.290916  8596 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
W20260709 12:23:46.300105  8550 consensus_meta.cc:287] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.536s	user 0.008s	sys 0.004s
I20260709 12:23:46.295804  8173 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:23:46.309938  8598 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:23:46.271741  8605 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.325539 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.328297 27657 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:46.329335 27628 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:46.327514  8173 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.331597  8297 consensus_queue.cc:1048] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.003s
I20260709 12:23:46.334569  8430 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:23:46.336656 27475 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.278s	user 0.000s	sys 0.000s
W20260709 12:23:46.327613  8606 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:23:46.337212 27656 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:46.341322  8572 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.322225  8587 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:46.337913  8173 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.356835  8561 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.372416 27580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56296 (request call id 2404) took 1298 ms. Trace:
I20260709 12:23:46.372849 27580 rpcz_store.cc:276] 0709 12:23:45.074444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.074743 (+   299us) service_pool.cc:224] Handling call
0709 12:23:45.075064 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.195836 (+120772us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:23:45.802742 (+606906us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.236428 (+433686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.236496 (+    68us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:46.236558 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.372196 (+135638us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54001243d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:46.372241 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.372361 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":461109,"prepare.queue_time_us":501757,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":2970,"raft.queue_time_us":1512,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":12352,"spinlock_wait_cycles":277259136,"thread_start_us":545032,"threads_started":2,"wal-append.queue_time_us":490970}
I20260709 12:23:46.375526  8377 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.376330  8377 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.379720  8597 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.380342  8597 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.382881  8597 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.404630  8498 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:46.406145 27584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39856 (request call id 5281) took 1881 ms (client timeout 1998 ms). Trace:
W20260709 12:23:46.406708 27584 rpcz_store.cc:269] 0709 12:23:44.524372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.569760 (+ 45388us) service_pool.cc:224] Handling call
0709 12:23:46.405890 (+1836130us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b540008a7d4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:46.406068 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1805996800}
W20260709 12:23:46.407648  8615 log.cc:927] Time spent T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:23:46.358167  8344 consensus_queue.cc:1059] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 41.26, Next index: 27, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:46.360008  8320 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.362625  8605 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:46.364032  8610 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:23:46.363978  8424 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:46.364807  8506 consensus_meta.cc:287] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.881s	user 0.012s	sys 0.000s
I20260709 12:23:46.365229  8515 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.353525  8587 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.322378  8550 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.353557  8572 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.439098 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.379251  8377 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.441468 27588 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:46.424804  8344 raft_consensus.cc:3055] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:46.425017  8362 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.446630  8550 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.440146  8572 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.448958 27735 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
W20260709 12:23:46.450569  8608 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.214s	user 0.003s	sys 0.000s
I20260709 12:23:46.442956  8344 raft_consensus.cc:740] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:46.458321  8572 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.461002 27583 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:46.461498 27583 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:46.453052  8542 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.465436  8587 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.466871  8542 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.469010 28330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.471496  8344 consensus_queue.cc:260] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.26, Last appended by leader: 26, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.472740  8344 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:46.453707  8362 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.487504  8362 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.492784 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:46.494953 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.495754 27657 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:46.497853 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.496580 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.462360 27583 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.503350 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:46.499037 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.507479 27734 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:46.511613 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:46.504284 27584 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:46.514078 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:46.514616 27734 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:46.514675 27730 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: [41.23-41.23]
I20260709 12:23:46.517602 27580 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:23:46.518262 27660 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:46.514678 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:46.547827 27587 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:46.548456 27587 raft_consensus.cc:606] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:46.540534 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:46.550730 27731 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 41: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:46.551101 27734 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:46.552021 27737 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:46.512940  8362 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:46.517906  8572 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.569140  8572 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.565004  8542 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.573097  8572 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.522607  8607 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.039s
I20260709 12:23:46.574174 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:46.574950 27581 raft_consensus.cc:2393] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 41: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:46.553115  8353 raft_consensus.cc:493] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:46.578576  8353 raft_consensus.cc:515] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.581621  8353 leader_election.cc:290] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.586548 27583 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:46.524227 27663 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:23:46.539408  8574 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:46.517589  8531 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.617123 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:46.630256 27588 raft_consensus.cc:2368] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:46.624979 27582 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Deduplicated request from leader. Original: 34.19->[36.20-38.21]   Dedup: 36.20->[38.21-38.21]
I20260709 12:23:46.629945  8531 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.588649  8362 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.632630  8531 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.521512  8550 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.618263 27732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34464 (request call id 4550) took 1059 ms. Trace:
I20260709 12:23:46.633262  8362 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.630791 27582 raft_consensus.cc:1240] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: [36.20-38.21]
I20260709 12:23:46.634550  8574 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:46.635017  8574 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:46.595593  8556 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.639498 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:46.640554 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51630.9us
I20260709 12:23:46.643580 27583 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 42 index: 23. (term mismatch)
I20260709 12:23:46.643965 27583 pending_rounds.cc:85] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 22
I20260709 12:23:46.644186 27583 pending_rounds.cc:107] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 41.23
I20260709 12:23:46.644433 27583 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:46.645006  8546 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 42 current term: 42 is higher.
I20260709 12:23:46.612989 27733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4549) took 1055 ms. Trace:
I20260709 12:23:46.648388 27733 rpcz_store.cc:276] 0709 12:23:45.558036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.558567 (+   531us) service_pool.cc:224] Handling call
0709 12:23:45.558860 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.632642 (+ 73782us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.632674 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.066702 (+434028us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b2c0010b5c8. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e98c41a11 00007f0e98c51372 00007f0e9a383a01 00007f0e9a34843a 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2
0709 12:23:46.088817 (+ 22115us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.126702 (+ 37885us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:23:46.126767 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.612697 (+485930us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b54001d2ed4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a348e01 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c 00007f0e98882cc2 00007f0e9868348c 00007f0e986828c6
0709 12:23:46.612748 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.612792 (+    44us) raft_consensus.cc:1685] finished
0709 12:23:46.612806 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.612933 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44424,"raft.queue_time_us":22193,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":10110,"spinlock_wait_cycles":127566848,"thread_start_us":23706,"threads_started":1,"wal-append.queue_time_us":129987}
I20260709 12:23:46.651557 27660 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:23:46.652086  8436 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.623804 27585 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:23:46.660650 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.633736 27732 rpcz_store.cc:276] 0709 12:23:45.559307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.582373 (+ 23066us) service_pool.cc:224] Handling call
0709 12:23:46.618131 (+1035758us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b5400317254. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a34c2fa 00007f0e9f30db17 00007f0e988886d4 00007f0e98888662 00007f0e98888591 00007f0e9888850c 00007f0e988877c2 00007f0e9868348c 00007f0e986828c6 00007f0e9868226f
0709 12:23:46.618214 (+    83us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":764977024}
I20260709 12:23:46.663555 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.664390 28337 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:23:46.665745 28292 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:46.667037 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.667647 28335 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:46.673403 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:46.674463 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.675050 28335 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:46.679410  8341 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.679965  8341 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.680603  8434 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:46.681129  8434 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.682192  8341 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.682709  8341 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:46.683161  8341 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.685237 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:46.683583  8446 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:46.686048 27736 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:46.686273  8446 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:46.686812  8446 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:46.689265 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.690303 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.692875 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:46.691291 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.693938 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:46.695443 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.698134 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.696545 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62a7e798fd5e41e0bb2d9f5c7dab1adc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.693473 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.701562 27732 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 44: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:46.704159 27733 raft_consensus.cc:2393] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:46.708720 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.709362 27656 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:46.710098 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.683087  8434 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:46.714646 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.694172  8426 raft_consensus.cc:2804] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:46.716710 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.714996 27736 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:46.633466  8550 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.565116  8491 consensus_queue.cc:260] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.731511 27729 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:23:46.722777  8426 raft_consensus.cc:697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:23:46.735031 27729 raft_consensus.cc:1240] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Rejecting Update request from peer 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:46.741539 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.743152 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:46.755466  8434 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:46.756155  8434 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.756634  8434 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:46.757314  8446 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.759977  8446 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.761996 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:46.762758 27661 raft_consensus.cc:2393] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 41: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:46.767555  8341 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.748785 27702 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:46.771414 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:46.771898  8434 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.782064  8370 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Requested vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.784062  8593 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:46.784688  8593 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.786342  8438 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.788133  8434 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:46.789352  8593 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.795136  8567 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:46.795675  8567 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.796062  8567 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:46.773375  8508 raft_consensus.cc:2764] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election decision vote started in defunct term 41: won
I20260709 12:23:46.801563  8426 consensus_queue.cc:237] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.806010  8491 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:46.805941 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:46.811511  8550 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:46.819691 28337 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 42.
I20260709 12:23:46.821209 27628 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:46.822551  8607 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:46.823138  8607 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:46.824177  8607 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.26, Last appended by leader: 26, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:46.829633 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.834636 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.841010 27658 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:46.841830 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.844388 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.845117 27659 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 31.
I20260709 12:23:46.832556  8574 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:46.849860 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:46.854622 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:46.855399 28336 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:46.851507 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:46.858379 27730 raft_consensus.cc:2368] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 41. Current term is 42.
I20260709 12:23:46.844188  8495 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:46.873661 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:46.870146  8495 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:46.951597 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.974727 27583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39856 (request call id 5296) took 1193 ms. Trace:
I20260709 12:23:46.974637 27588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:46.976032 27585 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:23:46.976070 27588 raft_consensus.cc:606] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:46.975600 27583 rpcz_store.cc:276] 0709 12:23:45.781284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.781435 (+   151us) service_pool.cc:224] Handling call
0709 12:23:45.781740 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.460868 (+679128us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b54001714a0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2bfdc0 00007f0e9a34a1c0 00007f0e9a34b231 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4 00007f0e98883b62 00007f0e98883a91 00007f0e98883a0c
0709 12:23:46.586356 (+125488us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b58005b88d0. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a2a40aa 00007f0e9a326447 00007f0e9a33efe7 00007f0e9a3407de 00007f0e9a34ab49 00007f0e9a34b416 00007f0e9a347d8f 00007f0e9a34783e 00007f0e9f30c419 00007f0e98883bd4
0709 12:23:46.644837 (+ 58481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.974487 (+329650us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":196496384}
I20260709 12:23:46.977180 27588 raft_consensus.cc:614] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:46.978551 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:46.985968 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.989559 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.991912 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:46.996534 27655 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 45 because replica has last-logged OpId of term: 43 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
I20260709 12:23:47.002045 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.003187 27664 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
W20260709 12:23:47.004387 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 143918us
I20260709 12:23:47.008896 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.010824 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.011883 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.014912 27664 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:23:47.016170 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.011176 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.020442 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:47.021055 27662 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:47.023972 28328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:47.030150 28292 leader_election.cc:400] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 41. Current term is 42.
I20260709 12:23:47.030720 28292 leader_election.cc:403] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:23:47.031688 27627 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:47.055146  8344 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:47.056299  8344 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.060616  8531 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.061700  8531 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.064677  8546 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.065433  8546 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.094710  8344 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:47.113875 27552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.007s	sys 0.011s
W20260709 12:23:47.114480 27552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.008s	sys 0.012s
I20260709 12:23:47.133785  8495 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:47.098300  8377 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:47.144980 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81157.3us
W20260709 12:23:47.136533 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 582143us
I20260709 12:23:47.154556  8574 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.154507 28295 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.156050 27475 maintenance_manager.cc:643] P 04e0ca717d6b41608e4d35acaee1cac8: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 2.974s	user 0.140s	sys 0.112s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":25132,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":290130,"dirs.run_cpu_time_us":635,"dirs.run_wall_time_us":22003,"lbm_read_time_us":3658,"lbm_reads_lt_1ms":26,"lbm_write_time_us":1519,"lbm_writes_lt_1ms":10,"peak_mem_usage":265487,"reinsert_count":0,"thread_start_us":583636,"threads_started":2,"update_count":188}
I20260709 12:23:47.158486  8426 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:23:47.168354 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70159.4us
I20260709 12:23:47.170670 27630 leader_election.cc:400] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:47.171095 27630 leader_election.cc:403] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Cancelling election due to peer responding with higher term
I20260709 12:23:47.172606  8390 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.173442  8390 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.173626  8377 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.177562  8440 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:47.178467 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:47.179818  8638 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:47.182150 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:47.184094 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:47.186637 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 158225us
I20260709 12:23:47.187505  8529 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:47.188372 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.188736  8529 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.191474  8417 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:47.192519  8417 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.193758 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:47.194872 28333 raft_consensus.cc:2393] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 42: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
I20260709 12:23:47.197438  8607 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:47.197842  7968 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:47.198271  8531 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.178385  8440 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.190984 27627 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.120769  8567 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.198835  7968 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.203598  8377 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:47.241716 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:47.207762  8546 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:47.246353 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.249620 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 220130us
W20260709 12:23:47.251940 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 81618.9us
I20260709 12:23:47.263674  8495 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election lost for term 40. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41.
I20260709 12:23:47.264572  8574 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:47.268936 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.273343 27662 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:47.283458 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.283766 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:47.287241  8546 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:47.288475  8546 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.289346 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.290860  8546 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:47.291913 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101173us
I20260709 12:23:47.279479 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.296995 27584 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:52220
W20260709 12:23:47.305090 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 119224us
I20260709 12:23:47.317364 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:47.314688 27584 raft_consensus.cc:606] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.320057 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.326354 27580 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
W20260709 12:23:47.326448 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47960.8us
I20260709 12:23:47.330601 28292 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.328378  8390 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.316051  8529 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.327057  8417 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:47.349864  7968 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:47.345264  8440 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:47.343843  8593 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.023s
I20260709 12:23:47.370751 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.364035  8567 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:47.381865 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:47.370429  7968 raft_consensus.cc:493] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:47.384359 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.391579  8607 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:47.391196 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.387192 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.377279  8417 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:47.405200 27479 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 34 to 39. New cstate: current_term: 39 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.415544 27588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:47.382874  8644 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:47.418751 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.419675 27580 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:47.426585 27586 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:23:47.431756  8560 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:47.432060  8508 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.432699  8560 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.432843  8508 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.423125 27588 raft_consensus.cc:606] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.419312 27479 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 32 to 35. New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.437675 27479 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 40. New cstate: current_term: 40 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.440068  8508 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:47.424201  8574 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:47.446208 27588 raft_consensus.cc:614] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:47.454535 27552 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:47.455446  8560 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.455917 27552 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:47.456218  8560 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:47.456947  8529 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:47.458132  8297 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.049s
W20260709 12:23:47.460427  8630 log.cc:927] Time spent T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.214s	user 0.002s	sys 0.001s
I20260709 12:23:47.459156 27588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:47.461874 27588 raft_consensus.cc:606] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Received request to transfer leadership to TS 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.461575  8417 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.429394  8645 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:47.466459 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:47.468822  8529 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:47.473392  8560 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:47.474606 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:47.474275  8529 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.28, Last appended by leader: 28, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.479835  8560 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.483007 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.424903  7968 raft_consensus.cc:515] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.490558 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:47.492105 27730 raft_consensus.cc:1275] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
I20260709 12:23:47.499250 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.499826 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.505000  8417 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:47.508553  8491 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.702s	user 0.010s	sys 0.010s
I20260709 12:23:47.508718 28337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:47.508948 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:23:47.508961  8639 log.cc:927] Time spent T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:23:47.510388 27587 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:47.517104 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:47.521287 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.538566 27588 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 39: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:47.538460 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.539289 28334 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:47.540535  7554 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.472643  8645 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:47.530663 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 64423.7us
I20260709 12:23:47.531373  8440 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:23:47.532249 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 141338us
I20260709 12:23:47.540988  7554 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.470670 27479 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 34 to 38, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 38 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.548875 27479 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 42 to 45. New cstate: current_term: 45 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.550613 27479 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 35 to 37, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 37 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.551576 27659 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:47.552186  7554 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.563301 28334 raft_consensus.cc:2468] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:47.564451 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:47.551684 27630 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:47.557922  8644 consensus_queue.cc:1048] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.011s
I20260709 12:23:47.570019  7968 leader_election.cc:290] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:47.567221 27702 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:47.542661  8440 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:47.542912  8645 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:47.582140  7554 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:47.588116 27627 leader_election.cc:304] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.588434 27479 catalog_manager.cc:5697] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 24 to 33, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 33 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.583611  8440 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:47.590654 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.592402  7554 raft_consensus.cc:2749] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:47.598060  8491 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:23:47.598500 27627 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.599695  8297 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:47.600798 27627 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.602768  8491 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:47.603967 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.604848 27589 raft_consensus.cc:2410] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 46 because replica has last-logged OpId of term: 43 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 23.
I20260709 12:23:47.604693 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:47.607661 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:47.610711  7554 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:47.611399  7968 raft_consensus.cc:2804] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:47.612156  7968 raft_consensus.cc:697] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:47.615804 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82728.6us
I20260709 12:23:47.615792  8508 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:47.619763 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.635562 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.617066 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.590561 27479 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 37 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.648257 27630 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:47.632900  8432 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:47.650235  8297 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.634657 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.614913 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:47.651777  8390 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:47.653215 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.634259 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.626904 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:47.620322  8546 consensus_queue.cc:1059] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 45.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:23:47.632329 27584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.624369  8508 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:23:47.653826 28336 consensus_meta.cc:287] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent flushing consensus metadata: real 0.797s	user 0.007s	sys 0.008s
I20260709 12:23:47.641709 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74b289dab6a94832ad11d8ac1e30d802" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.656667 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:47.635620 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:47.618996  7968 consensus_queue.cc:237] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.28, Last appended by leader: 28, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:47.616291 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.686785 27656 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:23:47.687068 27662 raft_consensus.cc:2393] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 46: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:47.688190 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:47.690232  8297 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.691449  8574 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:47.692893  8297 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.693538 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.694221 27587 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:47.695778  8417 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:47.695022 27479 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 38, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 38 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.697328 27581 pending_rounds.cc:85] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 24
I20260709 12:23:47.697676 27581 pending_rounds.cc:107] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 42.25
I20260709 12:23:47.698395 27581 raft_consensus.cc:2889] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.697628 27479 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 35 to 41, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 41 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.701229 28336 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:47.712111  8641 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a for earlier term 41. Current term is 42.
I20260709 12:23:47.713326 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.710489  8531 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:23:47.709486 27479 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 38 to 39, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 39 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.704823  8574 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:47.729580 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 114923us
I20260709 12:23:47.717584  8531 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:47.735782 27628 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.746481  8574 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.747848 27736 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
W20260709 12:23:47.752632 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 99568.2us
I20260709 12:23:47.757263  8466 raft_consensus.cc:3055] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:23:47.757750  8466 raft_consensus.cc:740] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:47.761662  8400 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.762375  8400 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.764799  8400 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.767208 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:47.777225 27736 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 44: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:47.777271 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:47.778028 27729 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:23:47.739187  8531 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:47.780959 27734 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:23:47.740839  8417 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:23:47.784750 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 158129us
I20260709 12:23:47.787200 27552 leader_election.cc:304] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.780685  8400 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.792457 27553 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.787302  8417 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:47.806296 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:47.759001  8466 consensus_queue.cc:260] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.802302  8560 raft_consensus.cc:2749] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:47.809285  8417 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:47.831624 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:47.831777 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.839118 28337 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 41 index: 26. Preceding OpId from leader: term: 42 index: 27. (index mismatch)
I20260709 12:23:47.805894  8440 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.848313 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:47.859532  8377 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:23:47.862876  8276 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:23:47.867635  8297 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:47.868155  8297 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.868566  8297 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:47.832208  8400 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.868896  8550 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:23:47.870637  8400 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.872694 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.874185  8377 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:23:47.867646 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 80650.8us
I20260709 12:23:47.876320 27552 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.878818 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:47.879503  8550 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:47.881155 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.881961  8400 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:47.882611  8400 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:23:47.886721 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 56799.3us
I20260709 12:23:47.889533 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:47.892720 27580 raft_consensus.cc:1240] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 40. Current term is 41. Ops: []
I20260709 12:23:47.891913 27479 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 43 to 45. New cstate: current_term: 45 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:47.900008 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 107960us
I20260709 12:23:47.883522  8466 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:47.905737  8607 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:47.926776 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:47.883288  8645 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.955485 27587 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 44 because replica has last-logged OpId of term: 42 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 21.
I20260709 12:23:47.962172 27736 raft_consensus.cc:2410] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 44 because replica has last-logged OpId of term: 42 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 21.
I20260709 12:23:47.971613  8656 consensus_queue.cc:1059] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:47.972637  8656 raft_consensus.cc:3055] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:23:47.973044  8656 raft_consensus.cc:740] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:47.973980  8656 consensus_queue.cc:260] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:47.944924 27479 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 26 to 33, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 33 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.976231 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:47.978739 27479 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 36 to 38, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:47.974766  8531 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.001691  8546 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:48.001171 27479 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 29 to 40, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 40 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:48.002825  8560 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:23:47.981333 27704 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:48.004079  8390 raft_consensus.cc:993] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:48.007157  8645 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:47.975284  8656 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:48.028364 27627 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.032846  8390 raft_consensus.cc:1081] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Signalling peer 9c09719959a54e8eb66d2041dd9c3113 to start an election
I20260709 12:23:47.977008  8440 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:48.049000 27480 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 230) took 56491 ms (client timeout 15000 ms). Trace:
I20260709 12:23:48.039827  8567 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:48.052021  8567 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.054804  8567 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:47.979697  8297 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.057066  8297 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:48.049496 27480 rpcz_store.cc:269] 0709 12:22:51.557162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.227601 (+3670439us) service_pool.cc:224] Handling call
0709 12:23:48.048733 (+52821132us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:45.289847 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:45.631915 (+342068us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:45.637111 (+  5196us) write_op.cc:435] Acquired schema lock
0709 12:23:45.637133 (+    22us) tablet.cc:662] Decoding operations
0709 12:23:45.769452 (+132319us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:45.769513 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:45.769541 (+    28us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:23:45.770821 (+  1280us) tablet.cc:701] Row locks acquired
0709 12:23:45.770842 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:23:45.771176 (+   334us) write_op.cc:270] Start()
0709 12:23:45.771304 (+   128us) write_op.cc:276] Timestamp: P: 1783599825771107 usec, L: 0
0709 12:23:45.789941 (+ 18637us) op_driver.cc:348] REPLICATION: starting
0709 12:23:45.885629 (+ 95688us) log.cc:844] Serialized 6690 byte log entry
0709 12:23:46.356900 (+471271us) op_driver.cc:464] REPLICATION: finished
0709 12:23:46.575593 (+218693us) write_op.cc:301] APPLY: starting
0709 12:23:46.575728 (+   135us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:46.630277 (+ 54549us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:46.630447 (+   170us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:46.687232 (+ 56785us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a343e28 00007f0e9dd7e581 00007f0e9dd7dd58 00007f0e9dd829f1 00007f0e9dd829a9 00007f0e9dd82939 00007f0e9dd82901 00007f0e9dd81b2d 0000000000388634 0000000000388479
0709 12:23:46.756085 (+ 68853us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:46.756375 (+   290us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0709 12:23:46.756419 (+    44us) write_op.cc:312] APPLY: finished
0709 12:23:46.756761 (+   342us) log.cc:844] Serialized 156 byte log entry
0709 12:23:46.756938 (+   177us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:46.757126 (+   188us) write_op.cc:454] Released schema lock
0709 12:23:46.757186 (+    60us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:46.757781 (+   595us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:46.757813 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":41262,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":218373,"apply.run_cpu_time_us":28368,"apply.run_wall_time_us":182161,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75187,"cfile_init":1,"lbm_read_time_us":454,"lbm_reads_lt_1ms":4,"num_ops":17,"prepare.queue_time_us":188454,"prepare.run_cpu_time_us":23860,"prepare.run_wall_time_us":1406794,"raft.queue_time_us":133549,"raft.run_cpu_time_us":47049,"raft.run_wall_time_us":163420,"replication_time_us":566814,"spinlock_wait_cycles":61204224,"thread_start_us":384016,"threads_started":4,"wal-append.queue_time_us":234369,"wal-append.run_cpu_time_us":82454,"wal-append.run_wall_time_us":1634780}]]}
I20260709 12:23:48.064190 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:48.066552 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 127330us
I20260709 12:23:48.067070 28334 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: []
W20260709 12:23:48.037173  8653 log.cc:927] Time spent T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.108s	user 0.001s	sys 0.001s
I20260709 12:23:48.096056  8556 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.096711  8556 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.099450  8607 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:48.101271 27704 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:48.079380  8645 consensus_queue.cc:1059] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.025s
I20260709 12:23:48.104599  8508 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:48.105335  8508 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:48.077711  7968 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:48.112515  7968 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.113030  7968 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:48.117815  8667 log.cc:927] Time spent T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:48.120345 27479 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 32. New cstate: current_term: 32 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:48.087622 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:48.124080 27585 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:48.127818 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 326149us
I20260709 12:23:48.129448 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.131980 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb215cc72c384ec8b9421c81d953589b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.132715 27734 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:23:48.133492 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.133949 27736 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:23:48.134467 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe68c465ca4346b9b9db89a5be4e639c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.135016 27664 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:23:48.135236 27663 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:48.135818 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.168512 27730 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 44.
I20260709 12:23:48.136507 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.170245  8662 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:48.172780 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.137804  8508 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:48.141999  8417 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.144793  8531 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.122568 27479 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 37. New cstate: current_term: 37 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:48.148155 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:48.152737 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.154245 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.154688  8645 raft_consensus.cc:3055] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:48.155728 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:48.157895 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.158598 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.158758 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.161434 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.099081  8556 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.137194  8377 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:48.173928 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:48.106952 27704 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:48.212375 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.216852 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.224367 27655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.225270 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.229465 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.233062  8677 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:48.235086 27659 raft_consensus.cc:2393] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 40: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:48.235399  7554 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:48.235795 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.237447 27731 raft_consensus.cc:1240] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:48.207953  8276 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:48.210160  8645 raft_consensus.cc:740] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:48.251273 27730 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 41 index: 26. Preceding OpId from leader: term: 42 index: 27. (index mismatch)
I20260709 12:23:48.222188  8662 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:48.217353  8607 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:48.258366 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:48.258677 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 47252.9us
I20260709 12:23:48.259143 27735 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:48.261686 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:48.258416 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 85830.8us
I20260709 12:23:48.264983 27731 raft_consensus.cc:1240] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 LEADER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 42. Current term is 43. Ops: []
W20260709 12:23:48.286653 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 179772us
I20260709 12:23:48.289251 27704 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:48.285743  8607 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.293578  8607 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.297256 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.298355 27659 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 44.
I20260709 12:23:48.301823 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.302415 27661 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:48.303669 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:48.304533 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:48.305521 27660 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 43: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:48.308089 27662 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:48.311704 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.312369 27659 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:48.313272 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:48.313808 27658 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:48.315649 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:48.243094  8508 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:48.333185 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 44895us
I20260709 12:23:48.333406 27735 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:48.333201 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
W20260709 12:23:48.317800 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 143948us
W20260709 12:23:48.335436 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 46293.2us
I20260709 12:23:48.347111 27736 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:48.354291 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:48.275573  8276 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.353533  7554 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:48.363794 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 101189us
I20260709 12:23:48.397971 27656 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 41.
I20260709 12:23:48.400345 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:48.384418  7554 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.392608 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:48.362603 27660 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
 from {username='slave'} at 127.0.0.1:49998
I20260709 12:23:48.411888 27660 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:48.412281 27660 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:48.403766 27655 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
W20260709 12:23:48.421406 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 49066us
I20260709 12:23:48.290185  8645 consensus_queue.cc:260] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 42.29, Last appended by leader: 29, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:48.353025  8390 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:23:48.353863  8440 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.063s
W20260709 12:23:48.426162 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43767.4us
I20260709 12:23:48.473464 27585 raft_consensus.cc:2468] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:48.456876  8681 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:48.404253  8556 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:48.459460  8662 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:23:48.463406  8508 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:48.463904 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63695.6us
I20260709 12:23:48.424644  8531 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:48.447939  8682 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:48.469913 27658 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 47.
I20260709 12:23:48.470135 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:48.475246 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 48226.5us
I20260709 12:23:48.475809  7554 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.478699 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:48.480188 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 110403us
W20260709 12:23:48.481334 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.483574 27730 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:48.422268  8377 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:48.516601 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46557us
I20260709 12:23:48.464619  8276 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.483201  8662 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.581211 27589 raft_consensus.cc:1240] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 46. Current term is 47. Ops: [46.26-46.26]
I20260709 12:23:48.490734  8645 raft_consensus.cc:3060] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:23:48.585822 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 97421.4us
I20260709 12:23:48.487388  8531 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.577669 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.603264 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:48.606465 27701 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.484474  8556 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.585170  8662 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:48.613090  8655 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:48.616955 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.618662 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:48.591627  8531 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:48.607565  8491 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:48.642189  8683 log.cc:927] Time spent T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:23:48.648535  8679 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.638561  8655 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:48.667507 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 77107.3us
I20260709 12:23:48.675150 27660 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.656924  8431 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:48.672890  7968 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.559s	user 0.011s	sys 0.000s
I20260709 12:23:48.672452  8508 raft_consensus.cc:2749] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:23:48.679307  8697 consensus_queue.cc:1059] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.042s
I20260709 12:23:48.667429  8491 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:48.696611  8655 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.648702  8377 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:48.698369  8491 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:48.698796 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.699069  8508 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:23:48.699577  8508 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:48.700615  8508 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.26, Last appended by leader: 26, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.701951  8677 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:23:48.673822  8556 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.656292  8660 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:48.715914  8431 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:48.718791  8431 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:48.652053  8546 consensus_queue.cc:1059] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.042s
I20260709 12:23:48.723877  8660 raft_consensus.cc:3055] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:48.724258  8660 raft_consensus.cc:740] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:48.725081  8660 consensus_queue.cc:260] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.726331 27660 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:48.707511 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 93022.8us
I20260709 12:23:48.719267 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.705084 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.687701  8679 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.745180  8679 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.730489  8417 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:48.750972 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.753772 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
W20260709 12:23:48.756605 27554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.763820 27704 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:48.773959 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 184608us
W20260709 12:23:48.798167 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.799924 27586 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 43 index: 29. (term mismatch)
I20260709 12:23:48.800316 27586 pending_rounds.cc:85] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 28
I20260709 12:23:48.800549 27586 pending_rounds.cc:107] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 42.29
I20260709 12:23:48.800838 27586 raft_consensus.cc:2889] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:48.801623  8704 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:23:48.805387  8531 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.693715  7968 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.813269 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.782819  8662 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.813977 27738 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
W20260709 12:23:48.814954 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 112474us
I20260709 12:23:48.797047  8684 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:48.815603 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:48.817267 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 295469us
I20260709 12:23:48.820477 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.823258 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.826086 27735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.827957 27732 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:23:48.829413 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.835963  8685 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.024s
I20260709 12:23:48.837589 27729 raft_consensus.cc:2393] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:48.840215  8662 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:48.840831  8531 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:48.844816 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.848583 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.853713 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:48.854440 27731 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:48.857309  7968 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:48.859760 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.865618 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:48.866911 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:48.867758 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:48.869138 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:48.869313 28333 raft_consensus.cc:1275] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:23:48.889708  8679 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:48.890374  8679 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:48.892513  8679 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:48.895601 27552 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:48.897058  8531 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:48.897630  8531 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:48.898505  8531 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.899443 27552 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:48.900029  8466 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:48.900887  8700 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:48.901454  8700 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:48.903537  8677 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:23:48.902444  8700 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.909263 27734 raft_consensus.cc:1240] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 38. Current term is 39. Ops: [38.23-38.23]
I20260709 12:23:48.912284 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:48.912338 27552 leader_election.cc:304] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.913443  8645 raft_consensus.cc:2749] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:48.914464 27701 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:48.915766 27552 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:48.917160  8660 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:48.938812 27627 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:48.934463 27731 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
W20260709 12:23:48.934922 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 54695.7us
I20260709 12:23:48.923663  8508 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:48.942476 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.943850 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:48.944579 27589 raft_consensus.cc:3055] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:23:48.947175 27630 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:48.942651 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d7343bc8174a5f89f4652b4401c3ec" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:48.947989 27627 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:48.948381 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.944962 27589 raft_consensus.cc:740] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:48.950634 27627 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:48.953991 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.954793 27585 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 46.
I20260709 12:23:48.956421  8431 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.008s
I20260709 12:23:48.971431 27586 raft_consensus.cc:1240] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 44. Current term is 45. Ops: []
I20260709 12:23:48.975910  7554 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:48.975366 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:48.953197  8677 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:48.956593  8685 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:48.978866 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:48.979624  8431 consensus_queue.cc:1059] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.004s
I20260709 12:23:48.982249  8709 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:23:48.982501  8431 raft_consensus.cc:3055] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:48.981868  8432 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:48.983144  8431 raft_consensus.cc:740] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:48.984342  8431 consensus_queue.cc:260] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.985960  8431 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:23:48.987663 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9c09719959a54e8eb66d2041dd9c3113 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:48.977648 27589 consensus_queue.cc:260] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 44.23, Last appended by leader: 23, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:48.992483 27589 raft_consensus.cc:3060] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:48.995468  8699 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:23:48.983161  8690 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:48.951501  8684 consensus_queue.cc:1048] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.033s
W20260709 12:23:49.004935 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 90614.2us
I20260709 12:23:49.005820 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.008193  8677 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:49.010350  8685 raft_consensus.cc:697] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:23:49.011514 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 95841.5us
I20260709 12:23:48.982704  8709 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.012203  8709 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:49.014029 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:49.014412 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.014997  8699 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:49.015817 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.016631  8662 raft_consensus.cc:2804] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:49.017006  8662 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:49.011276  8685 consensus_queue.cc:237] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.017890 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.018086 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.021397 27630 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.021551 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.023686 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.024726 27627 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:49.024978 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.025463  7554 raft_consensus.cc:2804] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:49.025624 27627 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.026037  7554 raft_consensus.cc:697] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:49.017225  8699 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:23:49.017431  8662 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:49.032320 27701 leader_election.cc:304] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.030956 27630 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.022795  8684 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:49.036613  8560 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:49.022491  8531 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:49.038934  7554 consensus_queue.cc:237] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 44.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.033608  8656 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:23:49.030575  8645 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:49.052920 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:49.042546 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.029031  8697 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:49.029614  8700 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:49.065610 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:49.067895 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:49.068252 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.063194  8677 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.046619  8679 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:49.071313 27589 raft_consensus.cc:2468] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:23:49.048820 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.060312 27660 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:23:49.034963  8417 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:49.095372 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:49.060663  8704 raft_consensus.cc:2749] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:49.057158  8645 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:49.102454 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:49.094301 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45538.6us
I20260709 12:23:49.104642 27630 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.070155  8699 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:23:49.066485  8700 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:49.096377  8546 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:49.108163  8645 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:49.148777 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.097229  8677 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.151508  8440 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:49.151688  8680 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:23:49.151971  8440 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:49.152318  8680 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.152931  8680 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:49.114122  8699 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:49.123591  8574 raft_consensus.cc:2749] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:49.152501  8440 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:49.123839  8700 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:49.131558  8466 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:23:49.155673  8699 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:49.210158 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:49.159626  8390 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
W20260709 12:23:49.214834 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 118339us
I20260709 12:23:49.218492 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:49.212598  8466 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:49.224097  8656 consensus_queue.cc:1059] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:49.231290 27630 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.220383 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.236469  8720 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:49.210882  8701 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:49.239055 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:49.242480 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.219678  8677 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:49.247258 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.224449  8466 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.271941 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:49.250211  8495 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:49.252236  8704 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:49.252843  8297 raft_consensus.cc:3055] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:23:49.280556 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:49.230538  8390 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.301290  8446 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.301970  8446 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:49.307337  8446 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:49.307703 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:49.313239 27664 raft_consensus.cc:2410] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 39 because replica has last-logged OpId of term: 38 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 22.
I20260709 12:23:49.277432  8706 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:49.282338  8704 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:49.282879  8297 raft_consensus.cc:740] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:49.311672  8508 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:23:49.334900 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 74204.9us
I20260709 12:23:49.326320  8704 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:49.311290  8709 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.347672  8721 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:23:49.348666  8715 log.cc:927] Time spent T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: Append to log took a long time: real 0.206s	user 0.000s	sys 0.002s
I20260709 12:23:49.347896  8390 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:49.404945  8709 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:49.406085 27659 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 45. Current term is 46. Ops: [45.24-45.24]
I20260709 12:23:49.360839  8297 consensus_queue.cc:260] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.408210  8297 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:49.426271 28333 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
W20260709 12:23:49.473569  8716 log.cc:927] Time spent T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.337s	user 0.002s	sys 0.000s
W20260709 12:23:49.488703  8722 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.189s	user 0.002s	sys 0.000s
I20260709 12:23:49.528318 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
W20260709 12:23:49.532032 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 177954us
I20260709 12:23:49.545313 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:49.545992 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:49.549079 27655 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 21. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:23:49.552922 27729 raft_consensus.cc:1240] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:23:49.546839 27659 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:23:49.567771 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:49.570969 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:49.591984 27660 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
W20260709 12:23:49.553534  8719 log.cc:927] Time spent T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.199s	user 0.000s	sys 0.002s
I20260709 12:23:49.602651 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.619812 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:49.542058  8440 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.652431  7554 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:49.593353  8546 consensus_queue.cc:1059] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.053s
I20260709 12:23:49.630828  8344 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:49.653190  7554 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.593103  8726 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.051s
W20260709 12:23:49.649108  8662 consensus_meta.cc:287] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.599s	user 0.011s	sys 0.000s
I20260709 12:23:49.636706  8400 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.043s
I20260709 12:23:49.670464  8344 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.686000 27730 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:23:49.673206  8662 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.703292  8344 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:23:49.738265  8731 log.cc:927] Time spent T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:23:49.746150 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 95274.8us
I20260709 12:23:49.760389 27737 raft_consensus.cc:3060] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:49.761611  8440 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:49.761993  8662 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:49.762959  8699 consensus_meta.cc:287] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.538s	user 0.010s	sys 0.000s
I20260709 12:23:49.770012 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:23:49.773268 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 144685us
I20260709 12:23:49.779982 27730 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 42 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
W20260709 12:23:49.794880  8700 consensus_meta.cc:287] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.590s	user 0.003s	sys 0.005s
W20260709 12:23:49.795360  8722 log.cc:927] Time spent T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:23:49.819767 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 162095us
I20260709 12:23:49.832324  7554 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:49.837685 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 190134us
I20260709 12:23:49.838757 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:49.843567 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:49.858950  8684 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.815s	user 0.007s	sys 0.003s
I20260709 12:23:49.898572 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:49.887104 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 43630.4us
I20260709 12:23:49.859831 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:49.850147 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:23:49.861198  8680 consensus_meta.cc:287] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: Time spent flushing consensus metadata: real 0.708s	user 0.010s	sys 0.003s
W20260709 12:23:49.862586  8704 consensus_meta.cc:287] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: Time spent flushing consensus metadata: real 0.517s	user 0.005s	sys 0.005s
I20260709 12:23:49.874472  8720 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:49.887188 27583 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:49.825086  8699 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:49.894249 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 124348us
I20260709 12:23:49.891124 27588 raft_consensus.cc:2368] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40.
I20260709 12:23:49.904335 27738 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
W20260709 12:23:49.906831 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70920.1us
W20260709 12:23:49.908215 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 58154.6us
I20260709 12:23:49.865975 27659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:49.932711 27584 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 44 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:23:49.933480 27735 raft_consensus.cc:1275] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 44 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:23:49.936004 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:49.920220  8440 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:23:49.936618 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:49.928053  8704 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.939359 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:49.904976  7968 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:49.940860 27732 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 38. Current term is 39. Ops: []
I20260709 12:23:49.942402  8466 consensus_queue.cc:1059] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:23:49.921595  8680 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.943449  8390 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:49.944646  8680 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:49.945061  8400 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:23:49.916427  8706 consensus_meta.cc:287] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.593s	user 0.011s	sys 0.000s
I20260709 12:23:49.936653 27585 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:49.952268 27580 raft_consensus.cc:1275] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:23:49.954248 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:49.956529 27588 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:49.907702  8432 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:23:49.909181  8684 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:49.973649 27587 raft_consensus.cc:1240] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 41. Current term is 42. Ops: []
I20260709 12:23:49.968770 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:49.976979 27585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:49.970422 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:49.987428 27656 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:49.903440  8700 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.992463 27732 raft_consensus.cc:3060] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:23:49.993536 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 61409.2us
I20260709 12:23:49.947088  8400 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.033s
I20260709 12:23:49.996227 27704 leader_election.cc:400] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40.
I20260709 12:23:49.996549 27704 leader_election.cc:403] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20260709 12:23:49.997409  7554 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:49.997789  7554 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:49.998124  7554 raft_consensus.cc:3060] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:49.978695  8706 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:23:50.010236 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:50.018450 27581 raft_consensus.cc:1240] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 44. Current term is 45. Ops: []
W20260709 12:23:50.022193 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50115.2us
I20260709 12:23:49.990651  8704 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:50.022171  8645 consensus_meta.cc:287] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: Time spent flushing consensus metadata: real 0.826s	user 0.007s	sys 0.006s
I20260709 12:23:50.029798  8645 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:49.985258  8297 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.036s
I20260709 12:23:50.016532  8508 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 39. Current term is 40.
I20260709 12:23:50.049824 27589 raft_consensus.cc:1240] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
W20260709 12:23:50.032507  8715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.009s	sys 0.021s
I20260709 12:23:50.061926 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:50.066084 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 124131us
W20260709 12:23:50.063469  8715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.010s	sys 0.022s
W20260709 12:23:50.073642 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 429044us
I20260709 12:23:50.039775  8697 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.040s
I20260709 12:23:50.031656  8730 consensus_queue.cc:1059] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.041s
I20260709 12:23:50.048905  8656 consensus_queue.cc:1048] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:23:50.087178 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:50.086947 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:50.050269  8681 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:23:50.074218  8466 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.025s
I20260709 12:23:50.108412  8700 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:50.108520  8645 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:50.109349 27588 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 47.
I20260709 12:23:50.094012  8656 consensus_queue.cc:1243] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer 3f90188458da46f6ba138d5f82c55b69 log is divergent from this leader: its last log entry 37.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:23:50.125618  8344 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:50.130882  8725 raft_consensus.cc:3055] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:23:50.133659 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:50.134645  7554 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.187989 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:50.134164  8344 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:50.152509 27657 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:23:50.158320 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:50.118266  8692 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:23:50.135380  8660 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:23:50.172959 27732 raft_consensus.cc:2468] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 47.
I20260709 12:23:50.114991  8531 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:23:50.177820  8745 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:23:50.208384 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:50.090126  8495 consensus_queue.cc:1048] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.030s
I20260709 12:23:50.209537 27737 raft_consensus.cc:2393] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:50.161316 27630 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:50.153304  8377 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:50.215924  8377 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:50.139038  8662 raft_consensus.cc:3055] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:23:50.217605  8662 raft_consensus.cc:740] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:50.174151  8699 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:50.218626  8662 consensus_queue.cc:260] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.220463  8662 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:50.206485  7554 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:50.174598  8725 raft_consensus.cc:740] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:50.234663  7968 consensus_queue.cc:1059] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 44.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:50.235229  8747 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:50.235808  8747 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:50.235915  7968 raft_consensus.cc:3055] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:23:50.236331  7968 raft_consensus.cc:740] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:50.236749  8377 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.237468  7968 consensus_queue.cc:260] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 44.22, Last appended by leader: 22, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:50.249243 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 88065.6us
I20260709 12:23:50.253300 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:50.254526 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:50.257898 27580 raft_consensus.cc:1217] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Deduplicated request from leader. Original: 34.19->[36.20-40.22]   Dedup: 36.20->[38.21-40.22]
I20260709 12:23:50.258596 27580 pending_rounds.cc:85] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting all ops after (but not including) 20
I20260709 12:23:50.258867 27580 pending_rounds.cc:107] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Aborting uncommitted NO_OP operation due to leader change: 37.21
I20260709 12:23:50.259140 27580 raft_consensus.cc:2889] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:50.260313 27737 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 47 index: 26. (term mismatch)
I20260709 12:23:50.260950 27737 pending_rounds.cc:85] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting all ops after (but not including) 25
I20260709 12:23:50.261359 27737 pending_rounds.cc:107] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: Aborting uncommitted NO_OP operation due to leader change: 46.26
I20260709 12:23:50.261760 27737 raft_consensus.cc:2889] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:50.267426 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:50.205163  8660 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:50.277513 27730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:50.279148 27583 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: []
I20260709 12:23:50.225020  8508 consensus_queue.cc:1059] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.052s
I20260709 12:23:50.272652  8656 raft_consensus.cc:2804] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:23:50.298034  8656 raft_consensus.cc:697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:50.302114 28332 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: [42.25-42.25]
I20260709 12:23:50.314615 27588 raft_consensus.cc:2468] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:50.287508  7968 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:23:50.318089 27704 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.318660 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:50.338702 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:50.319435 27664 raft_consensus.cc:1240] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 45. Current term is 46. Ops: [45.24-45.24]
I20260709 12:23:50.339709 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
W20260709 12:23:50.360013 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 44555us
I20260709 12:23:50.366707  8390 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:50.353585  8737 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:23:50.410480 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47418.3us
I20260709 12:23:50.362172  8704 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.312719  8725 consensus_queue.cc:260] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.410125  8432 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.459860  8432 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.402160  8645 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.065s
I20260709 12:23:50.407606  8508 consensus_queue.cc:1059] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.055s
I20260709 12:23:50.462162  8432 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:50.463485 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 63239us
W20260709 12:23:50.463693  8756 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:23:50.458312 27630 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:50.479432  8761 raft_consensus.cc:2804] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:23:50.483861 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 72744.2us
I20260709 12:23:50.515076 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:50.515736 27588 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:50.358757  8660 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.24, Last appended by leader: 24, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.531924  8377 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:50.535969  8747 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:50.537575  8754 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
W20260709 12:23:50.539206  8744 log.cc:927] Time spent T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.475s	user 0.012s	sys 0.006s
W20260709 12:23:50.542843  8753 log.cc:927] Time spent T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 12:23:50.548069  8720 raft_consensus.cc:3055] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:50.548599  8720 raft_consensus.cc:740] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:50.537611  8747 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:50.549374  8720 consensus_queue.cc:260] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:50.552953 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 61729.5us
I20260709 12:23:50.467099  8704 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:50.562304 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 208122us
W20260709 12:23:50.564555 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 202619us
W20260709 12:23:50.565980 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 212917us
I20260709 12:23:50.519405  8761 raft_consensus.cc:697] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:50.570086 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:50.570154 27731 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 42.
I20260709 12:23:50.529444  8682 raft_consensus.cc:3055] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:23:50.570794 27736 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:50.474987  8656 consensus_queue.cc:237] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.565264  8660 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:50.577231  8720 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:50.579138  8704 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:50.581142  8761 consensus_queue.cc:237] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:50.586726 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9c09719959a54e8eb66d2041dd9c3113 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:50.588505 27627 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:50.592517 27553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.594463 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:50.595968 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:50.572184  8682 raft_consensus.cc:740] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
W20260709 12:23:50.563596  8546 consensus_queue.cc:1175] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9c09719959a54e8eb66d2041dd9c3113" responder_term: 46 status { last_received { term: 43 index: 23 } last_committed_idx: 23 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3f90188458da46f6ba138d5f82c55b69 for earlier term 45. Current term is 46. Ops: [45.24-45.24]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:50.597513  8431 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:23:50.679256 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 87420.4us
W20260709 12:23:50.721196 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 125809us
I20260709 12:23:50.719830  8431 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.723855  8431 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:23:50.725337 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 126229us
W20260709 12:23:50.726217 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 126979us
I20260709 12:23:50.742717 27701 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:50.744220  8682 consensus_queue.cc:260] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:50.745519  8682 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:50.763401  8556 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:50.764204  8556 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.767053  8556 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:50.799844 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 57216.6us
W20260709 12:23:50.808881  8773 log.cc:927] Time spent T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:23:50.825531 27730 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 45.
I20260709 12:23:50.828989 27581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:50.829586 27581 raft_consensus.cc:606] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:50.829914 27581 raft_consensus.cc:614] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:50.891669 27581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:50.892517 27581 raft_consensus.cc:606] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:50.892964 27581 raft_consensus.cc:614] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:50.879881  8431 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:50.949270  8431 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:50.995787 27581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:50.999974 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:51.000743 27586 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:51.000299  8704 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:51.015997 27582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:51.020193 27582 raft_consensus.cc:606] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.020612 27582 raft_consensus.cc:614] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.035880 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.038628  8747 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:23:51.046295  8760 log.cc:927] Time spent T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.506s	user 0.000s	sys 0.011s
I20260709 12:23:51.047880 27586 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:51.053341 27586 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:51.054119 27586 raft_consensus.cc:606] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.054582 27586 raft_consensus.cc:614] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.081698  8725 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:23:51.087146 27583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:51.087878 27583 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.088204 27583 raft_consensus.cc:614] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.093817 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:23:51.106197  8757 log.cc:927] Time spent T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.565s	user 0.002s	sys 0.008s
I20260709 12:23:51.107077  8417 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:51.107944  8417 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:23:51.109530 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 97009us
I20260709 12:23:51.124877  8417 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:51.125751 27734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34464 (request call id 4615) took 1032 ms. Trace:
I20260709 12:23:51.127187 27734 rpcz_store.cc:276] 0709 12:23:50.093364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.093503 (+   139us) service_pool.cc:224] Handling call
0709 12:23:50.093860 (+   357us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:50.112852 (+ 18992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.112883 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:50.117286 (+  4403us) log.cc:844] Serialized 52 byte log entry
0709 12:23:50.324948 (+207662us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:23:50.324974 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:50.325014 (+    40us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:51.105380 (+780366us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e99ee386e 00007f0e99ee2bb6 00007f0e99ee27d8 00007f0e99eec221 00007f0e99eec1d9 00007f0e99eec169
0709 12:23:51.125470 (+ 20090us) raft_consensus.cc:1685] finished
0709 12:23:51.125490 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:51.125695 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6593,"raft.queue_time_us":20463,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3129,"spinlock_wait_cycles":27866496,"thread_start_us":119642,"threads_started":1,"wal-append.queue_time_us":314237}
W20260709 12:23:51.132810 27702 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.134982 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.140209 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.142206 28336 raft_consensus.cc:1240] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Rejecting Update request from peer ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 42. Current term is 43. Ops: [42.25-42.25]
I20260709 12:23:51.143272 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:51.148320 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:51.149009 27729 raft_consensus.cc:3060] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:51.150291 27702 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:51.152181  8691 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:51.154259  8417 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:51.154898  8417 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:51.158190 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.161160 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:51.163476 27729 raft_consensus.cc:2468] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:51.163723  8417 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:51.170265 27552 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:51.172235  8767 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:51.174793  8682 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
W20260709 12:23:51.175037  8764 log.cc:927] Time spent T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.613s	user 0.000s	sys 0.003s
W20260709 12:23:51.175994 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 62473.9us
I20260709 12:23:51.176295 27552 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:51.182014 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:51.187839  8645 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:23:51.188496  8645 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:51.189483  8645 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.178912 27630 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:51.174858  8767 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:51.192471  8767 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 44.22, Last appended by leader: 22, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.195775 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:51.175622  8682 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.197741  8682 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:51.199422 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.204030 27588 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 45: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:51.208032 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:23:51.208777  8692 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:23:51.212286 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 50063.2us
I20260709 12:23:51.215133  8656 raft_consensus.cc:2804] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:23:51.215719  8701 raft_consensus.cc:493] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:51.216343  8701 raft_consensus.cc:515] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:51.218618  8701 leader_election.cc:290] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:51.215760  8656 raft_consensus.cc:697] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:51.220230  8656 consensus_queue.cc:237] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.231668 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:51.237318 27627 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.250216 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:51.241312  8701 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:23:51.271942 27479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 135) took 20748 ms (client timeout 15000 ms). Trace:
I20260709 12:23:51.293064 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22597fcae204491f84839e680c083ecc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.294023 27734 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
W20260709 12:23:51.272505 27479 rpcz_store.cc:269] 0709 12:23:30.523567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.468896 (+12945329us) service_pool.cc:224] Handling call
0709 12:23:51.271837 (+7802941us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:48.901249 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:48.901938 (+   689us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:48.901995 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:23:48.902014 (+    19us) tablet.cc:662] Decoding operations
0709 12:23:48.910497 (+  8483us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:48.910591 (+    94us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:48.910619 (+    28us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:23:48.912011 (+  1392us) tablet.cc:701] Row locks acquired
0709 12:23:48.912059 (+    48us) write_op.cc:260] PREPARE: finished
0709 12:23:48.912215 (+   156us) write_op.cc:270] Start()
0709 12:23:48.912389 (+   174us) write_op.cc:276] Timestamp: P: 1783599828912159 usec, L: 0
0709 12:23:48.912430 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:23:48.913107 (+   677us) log.cc:844] Serialized 6672 byte log entry
0709 12:23:50.132089 (+1218982us) op_driver.cc:464] REPLICATION: finished
0709 12:23:50.300255 (+168166us) write_op.cc:301] APPLY: starting
0709 12:23:50.300375 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:50.310650 (+ 10275us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:50.310771 (+   121us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:50.479438 (+168667us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:50.479827 (+   389us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0709 12:23:50.479873 (+    46us) write_op.cc:312] APPLY: finished
0709 12:23:50.611029 (+131156us) log.cc:844] Serialized 156 byte log entry
0709 12:23:50.740036 (+129007us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:50.744400 (+  4364us) write_op.cc:454] Released schema lock
0709 12:23:50.744490 (+    90us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:50.744972 (+   482us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:50.744999 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":539736,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":166930,"apply.run_cpu_time_us":23976,"apply.run_wall_time_us":445593,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75187,"num_ops":17,"prepare.queue_time_us":215520,"prepare.run_cpu_time_us":37991,"prepare.run_wall_time_us":122298,"raft.queue_time_us":731325,"raft.run_cpu_time_us":34311,"raft.run_wall_time_us":203693,"replication_time_us":1182794,"thread_start_us":1112559,"threads_started":4,"wal-append.queue_time_us":116277}]]}
I20260709 12:23:51.325405 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:51.326123 27662 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
I20260709 12:23:51.336853 27627 leader_election.cc:304] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.338223 27580 raft_consensus.cc:1275] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
I20260709 12:23:51.341650  8787 raft_consensus.cc:2749] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:51.344014 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:51.350119  8682 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:23:51.350731  8682 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:51.351178  8682 raft_consensus.cc:3060] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:51.338624  8762 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.017s
I20260709 12:23:51.368144  8762 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:51.368701  8762 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.371186  8762 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.371858 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.372741 27729 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:51.372278 27483 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 32 to 33, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 33 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.382208 27483 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 29 to 33, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 33 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:51.384158 27483 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 27 to 32. New cstate: current_term: 32 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.399715 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.401008 27580 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:51.400365 27587 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:51.398020  8762 consensus_queue.cc:1048] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.003s
I20260709 12:23:51.402782 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.404419  8762 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:23:51.404847  8762 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:51.405256  8762 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:51.411080 27731 raft_consensus.cc:1275] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:51.412076 27483 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 33 to 38. New cstate: current_term: 38 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.416803  8779 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:51.421700 27483 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 37 to 39, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.423746  8752 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.427291  8752 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.430037  8752 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.432125 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:51.438071 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.438877 27731 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 32.
I20260709 12:23:51.427193 27483 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:51.440296 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:51.443482 27483 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.453084  8432 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:51.453195 27483 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 32 to 34, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 34 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.453809  8432 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.455926  8432 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.456204  8752 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:23:51.456748  8752 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.457149  8752 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:51.460448  8432 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:51.461066  8432 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.440579  8682 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.466329 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.467424 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.472584 27586 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:51.468578 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.469486 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.473492 27731 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:51.474653 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:51.475687  7554 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:23:51.476071  7554 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting leader election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:51.476434  7554 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:23:51.476797 27483 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 32 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 37 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.481709 27734 raft_consensus.cc:2393] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 45: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:51.480363 27483 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 31 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 37 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:51.483168 27627 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.463246  8432 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.488139  8787 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:51.492022  8431 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:23:51.495854  8787 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.503942 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.506978 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.511089 27734 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 42: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:51.479750  8707 consensus_queue.cc:1048] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:23:51.515110  8682 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:51.511801  8645 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.520488 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.526089 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:51.526607  8736 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:23:51.517359 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:51.509670  8787 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.541450  7554 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.553153 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:51.553908 27586 raft_consensus.cc:2393] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:51.513073  8762 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.536235  8645 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.570896  7554 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.572192 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:51.578053 27588 raft_consensus.cc:2393] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 45: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
I20260709 12:23:51.583920 27662 raft_consensus.cc:2468] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
I20260709 12:23:51.585523 27701 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:51.594478  8796 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:23:51.593226 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:51.596195  8691 raft_consensus.cc:2804] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:23:51.613514 27586 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:23:51.618649 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:51.615211 27630 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:51.619624 27729 raft_consensus.cc:2393] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 39: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:23:51.622212  8707 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:51.627710 27734 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 44 index: 22. Preceding OpId from leader: term: 45 index: 23. (index mismatch)
I20260709 12:23:51.629161  7554 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.629823  7554 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.619809  8762 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:23:51.648110  8795 log.cc:927] Time spent T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:23:51.652055  7554 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.652487  8645 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.623942  8691 raft_consensus.cc:697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:51.654147  8691 consensus_queue.cc:237] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.618901  8390 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.655625 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.655895  8390 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.656455  8739 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:23:51.658802 27588 raft_consensus.cc:2410] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for term 47 because replica has last-logged OpId of term: 45 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 23.
I20260709 12:23:51.658643  8390 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.662251  8752 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.663769 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.663897  8431 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:51.666776 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:23:51.672955 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9c09719959a54e8eb66d2041dd9c3113 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:51.673934 27731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.680796  8707 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:51.688345  8707 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.694275  8736 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:23:51.701836 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47536.7us
I20260709 12:23:51.702078 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:23:51.726202 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:51.728474 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:51.729566 27660 raft_consensus.cc:2410] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for term 46 because replica has last-logged OpId of term: 44 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 23.
I20260709 12:23:51.733412 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:51.734084 27662 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 46.
I20260709 12:23:51.736184  8752 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.739569 27630 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.744345  8736 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:23:51.751793  8495 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:23:51.752434  8495 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:51.753094 27732 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:23:51.753456  8495 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.785079 27586 raft_consensus.cc:1275] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:23:51.805389  8431 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.038s
I20260709 12:23:51.855507  8779 consensus_queue.cc:1048] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:51.859517  7554 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:51.860319  7554 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.883527 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:51.884462 27733 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:23:51.894722 27733 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:51.927945  7554 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:51.927194 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.933756 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:51.934597 27732 raft_consensus.cc:2368] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:51.946887 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:51.948246 27627 leader_election.cc:400] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:51.948635 27627 leader_election.cc:403] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:51.950789  8752 raft_consensus.cc:2749] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:51.964852 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:51.965726 27588 raft_consensus.cc:2368] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:51.976775 27738 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:23:51.982285  8679 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:23:51.986382 27630 leader_election.cc:400] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
I20260709 12:23:51.977995  8752 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.980983  8725 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:23:52.008020  8805 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.019716  8805 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:51.977591  8508 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:23:52.026958  8752 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.027647  8508 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.029476  8752 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:52.014930  8725 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.034629  8725 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:52.051461  8679 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 32. Current term is 33.
W20260709 12:23:52.053115 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 104988us
I20260709 12:23:52.054476 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:52.059294 27662 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
W20260709 12:23:52.059649 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78016.9us
I20260709 12:23:52.061712 27663 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 44 index: 22. Preceding OpId from leader: term: 45 index: 23. (index mismatch)
W20260709 12:23:52.068540 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 76688.7us
I20260709 12:23:52.069377 27552 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.099416  8772 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.099901  8739 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:52.100350  8772 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:52.087951  8645 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:23:52.103744  8645 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:52.104682  8645 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.111111 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:52.112018 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 43.
I20260709 12:23:52.116041 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.124842 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:52.126273  8805 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:52.132620 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:52.133908 28333 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:52.136562  8725 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.108688  8772 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:52.141390 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 33.
I20260709 12:23:52.154942 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:23:52.150605  8797 log.cc:927] Time spent T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.299s	user 0.002s	sys 0.002s
I20260709 12:23:52.167419  8508 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:52.178999 27731 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:52.158548  8645 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:52.190094 27587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:52.191121 27587 raft_consensus.cc:2368] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 46. Current term is 47.
I20260709 12:23:52.191157 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:52.213979 27580 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:23:52.219573 27704 leader_election.cc:400] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 46. Current term is 47.
I20260709 12:23:52.220158 27704 leader_election.cc:403] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:52.221788  8725 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:52.222733  8816 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.223280  8816 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.225415  8816 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:52.225900 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 106598us
I20260709 12:23:52.227414 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:52.187690 27702 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.235628 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:52.237214  8682 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:52.237690  8682 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.238039  8682 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.238651  8816 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 47
I20260709 12:23:52.216079  8645 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.240681  8645 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:23:52.243625 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 127645us
I20260709 12:23:52.247598 27658 raft_consensus.cc:2393] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 46: Already voted for candidate 9c09719959a54e8eb66d2041dd9c3113 in this term.
I20260709 12:23:52.247834 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:52.228238  8736 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:52.284221 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:52.284843 27658 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:23:52.279332  8807 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.011s
W20260709 12:23:52.291553 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 45267.5us
I20260709 12:23:52.294008 27704 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
W20260709 12:23:52.290058  8818 log.cc:927] Time spent T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:23:52.300599 27589 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:23:52.301113 27580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:52.304080 27580 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:23:52.305989 27588 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:23:52.277794  8801 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.023s
W20260709 12:23:52.308005 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 58734.1us
I20260709 12:23:52.282099  8736 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:52.312920 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 61777.6us
I20260709 12:23:52.284636  8660 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.021s
I20260709 12:23:52.308854 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.310590  8817 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:23:52.316282  8709 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:52.365372  8736 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:52.372931  8645 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:52.398170 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 89403us
I20260709 12:23:52.406843 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:52.375286 27588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:52.377892  8816 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 46. Reason: Vote denied by peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) with higher term. Message: Invalid argument: T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for earlier term 46. Current term is 47.
I20260709 12:23:52.408844 27660 raft_consensus.cc:1275] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 44 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:23:52.391294  8817 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.412560  8817 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:52.413641  8682 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:52.415458  8574 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:52.416042  8574 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:52.418344  8574 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:52.421995  8645 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:52.404505  8704 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.022s
I20260709 12:23:52.426129  8682 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:52.426676  8817 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.431289 27588 raft_consensus.cc:2393] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate 3f90188458da46f6ba138d5f82c55b69 in this term.
W20260709 12:23:52.387883 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 78625.2us
I20260709 12:23:52.374279 27658 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 47.
I20260709 12:23:52.407712 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:23:52.432920 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:52.436538 27662 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.434582 27580 raft_consensus.cc:1240] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44. Ops: []
I20260709 12:23:52.403066  8823 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:23:52.439761  8823 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.440219  8823 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.453388 27662 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:52.458997 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:52.460525 28334 raft_consensus.cc:3060] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.461455 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:52.462567 27738 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 47.
I20260709 12:23:52.458112 27701 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.466533  8417 raft_consensus.cc:2804] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:52.467059  8417 raft_consensus.cc:697] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:52.467989  8417 consensus_queue.cc:237] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:52.480806 28334 raft_consensus.cc:2468] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:52.435249  8816 consensus_queue.cc:1048] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.049s
I20260709 12:23:52.484936 27732 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:23:52.489163 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:52.512178 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:52.513115 27738 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.513942 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:52.525271 27552 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:52.533555 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:52.542685 27733 raft_consensus.cc:1240] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Rejecting Update request from peer 9c09719959a54e8eb66d2041dd9c3113 for earlier term 39. Current term is 40. Ops: [39.25-39.25]
I20260709 12:23:52.546943  8819 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:23:52.547799  8819 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:52.549023  8819 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.24, Last appended by leader: 24, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.557164  8823 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.558877 27732 raft_consensus.cc:2368] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44.
I20260709 12:23:52.571532 27738 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 44.
I20260709 12:23:52.502045  8782 consensus_queue.cc:1059] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.043s
I20260709 12:23:52.576712 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "456e2eac35d647a3bf9eeb45a3d7596f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:52.576646 27552 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.581676  8782 raft_consensus.cc:3055] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:23:52.582129  8782 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:52.582967  8782 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.584177  8782 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.587935  8390 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:52.588490  8390 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:52.589412  8390 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.601011 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:52.608683  8811 consensus_queue.cc:1059] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:52.613374  8827 raft_consensus.cc:3055] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:23:52.613864  8827 raft_consensus.cc:740] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:52.614771  8827 consensus_queue.cc:260] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.616621  8827 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:23:52.620683 27628 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:52.622223  8779 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:52.629304  8817 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:52.634948 27732 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:23:52.637079 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:52.639158 27627 leader_election.cc:400] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44.
I20260709 12:23:52.639652 27627 leader_election.cc:403] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:52.660365  8709 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.688558 27656 raft_consensus.cc:2410] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 for term 40 because replica has last-logged OpId of term: 39 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 24.
I20260709 12:23:52.690570 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:52.654592  8725 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:23:52.693207  8826 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:23:52.721588 27664 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:23:52.724706  8823 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:52.725709 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:52.727160  8725 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:23:52.740216 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:52.741055 27583 raft_consensus.cc:3060] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:23:52.759450  8709 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 43. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 for earlier term 43. Current term is 44.
I20260709 12:23:52.761559 27583 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:52.762742 27704 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:52.763909  8817 raft_consensus.cc:2804] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:23:52.764377  8817 raft_consensus.cc:697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:52.765172  8817 consensus_queue.cc:237] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:52.767020 27664 raft_consensus.cc:2468] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:23:52.800275 27583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:52.801192 27583 raft_consensus.cc:606] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:52.801587 27583 raft_consensus.cc:614] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:52.808564 27487 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 31 to 39. New cstate: current_term: 39 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.810654 27487 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 24 to 33. New cstate: current_term: 33 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:52.813102 27487 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 37 to 39. New cstate: current_term: 39 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.824301 27487 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 32 to 40, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.828207 27487 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 33 to 34, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 34 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:23:52.827036 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 240) took 52232 ms (client timeout 15000 ms). Trace:
I20260709 12:23:52.832882 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 35 to 37. New cstate: current_term: 37 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:23:52.830272 27483 rpcz_store.cc:269] 0709 12:23:00.609298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.286661 (+4677363us) service_pool.cc:224] Handling call
0709 12:23:52.826952 (+47540291us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:51.935693 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:51.957496 (+ 21803us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:51.957537 (+    41us) write_op.cc:435] Acquired schema lock
0709 12:23:51.957557 (+    20us) tablet.cc:662] Decoding operations
0709 12:23:51.958553 (+   996us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:51.958632 (+    79us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:51.958661 (+    29us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:23:51.959463 (+   802us) tablet.cc:701] Row locks acquired
0709 12:23:51.959482 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:23:51.959622 (+   140us) write_op.cc:270] Start()
0709 12:23:51.959747 (+   125us) write_op.cc:276] Timestamp: P: 1783599831959570 usec, L: 0
0709 12:23:51.959775 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:23:51.960492 (+   717us) log.cc:844] Serialized 4004 byte log entry
0709 12:23:52.052675 (+ 92183us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b4400022b4c. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e970e3db1 00007f0e97181c85 00007f0e971826a2 00007f0e99ef41db 00007f0e99eefaaf 00007f0e9723e0ac 00007f0ea00989a9 00007f0e9725a69b 00007f0e9725893f 00007f0e9725ac02 00007f0e99ee220b
0709 12:23:52.274505 (+221830us) op_driver.cc:464] REPLICATION: finished
0709 12:23:52.432062 (+157557us) write_op.cc:301] APPLY: starting
0709 12:23:52.432154 (+    92us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:52.497786 (+ 65632us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:52.501392 (+  3606us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:52.710325 (+208933us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:52.710610 (+   285us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 12:23:52.710663 (+    53us) write_op.cc:312] APPLY: finished
0709 12:23:52.710984 (+   321us) log.cc:844] Serialized 97 byte log entry
0709 12:23:52.713358 (+  2374us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:52.713510 (+   152us) write_op.cc:454] Released schema lock
0709 12:23:52.713591 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:52.719554 (+  5963us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:52.719595 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":39,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":157297,"apply.run_cpu_time_us":16458,"apply.run_wall_time_us":287457,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45927,"num_ops":10,"prepare.queue_time_us":319891,"prepare.run_cpu_time_us":19630,"prepare.run_wall_time_us":259408,"raft.queue_time_us":30850,"raft.run_cpu_time_us":21030,"raft.run_wall_time_us":117054,"replication_time_us":314522,"spinlock_wait_cycles":44475648,"thread_start_us":603266,"threads_started":4,"wal-append.queue_time_us":271832}]]}
I20260709 12:23:52.852756 27583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:52.853677 27583 raft_consensus.cc:606] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:52.854030 27583 raft_consensus.cc:614] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:52.869640 27487 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 38 to 40. New cstate: current_term: 40 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.876171 27487 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 31 to 35, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 35 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.913157 27487 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 31 to 35, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 35 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.915478 27487 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 30 to 34, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 34 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.920408 27487 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 34 to 36, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 36 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.924561 27487 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 35 to 39. New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.928450 27487 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 23 to 27. New cstate: current_term: 27 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:52.980942 27487 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 32 to 34, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 34 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:53.005713 28334 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:23:53.008284  8816 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:53.018185 27662 raft_consensus.cc:1275] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:23:53.019510  8816 consensus_queue.cc:1048] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:53.040750 27664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:41556
I20260709 12:23:53.041370 27664 raft_consensus.cc:606] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:53.053498 27589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:53.054185 27589 raft_consensus.cc:606] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:53.054538 27589 raft_consensus.cc:614] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:53.057222 27589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:53.057783 27589 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:53.058122 27589 raft_consensus.cc:614] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:53.059224 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:53.061235 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:53.062492 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:53.081974 27664 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 44 index: 25. (term mismatch)
I20260709 12:23:53.082512 27664 pending_rounds.cc:85] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Aborting all ops after (but not including) 24
I20260709 12:23:53.091265 27664 pending_rounds.cc:107] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Aborting uncommitted NO_OP operation due to leader change: 43.25
I20260709 12:23:53.091634 27664 raft_consensus.cc:2889] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:53.093557  8819 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:53.095227  8736 raft_consensus.cc:993] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:53.095934  8736 raft_consensus.cc:1081] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:53.109596 27586 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:23:53.110396 27586 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:53.110854 27586 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:53.120769 27738 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:23:53.129654  8725 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:23:53.131022 27664 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 47 index: 25. (index mismatch)
I20260709 12:23:53.139094  8739 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:53.172873 27586 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:53.182783 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:53.183351 27664 raft_consensus.cc:3055] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:53.183601 27664 raft_consensus.cc:740] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:53.188409 27664 consensus_queue.cc:260] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 38.27, Last appended by leader: 27, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:53.189610 27664 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:53.190693 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:53.188431 27586 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:53.191494 28334 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:53.210484 27664 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:53.214751 27552 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:23:53.218932  8831 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:53.219570  8831 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:53.220546  8831 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 38.27, Last appended by leader: 27, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:53.259130 28334 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:53.277299 27736 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 47 index: 25. (index mismatch)
I20260709 12:23:53.279786  8725 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:23:53.379215 27589 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:23:53.383951  8816 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:23:53.515477 27664 raft_consensus.cc:1275] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
W20260709 12:23:53.623025 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 94234.7us
I20260709 12:23:53.641021  8858 consensus_queue.cc:1048] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.012s
W20260709 12:23:53.709488 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 46549.8us
W20260709 12:23:53.787902 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 73289.7us
W20260709 12:23:53.796285 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9c09719959a54e8eb66d2041dd9c3113 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:53.785524  8736 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.814467  8736 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:53.830297  8736 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:53.835808 27589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69" is_pre_election: true
I20260709 12:23:53.841281 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:53.842218 27738 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:23:53.850317 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:53.861584  8736 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:23:53.862067  8736 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.862473  8736 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:23:53.870955  8831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.102s	sys 0.117s
W20260709 12:23:53.871395  8831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.103s	sys 0.117s
I20260709 12:23:53.874007  8736 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:53.916180  8736 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:53.918735 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:53.920627 27738 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:23:53.927407 27583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:53.948968 27738 raft_consensus.cc:2468] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 41.
I20260709 12:23:53.950570 27627 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:53.951974  8861 raft_consensus.cc:2804] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:23:53.952572  8861 raft_consensus.cc:697] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:53.953814  8861 consensus_queue.cc:237] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:53.972880  8875 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:53.973678  8875 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:53.976084  8875 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:53.980108 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:53.982930 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:53.983796 27738 raft_consensus.cc:2368] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41.
I20260709 12:23:53.991588 27552 leader_election.cc:400] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41.
I20260709 12:23:53.992313 27552 leader_election.cc:403] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:53.992952  8801 raft_consensus.cc:3060] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Advancing to term 41
I20260709 12:23:54.031611  8801 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 40. Reason: Vote denied by peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211) with higher term. Message: Invalid argument: T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f90188458da46f6ba138d5f82c55b69 for earlier term 40. Current term is 41.
W20260709 12:23:54.037925 27487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 245) took 32733 ms (client timeout 15000 ms). Trace:
W20260709 12:23:54.038407 27487 rpcz_store.cc:269] 0709 12:23:21.323555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.147252 (+12823697us) service_pool.cc:224] Handling call
0709 12:23:34.665389 (+518137us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b2090 00007f0e9a350f3a 00007f0ea0106e9c 00007f0ea020504e 00007f0e9e6c62a4 00007f0e9e6c6232 00007f0e9e6c6161 00007f0e9e6c60dc 00007f0e9e6c5392 00007f0e9868348c 00007f0e986828c6
0709 12:23:54.037812 (+19372423us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:53.262848 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:53.294164 (+ 31316us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:53.294202 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:23:53.294222 (+    20us) tablet.cc:662] Decoding operations
0709 12:23:53.310836 (+ 16614us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:53.310903 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:53.310949 (+    46us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:23:53.312191 (+  1242us) tablet.cc:701] Row locks acquired
0709 12:23:53.312214 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:23:53.312371 (+   157us) write_op.cc:270] Start()
0709 12:23:53.312515 (+   144us) write_op.cc:276] Timestamp: P: 1783599833312317 usec, L: 0
0709 12:23:53.312538 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:23:53.313349 (+   811us) log.cc:844] Serialized 5525 byte log entry
0709 12:23:53.666991 (+353642us) op_driver.cc:464] REPLICATION: finished
0709 12:23:53.817296 (+150305us) write_op.cc:301] APPLY: starting
0709 12:23:53.836168 (+ 18872us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:53.897015 (+ 60847us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:53.897122 (+   107us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:53.967007 (+ 69885us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:53.973373 (+  6366us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:23:53.973431 (+    58us) write_op.cc:312] APPLY: finished
0709 12:23:53.973824 (+   393us) log.cc:844] Serialized 129 byte log entry
0709 12:23:53.974049 (+   225us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:53.974223 (+   174us) write_op.cc:454] Released schema lock
0709 12:23:53.974296 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:53.983443 (+  9147us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:53.983479 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":31793,"reported_tablets":51,"spinlock_wait_cycles":8239488,"child_traces":[["op",{"apply.queue_time_us":141036,"apply.run_cpu_time_us":20359,"apply.run_wall_time_us":157657,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62647,"num_ops":14,"prepare.queue_time_us":205621,"prepare.run_cpu_time_us":5298,"prepare.run_wall_time_us":108846,"replication_time_us":334422,"thread_start_us":210590,"threads_started":2}]]}
I20260709 12:23:54.241489 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:54.253577 27481 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 36 to 38, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 38 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.255349 27481 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 33 to 37. New cstate: current_term: 37 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:54.260867 27481 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 35 to 37. New cstate: current_term: 37 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:54.270735 27481 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 32. New cstate: current_term: 32 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.278688 27481 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 33 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 37 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:54.288280 27481 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 34 to 38. New cstate: current_term: 38 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.289829 27481 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 31 to 35. New cstate: current_term: 35 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.292305 27481 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 26 to 30, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 30 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:54.295176 27481 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 reported cstate change: term changed from 30 to 34, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 34 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:54.383189 27582 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:23:54.391114  8861 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:54.394418 27736 raft_consensus.cc:1275] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
W20260709 12:23:54.394929 27628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48002.2us
I20260709 12:23:54.420815  8736 consensus_queue.cc:1048] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:23:54.559283 27662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c3943cf144748fca8d2892751757c1a"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:41556
I20260709 12:23:54.559993 27662 raft_consensus.cc:606] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:54.560379 27662 raft_consensus.cc:614] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:54.582971 27582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:54.563686 27487 catalog_manager.cc:5697] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 39 to 41. New cstate: current_term: 41 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.583642 27582 raft_consensus.cc:606] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:54.587638 27582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:54.588194 27582 raft_consensus.cc:606] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:54.588549 27582 raft_consensus.cc:614] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:54.591922 27487 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 35 to 36, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 36 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.594830 27582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:54.595455 27582 raft_consensus.cc:606] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:54.595777 27582 raft_consensus.cc:614] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:54.598668 27481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 244) took 58772 ms (client timeout 15000 ms). Trace:
W20260709 12:23:54.599166 27481 rpcz_store.cc:269] 0709 12:22:55.826406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.764899 (+7938493us) service_pool.cc:224] Handling call
0709 12:23:54.598589 (+50833690us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:54.431510 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:54.432032 (+   522us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:54.432072 (+    40us) write_op.cc:435] Acquired schema lock
0709 12:23:54.432090 (+    18us) tablet.cc:662] Decoding operations
0709 12:23:54.433028 (+   938us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:54.433113 (+    85us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:54.433144 (+    31us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:23:54.433975 (+   831us) tablet.cc:701] Row locks acquired
0709 12:23:54.433989 (+    14us) write_op.cc:260] PREPARE: finished
0709 12:23:54.434126 (+   137us) write_op.cc:270] Start()
0709 12:23:54.434237 (+   111us) write_op.cc:276] Timestamp: P: 1783599834434078 usec, L: 0
0709 12:23:54.434254 (+    17us) op_driver.cc:348] REPLICATION: starting
0709 12:23:54.456051 (+ 21797us) log.cc:844] Serialized 3619 byte log entry
0709 12:23:54.468701 (+ 12650us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b54003c9bd4. stack: 00007f0e97229343 00007f0e97229240 00007f0e96bb81d4 00007f0ea08f6304 00007f0ea08f6299 00007f0e9a2b1571 00007f0e9a343e28 00007f0e9dd7e581 00007f0e9dd7dd58 00007f0e9dd829f1 00007f0e9dd829a9 00007f0e9dd82939 00007f0e9dd82901 00007f0e9dd81b2d 0000000000388634 0000000000388479
0709 12:23:54.478119 (+  9418us) op_driver.cc:464] REPLICATION: finished
0709 12:23:54.478655 (+   536us) write_op.cc:301] APPLY: starting
0709 12:23:54.478739 (+    84us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:54.486484 (+  7745us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:54.486508 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:54.546055 (+ 59547us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:54.546199 (+   144us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:23:54.546230 (+    31us) write_op.cc:312] APPLY: finished
0709 12:23:54.546445 (+   215us) log.cc:844] Serialized 89 byte log entry
0709 12:23:54.546642 (+   197us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:54.546775 (+   133us) write_op.cc:454] Released schema lock
0709 12:23:54.546828 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:54.584379 (+ 37551us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:54.586048 (+  1669us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":412662,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":277,"apply.run_cpu_time_us":12095,"apply.run_wall_time_us":68535,"cfile_cache_hit":36,"cfile_cache_hit_bytes":37620,"num_ops":9,"prepare.queue_time_us":127957,"prepare.run_cpu_time_us":6077,"prepare.run_wall_time_us":37384,"replication_time_us":43698,"spinlock_wait_cycles":48549248,"thread_start_us":129070,"threads_started":1}]]}
I20260709 12:23:54.617197 27487 catalog_manager.cc:5697] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 40 to 42. New cstate: current_term: 42 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.617941 27586 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd03dfdd894243c3af75d463b6913332"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:54.627743 27586 raft_consensus.cc:606] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:54.628161 27586 raft_consensus.cc:614] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:54.628459 27487 catalog_manager.cc:5697] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 45 to 47. New cstate: current_term: 47 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.629576 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:54.631834 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:54.632814 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:54.640134 27487 catalog_manager.cc:5697] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 39, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 39 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.642135 27487 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 39, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.648931 27487 catalog_manager.cc:5697] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 33 to 35, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.651847 27487 catalog_manager.cc:5697] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 41 to 43, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 43 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.668232 27487 catalog_manager.cc:5697] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 42, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 42 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.669767 27487 catalog_manager.cc:5697] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 27 to 34, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 34 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.671144 27487 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 45 to 48, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 48 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.676398 27487 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 31 to 36. New cstate: current_term: 36 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.677717 27487 catalog_manager.cc:5697] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 33 to 35, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.688006 27487 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 38 to 41, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 41 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.696142 27487 catalog_manager.cc:5697] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 34 to 40, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.698292 27487 catalog_manager.cc:5697] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 30 to 35. New cstate: current_term: 35 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.700088 27487 catalog_manager.cc:5697] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 36 to 42. New cstate: current_term: 42 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.708067 27487 catalog_manager.cc:5697] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 40 to 42. New cstate: current_term: 42 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.712173 27487 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 36 to 38, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.724057 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 36 to 40. New cstate: current_term: 40 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.725987 27487 catalog_manager.cc:5697] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 32 to 34, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 34 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.729481 27487 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 37 to 39. New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.730988 27487 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 32 to 36, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 36 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:54.788637  8739 raft_consensus.cc:993] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:23:54.789165  8739 raft_consensus.cc:1081] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Signalling peer ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:23:54.802124 27736 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:34464
I20260709 12:23:54.802706 27736 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:54.803107 27736 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:54.852551 27736 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:54.881772 27582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:54.882493 27582 raft_consensus.cc:3055] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:23:54.882829 27582 raft_consensus.cc:740] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:54.883790 27582 consensus_queue.cc:260] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:54.885473 27582 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:54.891747 27736 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:54.893153 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:54.893821 27662 raft_consensus.cc:3060] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:54.897444 27582 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 48.
I20260709 12:23:54.899025 27704 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:54.899955  8826 raft_consensus.cc:2804] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:23:54.900490  8826 raft_consensus.cc:697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:54.901360  8826 consensus_queue.cc:237] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:54.907608 27662 raft_consensus.cc:2468] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 48.
I20260709 12:23:54.944916 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51028
I20260709 12:23:54.945720 27736 raft_consensus.cc:606] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Received request to transfer leadership
I20260709 12:23:54.951932 27736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51028
I20260709 12:23:54.952580 27736 raft_consensus.cc:606] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Received request to transfer leadership
W20260709 12:23:54.954707 27471 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 8355fe4a548a43f8b5215ea9ae7ba8b2 already in progress [suppressed 2 similar messages]
I20260709 12:23:54.971869 27738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51028
I20260709 12:23:54.972483 27738 raft_consensus.cc:606] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Received request to transfer leadership
I20260709 12:23:54.995832 27487 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45922 (request call id 136) took 8357 ms. Trace:
I20260709 12:23:54.999929 27480 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 33 to 39. New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:54.996264 27487 rpcz_store.cc:276] 0709 12:23:46.638727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.042543 (+7403816us) service_pool.cc:224] Handling call
0709 12:23:54.995748 (+953205us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:54.779890 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:54.780422 (+   532us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:54.780459 (+    37us) write_op.cc:435] Acquired schema lock
0709 12:23:54.780476 (+    17us) tablet.cc:662] Decoding operations
0709 12:23:54.784614 (+  4138us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:54.786007 (+  1393us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:54.792618 (+  6611us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:23:54.836015 (+ 43397us) tablet.cc:701] Row locks acquired
0709 12:23:54.836035 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:23:54.836201 (+   166us) write_op.cc:270] Start()
0709 12:23:54.836310 (+   109us) write_op.cc:276] Timestamp: P: 1783599834836155 usec, L: 0
0709 12:23:54.836331 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:23:54.837025 (+   694us) log.cc:844] Serialized 9018 byte log entry
0709 12:23:54.852859 (+ 15834us) op_driver.cc:464] REPLICATION: finished
0709 12:23:54.853238 (+   379us) write_op.cc:301] APPLY: starting
0709 12:23:54.853333 (+    95us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:54.876439 (+ 23106us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:54.876555 (+   116us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:54.930792 (+ 54237us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:54.931157 (+   365us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=0,mrs_lookups=0
0709 12:23:54.931238 (+    81us) write_op.cc:312] APPLY: finished
0709 12:23:54.931903 (+   665us) log.cc:844] Serialized 204 byte log entry
0709 12:23:54.932766 (+   863us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:54.932998 (+   232us) write_op.cc:454] Released schema lock
0709 12:23:54.933074 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:54.960858 (+ 27784us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:54.962647 (+  1789us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":35567,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":130,"apply.run_cpu_time_us":28147,"apply.run_wall_time_us":106363,"cfile_cache_hit":92,"cfile_cache_hit_bytes":96140,"num_ops":23,"prepare.queue_time_us":575,"prepare.run_cpu_time_us":5956,"prepare.run_wall_time_us":67734,"replication_time_us":16307}]]}
I20260709 12:23:55.002585 27480 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 32 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 37 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.011152 27480 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 37 to 42, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 42 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.012930 27480 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 29 to 39. New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.014689 27480 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 34 to 40. New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.016044 27480 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 41. New cstate: current_term: 41 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.019032 27480 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 35 to 38, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.024432 27480 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 33 to 35, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 35 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.026146 27480 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 32 to 36, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 36 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.027980 27480 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.031176 27480 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 36 to 38, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 38 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.033646 27480 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 37 to 40. New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.036587 27480 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 27 to 28, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 28 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.038611 27480 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 38 to 43. New cstate: current_term: 43 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.119232  8826 raft_consensus.cc:993] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:55.119798  8826 raft_consensus.cc:1081] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:55.121812 27586 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39856
I20260709 12:23:55.122488 27586 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:55.122900 27586 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:55.138834 27586 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:55.142513 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:55.142755 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:55.143105 27662 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:55.143297 27738 raft_consensus.cc:3055] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:23:55.143674 27738 raft_consensus.cc:740] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:55.144480 27738 consensus_queue.cc:260] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:55.145912 27738 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:55.184789 27662 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:55.186040 27738 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:55.188242 27552 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:55.189623 27487 catalog_manager.cc:5697] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 38 to 43, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 43 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:55.190914  8886 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:23:55.191502  8886 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:55.192550  8886 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 39, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:55.191191 27487 catalog_manager.cc:5697] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 39 to 41. New cstate: current_term: 41 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:55.195748 27535 auto_rebalancer.cc:893] Removing replace marker failed: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 39 committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:23:55.196219 27535 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 8355fe4a548a43f8b5215ea9ae7ba8b2 already in progress
W20260709 12:23:55.196528 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 8355fe4a548a43f8b5215ea9ae7ba8b2 already in progress
I20260709 12:23:55.195272 27487 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 38 to 39, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:55.224479 27487 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 41 to 42, leader changed from 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 42 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:55.226011 27487 catalog_manager.cc:5697] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 40 to 44. New cstate: current_term: 44 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:55.232106 27586 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 39 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:23:55.233559 27474 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 6b423a3b3ee1452fb419df4c9c0a8d67 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 3f90188458da46f6ba138d5f82c55b69 from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 36 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:55.234283 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: true dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:23:55.234913 28334 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:23:55.249660 28334 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 39.
I20260709 12:23:55.313133 27582 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:23:55.315958  8826 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
W20260709 12:23:55.387183 27480 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 241) took 21555 ms (client timeout 15000 ms). Trace:
W20260709 12:23:55.387743 27480 rpcz_store.cc:269] 0709 12:23:33.831887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.063202 (+14231315us) service_pool.cc:224] Handling call
0709 12:23:55.387094 (+7323892us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:55.044709 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:55.045345 (+   636us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:55.045394 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:23:55.045419 (+    25us) tablet.cc:662] Decoding operations
0709 12:23:55.047102 (+  1683us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:55.047172 (+    70us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:55.047207 (+    35us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:23:55.048384 (+  1177us) tablet.cc:701] Row locks acquired
0709 12:23:55.048401 (+    17us) write_op.cc:260] PREPARE: finished
0709 12:23:55.048553 (+   152us) write_op.cc:270] Start()
0709 12:23:55.048662 (+   109us) write_op.cc:276] Timestamp: P: 1783599835048495 usec, L: 0
0709 12:23:55.048678 (+    16us) op_driver.cc:348] REPLICATION: starting
0709 12:23:55.049396 (+   718us) log.cc:844] Serialized 5540 byte log entry
0709 12:23:55.068640 (+ 19244us) op_driver.cc:464] REPLICATION: finished
0709 12:23:55.073264 (+  4624us) write_op.cc:301] APPLY: starting
0709 12:23:55.073389 (+   125us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:55.090268 (+ 16879us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:55.090389 (+   121us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:55.120332 (+ 29943us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:55.120609 (+   277us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:23:55.120659 (+    50us) write_op.cc:312] APPLY: finished
0709 12:23:55.121042 (+   383us) log.cc:844] Serialized 129 byte log entry
0709 12:23:55.182054 (+ 61012us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:55.182230 (+   176us) write_op.cc:454] Released schema lock
0709 12:23:55.182293 (+    63us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:55.183780 (+  1487us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:55.183809 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5637,"reported_tablets":52,"spinlock_wait_cycles":71936,"child_traces":[["op",{"apply.queue_time_us":4281,"apply.run_cpu_time_us":19251,"apply.run_wall_time_us":110553,"cfile_cache_hit":56,"cfile_cache_hit_bytes":58520,"num_ops":14,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":5044,"prepare.run_wall_time_us":40413,"replication_time_us":19759}]]}
I20260709 12:23:55.426265 27662 raft_consensus.cc:1275] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:23:55.427733  8826 consensus_queue.cc:1048] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:23:55.556294 27481 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 41. New cstate: current_term: 41 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.567909 27481 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 36 to 45. New cstate: current_term: 45 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.569728 27481 catalog_manager.cc:5697] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 34 to 40. New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.572146 27481 catalog_manager.cc:5697] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 40 to 42. New cstate: current_term: 42 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.585553 27662 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:23:55.587670  8876 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:55.593277 28334 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:23:55.598459  8885 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:23:55.573509 27481 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 43. New cstate: current_term: 43 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.610778 27481 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 35 to 44, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 44 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.615178 27738 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:23:55.617179  8885 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:23:55.612740 27481 catalog_manager.cc:5697] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 35, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 35 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.620198 27481 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 40 to 42, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 42 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.622102 27481 catalog_manager.cc:5697] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 43, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 43 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.623519 27481 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 41 to 43, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 43 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.625322 27481 catalog_manager.cc:5697] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 38 to 40. New cstate: current_term: 40 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.648061 27481 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 35 to 38, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 38 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.650909 27481 catalog_manager.cc:5697] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 42, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 42 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.659152 27481 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 32 to 39, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.660854 27481 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 40 to 44. New cstate: current_term: 44 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.662133 27481 catalog_manager.cc:5697] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 43. New cstate: current_term: 43 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.664336 27481 catalog_manager.cc:5697] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 30 to 37. New cstate: current_term: 37 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.665529 27481 catalog_manager.cc:5697] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 37 to 45, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 45 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.666801 27481 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 28 to 29, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 29 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.680496 27481 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 37 to 45, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 45 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:55.681993 27481 catalog_manager.cc:5697] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 43 to 44, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 44 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:55.790957 27487 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 38 to 39, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:55.837987 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:55.926731 27481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 242) took 4755 ms. Trace:
I20260709 12:23:55.942133 27481 rpcz_store.cc:276] 0709 12:23:51.171200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.603075 (+3431875us) service_pool.cc:224] Handling call
0709 12:23:55.926644 (+1323569us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:55.703653 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:55.704252 (+   599us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:55.704298 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:23:55.704317 (+    19us) tablet.cc:662] Decoding operations
0709 12:23:55.706338 (+  2021us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:55.706404 (+    66us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:55.706443 (+    39us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:23:55.713953 (+  7510us) tablet.cc:701] Row locks acquired
0709 12:23:55.713973 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:23:55.714178 (+   205us) write_op.cc:270] Start()
0709 12:23:55.714307 (+   129us) write_op.cc:276] Timestamp: P: 1783599835714119 usec, L: 0
0709 12:23:55.714327 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:23:55.715130 (+   803us) log.cc:844] Serialized 8235 byte log entry
0709 12:23:55.719018 (+  3888us) op_driver.cc:464] REPLICATION: finished
0709 12:23:55.719566 (+   548us) write_op.cc:301] APPLY: starting
0709 12:23:55.719655 (+    89us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:55.730748 (+ 11093us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:55.730864 (+   116us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:55.779872 (+ 49008us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:55.780200 (+   328us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=0,mrs_lookups=0
0709 12:23:55.780250 (+    50us) write_op.cc:312] APPLY: finished
0709 12:23:55.780667 (+   417us) log.cc:844] Serialized 188 byte log entry
0709 12:23:55.788167 (+  7500us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:55.788358 (+   191us) write_op.cc:454] Released schema lock
0709 12:23:55.788420 (+    62us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:55.839775 (+ 51355us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:55.839812 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":232,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":26128,"apply.run_wall_time_us":120196,"cfile_cache_hit":84,"cfile_cache_hit_bytes":87780,"num_ops":21,"prepare.queue_time_us":10302,"prepare.run_cpu_time_us":6090,"prepare.run_wall_time_us":11796,"replication_time_us":4379}]]}
I20260709 12:23:56.036792 27484 catalog_manager.cc:5244] ChangeConfig:REMOVE_PEER RPC for tablet 7274fd8a2115405c9b78c61214bff736 with cas_config_opid_index 5: aborting the task: latest config opid_index 11; task opid_index 5
W20260709 12:23:56.042568 27484 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 240) took 182350 ms (client timeout 15000 ms). Trace:
W20260709 12:23:56.057768 27484 rpcz_store.cc:269] 0709 12:20:53.692271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.692524 (+   253us) service_pool.cc:224] Handling call
0709 12:23:56.042432 (+182349908us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":639191,"reported_tablets":51}
I20260709 12:23:56.098289 27586 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:52220
I20260709 12:23:56.098919 27586 raft_consensus.cc:606] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:56.103217 27662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:41556
I20260709 12:23:56.103927 27662 raft_consensus.cc:606] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Received request to transfer leadership to TS ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:56.109124 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:41556
I20260709 12:23:56.109819 27663 raft_consensus.cc:606] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:56.110178 27663 raft_consensus.cc:614] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:56.113720 27663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01"
dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
mode: GRACEFUL
new_leader_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:41556
I20260709 12:23:56.114428 27663 raft_consensus.cc:606] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Received request to transfer leadership to TS 3f90188458da46f6ba138d5f82c55b69
I20260709 12:23:56.120532 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:23:56.122598 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:56.123588 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:23:56.146689 27485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 241) took 156198 ms (client timeout 15000 ms). Trace:
W20260709 12:23:56.148492 27485 rpcz_store.cc:269] 0709 12:21:19.948592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.993795 (+ 45203us) service_pool.cc:224] Handling call
0709 12:23:56.146466 (+156152671us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17966,"reported_tablets":51}
I20260709 12:23:56.171446 27478 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 38 to 41, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 41 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:56.259876 27485 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 45 to 47. New cstate: current_term: 47 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.261861 27485 catalog_manager.cc:5697] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 34 to 40. New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.369860 27485 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 42 to 44, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 44 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.371644 27485 catalog_manager.cc:5697] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 37 to 39, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 39 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.372991  8801 raft_consensus.cc:993] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: : Instructing follower ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:23:56.373522  8801 raft_consensus.cc:1081] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Signalling peer ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:23:56.373026 27485 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 38 to 42, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 42 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.376513 27485 catalog_manager.cc:5697] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 41 to 48, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 48 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:56.384536 27485 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 44 to 45, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 45 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.385995 27485 catalog_manager.cc:5697] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 42 to 44. New cstate: current_term: 44 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:56.387497 27485 catalog_manager.cc:5697] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 47, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 47 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.403378  8736 raft_consensus.cc:993] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:23:56.407191 27485 catalog_manager.cc:5697] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 44 to 45, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 45 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:56.409421 27478 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 231) took 38582 ms (client timeout 15000 ms). Trace:
I20260709 12:23:56.420621  8709 raft_consensus.cc:1081] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Signalling peer ce69a3f9261a47a7a22f02f9c1f07964 to start an election
I20260709 12:23:56.427853 27485 catalog_manager.cc:5697] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 36 to 41, leader changed from ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 41 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.441444 27485 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 29 to 33, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 33 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:56.443043 27485 catalog_manager.cc:5697] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 40 to 45, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 45 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:23:56.417960 27478 rpcz_store.cc:269] 0709 12:23:17.826948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.575504 (+9748556us) service_pool.cc:224] Handling call
0709 12:23:56.197271 (+28621767us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:56.197309 (+    38us) write_op.cc:454] Released schema lock
0709 12:23:56.409323 (+212014us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:56.187693 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:56.188337 (+   644us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:56.188384 (+    47us) write_op.cc:435] Acquired schema lock
0709 12:23:56.188408 (+    24us) tablet.cc:662] Decoding operations
0709 12:23:56.188961 (+   553us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:56.189027 (+    66us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:56.189076 (+    49us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:23:56.189337 (+   261us) tablet.cc:701] Row locks acquired
0709 12:23:56.189363 (+    26us) write_op.cc:260] PREPARE: finished
0709 12:23:56.189531 (+   168us) write_op.cc:270] Start()
0709 12:23:56.189667 (+   136us) write_op.cc:276] Timestamp: P: 1783599836189469 usec, L: 0
0709 12:23:56.189693 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:23:56.190390 (+   697us) log.cc:844] Serialized 538 byte log entry
0709 12:23:56.192868 (+  2478us) op_driver.cc:464] REPLICATION: finished
0709 12:23:56.193351 (+   483us) write_op.cc:301] APPLY: starting
0709 12:23:56.193448 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:56.194405 (+   957us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:56.194528 (+   123us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:56.195356 (+   828us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:56.195556 (+   200us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:23:56.195614 (+    58us) write_op.cc:312] APPLY: finished
0709 12:23:56.195818 (+   204us) log.cc:844] Serialized 25 byte log entry
0709 12:23:56.195997 (+   179us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:56.196091 (+    94us) write_op.cc:454] Released schema lock
0709 12:23:56.196142 (+    51us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":24189,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":271,"apply.run_cpu_time_us":3108,"apply.run_wall_time_us":3104,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4180,"num_ops":1,"prepare.queue_time_us":3026,"prepare.run_cpu_time_us":3066,"prepare.run_wall_time_us":4957,"replication_time_us":3049}]]}
W20260709 12:23:56.440457 27535 auto_rebalancer.cc:629] Failed to schedule move for tablet 37a0fdc57f6240869f0977e359f258c0: Illegal state: Replica ce69a3f9261a47a7a22f02f9c1f07964 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 42 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
W20260709 12:23:56.471282 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 60664.3us
I20260709 12:23:56.473500 27732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:34476
I20260709 12:23:56.474141 27732 raft_consensus.cc:493] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:56.474507 27732 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:56.492892 27732 raft_consensus.cc:515] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.497920 27738 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1"
dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
 from {username='slave'} at 127.0.0.1:34464
I20260709 12:23:56.498574 27738 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:56.498950 27738 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:56.499773 27732 leader_election.cc:290] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:23:56.517287 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 45072.3us
I20260709 12:23:56.527284 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:56.528009 27662 raft_consensus.cc:3055] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:23:56.528312 27662 raft_consensus.cc:740] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:56.529028 27662 consensus_queue.cc:260] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.530112 27662 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:56.537459 27662 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:56.552155 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39fef9cf211c4b858a135faa1b00e1b1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:56.552872 27586 raft_consensus.cc:3060] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:23:56.562848 27586 raft_consensus.cc:2468] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 30.
I20260709 12:23:56.571856 27738 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.564178 27704 leader_election.cc:304] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:56.574793  8855 raft_consensus.cc:2804] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:23:56.575390  8855 raft_consensus.cc:697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:56.576249  8855 consensus_queue.cc:237] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.593761  8907 raft_consensus.cc:993] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: : Instructing follower 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:56.594365  8907 raft_consensus.cc:1081] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Signalling peer 3f90188458da46f6ba138d5f82c55b69 to start an election
I20260709 12:23:56.600299 27582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01"
dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
 from {username='slave'} at 127.0.0.1:39872
I20260709 12:23:56.600847 27582 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:56.601239 27582 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:56.622675 27582 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.626457 27586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: true dest_uuid: "3f90188458da46f6ba138d5f82c55b69"
I20260709 12:23:56.626718 27738 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:56.627086 27586 raft_consensus.cc:3055] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:23:56.627425 27586 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:56.628144 27586 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.25, Last appended by leader: 25, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.629372 27586 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:56.633114 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:56.633755 27663 raft_consensus.cc:3055] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:23:56.634016 27663 raft_consensus.cc:740] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:23:56.634670 27663 consensus_queue.cc:260] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.636071 27663 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:56.640621 27738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: true dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:23:56.641172 27738 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:23:56.647425 27582 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:56.650593 27662 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:23:56.654357 27586 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 48.
I20260709 12:23:56.655539 27704 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:56.656291  8826 raft_consensus.cc:2804] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:23:56.656847  8826 raft_consensus.cc:697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:23:56.657725  8826 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.25, Last appended by leader: 25, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:56.666395 27656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e76c3cbbc042dcbbbfe8b1ab6948e1" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: true dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:23:56.667140 27656 raft_consensus.cc:3060] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:23:56.675141 27656 raft_consensus.cc:2468] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 48.
I20260709 12:23:56.675498 27738 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 46.
I20260709 12:23:56.685807 27485 catalog_manager.cc:5697] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 29 to 30, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 30 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:56.696126 27552 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:23:56.697283  8906 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:23:56.697878  8906 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 46 LEADER]: Becoming Leader. State: Replica: 3f90188458da46f6ba138d5f82c55b69, State: Running, Role: LEADER
I20260709 12:23:56.698953  8906 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:23:56.716758 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 50804.3us
I20260709 12:23:56.727937 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 7274fd8a2115405c9b78c61214bff736 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:51028
I20260709 12:23:56.741879 27663 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f90188458da46f6ba138d5f82c55b69 in term 46.
I20260709 12:23:56.754583 27586 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
W20260709 12:23:56.757920 27482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38758 (request call id 239) took 77680 ms (client timeout 15000 ms). Trace:
W20260709 12:23:56.758508 27482 rpcz_store.cc:269] 0709 12:22:39.064554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.015499 (+3950945us) service_pool.cc:224] Handling call
0709 12:23:56.737890 (+73722391us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11818,"reported_tablets":52,"spinlock_wait_cycles":1879936}
I20260709 12:23:56.760792  8908 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:23:56.776015 27628 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 073b12eba02f4990a4fbe7080d8b8aff. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:56.796909 27479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 232) took 16557 ms (client timeout 15000 ms). Trace:
W20260709 12:23:56.797804 27479 rpcz_store.cc:269] 0709 12:23:40.240130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.359507 (+11119377us) service_pool.cc:224] Handling call
0709 12:23:56.796835 (+5437328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25950,"reported_tablets":51}
W20260709 12:23:56.797250  8916 log.cc:927] Time spent T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:23:56.812371 27738 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:23:56.828994  8908 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Committing config change with OpId 35.20: config changed from index -1 to 20, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:56.815828 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 7274fd8a2115405c9b78c61214bff736 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
I20260709 12:23:56.859545  8907 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:56.897313 27738 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Committing config change with OpId 35.20: config changed from index -1 to 20, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:56.993474 27479 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 47 to 48, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 48 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:57.042995 27663 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 47 index: 25. Preceding OpId from leader: term: 48 index: 26. (index mismatch)
I20260709 12:23:57.049312  8903 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:23:57.353075 27656 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:23:57.377509  8903 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:23:57.431551 27555 thread.cc:641] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.028s	sys 0.063s
W20260709 12:23:57.432374 27555 thread.cc:608] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.031s	sys 0.064s
I20260709 12:23:57.417191 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:57.489605 27586 raft_consensus.cc:1275] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:23:57.499549 27586 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 47 index: 25. Preceding OpId from leader: term: 48 index: 26. (index mismatch)
I20260709 12:23:57.644229  8914 consensus_queue.cc:1048] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:23:57.646481  8925 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:23:57.692759 27486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 227) took 146625 ms (client timeout 15000 ms). Trace:
W20260709 12:23:57.693490 27486 rpcz_store.cc:269] 0709 12:21:31.067073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.075561 (+  8488us) service_pool.cc:224] Handling call
0709 12:23:57.692506 (+146616945us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":162333,"reported_tablets":51}
I20260709 12:23:57.745364 27732 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:23:57.802143  8910 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:57.803200  8910 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:57.806412  8910 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:57.839699 27586 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Committing config change with OpId 35.20: config changed from index -1 to 20, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:23:57.874910 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 75360us
I20260709 12:23:57.900544  8946 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:23:57.907299 27483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38754 (request call id 246) took 13567 ms (client timeout 15000 ms). Trace:
W20260709 12:23:57.909086 27483 rpcz_store.cc:269] 0709 12:23:44.354782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.910003 (+8555221us) service_pool.cc:224] Handling call
0709 12:23:57.907050 (+4997047us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":438468,"reported_tablets":51,"spinlock_wait_cycles":1920}
I20260709 12:23:57.911448 27656 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:23:57.911206 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:57.932866  8949 ts_tablet_manager.cc:933] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:23:57.910360 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:57.956696 27583 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:57.975265 27588 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:23:57.978364  8906 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:57.970382 27483 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index -1 to 20, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 35 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:58.026208 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:58.027601  8945 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:23:58.033583 27662 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:58.037845  8949 tablet_copy_client.cc:323] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
W20260709 12:23:58.047641 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 71332.6us
I20260709 12:23:58.048740 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 073b12eba02f4990a4fbe7080d8b8aff from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:23:58.051759 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 073b12eba02f4990a4fbe7080d8b8aff from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-073b12eba02f4990a4fbe7080d8b8aff
I20260709 12:23:58.057544  8934 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:23:58.058564 27480 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38756 (request call id 233) took 2467 ms. Trace:
I20260709 12:23:58.059096 27480 rpcz_store.cc:276] 0709 12:23:55.577230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.577414 (+   184us) service_pool.cc:224] Handling call
0709 12:23:58.042257 (+2464843us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":611553,"reported_tablets":51}
I20260709 12:23:58.078578 27674 tablet_copy_source_session.cc:215] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:23:58.099182 28334 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:23:58.101158  8939 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:23:58.138103 27553 consensus_peers.cc:597] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83433b8ed26448aca1d62a3b80ba4198. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.139972  8949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b12eba02f4990a4fbe7080d8b8aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:58.162221 27657 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:23:58.164245  8910 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:23:58.217559  8945 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Committing config change with OpId 39.23: config changed from index 13 to 23, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:58.249322 27732 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Refusing update from remote peer 3f90188458da46f6ba138d5f82c55b69: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:23:58.261350 27663 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Committing config change with OpId 39.23: config changed from index 13 to 23, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:23:58.297856 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08001fdb40 after lost signal to thread 26411
W20260709 12:23:58.299379 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080028fd00 after lost signal to thread 27521
W20260709 12:23:58.300132 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08003f7520 after lost signal to thread 27535
I20260709 12:23:58.287731  8910 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.013s
W20260709 12:23:58.301934 27759 debug-util.cc:398] Leaking SignalData structure 0x7b08003d89c0 after lost signal to thread 27613
W20260709 12:23:58.303455 27759 debug-util.cc:398] Leaking SignalData structure 0x7b080029d900 after lost signal to thread 27688
W20260709 12:23:58.304740 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800352c00 after lost signal to thread 27762
W20260709 12:23:58.306097 27759 debug-util.cc:398] Leaking SignalData structure 0x7b0800398c40 after lost signal to thread 28361
I20260709 12:23:58.323289  8946 raft_consensus.cc:2955] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Committing config change with OpId 44.26: config changed from index -1 to 26, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:58.340171 27474 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 10)
I20260709 12:23:58.349452 28334 raft_consensus.cc:2955] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Committing config change with OpId 39.23: config changed from index 13 to 23, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:58.374295 27657 raft_consensus.cc:2955] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Committing config change with OpId 44.26: config changed from index -1 to 26, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:23:58.438079 27553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 56793.3us
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:461: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
I20260709 12:23:58.464557 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
I20260709 12:23:58.469381 27486 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: config changed from index 13 to 23, VOTER ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3) evicted. New cstate: current_term: 39 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:58.464663 27733 raft_consensus.cc:2955] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Committing config change with OpId 44.26: config changed from index -1 to 26, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:58.491719  8949 tablet_copy_client.cc:806] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:23:58.492446  8949 tablet_copy_client.cc:670] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:23:58.493881 27729 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 48.26, Last appended by leader: 25, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:23:58.501837 27660 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 48 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:23:58.503947  8826 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
W20260709 12:23:58.514040 27702 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31e76c3cbbc042dcbbbfe8b1ab6948e1. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.505470  8949 tablet_copy_client.cc:538] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:23:58.549250 27480 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 45 to 46, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1). New cstate: current_term: 46 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } }
I20260709 12:23:58.552416 27728 tablet_service.cc:1558] Processing DeleteTablet for tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 with delete_type TABLET_DATA_TOMBSTONED (TS ce69a3f9261a47a7a22f02f9c1f07964 not found in new config with opid_index 23) from {username='slave'} at 127.0.0.1:51028
W20260709 12:23:58.572664 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 49596.7us
W20260709 12:23:58.575219 27704 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.568439  8975 tablet_replica.cc:333] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:23:58.602404  8826 raft_consensus.cc:2955] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 LEADER]: Committing config change with OpId 48.27: config changed from index -1 to 27, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:58.604848 27660 raft_consensus.cc:2955] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Committing config change with OpId 48.27: config changed from index -1 to 27, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:23:58.612237 27704 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.623436 27704 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.624218 27704 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.625195 27479 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 26, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 44 leader_uuid: "3f90188458da46f6ba138d5f82c55b69" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
I20260709 12:23:58.637776  8949 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
W20260709 12:23:58.665974 27704 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.669533 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 110254us
I20260709 12:23:58.692178  8975 raft_consensus.cc:2243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:58.693033  8975 raft_consensus.cc:2272] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:58.696031  8974 ts_tablet_manager.cc:933] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
W20260709 12:23:58.697500 27630 consensus_peers.cc:597] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.698199 27630 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.698820 27630 consensus_peers.cc:597] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.699419 27630 consensus_peers.cc:597] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.700057 27630 consensus_peers.cc:597] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.712209  8975 ts_tablet_manager.cc:1916] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:23:58.719882 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.722028 27630 consensus_peers.cc:597] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.731894 27479 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index -1 to 27, NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) added. New cstate: current_term: 48 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } }
W20260709 12:23:58.731931 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.731355 27630 consensus_peers.cc:597] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.737295 27630 consensus_peers.cc:597] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.741022 27630 consensus_peers.cc:597] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.741559 27630 consensus_peers.cc:597] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.758004 27704 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.772559  8974 tablet_copy_client.cc:323] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.1:41657
W20260709 12:23:58.776160 27630 consensus_peers.cc:597] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.782538 27535 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:23:58.783016 27535 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:23:58.783308 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:23:58.794270 27704 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.795691 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.814368 27630 consensus_peers.cc:597] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.821588 27630 consensus_peers.cc:597] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.824605 27630 consensus_peers.cc:597] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.831851 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:58.954119 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 70423.7us
I20260709 12:23:58.983740  8975 ts_tablet_manager.cc:1929] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 39.22
I20260709 12:23:58.984413  8975 log.cc:1199] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P ce69a3f9261a47a7a22f02f9c1f07964: Deleting WAL directory at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/wals/8355fe4a548a43f8b5215ea9ae7ba8b2
W20260709 12:23:58.984655 27704 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:58.986909 27471 catalog_manager.cc:5028] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): tablet 8355fe4a548a43f8b5215ea9ae7ba8b2 (table test-workload [id=7d627b8d55164602aecda877945cd940]) successfully deleted
W20260709 12:23:59.012419 27704 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:59.016732  8949 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:23:59.018157  8949 tablet_bootstrap.cc:492] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:23:59.020460  8988 ts_tablet_manager.cc:933] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.019126  8949 ts_tablet_manager.cc:1403] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.396s	user 0.064s	sys 0.022s
I20260709 12:23:59.034061  8988 tablet_copy_client.cc:323] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.3:41211
I20260709 12:23:59.036402 27748 tablet_copy_service.cc:140] P ce69a3f9261a47a7a22f02f9c1f07964: Received BeginTabletCopySession request for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:47076)
I20260709 12:23:59.037258 27748 tablet_copy_service.cc:161] P ce69a3f9261a47a7a22f02f9c1f07964: Beginning new tablet copy session on tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:47076: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-31e76c3cbbc042dcbbbfe8b1ab6948e1
W20260709 12:23:59.038523 27704 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:59.048406 27748 tablet_copy_source_session.cc:215] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 12:23:59.069857 27630 consensus_peers.cc:597] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:59.030699  8949 raft_consensus.cc:359] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:23:59.074512  8949 raft_consensus.cc:740] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
W20260709 12:23:59.075417 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:59.076454 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:59.079838  8988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:59.075265  8949 consensus_queue.cc:260] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:23:59.086544  8949 ts_tablet_manager.cc:1434] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.060s	user 0.009s	sys 0.000s
W20260709 12:23:59.087612 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 145755us
I20260709 12:23:59.100142 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-073b12eba02f4990a4fbe7080d8b8aff received from {username='slave'} at 127.0.0.1:36354
W20260709 12:23:59.105868 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:59.113359 27630 consensus_peers.cc:597] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:59.140751 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-073b12eba02f4990a4fbe7080d8b8aff on tablet 073b12eba02f4990a4fbe7080d8b8aff with peer 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:23:59.142163 27630 consensus_peers.cc:597] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:59.171787 28333 raft_consensus.cc:1217] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEARNER]: Deduplicated request from leader. Original: 35.19->[35.20-35.20]   Dedup: 35.20->[]
W20260709 12:23:59.236832 27471 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6d40162b2bb4b61adc69c9fa3f03326 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 9c09719959a54e8eb66d2041dd9c3113 from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 33 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:23:59.249672 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:23:59.254531 26408 tablet_replica.cc:333] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.255721 26408 raft_consensus.cc:2243] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.269189 26408 raft_consensus.cc:2272] T 39fef9cf211c4b858a135faa1b00e1b1 P 3f90188458da46f6ba138d5f82c55b69 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.272392 26408 tablet_replica.cc:333] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.278528 26408 raft_consensus.cc:2243] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.295882 26408 raft_consensus.cc:2272] T eb215cc72c384ec8b9421c81d953589b P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.313961  8988 tablet_copy_client.cc:806] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:23:59.314594  8988 tablet_copy_client.cc:670] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
I20260709 12:23:59.322402  8988 tablet_copy_client.cc:538] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:23:59.333515 26408 tablet_replica.cc:333] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.334499 26408 raft_consensus.cc:2243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.340333  8946 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.340968  8945 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.340850  8946 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.341512  8945 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:23:59.342934  8946 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.344154  8945 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.371486 26408 raft_consensus.cc:2272] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.373157 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.374989 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b187795e304ffd9c1293d922ff5b38" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.381376 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.379428 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.404104 26408 tablet_replica.cc:333] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.430735  8988 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
I20260709 12:23:59.436564 26408 raft_consensus.cc:2243] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.438787 27552 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.463737 26408 raft_consensus.cc:2272] T 5865d310fddd49789e9de0a6a7bb260a P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.464661 27552 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.480536  8970 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:59.467401  8936 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:23:59.467903  8946 raft_consensus.cc:493] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:59.492300  8946 raft_consensus.cc:515] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.494510  8946 leader_election.cc:290] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.494926 26408 tablet_replica.cc:333] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.497242 26408 raft_consensus.cc:2243] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:23:59.498639 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.499818 26408 raft_consensus.cc:2272] T 7efb19c4fe844c2ea043bd3638faae4e P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.502868 26408 tablet_replica.cc:333] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.508333 26408 raft_consensus.cc:2243] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.517367 26408 raft_consensus.cc:2272] T fe68c465ca4346b9b9db89a5be4e639c P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.515616 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0b3d61cfc0d48f98b3cc7cc7cb63370" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.549630  8936 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.550259  8936 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.552608  8936 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.554443  8946 raft_consensus.cc:493] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:59.554991  8946 raft_consensus.cc:515] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.557134  8946 leader_election.cc:290] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.560462 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.562624 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.560676 26408 tablet_replica.cc:333] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.564193 26408 raft_consensus.cc:2243] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.564738 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c596ddd50b894549b567a5b66ac1a31e" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.564953 26408 raft_consensus.cc:2272] T fd03dfdd894243c3af75d463b6913332 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.568430 26408 tablet_replica.cc:333] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.569404 26408 raft_consensus.cc:2243] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:23:59.570503 26408 raft_consensus.cc:2272] T 5357eb349b4b4934a5b0d36f90884fc6 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.571587  8906 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.572196  8906 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.574898  8906 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.575529 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.576689 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c52fdc88d8646dd8e6584f8dbd137a3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.587765 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3d15932c0a64b668179d6868ab4707c" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.611516  8970 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:59.612062  8970 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.612859  8981 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.613415  8981 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.614276  8936 raft_consensus.cc:493] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:59.614806  8936 raft_consensus.cc:515] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.616813  8936 leader_election.cc:290] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.617480  8970 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.622987  8970 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.623615  8970 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.625687  8970 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.629153 27552 leader_election.cc:304] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.630604 27552 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.631278  8996 raft_consensus.cc:2749] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:23:59.631979  8996 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:23:59.648865  8981 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.650367 27552 leader_election.cc:304] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.651608 27552 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.669914 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.670946 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.671813 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.671972 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.657843  8906 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.674719 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1ffc143bb914f039c5b8bb615692aa3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:59.660789  8946 raft_consensus.cc:2749] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:23:59.686491 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f34c916e524bfa9fcae2044da089fc" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.680825  8945 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:23:59.690380 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7274fd8a2115405c9b78c61214bff736" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:59.677253  8996 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.707015 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ac8041c87e49308784e1cad6aeec03" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.682410  8906 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.713424  8906 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.726104  8996 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.728606  8996 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.729187  8996 raft_consensus.cc:493] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:59.729822  8996 raft_consensus.cc:515] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.736758  8936 raft_consensus.cc:493] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:59.737447  8936 raft_consensus.cc:515] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.686820  8934 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.747453  8934 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.750165  8934 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.800817  8906 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.801620  8906 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.831624  8996 leader_election.cc:290] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.831590  8906 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.832504  8936 leader_election.cc:290] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.836489 26408 tablet_replica.cc:333] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.837837 26408 raft_consensus.cc:2243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:23:59.848825  8928 raft_consensus.cc:1064] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:23:59.853637  8928 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 18, Current term: 35, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.859252 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.859767 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d80be98cdd043ffa8d122ba0ac29447" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.864835  8999 raft_consensus.cc:493] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:59.865649  8999 raft_consensus.cc:515] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.868368  8999 leader_election.cc:290] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.885412  9000 raft_consensus.cc:493] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:59.888741 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:59.886166  9000 raft_consensus.cc:515] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.891085 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:23:59.892086  9000 leader_election.cc:290] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.893826  8999 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.894438  8999 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.896488  8999 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.897466  8936 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.897984  8936 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.900172  8936 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.911125 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.923491 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:23:59.839262 26408 raft_consensus.cc:2272] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:59.939127  8936 raft_consensus.cc:493] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4aa1523b9af84c3db99ba394e2b9aa9a)
I20260709 12:23:59.939821  8936 raft_consensus.cc:515] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.942049  8936 leader_election.cc:290] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.944388  9000 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.944973  9000 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:23:59.947474 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70b48af2ec82404986e5e18d544db5b4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.949916 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f21db64000c434ca36b2b9021c13a44" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:23:59.951815 26408 tablet_replica.cc:333] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:23:59.952768 26408 raft_consensus.cc:2243] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:59.953519 26408 raft_consensus.cc:2272] T 4811ee81866e469abca31f212c08ace2 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus is shut down!
W20260709 12:23:59.955154 28295 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 91 similar messages]
I20260709 12:23:59.958398  9000 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:23:59.969563  9000 raft_consensus.cc:493] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:23:59.969740 27553 leader_election.cc:304] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:23:59.970178  9000 raft_consensus.cc:515] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.971650  8983 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:23:59.972265  8983 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:23:59.974553  8983 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.975134  8983 raft_consensus.cc:2749] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:23:59.980563  9000 leader_election.cc:290] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:23:59.981225 27552 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.982520 27552 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.983273  8906 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:23:59.985116 27552 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:23:59.984169  8906 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:24:00.004669 26408 tablet_replica.cc:333] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.008580 28334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:00.018648 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:00.019143 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:00.020537 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:00.027148 28335 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:24:00.028038  8906 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:00.030646 26408 raft_consensus.cc:2243] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus shutting down.
W20260709 12:24:00.029306 27552 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 44315.7us
I20260709 12:24:00.032492 27737 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:24:00.032199 27552 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.045215  8934 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:00.054605 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.054669 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.054565 26408 raft_consensus.cc:2272] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.055903 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.057297 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdb219c7e3524fc9b4a91a96610cee8b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.057229 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e6b2dabf8f4e6cb549ca250430e568" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.060914 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74155a9e0ffe48e5ac2f5dd6d0e8aac8" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.061476 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e634fbff5fb24e2fa133b95599b0db1f" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.064304 27736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b7862879764c33b1db83d8596e3e87" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.066442 26408 tablet_replica.cc:333] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.067376 26408 raft_consensus.cc:2243] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.068050 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdcf17dc981450f989b93d262e9a5d0" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.072438 27658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f392cf87198499d9c064d1ec232d3d7" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.073007 27660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b881b4e55d543f9ade316642ea1b7e3" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.074025 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68dbab4c1e1442b699465f7b3e728acd" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.043439  8928 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.044s
I20260709 12:24:00.090262  8986 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:24:00.097975 27630 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:00.100380 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.100906 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4e3fe6577354786a64bed5843d9a6a4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.102458 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.104141 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70329e70bb724d43a3185508e1186f4b" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.111348 26408 raft_consensus.cc:2272] T 83433b8ed26448aca1d62a3b80ba4198 P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.114447 26408 tablet_replica.cc:333] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.124298 26408 raft_consensus.cc:2243] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.145634  8906 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:00.150323  8906 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:00.153455  8906 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:00.154399 28333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:00.156162  8936 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:00.156620  8906 raft_consensus.cc:493] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:00.156728  8936 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:00.157207  8906 raft_consensus.cc:515] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:00.158912  8936 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:00.160702 27552 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.161525  8983 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:24:00.162024  8906 leader_election.cc:290] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:00.162696 26408 raft_consensus.cc:2272] T 27b4972f682948f4981cb0dd1a74b4f5 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.164798 27552 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.166142  8936 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:00.175992 26408 tablet_replica.cc:333] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.177029 26408 raft_consensus.cc:2243] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.190997 26408 raft_consensus.cc:2272] T b8ac8041c87e49308784e1cad6aeec03 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.203722 27552 leader_election.cc:304] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:00.207404 27552 leader_election.cc:304] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:00.209627  8928 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Committing config change with OpId 35.21: config changed from index 20 to 21, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.211882 26408 tablet_replica.cc:333] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.212821 26408 raft_consensus.cc:2243] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.213714 26408 raft_consensus.cc:2272] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.222723 28335 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Committing config change with OpId 35.21: config changed from index 20 to 21, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.224720  8936 raft_consensus.cc:2749] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:24:00.225736 27552 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:00.226382 27552 raft_consensus.cc:2686] T 81e6b2dabf8f4e6cb549ca250430e568 P 3f90188458da46f6ba138d5f82c55b69: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:24:00.227185 27552 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:00.232789  8936 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:24:00.234126 27552 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:00.208966  8906 raft_consensus.cc:2749] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:00.232245 26408 tablet_replica.cc:333] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.236843  8936 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:00.240720 26408 raft_consensus.cc:2243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.246480 26408 raft_consensus.cc:2272] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.250888 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:24:00.252034 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.251641 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.252710 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.266610 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76581eba9454a229c556095ac8b3e23" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.267198 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.269363 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07754f58c3a421ba559f56d67d87bb4" candidate_uuid: "3f90188458da46f6ba138d5f82c55b69" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:00.278749 27732 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Committing config change with OpId 35.21: config changed from index 20 to 21, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.282128 26408 tablet_replica.cc:333] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.283154 26408 raft_consensus.cc:2243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.284178 26408 raft_consensus.cc:2272] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.304395 27552 leader_election.cc:304] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.305788 26408 tablet_replica.cc:333] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.307157  8906 raft_consensus.cc:2749] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:24:00.309585 27552 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.321285  8936 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:24:00.323257 27479 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 20 to 21, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 35 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:00.325266 27552 leader_election.cc:304] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.326396  8906 raft_consensus.cc:2749] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:24:00.327828 27552 leader_election.cc:304] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:24:00.328576 27552 raft_consensus.cc:2686] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 3f90188458da46f6ba138d5f82c55b69: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:24:00.306805 26408 raft_consensus.cc:2243] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.332060 27552 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.333847  8906 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:24:00.333134 26408 raft_consensus.cc:2272] T 6f3a9a00f0db424982bf0ea90e7ae286 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.341225 27552 leader_election.cc:304] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:24:00.357695 27552 raft_consensus.cc:2686] T 70329e70bb724d43a3185508e1186f4b P 3f90188458da46f6ba138d5f82c55b69: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:24:00.360673 26408 tablet_replica.cc:333] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.377645 27552 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.379940  8906 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:00.379101 26408 raft_consensus.cc:2243] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.384747 26408 raft_consensus.cc:2272] T d3d15932c0a64b668179d6868ab4707c P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.392469  8988 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=2 applied=27 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:00.393539  8988 tablet_bootstrap.cc:492] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:24:00.401072 26408 tablet_replica.cc:333] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.404637 26408 raft_consensus.cc:2243] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.407287 27552 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.410992  8906 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:00.415437 27552 leader_election.cc:304] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f90188458da46f6ba138d5f82c55b69; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:00.419097  8906 raft_consensus.cc:2749] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:00.410141 26408 raft_consensus.cc:2272] T c596ddd50b894549b567a5b66ac1a31e P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.438050  8988 ts_tablet_manager.cc:1403] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 1.043s	user 0.084s	sys 0.012s
I20260709 12:24:00.442173  8988 raft_consensus.cc:359] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:00.457988 27662 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:00.461570 26408 tablet_replica.cc:333] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.453948  8988 raft_consensus.cc:740] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
I20260709 12:24:00.466626  8988 consensus_queue.cc:260] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 48.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:00.474319 26408 raft_consensus.cc:2243] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.476840 27732 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:24:00.479446 26408 raft_consensus.cc:2272] T dbdcf17dc981450f989b93d262e9a5d0 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.481346 28335 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:24:00.489531  8928 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:24:00.491041 26408 tablet_replica.cc:333] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.491523  8986 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:24:00.485208  8988 ts_tablet_manager.cc:1434] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.047s	user 0.000s	sys 0.007s
I20260709 12:24:00.498919  8986 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Committing config change with OpId 35.22: config changed from index 21 to 22, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.502388 27748 tablet_copy_service.cc:342] P ce69a3f9261a47a7a22f02f9c1f07964: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-31e76c3cbbc042dcbbbfe8b1ab6948e1 received from {username='slave'} at 127.0.0.1:47076
I20260709 12:24:00.502943 27748 tablet_copy_service.cc:434] P ce69a3f9261a47a7a22f02f9c1f07964: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-31e76c3cbbc042dcbbbfe8b1ab6948e1 on tablet 31e76c3cbbc042dcbbbfe8b1ab6948e1 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:24:00.503337 28335 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Committing config change with OpId 35.22: config changed from index 21 to 22, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.491927 26408 raft_consensus.cc:2243] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.508435 26408 raft_consensus.cc:2272] T dad2444fb3474aee8e7dbbd7e673ab01 P 3f90188458da46f6ba138d5f82c55b69 [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.508522 27737 raft_consensus.cc:2955] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Committing config change with OpId 35.22: config changed from index 21 to 22, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.524180 26408 tablet_replica.cc:333] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.526789 26408 raft_consensus.cc:2243] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.527429 26408 raft_consensus.cc:2272] T bdb219c7e3524fc9b4a91a96610cee8b P 3f90188458da46f6ba138d5f82c55b69 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.531677 26408 tablet_replica.cc:333] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.532557 26408 raft_consensus.cc:2243] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.532078 27483 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: config changed from index 21 to 22, VOTER 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) evicted. New cstate: current_term: 35 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:00.534024 26408 raft_consensus.cc:2272] T a6d40162b2bb4b61adc69c9fa3f03326 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.537753 26408 tablet_replica.cc:333] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.538712 26408 raft_consensus.cc:2243] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.539896 26408 raft_consensus.cc:2272] T 4ab0ff10d53741dea50e33f971484854 P 3f90188458da46f6ba138d5f82c55b69 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.542987 26408 tablet_replica.cc:333] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.543926 26408 raft_consensus.cc:2243] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.544718 26408 raft_consensus.cc:2272] T 073b12eba02f4990a4fbe7080d8b8aff P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.547349 26408 tablet_replica.cc:333] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.549582 26408 raft_consensus.cc:2243] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.550169 26408 raft_consensus.cc:2272] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 3f90188458da46f6ba138d5f82c55b69 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.576260 26408 tablet_replica.cc:333] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.577138 26408 raft_consensus.cc:2243] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.577809 26408 raft_consensus.cc:2272] T 7274fd8a2115405c9b78c61214bff736 P 3f90188458da46f6ba138d5f82c55b69 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.599009 26408 tablet_replica.cc:333] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.599926 26408 raft_consensus.cc:2243] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.612352 26408 raft_consensus.cc:2272] T c3f34c916e524bfa9fcae2044da089fc P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.615119 26408 tablet_replica.cc:333] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
W20260709 12:24:00.615418 27474 catalog_manager.cc:4755] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 073b12eba02f4990a4fbe7080d8b8aff: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 12:24:00.615943 26408 raft_consensus.cc:2243] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.617019 26408 raft_consensus.cc:2272] T 42e7d195f3f545b98a8dedb26c479c9c P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.619812 26408 tablet_replica.cc:333] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.620616 26408 raft_consensus.cc:2243] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.621613 26408 raft_consensus.cc:2272] T 7c204963842941859d4313114fd92c93 P 3f90188458da46f6ba138d5f82c55b69 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.624291 26408 tablet_replica.cc:333] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.624652 27471 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 073b12eba02f4990a4fbe7080d8b8aff with cas_config_opid_index 21: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:00.625186 26408 raft_consensus.cc:2243] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.625823 26408 raft_consensus.cc:2272] T d2b187795e304ffd9c1293d922ff5b38 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.628713 26408 tablet_replica.cc:333] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.629582 26408 raft_consensus.cc:2243] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.630340 26408 raft_consensus.cc:2272] T a07754f58c3a421ba559f56d67d87bb4 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.633224 26408 tablet_replica.cc:333] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.634001 26408 raft_consensus.cc:2243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.634518 26408 raft_consensus.cc:2272] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 3f90188458da46f6ba138d5f82c55b69 [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.636992 26408 tablet_replica.cc:333] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.637673 26408 raft_consensus.cc:2243] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.638201 26408 raft_consensus.cc:2272] T b4e3fe6577354786a64bed5843d9a6a4 P 3f90188458da46f6ba138d5f82c55b69 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.641754 26408 tablet_replica.cc:333] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.642493 26408 raft_consensus.cc:2243] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.643151 26408 raft_consensus.cc:2272] T e634fbff5fb24e2fa133b95599b0db1f P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.645848 26408 tablet_replica.cc:333] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.646610 26408 raft_consensus.cc:2243] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.647181 26408 raft_consensus.cc:2272] T 0d80be98cdd043ffa8d122ba0ac29447 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.648679  8928 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:00.650000 26408 tablet_replica.cc:333] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.649340  8928 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:00.650835 26408 raft_consensus.cc:2243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.651970 26408 raft_consensus.cc:2272] T 310b1b37298244c0bd5bbe2b0c7328d7 P 3f90188458da46f6ba138d5f82c55b69 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.654711 26408 tablet_replica.cc:333] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.655565 26408 raft_consensus.cc:2243] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.656608 26408 raft_consensus.cc:2272] T ecd16aab46634913b20f3501e9a6389f P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.659274 26408 tablet_replica.cc:333] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.660048 26408 raft_consensus.cc:2243] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.661135 26408 raft_consensus.cc:2272] T 6491d84db64f4d48b3b3d5be8aec385b P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.664079 26408 tablet_replica.cc:333] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
W20260709 12:24:00.664388 27630 leader_election.cc:336] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:00.664829 26408 raft_consensus.cc:2243] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.665421 26408 raft_consensus.cc:2272] T 5f21db64000c434ca36b2b9021c13a44 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.667960 26408 tablet_replica.cc:333] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.668699 26408 raft_consensus.cc:2243] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.669329 26408 raft_consensus.cc:2272] T 68dbab4c1e1442b699465f7b3e728acd P 3f90188458da46f6ba138d5f82c55b69 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.683046 26408 tablet_replica.cc:333] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.683351 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:00.683892 26408 raft_consensus.cc:2243] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.684572 26408 raft_consensus.cc:2272] T e1ffc143bb914f039c5b8bb615692aa3 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.685128 27628 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:24:00.686362  8928 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:00.686414  8985 raft_consensus.cc:2749] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:24:00.687379 26408 tablet_replica.cc:333] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.688216 26408 raft_consensus.cc:2243] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.689353 26408 raft_consensus.cc:2272] T 22efa3bef6634e7a811baa3c42d9cd18 P 3f90188458da46f6ba138d5f82c55b69 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.709007 26408 tablet_replica.cc:333] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.709941 26408 raft_consensus.cc:2243] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.710602 26408 raft_consensus.cc:2272] T 85b7862879764c33b1db83d8596e3e87 P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.725728 26408 tablet_replica.cc:333] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.739648 26408 raft_consensus.cc:2243] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.740643 26408 raft_consensus.cc:2272] T e76581eba9454a229c556095ac8b3e23 P 3f90188458da46f6ba138d5f82c55b69 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.744692 26408 tablet_replica.cc:333] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.745461 26408 raft_consensus.cc:2243] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.746279 26408 raft_consensus.cc:2272] T 3b881b4e55d543f9ade316642ea1b7e3 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.749362 26408 tablet_replica.cc:333] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.750249 26408 raft_consensus.cc:2243] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.750999 26408 raft_consensus.cc:2272] T 4f392cf87198499d9c064d1ec232d3d7 P 3f90188458da46f6ba138d5f82c55b69 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.756014 26408 tablet_replica.cc:333] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.763123 26408 raft_consensus.cc:2243] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:00.763940 26408 raft_consensus.cc:2272] T 70b48af2ec82404986e5e18d544db5b4 P 3f90188458da46f6ba138d5f82c55b69 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.768916 26408 tablet_replica.cc:333] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
I20260709 12:24:00.772555  8932 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:00.776356 26408 raft_consensus.cc:2243] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.776044  8932 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:00.777685 26408 raft_consensus.cc:2272] T 37a0fdc57f6240869f0977e359f258c0 P 3f90188458da46f6ba138d5f82c55b69 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.778709  8932 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:00.780771 26408 tablet_replica.cc:333] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69: stopping tablet replica
W20260709 12:24:00.781749 27704 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:00.785462 26408 raft_consensus.cc:2243] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:24:00.782655 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.786885 26408 raft_consensus.cc:2272] T 5accfdbc68cd4cf399c98e2a533f500e P 3f90188458da46f6ba138d5f82c55b69 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:00.787755 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:00.792573  8984 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:24:00.803625 27630 consensus_peers.cc:597] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:00.819586  8932 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:00.820268  8932 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:00.822566  8932 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:00.824747 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:24:00.828910 27704 leader_election.cc:336] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:00.829370 27704 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:24:00.830471  8932 raft_consensus.cc:2749] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:24:00.832152 27555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.837170 27630 consensus_peers.cc:597] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:00.856536 27630 consensus_peers.cc:597] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:00.918355 27704 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:00.928963  8990 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:00.929734  8990 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:00.935686 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:00.936347 27662 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:24:00.939055 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:24:00.939940  8943 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:24:00.940384  8943 raft_consensus.cc:493] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:00.940766  8943 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:24:00.945641 28295 leader_election.cc:336] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:00.948125  8990 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:00.968379 27630 consensus_peers.cc:597] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:00.971423 27630 consensus_peers.cc:597] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:00.974237  8943 raft_consensus.cc:515] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:00.976552  8943 leader_election.cc:290] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:00.982787 28295 leader_election.cc:336] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:00.990410 28335 raft_consensus.cc:1217] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 LEARNER]: Deduplicated request from leader. Original: 48.26->[48.27-48.27]   Dedup: 48.27->[]
W20260709 12:24:01.010552 27704 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.026588 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7efb19c4fe844c2ea043bd3638faae4e" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 28 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:01.029723 27657 raft_consensus.cc:3060] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:24:01.074891 27630 consensus_peers.cc:597] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.093959 27657 raft_consensus.cc:2468] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:24:01.101601 28292 leader_election.cc:304] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.102573  9010 raft_consensus.cc:2804] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:24:01.103195  9010 raft_consensus.cc:697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:24:01.104317  9010 consensus_queue.cc:237] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 39.28, Last appended by leader: 28, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:01.121757 27630 consensus_peers.cc:597] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.125316 27704 connection.cc:573] client connection to 127.25.202.1:41657 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260709 12:24:01.128494  8990 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.129786  8990 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.132783  8990 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.137373 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.137308 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
W20260709 12:24:01.142496 27630 consensus_peers.cc:597] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.147746 28295 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:01.149565 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:01.152323  8943 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:24:01.157264 27483 catalog_manager.cc:5697] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 39 to 40, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 40 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:01.151510  9010 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.168947  9010 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:01.177858 27704 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.184989 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.197305 28295 leader_election.cc:336] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.198453  9010 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.199618 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.200341 27732 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:24:01.201493 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.202673  9010 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:24:01.203120  9010 raft_consensus.cc:493] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.203574  9010 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:24:01.216321  8986 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.216934  8986 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.219079  8986 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.220582 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:24:01.230126 27630 consensus_peers.cc:597] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.230810 27630 consensus_peers.cc:597] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.241295  9010 raft_consensus.cc:515] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:01.249289 27704 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.245947 27630 leader_election.cc:336] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.253341 27630 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:01.254261  8986 raft_consensus.cc:2749] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:01.257733 27630 consensus_peers.cc:597] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.264583 28295 leader_election.cc:336] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.269127 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5357eb349b4b4934a5b0d36f90884fc6" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:01.269780 27737 raft_consensus.cc:3060] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:24:01.275823 27704 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.290337 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.294857 27737 raft_consensus.cc:2468] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:24:01.298548 28292 leader_election.cc:304] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.299853  8990 raft_consensus.cc:2804] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:24:01.300485  8990 raft_consensus.cc:697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:24:01.301571  8990 consensus_queue.cc:237] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.318342  9010 leader_election.cc:290] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:01.326663 27704 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.335810 27630 consensus_peers.cc:597] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.343786 27483 catalog_manager.cc:5697] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 39 to 40, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 40 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:01.346912  8974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58264 (request call id 2107) took 2835 ms. Trace:
I20260709 12:24:01.347414  8974 rpcz_store.cc:276] 0709 12:23:58.511675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.517634 (+  5959us) service_pool.cc:224] Handling call
0709 12:23:58.696372 (+178738us) ts_tablet_manager.cc:934] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657)
0709 12:24:01.346783 (+2650411us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":37045,"tablet-copy.queue_time_us":177803,"thread_start_us":190183,"threads_started":2}
W20260709 12:24:01.349785 28293 connection.cc:466] server connection from 127.0.0.1:58264 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58264 (request call id 2107) could send its response
W20260709 12:24:01.366348 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.367759  8928 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.368383  8928 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.372226 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:01.372822 28335 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 36.
I20260709 12:24:01.374125  8928 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:01.377225 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:24:01.387639 27630 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.388116 27630 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:01.389768  8928 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:24:01.400231 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.400960 27704 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.417745 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.418447 27630 consensus_peers.cc:597] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.421806  9011 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.422724  9011 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:01.431342 27704 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.434051 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:01.434818 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:24:01.438218 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:01.438885 28335 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:24:01.440421 27702 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.441308  8932 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:24:01.441699  8932 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.442113  8932 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:01.465061  9011 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:24:01.466161 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.478544  8932 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:01.489061 27704 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:01.499457 27630 consensus_peers.cc:597] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.500001 27630 consensus_peers.cc:597] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.502038 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:01.502701 27657 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:01.513728 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b423a3b3ee1452fb419df4c9c0a8d67" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:24:01.514458 28335 raft_consensus.cc:3060] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:01.517068  9010 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.517679  9010 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.521088 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:01.521204  9010 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:01.526765 28335 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
W20260709 12:24:01.528934 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:01.534700 28295 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.535259 28295 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:01.536027  9010 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:24:01.537087 27630 consensus_peers.cc:597] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.513592  8932 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:01.531303 27657 raft_consensus.cc:2468] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:24:01.539290 27701 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.540683  8932 raft_consensus.cc:2804] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:24:01.541213  8932 raft_consensus.cc:697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:01.542285  8932 consensus_queue.cc:237] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 37, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.552215 27657 raft_consensus.cc:1275] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 28. Preceding OpId from leader: term: 40 index: 29. (index mismatch)
I20260709 12:24:01.553817  9010 consensus_queue.cc:1048] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:24:01.558583  8984 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.558846  9011 raft_consensus.cc:1064] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:24:01.559139  8984 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.561475  8984 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:01.566171 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:01.564260  9011 consensus_queue.cc:237] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 48.27, Last appended by leader: 25, Current term: 48, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.603641  8986 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.604593  9024 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.605305  9024 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.604331  8986 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:01.607992  8984 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.608559  8984 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.611397 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.612035 27732 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:24:01.611346  8984 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:01.609278  8986 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.612818  9024 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:01.617730 27704 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.618448 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:01.620714 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:01.621992 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:01.622536 27662 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
I20260709 12:24:01.622742  9024 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:24:01.646114 28295 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:01.647089 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:01.648439  8928 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:24:01.648595 27630 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.670572 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:01.673310  8984 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:24:01.674445  8984 raft_consensus.cc:493] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.674830  8984 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:24:01.668720  8928 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:24:01.682423 27704 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.684233 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:01.684446  8928 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:24:01.695714  9030 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:01.694852  9028 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.697942  9028 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:01.693854 27704 leader_election.cc:336] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.700171  9028 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:01.706588 27704 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:01.714670 28335 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 LEARNER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 48 index: 28. (index mismatch)
W20260709 12:24:01.724826 27704 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:01.735791 27662 raft_consensus.cc:1275] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 48 index: 28. (index mismatch)
I20260709 12:24:01.736927 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.737669 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 43.
I20260709 12:24:01.739065 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:01.732966 27483 catalog_manager.cc:5697] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 36 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 37 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:01.754212  8985 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:24:01.754693  8985 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.755080  8985 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:24:01.755549 27630 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.749239  8984 raft_consensus.cc:515] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.726034  9026 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.034s
I20260709 12:24:01.779723  9028 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.780375  9028 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.782425  9028 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.762127  8928 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:01.785356  8928 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.787295  8984 leader_election.cc:290] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:01.790136 27704 leader_election.cc:336] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.791975 27657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b4972f682948f4981cb0dd1a74b4f5" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:01.796293 27657 raft_consensus.cc:3060] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:24:01.787882 27630 leader_election.cc:336] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.794169 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.794865 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:01.797431 27733 raft_consensus.cc:3060] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:24:01.775436  9011 consensus_queue.cc:1048] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.020s
I20260709 12:24:01.804286 27627 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:01.815574  9010 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.820941 27657 raft_consensus.cc:2468] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 46.
I20260709 12:24:01.825284 27701 leader_election.cc:304] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.826538  8932 raft_consensus.cc:2804] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:24:01.812847  8986 raft_consensus.cc:2749] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:24:01.826990  8985 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.829878  8985 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.827028  8932 raft_consensus.cc:697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:01.831511  8932 consensus_queue.cc:237] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.817298  9010 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.841964  9010 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.843487 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f3a9a00f0db424982bf0ea90e7ae286" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:01.844091 27732 raft_consensus.cc:3060] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:24:01.845257 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.845432 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 45.
I20260709 12:24:01.847579 27627 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:01.859242  9036 raft_consensus.cc:2804] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:24:01.859848  9036 raft_consensus.cc:697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
W20260709 12:24:01.860577 27630 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:01.860857  9036 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:01.873298 27732 raft_consensus.cc:2468] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 44.
I20260709 12:24:01.875092 27627 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:01.881456  9037 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.896536  9037 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.897188  9034 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.897825  9034 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:01.899999  9034 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:24:01.907783 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 53175.4us
I20260709 12:24:01.887590  9028 raft_consensus.cc:2804] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:24:01.907208  9037 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.923410 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:01.924263 27732 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
I20260709 12:24:01.906821  9031 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.910182  9026 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.914119  9011 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.913233  8986 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.927874  9024 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.926621  9028 raft_consensus.cc:697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:24:01.977511  9028 consensus_queue.cc:237] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:01.958350  9036 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.959116  8932 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.927105 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:01.970279  8984 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.977998  9030 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.967634 27483 catalog_manager.cc:5697] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 45 to 46, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 46 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:01.979480 27479 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 44 to 45, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 45 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:01.994894 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 67903.5us
I20260709 12:24:01.997740  8990 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:24:01.998210  8990 raft_consensus.cc:493] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:01.998576  8990 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:01.990307  9024 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.001564  9024 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:01.947811  8986 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.005225 28335 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:24:02.006258 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:01.944440  9011 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.006920 28332 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 39.
I20260709 12:24:02.008870  9011 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:01.942019  9026 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.010931  9042 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:24:02.010994 27704 leader_election.cc:336] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.011670  9042 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.012624  9026 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:01.940932  9031 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.013836  9042 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:02.015031  9031 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:01.990811  9036 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.018015  9036 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:02.018678 27704 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.019622 27704 leader_election.cc:336] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.021474  9026 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:24:02.022338 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:01.994992  8984 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.022953 27737 raft_consensus.cc:1275] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:24:02.024895  8984 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:02.025422  9030 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.027695  9030 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.027129  8990 raft_consensus.cc:515] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.029104 28335 raft_consensus.cc:2393] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 37: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:24:02.031711 27704 leader_election.cc:336] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.033833  8990 leader_election.cc:290] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:02.037165 27704 leader_election.cc:336] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.043705 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 47487.7us
I20260709 12:24:01.990566  8932 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:02.044528 27630 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.046144  8932 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:24:02.049201 27630 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.008581 28295 leader_election.cc:336] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.055058 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:02.006361  8986 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:02.057235 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 48686.6us
I20260709 12:24:02.061556 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:02.062285 27733 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 39.
I20260709 12:24:02.062680  9026 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.059397  8990 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:24:02.071096 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42e7d195f3f545b98a8dedb26c479c9c" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:02.071780 27733 raft_consensus.cc:3060] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:02.073761 27628 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:01.992349  8928 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.080543  8928 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.082708  8928 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:02.078624  9047 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:24:02.083382  9047 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.083822  9047 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:24:02.063220  9026 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.087205 27733 raft_consensus.cc:2468] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 37.
W20260709 12:24:02.088189 27704 leader_election.cc:336] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.091697 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:02.093021 28332 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:24:02.095279 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:02.100111 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:02.116888  9035 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:24:02.117488  9035 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.117988  9035 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:24:02.128762  9026 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.113698 28332 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 36.
I20260709 12:24:02.101078 27702 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:24:02.094712  9024 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
W20260709 12:24:02.144775 27630 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.144584  9024 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:02.146445 27630 leader_election.cc:336] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.147509 27630 leader_election.cc:336] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.149362 27702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39321, user_credentials={real_user=slave}} blocked reactor thread for 48453.7us
I20260709 12:24:02.150090 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.160356  9047 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.165555  9047 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:02.166708 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8355fe4a548a43f8b5215ea9ae7ba8b2" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:24:02.167387 28336 raft_consensus.cc:3060] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:24:02.172062 26408 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 27610)
I20260709 12:24:02.090588  9043 consensus_queue.cc:1048] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.057s
W20260709 12:24:02.172909 27704 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.181087 28295 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:02.178790 27479 catalog_manager.cc:5697] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 43 to 44, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 44 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:02.182116 28295 leader_election.cc:336] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.184930  8986 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.185443  8986 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.187433  8986 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:02.173444  9027 raft_consensus.cc:2749] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:24:02.191537 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:02.192154 27664 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 38.
I20260709 12:24:02.192909 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:24:02.193090 27474 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 108313us
I20260709 12:24:02.177040 28336 raft_consensus.cc:2468] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 40.
I20260709 12:24:02.178436  8984 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:24:02.199193 28292 leader_election.cc:304] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:02.203464  9040 raft_consensus.cc:2804] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:24:02.204006  9040 raft_consensus.cc:697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:24:02.204847  9040 consensus_queue.cc:237] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.206641  9035 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.198096  8932 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:24:02.212177  8932 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.212577  8932 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:02.212814 27628 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113; no voters: 
I20260709 12:24:02.206686  9053 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.213706  9050 raft_consensus.cc:2804] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:24:02.213493  9053 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.214226  9050 raft_consensus.cc:697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:24:02.215269  9050 consensus_queue.cc:237] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.213201  9035 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:02.215763  9053 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:02.200157  8984 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.217545  8984 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:24:02.217612 28295 leader_election.cc:336] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.218410 28295 leader_election.cc:336] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.230279 27704 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:02.191558  9024 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.233171  9024 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.233665  9024 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:02.235379 28295 leader_election.cc:336] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.236299  9024 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.252139 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:02.252942 27664 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:24:02.257377 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:02.258279 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
I20260709 12:24:02.267820 27664 raft_consensus.cc:1275] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:24:02.250805  8984 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:02.277062 27630 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.265286  8932 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:02.292168 27474 catalog_manager.cc:4755] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 073b12eba02f4990a4fbe7080d8b8aff: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.293286 27704 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.294049 27704 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:02.294543 27704 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:02.300576 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:02.301285 27664 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 41.
I20260709 12:24:02.302587 27729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:02.308297 27483 catalog_manager.cc:5697] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 39 to 40, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 40 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:02.311440 27729 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aa1523b9af84c3db99ba394e2b9aa9a in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:24:02.317444 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:02.318200 27664 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:24:02.323856 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:02.332326  8932 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:02.334107 28336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "310b1b37298244c0bd5bbe2b0c7328d7" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
W20260709 12:24:02.335465 27630 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.336400 28336 raft_consensus.cc:3060] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:24:02.338621  9040 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
W20260709 12:24:02.351224 27704 leader_election.cc:336] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.352042 27704 leader_election.cc:336] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:02.352859 27628 consensus_peers.cc:597] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321): Couldn't send request to peer 4aa1523b9af84c3db99ba394e2b9aa9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83433b8ed26448aca1d62a3b80ba4198. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:02.353808 27630 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.354918 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113" is_pre_election: true
I20260709 12:24:02.355710 27662 raft_consensus.cc:1275] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 45 index: 23. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
I20260709 12:24:02.355729 27479 catalog_manager.cc:5697] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 36 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 37 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.359179 27664 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:24:02.378974  8984 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:02.382617 28332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a"
I20260709 12:24:02.383574 28332 raft_consensus.cc:2393] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate ce69a3f9261a47a7a22f02f9c1f07964 in current term 40: Already voted for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in this term.
W20260709 12:24:02.386365 27704 leader_election.cc:336] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.390606 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
W20260709 12:24:02.391709 27630 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.396394  9030 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:24:02.399724  9030 raft_consensus.cc:493] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.400254  9030 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:24:02.397858 27702 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 4aa1523b9af84c3db99ba394e2b9aa9a
I20260709 12:24:02.398526 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.408627  9026 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:24:02.409657  9026 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.410012  9026 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:24:02.417588  9061 raft_consensus.cc:2749] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:24:02.433990 28336 raft_consensus.cc:2468] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 37.
I20260709 12:24:02.441931  9026 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.445513 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.447434  8932 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:24:02.448010  8984 consensus_queue.cc:1048] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:24:02.448055  8932 raft_consensus.cc:493] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.449107  8932 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:24:02.450726 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.451957 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.452831  9060 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:24:02.453294  9060 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.453685  9060 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:24:02.455530  9061 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:24:02.456003  9061 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.456410  9061 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:24:02.460711  8932 raft_consensus.cc:515] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.462875  9027 consensus_queue.cc:1048] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:24:02.469821  8932 leader_election.cc:290] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.463734 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.475838  9027 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:24:02.476003 27704 leader_election.cc:336] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.476337  9027 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.476833  9027 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:24:02.491959 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:24:02.492636 27630 consensus_peers.cc:597] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:02.492666 27733 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 46.
I20260709 12:24:02.489621  9058 raft_consensus.cc:2955] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 LEADER]: Committing config change with OpId 48.28: config changed from index 27 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:02.501331 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:02.502367 27733 raft_consensus.cc:2393] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 34: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:24:02.508885 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:02.510746 27664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c204963842941859d4313114fd92c93" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:02.511404 27664 raft_consensus.cc:3060] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:24:02.518414 27662 raft_consensus.cc:2955] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Committing config change with OpId 48.28: config changed from index 27 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:02.525285 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:02.531798  9030 raft_consensus.cc:515] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.562711 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:02.563784 27737 raft_consensus.cc:2393] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 40: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:24:02.564849  9030 leader_election.cc:290] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:02.569869 27704 leader_election.cc:336] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.584573 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ab0ff10d53741dea50e33f971484854" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:02.585285 27661 raft_consensus.cc:3060] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:24:02.589393 27664 raft_consensus.cc:2468] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 34.
W20260709 12:24:02.590695 27704 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.585642  9026 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.610157 27479 catalog_manager.cc:5697] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 27 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 48 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:02.612608 27661 raft_consensus.cc:2468] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 39.
I20260709 12:24:02.620203 27737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:02.633678  9060 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.635128 27737 raft_consensus.cc:2393] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 42: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:24:02.614298 27701 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.645474 27734 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:24:02.646392  9065 raft_consensus.cc:2804] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:24:02.613453  9027 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.647737  9065 raft_consensus.cc:697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:02.648561 27733 raft_consensus.cc:2393] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 43: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:24:02.648869  9065 consensus_queue.cc:237] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.664108 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.665359  9047 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:24:02.665730 27704 leader_election.cc:336] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.665778  9047 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 3f90188458da46f6ba138d5f82c55b69)
I20260709 12:24:02.666524  9047 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:24:02.668781  9061 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.692361 27732 raft_consensus.cc:2393] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c09719959a54e8eb66d2041dd9c3113 in current term 44: Already voted for candidate ce69a3f9261a47a7a22f02f9c1f07964 in this term.
I20260709 12:24:02.673216  9027 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.676126 27627 leader_election.cc:304] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:02.661533 27701 leader_election.cc:304] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.694056 27627 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:24:02.682236 27704 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.695502  9028 raft_consensus.cc:2749] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:02.695688  9030 raft_consensus.cc:2804] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:24:02.696331  9030 raft_consensus.cc:697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:02.696851  9028 raft_consensus.cc:3060] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:24:02.697297  9030 consensus_queue.cc:237] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.697337 27483 catalog_manager.cc:5697] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 33 to 34, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 34 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.702204  9047 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:02.711709 27630 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.714058 27702 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.715423  9047 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:02.715858 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:02.716564 27663 raft_consensus.cc:3060] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:24:02.717057  9011 raft_consensus.cc:2804] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:24:02.717640  9011 raft_consensus.cc:697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:02.716244 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:02.718617  9011 consensus_queue.cc:237] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:02.724550 27627 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:02.730576 27663 raft_consensus.cc:2468] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 43.
I20260709 12:24:02.734139  9028 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:24:02.738900 28336 raft_consensus.cc:2955] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Committing config change with OpId 48.28: config changed from index 27 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
I20260709 12:24:02.747635 27732 raft_consensus.cc:1275] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:24:02.757198  9061 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.808499  8990 consensus_queue.cc:1048] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:24:02.774025 27704 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:02.786103  8986 raft_consensus.cc:3060] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:24:02.812034 27661 raft_consensus.cc:2468] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 40.
I20260709 12:24:02.822373  8986 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:24:02.825297 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 51341.3us
W20260709 12:24:02.785965 27627 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:41211, user_credentials={real_user=slave}} blocked reactor thread for 60080.2us
I20260709 12:24:02.836551 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:02.842094  9077 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:02.842598 27627 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:02.843514  9077 raft_consensus.cc:3060] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:24:02.837025  8986 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.007s
W20260709 12:24:02.852151 28295 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:02.860028 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dad2444fb3474aee8e7dbbd7e673ab01" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:02.860745 27734 raft_consensus.cc:3060] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:24:02.870612 27734 raft_consensus.cc:2468] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c09719959a54e8eb66d2041dd9c3113 in term 47.
I20260709 12:24:02.866250 27480 catalog_manager.cc:5697] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 38 to 39, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 39 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.879487 27480 catalog_manager.cc:5697] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 36 to 37, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 37 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:02.889384  9060 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.900532 27701 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.903709  9011 raft_consensus.cc:2804] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:24:02.904316  9011 raft_consensus.cc:697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:02.905270  9011 consensus_queue.cc:237] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.910095 27701 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:02.911139  9061 raft_consensus.cc:2804] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:24:02.911911  9061 raft_consensus.cc:697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:02.912843  9061 consensus_queue.cc:237] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:02.919240  9077 raft_consensus.cc:2749] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:02.934808 28335 raft_consensus.cc:1275] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:24:02.956418  9028 consensus_queue.cc:1048] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:24:02.959641 27630 consensus_peers.cc:597] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:02.965749 27663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:02.974201 27661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "9c09719959a54e8eb66d2041dd9c3113"
I20260709 12:24:02.979905  9081 ts_tablet_manager.cc:933] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Initiating tablet copy from peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:02.998268 27627 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:03.007099 27661 raft_consensus.cc:2468] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 42.
I20260709 12:24:03.008179 27733 raft_consensus.cc:1275] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:24:03.010062  8928 consensus_queue.cc:1048] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:24:03.015744 27663 raft_consensus.cc:2468] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 44.
I20260709 12:24:03.007836  9050 raft_consensus.cc:2804] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:24:03.025058  9050 raft_consensus.cc:697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 LEADER]: Becoming Leader. State: Replica: 9c09719959a54e8eb66d2041dd9c3113, State: Running, Role: LEADER
I20260709 12:24:03.025977  9050 consensus_queue.cc:237] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:03.031749 27701 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:03.032606  9065 raft_consensus.cc:2804] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:24:03.033138  9065 raft_consensus.cc:697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:03.034070  9065 consensus_queue.cc:237] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:03.040714 27630 consensus_peers.cc:597] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.071913  9081 tablet_copy_client.cc:323] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Beginning tablet copy session from remote peer at address 127.25.202.2:33051
I20260709 12:24:03.078930 27674 tablet_copy_service.cc:140] P 9c09719959a54e8eb66d2041dd9c3113: Received BeginTabletCopySession request for tablet 83433b8ed26448aca1d62a3b80ba4198 from peer 4aa1523b9af84c3db99ba394e2b9aa9a ({username='slave'} at 127.0.0.1:36354)
I20260709 12:24:03.079875 27674 tablet_copy_service.cc:161] P 9c09719959a54e8eb66d2041dd9c3113: Beginning new tablet copy session on tablet 83433b8ed26448aca1d62a3b80ba4198 from peer 4aa1523b9af84c3db99ba394e2b9aa9a at {username='slave'} at 127.0.0.1:36354: session id = 4aa1523b9af84c3db99ba394e2b9aa9a-83433b8ed26448aca1d62a3b80ba4198
W20260709 12:24:03.103518 27704 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.144903 27480 catalog_manager.cc:5697] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 39 to 40, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 40 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:03.147789 27480 catalog_manager.cc:5697] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 42 to 43, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 43 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:03.151750 27701 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:03.152909  9065 raft_consensus.cc:2804] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:24:03.153635  9065 raft_consensus.cc:697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Becoming Leader. State: Replica: ce69a3f9261a47a7a22f02f9c1f07964, State: Running, Role: LEADER
I20260709 12:24:03.157742  9065 consensus_queue.cc:237] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:03.170938 27674 tablet_copy_source_session.cc:215] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 12:24:03.172572 26408 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 27610)
I20260709 12:24:03.195467  9081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83433b8ed26448aca1d62a3b80ba4198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:03.205866 27663 raft_consensus.cc:1275] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:24:03.219249  9061 consensus_queue.cc:1048] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:24:03.228163 27480 catalog_manager.cc:5697] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: term changed from 46 to 47, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2). New cstate: current_term: 47 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:03.252286  9081 tablet_copy_client.cc:806] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:03.254652  9081 tablet_copy_client.cc:670] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Starting download of 1 WAL segments...
W20260709 12:24:03.265167 27704 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.268692  9081 tablet_copy_client.cc:538] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:03.278200 28332 raft_consensus.cc:1275] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
W20260709 12:24:03.284897 27704 consensus_peers.cc:597] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.292663 27661 raft_consensus.cc:1275] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 38 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:24:03.266173 27483 catalog_manager.cc:5697] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 41 to 42, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 42 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:03.326666 27483 catalog_manager.cc:5697] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 reported cstate change: term changed from 43 to 44, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3). New cstate: current_term: 44 leader_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:03.333982  9027 consensus_queue.cc:1048] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:24:03.376951 27661 raft_consensus.cc:1275] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:24:03.386541  9030 consensus_queue.cc:1048] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:24:03.389338  8984 consensus_queue.cc:1048] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:24:03.406806  9081 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap starting.
W20260709 12:24:03.471343 27704 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.474628 27732 raft_consensus.cc:1275] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
I20260709 12:24:03.478483  9053 consensus_queue.cc:1048] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.016s
I20260709 12:24:03.518178 27661 raft_consensus.cc:1275] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 43 index: 26. (index mismatch)
I20260709 12:24:03.521071  9030 consensus_queue.cc:1048] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:24:03.536290 27630 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 50865us
W20260709 12:24:03.548846 27704 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.549386 27704 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:03.565469 27630 consensus_peers.cc:597] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.617750 27630 consensus_peers.cc:597] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.619102 27630 consensus_peers.cc:597] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.635515 27630 consensus_peers.cc:597] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.639307 27630 consensus_peers.cc:597] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.641178 27474 catalog_manager.cc:4755] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 073b12eba02f4990a4fbe7080d8b8aff: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:03.643875 27630 consensus_peers.cc:597] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.647598 27630 consensus_peers.cc:597] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.659667 27661 raft_consensus.cc:1275] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
W20260709 12:24:03.661433  9097 log.cc:927] Time spent T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.066s	user 0.007s	sys 0.000s
I20260709 12:24:03.661284  9027 consensus_queue.cc:1048] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:24:03.680137 27704 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:03.683254 27658 raft_consensus.cc:1275] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Refusing update from remote peer ce69a3f9261a47a7a22f02f9c1f07964: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:24:03.688105  9011 consensus_queue.cc:1048] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:24:03.700747 27630 consensus_peers.cc:597] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.707486 27704 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:03.715574 27630 consensus_peers.cc:597] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.715870 27704 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.716619 27704 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.725392 27630 consensus_peers.cc:597] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.749842 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.799847 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.802603 27630 consensus_peers.cc:597] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.879205  9081 log.cc:927] Time spent T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Append to log took a long time: real 0.077s	user 0.003s	sys 0.001s
W20260709 12:24:03.886193 27630 consensus_peers.cc:597] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.894167 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.897665 27630 consensus_peers.cc:597] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.903959 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.920614 27630 consensus_peers.cc:597] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.930244 27704 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:03.930855 27704 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:03.952405  9107 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.953110  9107 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:03.959925 28295 leader_election.cc:336] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:03.964110  9107 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:03.977916 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:03.978636 27732 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 40.
I20260709 12:24:03.985307 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69
I20260709 12:24:03.986555  9107 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:24:03.986994  9107 raft_consensus.cc:493] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.987392  9107 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:24:04.012777 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:04.031206  9107 raft_consensus.cc:515] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:04.034597 27734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6491d84db64f4d48b3b3d5be8aec385b" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:04.035198 27734 raft_consensus.cc:3060] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:24:04.043622 27734 raft_consensus.cc:2468] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 41.
I20260709 12:24:04.044982 28292 leader_election.cc:304] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:04.045917  9109 raft_consensus.cc:2804] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:24:04.046496  9109 raft_consensus.cc:697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
W20260709 12:24:04.048005 28295 leader_election.cc:336] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:04.048118  9109 consensus_queue.cc:237] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:04.072902 27480 catalog_manager.cc:5697] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 39 to 41, leader changed from 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 41 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:04.079428  9107 leader_election.cc:290] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 election: Requested vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:04.093588 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:04.094162 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:04.103821 27630 consensus_peers.cc:597] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:04.099087  9081 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=1 applied=26 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:24:04.111019  9081 tablet_bootstrap.cc:492] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Bootstrap complete.
I20260709 12:24:04.112358  9081 ts_tablet_manager.cc:1403] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent bootstrapping tablet: real 0.706s	user 0.077s	sys 0.024s
W20260709 12:24:04.109210 27704 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:04.121870  9081 raft_consensus.cc:359] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:04.129737  9081 raft_consensus.cc:740] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEARNER]: Becoming Follower/Learner. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Initialized, Role: LEARNER
W20260709 12:24:04.149488 27704 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:04.130764  9081 consensus_queue.cc:260] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } }
I20260709 12:24:04.163882  9081 ts_tablet_manager.cc:1434] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: Time spent starting tablet: real 0.051s	user 0.010s	sys 0.000s
I20260709 12:24:04.175109 27674 tablet_copy_service.cc:342] P 9c09719959a54e8eb66d2041dd9c3113: Request end of tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-83433b8ed26448aca1d62a3b80ba4198 received from {username='slave'} at 127.0.0.1:36354
I20260709 12:24:04.175637 27674 tablet_copy_service.cc:434] P 9c09719959a54e8eb66d2041dd9c3113: ending tablet copy session 4aa1523b9af84c3db99ba394e2b9aa9a-83433b8ed26448aca1d62a3b80ba4198 on tablet 83433b8ed26448aca1d62a3b80ba4198 with peer 4aa1523b9af84c3db99ba394e2b9aa9a
W20260709 12:24:04.184479 26408 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 27610)
W20260709 12:24:04.215205 27704 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:04.225790 27704 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:04.415216 28295 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:04.417232 27704 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:04.466503 27630 consensus_peers.cc:597] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:04.506017 28332 raft_consensus.cc:1217] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEARNER]: Deduplicated request from leader. Original: 44.26->[45.27-45.27]   Dedup: 45.27->[]
W20260709 12:24:04.544351 27630 consensus_peers.cc:597] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:04.587648 28295 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:04.594422 28295 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:04.633657 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
I20260709 12:24:04.653330 27734 raft_consensus.cc:1275] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:24:04.656129  9037 consensus_queue.cc:1048] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:24:04.779551 27704 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:04.814466 27630 consensus_peers.cc:597] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:04.853777 27704 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:04.923138  8986 raft_consensus.cc:1064] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: attempting to promote NON_VOTER 4aa1523b9af84c3db99ba394e2b9aa9a to VOTER
I20260709 12:24:04.946408  8986 consensus_queue.cc:237] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 26, Current term: 45, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:04.956981 27630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111) [suppressed 472 similar messages]
I20260709 12:24:05.106189 28332 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 LEARNER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
I20260709 12:24:05.107589  8986 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:24:05.123196 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
W20260709 12:24:05.132613 27701 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.136343 27733 raft_consensus.cc:1275] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Refusing update from remote peer 9c09719959a54e8eb66d2041dd9c3113: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
I20260709 12:24:05.158202  9077 consensus_queue.cc:1048] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20260709 12:24:05.180994 28292 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.186939 27630 consensus_peers.cc:597] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.172845 27701 consensus_peers.cc:597] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.191432 27701 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.199471 28292 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.200228 27701 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.203121 27701 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.211618 28292 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.237020 27701 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.259992  8986 raft_consensus.cc:2955] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Committing config change with OpId 45.28: config changed from index 26 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:24:05.277489 28292 consensus_peers.cc:597] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.282918 27701 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.283725 27732 raft_consensus.cc:2955] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Committing config change with OpId 45.28: config changed from index 26 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:24:05.289908 27701 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.290629 27701 consensus_peers.cc:597] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.300176 28335 raft_consensus.cc:2955] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Committing config change with OpId 45.28: config changed from index 26 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } } }
W20260709 12:24:05.310937 27701 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.312513 28292 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.336082 27701 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.343569 27480 catalog_manager.cc:5697] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 reported cstate change: config changed from index 26 to 28, 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4) changed from NON_VOTER to VOTER. New cstate: current_term: 45 leader_uuid: "9c09719959a54e8eb66d2041dd9c3113" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:05.326136 27630 consensus_peers.cc:597] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:05.366629 28292 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.377192 28292 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.382649 27701 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.388235 28292 consensus_peers.cc:597] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.410013 27704 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:05.473830 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:24:05.479296 27701 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.481684 26408 tablet_replica.cc:333] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.483291 26408 raft_consensus.cc:2243] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.486719 26408 raft_consensus.cc:2272] T 39fef9cf211c4b858a135faa1b00e1b1 P 9c09719959a54e8eb66d2041dd9c3113 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.490666 26408 tablet_replica.cc:333] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
W20260709 12:24:05.496990 28295 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:05.498509 27471 catalog_manager.cc:4755] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): ChangeConfig:REMOVE_PEER RPC failed for tablet 83433b8ed26448aca1d62a3b80ba4198: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:24:05.499073 26408 raft_consensus.cc:2243] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.500345 26408 raft_consensus.cc:2272] T eb215cc72c384ec8b9421c81d953589b P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.504542 28292 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.507920 27701 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.529461 26408 tablet_replica.cc:333] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
W20260709 12:24:05.530596 27701 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.538863 26408 raft_consensus.cc:2243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.544175 26408 raft_consensus.cc:2272] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.561053 27701 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:05.562902 26408 tablet_replica.cc:333] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.568157 26408 raft_consensus.cc:2243] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.569376 26408 raft_consensus.cc:2272] T fe68c465ca4346b9b9db89a5be4e639c P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.610282 27701 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.626597 27701 consensus_peers.cc:597] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.627148 27701 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:05.623603 27704 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:05.605520 27630 consensus_peers.cc:597] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:05.575516 26408 tablet_replica.cc:333] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.652974 26408 raft_consensus.cc:2243] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.653663 26408 raft_consensus.cc:2272] T 7efb19c4fe844c2ea043bd3638faae4e P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.661110 26408 tablet_replica.cc:333] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.662014 26408 raft_consensus.cc:2243] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.664558 26408 raft_consensus.cc:2272] T 83433b8ed26448aca1d62a3b80ba4198 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.672343 26408 tablet_replica.cc:333] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.677570 26408 raft_consensus.cc:2243] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.678614 26408 raft_consensus.cc:2272] T b8ac8041c87e49308784e1cad6aeec03 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.681308 26408 tablet_replica.cc:333] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.682052 26408 raft_consensus.cc:2243] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.682660 26408 raft_consensus.cc:2272] T 70329e70bb724d43a3185508e1186f4b P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.710227 27630 consensus_peers.cc:597] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:05.714737 26408 tablet_replica.cc:333] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.718075 26408 raft_consensus.cc:2243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.718868 26408 raft_consensus.cc:2272] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.731168 26408 tablet_replica.cc:333] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.732138 26408 raft_consensus.cc:2243] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.732841 26408 raft_consensus.cc:2272] T 27b4972f682948f4981cb0dd1a74b4f5 P 9c09719959a54e8eb66d2041dd9c3113 [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.735777 26408 tablet_replica.cc:333] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.736691 26408 raft_consensus.cc:2243] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.738849 26408 raft_consensus.cc:2272] T 4811ee81866e469abca31f212c08ace2 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.743376 26408 tablet_replica.cc:333] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.745074 26408 raft_consensus.cc:2243] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.746274 26408 raft_consensus.cc:2272] T 81e6b2dabf8f4e6cb549ca250430e568 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.751688 26408 tablet_replica.cc:333] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
W20260709 12:24:05.753326 27704 consensus_peers.cc:597] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:05.752534 26408 raft_consensus.cc:2243] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.755033 26408 raft_consensus.cc:2272] T 723b2024e4414796ba50ece654596156 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.758060 26408 tablet_replica.cc:333] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.758898 26408 raft_consensus.cc:2243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.760188 26408 raft_consensus.cc:2272] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.763221 26408 tablet_replica.cc:333] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.779973 26408 raft_consensus.cc:2243] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.781282 26408 raft_consensus.cc:2272] T 6f3a9a00f0db424982bf0ea90e7ae286 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.813298 26408 tablet_replica.cc:333] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.814303 26408 raft_consensus.cc:2243] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.815526 26408 raft_consensus.cc:2272] T d3d15932c0a64b668179d6868ab4707c P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.818913 26408 tablet_replica.cc:333] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.819765 26408 raft_consensus.cc:2243] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.820436 26408 raft_consensus.cc:2272] T c596ddd50b894549b567a5b66ac1a31e P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.823976 26408 tablet_replica.cc:333] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.824777 26408 raft_consensus.cc:2243] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 48 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.825938 26408 raft_consensus.cc:2272] T 73b2d9da2f874f958d24ff32c2097c0a P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.829107 26408 tablet_replica.cc:333] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.829955 26408 raft_consensus.cc:2243] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.831084 26408 raft_consensus.cc:2272] T dad2444fb3474aee8e7dbbd7e673ab01 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.838481 26408 tablet_replica.cc:333] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.839308 26408 raft_consensus.cc:2243] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.839974 26408 raft_consensus.cc:2272] T bdb219c7e3524fc9b4a91a96610cee8b P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.843091 26408 tablet_replica.cc:333] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.843986 26408 raft_consensus.cc:2243] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.844655 26408 raft_consensus.cc:2272] T dbdcf17dc981450f989b93d262e9a5d0 P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.855492 26408 tablet_replica.cc:333] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.856299 26408 raft_consensus.cc:2243] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.857682 26408 raft_consensus.cc:2272] T a6d40162b2bb4b61adc69c9fa3f03326 P 9c09719959a54e8eb66d2041dd9c3113 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.860525 26408 tablet_replica.cc:333] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.861304 26408 raft_consensus.cc:2243] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:24:05.862648 26408 raft_consensus.cc:2272] T d2b187795e304ffd9c1293d922ff5b38 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.865423 26408 tablet_replica.cc:333] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.866240 26408 raft_consensus.cc:2243] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.866995 26408 raft_consensus.cc:2272] T a07754f58c3a421ba559f56d67d87bb4 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.867547 27704 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:05.870087 26408 tablet_replica.cc:333] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.871029 26408 raft_consensus.cc:2243] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.871773 26408 raft_consensus.cc:2272] T 74b289dab6a94832ad11d8ac1e30d802 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:05.902547 26408 tablet_replica.cc:333] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
W20260709 12:24:05.919647 27630 consensus_peers.cc:597] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:05.950359 26408 raft_consensus.cc:2243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.956900 26408 raft_consensus.cc:2272] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P 9c09719959a54e8eb66d2041dd9c3113 [term 48 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.974002 27704 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:05.959774 26408 tablet_replica.cc:333] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.976171 26408 raft_consensus.cc:2243] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.976866 26408 raft_consensus.cc:2272] T 4ab0ff10d53741dea50e33f971484854 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:05.982299 27704 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:05.989490 26408 tablet_replica.cc:333] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:05.990484 26408 raft_consensus.cc:2243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:05.991438 26408 raft_consensus.cc:2272] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:06.013725 27704 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:06.024960 26408 tablet_replica.cc:333] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.025985 26408 raft_consensus.cc:2243] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.027148 26408 raft_consensus.cc:2272] T 073b12eba02f4990a4fbe7080d8b8aff P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.029971 26408 tablet_replica.cc:333] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.030731 26408 raft_consensus.cc:2243] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 LEADER]: Raft consensus shutting down.
W20260709 12:24:06.035389 27704 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.044054 27704 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:06.048285 26408 raft_consensus.cc:2272] T 8c52fdc88d8646dd8e6584f8dbd137a3 P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.063856 26408 tablet_replica.cc:333] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.064891 26408 raft_consensus.cc:2243] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Raft consensus shutting down.
W20260709 12:24:06.065296 27630 consensus_peers.cc:597] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.065635 26408 raft_consensus.cc:2272] T 22597fcae204491f84839e680c083ecc P 9c09719959a54e8eb66d2041dd9c3113 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.068807 26408 tablet_replica.cc:333] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.069634 26408 raft_consensus.cc:2243] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.070223 26408 raft_consensus.cc:2272] T 1c3943cf144748fca8d2892751757c1a P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.071341  9053 raft_consensus.cc:493] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:06.071890  9053 raft_consensus.cc:515] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.074939  9053 leader_election.cc:290] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:06.085366 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecd16aab46634913b20f3501e9a6389f" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:24:06.090152 27630 leader_election.cc:336] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.090660 27630 leader_election.cc:304] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:24:06.091651 27630 consensus_peers.cc:597] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.092160  9053 raft_consensus.cc:2749] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:06.095309 26408 tablet_replica.cc:333] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.096248 26408 raft_consensus.cc:2243] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.097004 26408 raft_consensus.cc:2272] T e634fbff5fb24e2fa133b95599b0db1f P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:06.110270 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.133047 26408 tablet_replica.cc:333] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.134021 26408 raft_consensus.cc:2243] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.142093 26408 raft_consensus.cc:2272] T 0d80be98cdd043ffa8d122ba0ac29447 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.145277 26408 tablet_replica.cc:333] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.149386 26408 raft_consensus.cc:2243] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.150049 26408 raft_consensus.cc:2272] T ecd16aab46634913b20f3501e9a6389f P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.160197 26408 tablet_replica.cc:333] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.161036 26408 raft_consensus.cc:2243] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.162137 26408 raft_consensus.cc:2272] T 5f21db64000c434ca36b2b9021c13a44 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.165141 26408 tablet_replica.cc:333] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.165949 26408 raft_consensus.cc:2243] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.166896 26408 raft_consensus.cc:2272] T 7274fd8a2115405c9b78c61214bff736 P 9c09719959a54e8eb66d2041dd9c3113 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:06.175243 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.178905 26408 tablet_replica.cc:333] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.179914 26408 raft_consensus.cc:2243] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.181074 26408 raft_consensus.cc:2272] T c3f34c916e524bfa9fcae2044da089fc P 9c09719959a54e8eb66d2041dd9c3113 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.184041 26408 tablet_replica.cc:333] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.185211 26408 raft_consensus.cc:2243] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.185863 26408 raft_consensus.cc:2272] T 7c204963842941859d4313114fd92c93 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.188691 26408 tablet_replica.cc:333] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.189625 26408 raft_consensus.cc:2243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.190414 26408 raft_consensus.cc:2272] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.194384 26408 tablet_replica.cc:333] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
W20260709 12:24:06.194814 27630 consensus_peers.cc:597] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.195155 26408 raft_consensus.cc:2243] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.196368 26408 raft_consensus.cc:2272] T b4e3fe6577354786a64bed5843d9a6a4 P 9c09719959a54e8eb66d2041dd9c3113 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.200739 26408 tablet_replica.cc:333] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.203485  9053 raft_consensus.cc:493] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:06.204182  9053 raft_consensus.cc:515] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.206598  9053 leader_election.cc:290] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:06.216055 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5accfdbc68cd4cf399c98e2a533f500e" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
W20260709 12:24:06.220671 27704 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.224663 27630 leader_election.cc:336] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.225198 27630 leader_election.cc:304] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:06.227424 26408 raft_consensus.cc:2243] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.228032  9078 raft_consensus.cc:2749] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:06.207103  9053 raft_consensus.cc:493] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:06.230369  9053 raft_consensus.cc:515] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.235723 26408 raft_consensus.cc:2272] T 456e2eac35d647a3bf9eeb45a3d7596f P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.238116  9053 leader_election.cc:290] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:06.241694 26408 tablet_replica.cc:333] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.241955 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22efa3bef6634e7a811baa3c42d9cd18" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:06.242650 26408 raft_consensus.cc:2243] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.243479 26408 raft_consensus.cc:2272] T 5accfdbc68cd4cf399c98e2a533f500e P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.246423 26408 tablet_replica.cc:333] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.247215 26408 raft_consensus.cc:2243] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.248024 26408 raft_consensus.cc:2272] T 27d7343bc8174a5f89f4652b4401c3ec P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.252787 26408 tablet_replica.cc:333] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.253559 26408 raft_consensus.cc:2243] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.254690 26408 raft_consensus.cc:2272] T 68dbab4c1e1442b699465f7b3e728acd P 9c09719959a54e8eb66d2041dd9c3113 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.257773 26408 tablet_replica.cc:333] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.258563 26408 raft_consensus.cc:2243] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:06.259239 26408 raft_consensus.cc:2272] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.262168 26408 tablet_replica.cc:333] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.263025 26408 raft_consensus.cc:2243] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 LEADER]: Raft consensus shutting down.
W20260709 12:24:06.263512 27630 leader_election.cc:336] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.264024 27630 leader_election.cc:304] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
W20260709 12:24:06.264712 27630 raft_consensus.cc:2686] T 22efa3bef6634e7a811baa3c42d9cd18 P 9c09719959a54e8eb66d2041dd9c3113: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:24:06.264163 26408 raft_consensus.cc:2272] T 3b881b4e55d543f9ade316642ea1b7e3 P 9c09719959a54e8eb66d2041dd9c3113 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.266848  9053 raft_consensus.cc:493] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:06.267428  9053 raft_consensus.cc:515] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.268646 26408 tablet_replica.cc:333] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.269501 26408 raft_consensus.cc:2243] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 LEADER]: Raft consensus shutting down.
W20260709 12:24:06.275113 27630 leader_election.cc:336] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.276160 26408 raft_consensus.cc:2272] T e76581eba9454a229c556095ac8b3e23 P 9c09719959a54e8eb66d2041dd9c3113 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.276827  9053 leader_election.cc:290] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:06.277119 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37a0fdc57f6240869f0977e359f258c0" candidate_uuid: "9c09719959a54e8eb66d2041dd9c3113" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:06.278769 27627 leader_election.cc:304] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c09719959a54e8eb66d2041dd9c3113; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:06.280031  9053 raft_consensus.cc:2749] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:06.283133 26408 tablet_replica.cc:333] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
W20260709 12:24:06.293774 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.297196 26408 raft_consensus.cc:2243] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:24:06.298856 26408 raft_consensus.cc:2272] T 70b48af2ec82404986e5e18d544db5b4 P 9c09719959a54e8eb66d2041dd9c3113 [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:06.308203 27704 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:06.308861 26408 tablet_replica.cc:333] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.315513 26408 raft_consensus.cc:2243] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 LEADER]: Raft consensus shutting down.
W20260709 12:24:06.329327 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.333415 26408 raft_consensus.cc:2272] T 4f392cf87198499d9c064d1ec232d3d7 P 9c09719959a54e8eb66d2041dd9c3113 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:06.336668 26408 tablet_replica.cc:333] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113: stopping tablet replica
I20260709 12:24:06.337399 26408 raft_consensus.cc:2243] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Raft consensus shutting down.
W20260709 12:24:06.337720 27704 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.338016 26408 raft_consensus.cc:2272] T 37a0fdc57f6240869f0977e359f258c0 P 9c09719959a54e8eb66d2041dd9c3113 [term 43 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:06.395263 27704 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.403012 27704 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.429376 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.438905 27704 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.483135 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.643242 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:06.683396 27704 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.705005  9109 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.705591  9109 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:06.709219  9109 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:06.716275 28295 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:06.719393 27471 catalog_manager.cc:4755] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): ChangeConfig:REMOVE_PEER RPC failed for tablet 83433b8ed26448aca1d62a3b80ba4198: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:06.732885 28292 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.733534 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:06.734380  9109 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:06.768168 28295 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:06.784956  9027 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.785547  9109 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.785655  9027 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:06.786319  9109 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:06.788916  9109 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:06.794018 27704 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.795903  9027 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:06.797802 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:06.801445 28335 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
W20260709 12:24:06.799715 28295 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.803107  9103 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.803974  9103 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.806419  9103 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:06.813635 27701 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.815887 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:06.815346 27704 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:06.817286  9131 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:06.824353 28292 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.825011 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:06.829308  9130 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:06.830814  9103 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.831897  9103 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:06.834700  9103 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:06.835867 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:24:06.841874 27701 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.842486 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:06.847462  9103 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:06.857774  9103 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.858439  9103 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:06.867681 27704 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:06.878320  9103 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:06.879117 27704 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:06.880411 27701 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.881897 27701 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:06.883535  9103 raft_consensus.cc:2749] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:24:06.889834 27701 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.891587 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:06.892413  9103 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:06.921213  9103 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.921734  9103 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:06.928565 27701 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:06.937785 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:06.939397 27702 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:06.940106  9103 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:06.942276  9131 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:24:06.952523 27704 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:06.960916 28295 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:06.983275  9131 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.983888  9131 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.986014  9131 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:06.991202  9103 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:06.991855  9103 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:06.993965  9103 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:06.990861 27701 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:07.029477 27701 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:07.029902 28295 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:07.030011  9103 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.032557  9103 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.035933 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
W20260709 12:24:07.031508 27704 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.037881 27704 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.039255  9131 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:07.040256 27704 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.043969 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.046294  9131 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:24:07.047456  9103 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:07.050062  9109 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.050663  9109 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:07.058604 27701 leader_election.cc:336] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:07.065084 28295 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:07.069396 28292 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.070446 27701 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.073099  9109 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:07.075201  9103 raft_consensus.cc:2749] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:07.075147 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:07.077476 27704 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:07.077746 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:07.078550 27733 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
I20260709 12:24:07.079901  9109 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:07.108649  9103 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.109357  9103 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:07.122643 27701 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:07.123617 27704 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.124087 27704 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.124935  9103 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.125537  9131 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:07.128062  9027 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.129268  9027 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.133716 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:07.134671  9027 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:07.140028  9027 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.143576  9109 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.144207  9109 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.141314  9027 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:07.147955 27701 leader_election.cc:336] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.149791 27701 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 4aa1523b9af84c3db99ba394e2b9aa9a, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.152851  9103 raft_consensus.cc:2749] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:24:07.154439  9027 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:07.162012 27704 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.163549  9109 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:07.164173 28292 leader_election.cc:336] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.167718 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:07.168447 27733 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 42.
I20260709 12:24:07.172336 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.175395  9109 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:24:07.175513 27701 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.176478 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.177346  9027 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:24:07.175956  9109 raft_consensus.cc:493] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.178580  9109 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:24:07.189800  9130 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.190413  9130 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.191617  9027 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.192196  9027 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.192626  9130 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:07.195428 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:07.196056 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 48.
W20260709 12:24:07.201960 27701 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.206449 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:07.207499  9130 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:24:07.208504  9130 raft_consensus.cc:493] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.209203  9130 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:24:07.210100  9027 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:07.210687 27704 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.211474 27704 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.215956  9027 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:07.228363 28292 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.186786  9109 raft_consensus.cc:515] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.235929  9109 leader_election.cc:290] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:07.241850  9027 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.248270  9027 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.250892  9027 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.225425  9130 raft_consensus.cc:515] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.244118 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "723b2024e4414796ba50ece654596156" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 43 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:07.252347 27733 raft_consensus.cc:3060] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:24:07.261005 27701 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.261145 27733 raft_consensus.cc:2468] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 43.
W20260709 12:24:07.262180 27704 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.262723 27704 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.264078  9027 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:24:07.264083  9130 leader_election.cc:290] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:07.275592 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73b2d9da2f874f958d24ff32c2097c0a" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:07.276750 27733 raft_consensus.cc:3060] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:24:07.298789 28292 leader_election.cc:304] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:07.299904  9130 raft_consensus.cc:2804] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:24:07.300500  9130 raft_consensus.cc:697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:24:07.301535  9130 consensus_queue.cc:237] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 42.27, Last appended by leader: 27, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.312974 27733 raft_consensus.cc:2468] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 49.
W20260709 12:24:07.316480 28292 leader_election.cc:336] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.323452 28292 leader_election.cc:304] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:07.325803  9109 raft_consensus.cc:2804] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:24:07.326433  9109 raft_consensus.cc:697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 49 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:24:07.327455  9109 consensus_queue.cc:237] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 48.32, Last appended by leader: 32, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.332357  9103 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.333082  9103 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.336061  9103 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:07.343523 28292 leader_election.cc:336] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.351742  9109 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.352500  9109 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.357483  9109 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:07.360674 27704 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.367619 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:07.369546 27733 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 35.
W20260709 12:24:07.371800 27701 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.373113 28292 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 
I20260709 12:24:07.373324  9027 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.379660  9027 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.379305 27483 catalog_manager.cc:5697] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 42 to 43, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 43 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:07.385921  9027 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.387225  9130 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:24:07.387764  9130 raft_consensus.cc:493] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.388242  9130 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:24:07.405978 28292 leader_election.cc:336] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.411707 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:07.428419 27701 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62836us
I20260709 12:24:07.429293  9027 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:07.426160  9130 raft_consensus.cc:515] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Starting leader election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.461272  9130 leader_election.cc:290] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Requested vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:07.524922 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 62745.1us
W20260709 12:24:07.529852 27704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 82541us
I20260709 12:24:07.530885  9027 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.532354  9027 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.539119  9027 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:07.551509 28292 leader_election.cc:336] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.572461 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "073b12eba02f4990a4fbe7080d8b8aff" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964"
I20260709 12:24:07.573906 27733 raft_consensus.cc:3060] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:24:07.614135 27704 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.614970  9142 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.616657  9142 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.619889  9142 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.615280 27483 catalog_manager.cc:5697] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 48 to 49, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 49 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:07.623191 27733 raft_consensus.cc:2468] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 36.
W20260709 12:24:07.625357 28292 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.627035 27704 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.629077  9103 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.629922  9103 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:07.632630  9103 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.635530  9130 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.636289  9130 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:07.644721 28295 leader_election.cc:336] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:07.646134 27701 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.646840 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.647873  9027 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:07.657840 27704 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:07.662606  9109 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.671749  9109 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.674530  9109 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:07.668615 28292 leader_election.cc:336] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.682607 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:07.689553 28295 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:07.690577  9109 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:24:07.670176 27701 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.695508 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.670701  9130 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.701859  9144 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:24:07.698902 27704 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:07.709152 27704 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:07.700325 28292 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:07.719349 28292 leader_election.cc:304] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.721791  9109 raft_consensus.cc:2804] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:24:07.722362 26408 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 27685)
I20260709 12:24:07.722397  9109 raft_consensus.cc:697] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Becoming Leader. State: Replica: 4aa1523b9af84c3db99ba394e2b9aa9a, State: Running, Role: LEADER
I20260709 12:24:07.724697  9109 consensus_queue.cc:237] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:07.725253  9027 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:07.726172  9027 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:07.728302 27701 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:07.728965  9027 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:07.762550 27483 catalog_manager.cc:5697] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a reported cstate change: term changed from 35 to 36, leader changed from 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2) to 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4). New cstate: current_term: 36 leader_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:07.758255 28292 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.779749 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.781934  9109 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:07.783207 27704 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:07.795570 28292 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.796324 28292 consensus_peers.cc:597] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:07.804702 27701 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.805289 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.811954  9144 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:24:07.813292 27701 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.813814 27701 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.814229 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.815030  9144 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:24:07.815420 28292 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.816947 27701 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.832934 27701 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:07.835264 27471 catalog_manager.cc:4755] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): ChangeConfig:REMOVE_PEER RPC failed for tablet 83433b8ed26448aca1d62a3b80ba4198: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:07.842363 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:07.843430  9144 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:24:07.864665 28292 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.875689 28292 consensus_peers.cc:597] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:07.878790 27701 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.879659 27701 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.880205 27701 consensus_peers.cc:597] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.882846 27701 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:07.887701 27732 raft_consensus.cc:1275] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 42 index: 27. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:24:07.889365  9109 consensus_queue.cc:1048] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:24:07.933079 27733 raft_consensus.cc:1275] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 49 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 48 index: 32. Preceding OpId from leader: term: 49 index: 33. (index mismatch)
I20260709 12:24:07.935751  9130 consensus_queue.cc:1048] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
W20260709 12:24:07.955674 28292 consensus_peers.cc:597] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.963614 28292 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:07.985101 28292 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.031219 27701 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.033815 28292 consensus_peers.cc:597] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.061551 27701 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.079676 28295 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.083250 27701 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.091436 27701 consensus_peers.cc:597] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.092090 27701 consensus_peers.cc:597] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.096094 27701 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.096674 27701 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.125641 27701 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.167570 27704 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.197755 27701 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.198388 27701 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.241827 28292 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:08.261010 27701 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:08.276111 27733 raft_consensus.cc:1275] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Refusing update from remote peer 4aa1523b9af84c3db99ba394e2b9aa9a: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:24:08.279188  9130 consensus_queue.cc:1048] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:24:08.291390 27704 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.294531 27704 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.315713 27701 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:08.372396 27704 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.392403 27704 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.401165 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:08.488653 27704 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:08.626132 27704 consensus_peers.cc:597] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:08.644469 27704 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:08.683167  9133 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.684015  9133 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:08.696131 27704 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:08.696265 28295 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:08.697192  9133 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:08.697654 28292 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.698189 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.699095  9133 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:08.722958 26408 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 27685)
I20260709 12:24:08.746944  9144 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.747746  9144 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:08.761544 27704 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:08.765753 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:08.766659 28335 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 45.
W20260709 12:24:08.775353 27701 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.781888  9144 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
W20260709 12:24:08.796000 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:08.775842 27701 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.801718  9144 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:08.802227 27704 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:08.821540 27704 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:08.838132  9130 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.842661  9144 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.843451  9144 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:08.843777  9130 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:08.845633 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:08.845876  9144 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:08.846298  9130 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:08.851977  9142 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.852964  9142 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:08.854050 27704 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:08.855084  9027 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.855749  9027 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:08.857900  9027 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:08.858446  9142 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:08.861259 28295 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:08.867029 28292 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.868050 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:08.868711 27701 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.869966 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.871277  9142 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:24:08.874573 27704 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:08.875494 27704 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:08.887854 27701 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.888378 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:08.889354 27701 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.890566 27701 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.889986  9142 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:08.891258  9130 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:08.894577  9144 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:24:08.904800 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:08.947050  9142 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.947844  9142 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:08.952530 28335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" is_pre_election: true
I20260709 12:24:08.953333 28335 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce69a3f9261a47a7a22f02f9c1f07964 in term 33.
W20260709 12:24:08.955106 27704 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:08.960610 27701 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.960709  9142 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), 4aa1523b9af84c3db99ba394e2b9aa9a (127.25.202.4:39321)
I20260709 12:24:08.961169 27701 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.962003  9142 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:08.962620  9144 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.963260  9027 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.963415  9144 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:08.963894  9027 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:08.965476 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:08.965726  9144 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:08.965835  9027 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:08.973803 27704 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:08.974434 27704 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:08.986637 27701 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.987229 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.988154  9027 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:08.988336 27701 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:08.988955 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:08.989753  9027 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:24:08.991737  9142 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:08.992268  9142 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:08.994489  9142 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:08.997452 27704 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:08.999953 27704 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:09.016953 27704 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:09.022761 27701 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.023398 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.024279  9027 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:24:09.096989  9142 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.097679  9142 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:09.100301  9142 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.110436 27704 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:09.114313 27701 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.115425 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.116729  9142 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:09.117602 27704 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:09.129870 27704 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:09.172047 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:09.172832 27704 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:09.174079  9142 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.174652  9142 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:09.177002  9142 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.188984 27704 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:09.190135 27701 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.190670 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.191952  9142 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:09.208717 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:09.271462 28295 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:09.316378 28295 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:09.386046 27704 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:09.425529 27704 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:09.521301  9027 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.521858  9027 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:09.524443  9027 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.530624 27701 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:09.531131 27704 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:09.531149  9027 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.531716 27704 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.533547  9144 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:24:09.533183  9027 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:09.541780  9027 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:09.546423  9144 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.547163  9144 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:09.547741 27704 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:09.549654  9144 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.555233 27704 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:09.567678 27701 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.568152 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.569204  9144 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:24:09.573369 27701 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.573948 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.574802  9027 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:09.623662 28295 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:09.699388  9027 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.700083  9027 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:09.702236  9027 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.712102 27704 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:09.713196 27701 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.713629 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.715183  9144 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:09.721381  9027 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.721958  9027 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:09.723624 26408 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 27685)
I20260709 12:24:09.724938  9027 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.729718 27704 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:09.731612  9027 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.732106  9027 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:09.733981  9027 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.741575 27701 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.743738 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:09.743191 27704 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:09.745059  9027 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:24:09.753777  9154 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.754351  9154 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:09.754949 27701 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:09.755505 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:09.756630  9154 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:09.760169  9027 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:24:09.762166 28295 leader_election.cc:336] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:09.759997  9154 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:09.764468  9154 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:09.771741 28295 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:09.772554  9154 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:09.875869 27474 catalog_manager.cc:4755] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 073b12eba02f4990a4fbe7080d8b8aff: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:09.958421 27701 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:09.978211 27701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111) [suppressed 701 similar messages]
W20260709 12:24:10.081663 28292 leader_election.cc:336] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:10.082855 28292 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:10.091631  9154 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:24:10.092980 28292 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:10.106004 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:10.113356  9154 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:10.146416 27701 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.183187 28292 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.247956 27704 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.318686 27701 consensus_peers.cc:597] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.367455 27701 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:10.407012  9144 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:10.407716  9144 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:10.418884  9144 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:10.425700 27701 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:10.426595 27701 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:10.493088  9154 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:10.493848  9154 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:10.497622  9154 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:10.500227 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:10.500979 27733 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
W20260709 12:24:10.503587 27704 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:10.506186 27701 consensus_peers.cc:597] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:10.510591 27704 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:10.532006 28295 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:10.535365  9027 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:24:10.542883 28292 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:10.538162 27701 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:10.554554  9027 raft_consensus.cc:493] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:10.555191  9027 raft_consensus.cc:515] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:10.561563  9027 leader_election.cc:290] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:10.552917 28292 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:10.582080  9154 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:10.583734 28292 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 50517.3us
W20260709 12:24:10.605247 27701 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.627592 27701 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:10.643376  9154 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:10.647419  9154 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:10.653527  9154 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:10.661293 27701 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.664290 27701 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.665139 28295 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:10.662963 27704 leader_election.cc:336] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:10.672302 27704 leader_election.cc:304] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:10.676568  9144 raft_consensus.cc:2749] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:24:10.677913 27704 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:10.686805  9144 raft_consensus.cc:493] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:10.687500  9144 raft_consensus.cc:515] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:10.689847  9144 leader_election.cc:290] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:10.706205 28292 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.686192 27701 consensus_peers.cc:597] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.724069 26408 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 27685)
W20260709 12:24:10.729879 27701 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.747072 28292 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.750676 27704 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:10.751248  9027 raft_consensus.cc:493] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:10.757936  9027 raft_consensus.cc:515] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:10.760167  9027 leader_election.cc:290] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:10.757035 28292 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:10.762804 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
W20260709 12:24:10.762321 27704 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:10.768167  9130 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:10.779927 27701 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.786494 27701 leader_election.cc:336] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:10.798468 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:24:10.800240 27701 leader_election.cc:304] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:10.807260  9027 raft_consensus.cc:2749] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:10.802588 27704 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:10.847584 28292 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.857960 28295 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.873957 27701 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.876102 27701 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.886065 27701 leader_election.cc:336] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:10.888454 27701 leader_election.cc:304] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:10.893396  9027 raft_consensus.cc:2749] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:24:10.911202 27704 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.916275 27704 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.923657 28292 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.929134 27704 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.933578 27701 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.938230 27701 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.939838 27701 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.952778 28292 consensus_peers.cc:597] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:10.954824 28292 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:10.958554 27704 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.961650 27704 consensus_peers.cc:597] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:10.976099 27701 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:11.053730 27704 consensus_peers.cc:597] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.124591 28292 consensus_peers.cc:597] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:24:11.154239  9130 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:11.155018  9130 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:11.162950 28295 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:11.172813  9130 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:11.182214 27733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6d40162b2bb4b61adc69c9fa3f03326" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:11.182957 27733 raft_consensus.cc:2468] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 33.
W20260709 12:24:11.207923 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.216148 28292 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:11.222075 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:11.224530  9130 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:24:11.218726 27704 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.235692 28292 consensus_peers.cc:597] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:11.316078 27704 consensus_peers.cc:597] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.362578 28292 consensus_peers.cc:597] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:11.373193 27704 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.391716 27704 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.435741 28292 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:11.453621 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.487231 27704 consensus_peers.cc:597] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:11.565024 28292 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:24:11.569216 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:24:11.578114  9154 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:11.578826  9154 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:11.585011  9154 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:11.590803 28295 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:11.595021 28292 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:11.596124 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:11.604380  9154 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:11.614135 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.687757 28295 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:11.688784 27704 consensus_peers.cc:597] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.724638 26408 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 27685)
W20260709 12:24:11.730007 27704 consensus_peers.cc:597] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:24:11.771775  9027 raft_consensus.cc:493] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:11.772388  9027 raft_consensus.cc:515] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:11.774505  9027 leader_election.cc:290] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:11.797338 27704 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:11.803664 27701 leader_election.cc:336] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:11.806278 27701 leader_election.cc:304] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:11.808460  9027 raft_consensus.cc:2749] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:11.867503 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:24:11.868251  9144 raft_consensus.cc:493] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:11.894536  9144 raft_consensus.cc:515] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:11.896703  9144 leader_election.cc:290] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:11.906044 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:11.909242 27704 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:24:11.916733  9027 raft_consensus.cc:493] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:11.917394  9027 raft_consensus.cc:515] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:11.921375  9027 leader_election.cc:290] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:11.930053 27704 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:11.933830 27701 leader_election.cc:336] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:11.935904 27701 leader_election.cc:304] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:11.939013  9027 raft_consensus.cc:2749] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:24:11.938265 27704 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:11.949868 27701 leader_election.cc:336] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:11.951543 27701 leader_election.cc:304] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:11.954440  9027 raft_consensus.cc:2749] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:24:11.962654 28295 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:11.994375 27704 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:12.004921  9027 raft_consensus.cc:493] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.005435  9027 raft_consensus.cc:515] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:12.016223  9027 leader_election.cc:290] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.032140 27704 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.043681 27701 leader_election.cc:336] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.044337 27701 leader_election.cc:304] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.045841  9144 raft_consensus.cc:2749] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:24:12.190459 28295 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:12.194242  9144 raft_consensus.cc:493] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.194890  9144 raft_consensus.cc:515] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:12.198217  9144 leader_election.cc:290] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.202459 27704 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.205437 27701 leader_election.cc:336] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.205986 27701 leader_election.cc:304] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.207285  9144 raft_consensus.cc:2749] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:24:12.245718 27704 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:24:12.363186  9144 raft_consensus.cc:493] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.373242  9144 raft_consensus.cc:515] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:12.377182  9144 leader_election.cc:290] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.387465 28295 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:41657, user_credentials={real_user=slave}} blocked reactor thread for 100360us
W20260709 12:24:12.408411 27704 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.413455 27701 leader_election.cc:336] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.416584 27701 leader_election.cc:304] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.417227  9144 raft_consensus.cc:493] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.425565  9144 raft_consensus.cc:515] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:12.427763  9144 leader_election.cc:290] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:12.428380  9027 raft_consensus.cc:2749] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:12.441098 27704 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.444413 27701 leader_election.cc:336] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.445003 27701 leader_election.cc:304] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.445855  9144 raft_consensus.cc:2749] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:12.490106  9154 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.490908  9154 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:12.494681  9154 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:12.496381 27732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83433b8ed26448aca1d62a3b80ba4198" candidate_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" is_pre_election: true
I20260709 12:24:12.497074 27732 raft_consensus.cc:2468] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa1523b9af84c3db99ba394e2b9aa9a in term 45.
W20260709 12:24:12.502596 28292 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:12.508646 28295 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:12.509155 28295 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a, ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.513715  9154 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:24:12.551240  9027 raft_consensus.cc:493] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.551990  9154 raft_consensus.cc:493] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.551791  9027 raft_consensus.cc:515] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:12.552546  9154 raft_consensus.cc:515] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:12.554133  9027 leader_election.cc:290] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
I20260709 12:24:12.554533  9154 leader_election.cc:290] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.557721 28292 leader_election.cc:336] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:12.566181 28295 leader_election.cc:336] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.566350 27704 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:12.566707 28295 leader_election.cc:304] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.567526  9154 raft_consensus.cc:2749] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:24:12.571753 27701 leader_election.cc:336] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.572281 27701 leader_election.cc:304] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.573063  9027 raft_consensus.cc:2749] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:12.659464 27701 consensus_peers.cc:597] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:12.674896 27701 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:12.726001 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
I20260709 12:24:12.727476  9027 raft_consensus.cc:493] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.728091  9027 raft_consensus.cc:515] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
W20260709 12:24:12.736553 27704 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:12.738752  9027 leader_election.cc:290] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.746721 27701 consensus_peers.cc:597] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:12.789521 27701 leader_election.cc:336] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.794695 27701 leader_election.cc:304] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.798429  9144 raft_consensus.cc:2749] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:24:12.799126  9144 raft_consensus.cc:493] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.799618  9144 raft_consensus.cc:515] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:12.801784  9144 leader_election.cc:290] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.825105 28292 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:12.817565 27704 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.841631 27701 consensus_peers.cc:597] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:12.842398 27701 leader_election.cc:336] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.842991 27701 leader_election.cc:304] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.843912  9027 raft_consensus.cc:2749] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:24:12.845973  9159 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:12.846586  9159 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:12.850884 27701 consensus_peers.cc:597] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:12.853993 28295 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:12.855904 27701 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:12.861449  9159 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:12.931921 28292 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:12.932598 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:12.933975  9154 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:24:12.992753 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
W20260709 12:24:13.045753 27701 consensus_peers.cc:597] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.048043  9144 raft_consensus.cc:493] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:13.048589  9144 raft_consensus.cc:515] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } }
I20260709 12:24:13.050644  9144 leader_election.cc:290] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:13.077936 27701 consensus_peers.cc:597] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.078662 27701 consensus_peers.cc:597] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.102210 27704 consensus_peers.cc:597] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:13.102972 27704 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:13.103832 27704 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:13.116597 28292 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.117283 28292 consensus_peers.cc:597] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.122948 27701 leader_election.cc:336] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:13.123553 27701 leader_election.cc:304] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce69a3f9261a47a7a22f02f9c1f07964; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:13.127219  9027 raft_consensus.cc:2749] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:24:13.128405 27701 consensus_peers.cc:597] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.141034 28292 consensus_peers.cc:597] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.142769 28292 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.142832 27704 consensus_peers.cc:597] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:13.143707 28292 consensus_peers.cc:597] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.144325 28292 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.159138 27701 consensus_peers.cc:597] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.164018 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:24:13.165120 26408 tablet_replica.cc:333] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.166117 26408 raft_consensus.cc:2243] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.167439 26408 raft_consensus.cc:2272] T 39fef9cf211c4b858a135faa1b00e1b1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.170552 26408 tablet_replica.cc:333] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.171572 26408 raft_consensus.cc:2243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 LEADER]: Raft consensus shutting down.
W20260709 12:24:13.173051 27701 consensus_peers.cc:597] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.173559 26408 raft_consensus.cc:2272] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.173542 28292 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:13.176779 26408 tablet_replica.cc:333] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.177748 26408 raft_consensus.cc:2243] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Raft consensus shutting down.
W20260709 12:24:13.177907 28292 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:13.178810 26408 raft_consensus.cc:2272] T 5865d310fddd49789e9de0a6a7bb260a P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.181500 26408 tablet_replica.cc:333] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.182543 26408 raft_consensus.cc:2243] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.183796 26408 raft_consensus.cc:2272] T fd03dfdd894243c3af75d463b6913332 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.187064 26408 tablet_replica.cc:333] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.187997 26408 raft_consensus.cc:2243] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.188689 26408 raft_consensus.cc:2272] T fe68c465ca4346b9b9db89a5be4e639c P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.188745 27701 consensus_peers.cc:597] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.189208 27704 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:13.192051 26408 tablet_replica.cc:333] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.193125 26408 raft_consensus.cc:2243] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.193943 26408 raft_consensus.cc:2272] T 5357eb349b4b4934a5b0d36f90884fc6 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.196390 27701 consensus_peers.cc:597] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.197256 26408 tablet_replica.cc:333] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.198271 26408 raft_consensus.cc:2243] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.199604 26408 raft_consensus.cc:2272] T 74155a9e0ffe48e5ac2f5dd6d0e8aac8 P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.202764 26408 tablet_replica.cc:333] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.203720 26408 raft_consensus.cc:2243] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.205072 26408 raft_consensus.cc:2272] T 83433b8ed26448aca1d62a3b80ba4198 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.208329 26408 tablet_replica.cc:333] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.209332 26408 raft_consensus.cc:2243] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.210101 26408 raft_consensus.cc:2272] T b8ac8041c87e49308784e1cad6aeec03 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.211200 27701 consensus_peers.cc:597] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.213286 26408 tablet_replica.cc:333] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.214208 26408 raft_consensus.cc:2243] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.215415 26408 raft_consensus.cc:2272] T 70329e70bb724d43a3185508e1186f4b P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.218266 26408 tablet_replica.cc:333] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.219187 26408 raft_consensus.cc:2243] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.220424 26408 raft_consensus.cc:2272] T 27b4972f682948f4981cb0dd1a74b4f5 P ce69a3f9261a47a7a22f02f9c1f07964 [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.223471 26408 tablet_replica.cc:333] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.224323 26408 raft_consensus.cc:2243] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.225574 26408 raft_consensus.cc:2272] T c596ddd50b894549b567a5b66ac1a31e P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.228561 26408 tablet_replica.cc:333] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.229419 26408 raft_consensus.cc:2243] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.230019 26408 raft_consensus.cc:2272] T 6f3a9a00f0db424982bf0ea90e7ae286 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.232683 26408 tablet_replica.cc:333] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.233513 26408 raft_consensus.cc:2243] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.234439 26408 raft_consensus.cc:2272] T d3d15932c0a64b668179d6868ab4707c P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.238718 26408 tablet_replica.cc:333] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.239459 26408 raft_consensus.cc:2243] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.239980 26408 raft_consensus.cc:2272] T 723b2024e4414796ba50ece654596156 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.242003 26408 tablet_replica.cc:333] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.242524 26408 raft_consensus.cc:2243] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 49 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.242973 26408 raft_consensus.cc:2272] T 73b2d9da2f874f958d24ff32c2097c0a P ce69a3f9261a47a7a22f02f9c1f07964 [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.245055 26408 tablet_replica.cc:333] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.245612 26408 raft_consensus.cc:2243] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.246342 26408 raft_consensus.cc:2272] T dad2444fb3474aee8e7dbbd7e673ab01 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.248602 26408 tablet_replica.cc:333] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.249192 26408 raft_consensus.cc:2243] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.250183 26408 raft_consensus.cc:2272] T bdb219c7e3524fc9b4a91a96610cee8b P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.252380 26408 tablet_replica.cc:333] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.252959 26408 raft_consensus.cc:2243] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.253855 26408 raft_consensus.cc:2272] T 4ab0ff10d53741dea50e33f971484854 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.256111 26408 tablet_replica.cc:333] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.256872 26408 raft_consensus.cc:2243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.257781 26408 raft_consensus.cc:2272] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P ce69a3f9261a47a7a22f02f9c1f07964 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.259639 28292 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:13.260535 26408 tablet_replica.cc:333] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.261272 26408 raft_consensus.cc:2243] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.262418 26408 raft_consensus.cc:2272] T 073b12eba02f4990a4fbe7080d8b8aff P ce69a3f9261a47a7a22f02f9c1f07964 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.265028 26408 tablet_replica.cc:333] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
W20260709 12:24:13.265057 27701 consensus_peers.cc:597] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.265749 26408 raft_consensus.cc:2243] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.266420 26408 raft_consensus.cc:2272] T 22597fcae204491f84839e680c083ecc P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.268543 26408 tablet_replica.cc:333] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.269093 26408 raft_consensus.cc:2243] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.269637 26408 raft_consensus.cc:2272] T 8c52fdc88d8646dd8e6584f8dbd137a3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.271649 26408 tablet_replica.cc:333] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.272172 26408 raft_consensus.cc:2243] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.273123 26408 raft_consensus.cc:2272] T 7c204963842941859d4313114fd92c93 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.275624 26408 tablet_replica.cc:333] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.276397 26408 raft_consensus.cc:2243] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.277094 26408 raft_consensus.cc:2272] T c3f34c916e524bfa9fcae2044da089fc P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.279224 26408 tablet_replica.cc:333] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.279830 26408 raft_consensus.cc:2243] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.280361 26408 raft_consensus.cc:2272] T 42e7d195f3f545b98a8dedb26c479c9c P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.282538 26408 tablet_replica.cc:333] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.283138 26408 raft_consensus.cc:2243] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.283895 26408 raft_consensus.cc:2272] T d2b187795e304ffd9c1293d922ff5b38 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.285840 26408 tablet_replica.cc:333] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.286343 26408 raft_consensus.cc:2243] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.287194 26408 raft_consensus.cc:2272] T a07754f58c3a421ba559f56d67d87bb4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.289057 26408 tablet_replica.cc:333] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.289606 26408 raft_consensus.cc:2243] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.290436 26408 raft_consensus.cc:2272] T 74b289dab6a94832ad11d8ac1e30d802 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.292570 26408 tablet_replica.cc:333] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.293090 26408 raft_consensus.cc:2243] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.294158 26408 raft_consensus.cc:2272] T b0b3d61cfc0d48f98b3cc7cc7cb63370 P ce69a3f9261a47a7a22f02f9c1f07964 [term 48 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.296027 27518 debug-util.cc:398] Leaking SignalData structure 0x7b080000b3c0 after lost signal to thread 26411
I20260709 12:24:13.296600 26408 tablet_replica.cc:333] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
W20260709 12:24:13.297247 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800326c20 after lost signal to thread 27521
I20260709 12:24:13.297387 26408 raft_consensus.cc:2243] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Raft consensus shutting down.
W20260709 12:24:13.298894 27518 debug-util.cc:398] Leaking SignalData structure 0x7b08003c5f80 after lost signal to thread 27762
I20260709 12:24:13.299096 26408 raft_consensus.cc:2272] T a6d40162b2bb4b61adc69c9fa3f03326 P ce69a3f9261a47a7a22f02f9c1f07964 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.301844 26408 tablet_replica.cc:333] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.302591 26408 raft_consensus.cc:2243] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.303715 26408 raft_consensus.cc:2272] T 1c3943cf144748fca8d2892751757c1a P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.304618 27518 debug-util.cc:398] Leaking SignalData structure 0x7b0800228f40 after lost signal to thread 28361
I20260709 12:24:13.306299 26408 tablet_replica.cc:333] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.307181 26408 raft_consensus.cc:2243] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.308112 26408 raft_consensus.cc:2272] T 0d80be98cdd043ffa8d122ba0ac29447 P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.311029 26408 tablet_replica.cc:333] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.311870 26408 raft_consensus.cc:2243] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.313160 26408 raft_consensus.cc:2272] T 310b1b37298244c0bd5bbe2b0c7328d7 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.316321 26408 tablet_replica.cc:333] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.317130 26408 raft_consensus.cc:2243] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.318389 26408 raft_consensus.cc:2272] T ecd16aab46634913b20f3501e9a6389f P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.321089 26408 tablet_replica.cc:333] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.321789 26408 raft_consensus.cc:2243] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.322717 26408 raft_consensus.cc:2272] T 6491d84db64f4d48b3b3d5be8aec385b P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.324849 28292 consensus_peers.cc:597] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:24:13.325651 26408 tablet_replica.cc:333] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.326278 26408 raft_consensus.cc:2243] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.326951 26408 raft_consensus.cc:2272] T 5f21db64000c434ca36b2b9021c13a44 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.329052 26408 tablet_replica.cc:333] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.329572 26408 raft_consensus.cc:2243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.330861 26408 raft_consensus.cc:2272] T 6b423a3b3ee1452fb419df4c9c0a8d67 P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.333061 26408 tablet_replica.cc:333] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.333585 26408 raft_consensus.cc:2243] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.334571 26408 raft_consensus.cc:2272] T b4e3fe6577354786a64bed5843d9a6a4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.337137 26408 tablet_replica.cc:333] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.337872 26408 raft_consensus.cc:2243] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.339025 26408 raft_consensus.cc:2272] T 456e2eac35d647a3bf9eeb45a3d7596f P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.341316 26408 tablet_replica.cc:333] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.341814 26408 raft_consensus.cc:2243] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.342725 26408 raft_consensus.cc:2272] T e1ffc143bb914f039c5b8bb615692aa3 P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.345261 26408 tablet_replica.cc:333] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.346025 26408 raft_consensus.cc:2243] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.347822 26408 raft_consensus.cc:2272] T 5accfdbc68cd4cf399c98e2a533f500e P ce69a3f9261a47a7a22f02f9c1f07964 [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.350174 26408 tablet_replica.cc:333] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.350749 26408 raft_consensus.cc:2243] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.351742 26408 raft_consensus.cc:2272] T 27d7343bc8174a5f89f4652b4401c3ec P ce69a3f9261a47a7a22f02f9c1f07964 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.353838 26408 tablet_replica.cc:333] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.354367 26408 raft_consensus.cc:2243] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.355077 26408 raft_consensus.cc:2272] T 68dbab4c1e1442b699465f7b3e728acd P ce69a3f9261a47a7a22f02f9c1f07964 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.357172 26408 tablet_replica.cc:333] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.357679 26408 raft_consensus.cc:2243] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.358711 26408 raft_consensus.cc:2272] T 22efa3bef6634e7a811baa3c42d9cd18 P ce69a3f9261a47a7a22f02f9c1f07964 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.360704 26408 tablet_replica.cc:333] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.361227 26408 raft_consensus.cc:2243] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.361796 26408 raft_consensus.cc:2272] T e76581eba9454a229c556095ac8b3e23 P ce69a3f9261a47a7a22f02f9c1f07964 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.363811 26408 tablet_replica.cc:333] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.364292 26408 raft_consensus.cc:2243] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.365079 26408 raft_consensus.cc:2272] T 70b48af2ec82404986e5e18d544db5b4 P ce69a3f9261a47a7a22f02f9c1f07964 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.367311 26408 tablet_replica.cc:333] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.367842 26408 raft_consensus.cc:2243] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:13.368680 26408 raft_consensus.cc:2272] T 37a0fdc57f6240869f0977e359f258c0 P ce69a3f9261a47a7a22f02f9c1f07964 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:13.370826 26408 tablet_replica.cc:333] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964: stopping tablet replica
I20260709 12:24:13.371285 26408 raft_consensus.cc:2243] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:13.371865 26408 raft_consensus.cc:2272] T 85b7862879764c33b1db83d8596e3e87 P ce69a3f9261a47a7a22f02f9c1f07964 [term 41 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:13.393934 28292 consensus_peers.cc:597] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.407022 28292 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:13.416781 28292 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.424013 28292 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.483008 28295 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:24:13.550046 28292 consensus_peers.cc:597] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Couldn't send request to peer ce69a3f9261a47a7a22f02f9c1f07964. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:13.560851  9159 raft_consensus.cc:493] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:13.561434  9159 raft_consensus.cc:515] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:13.564029  9159 leader_election.cc:290] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:13.570729 28295 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:24:13.575512 28295 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:13.584038 28292 leader_election.cc:336] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:13.584594 28292 leader_election.cc:304] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:13.585351  9159 raft_consensus.cc:2749] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:13.642503 28292 consensus_peers.cc:597] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:13.663169 28292 consensus_peers.cc:597] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.680181 28295 consensus_peers.cc:597] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:24:13.717135 28292 consensus_peers.cc:597] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:24:13.724612 28292 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.768002 28292 consensus_peers.cc:597] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.796783 28292 consensus_peers.cc:597] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:24:13.846779 28295 consensus_peers.cc:597] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:24:13.946435 28295 consensus_peers.cc:597] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:24:14.080725 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08000975a0 after lost signal to thread 26411
W20260709 12:24:14.082074 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb660 after lost signal to thread 27521
W20260709 12:24:14.083815 28358 debug-util.cc:398] Leaking SignalData structure 0x7b0800318ce0 after lost signal to thread 27762
W20260709 12:24:14.085243 28358 debug-util.cc:398] Leaking SignalData structure 0x7b08002357a0 after lost signal to thread 28361
W20260709 12:24:14.104017 27471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.25.202.2:33051, user_credentials={real_user=slave}} blocked reactor thread for 51163us
I20260709 12:24:14.111591  9154 raft_consensus.cc:493] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:14.112180  9154 raft_consensus.cc:515] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.114466  9154 leader_election.cc:290] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:14.125029 28295 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:14.130158 28295 consensus_peers.cc:597] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:24:14.139976 27471 catalog_manager.cc:4755] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): ChangeConfig:REMOVE_PEER RPC failed for tablet 83433b8ed26448aca1d62a3b80ba4198: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:14.147133 28292 leader_election.cc:336] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.148334 28292 leader_election.cc:304] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:14.149312  9154 raft_consensus.cc:2749] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:24:14.194253 28292 consensus_peers.cc:597] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Couldn't send request to peer 9c09719959a54e8eb66d2041dd9c3113. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:24:14.208196  9154 raft_consensus.cc:493] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.208969  9154 raft_consensus.cc:515] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:14.220988 28295 leader_election.cc:336] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:14.211063  9154 leader_election.cc:290] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.230697 28292 leader_election.cc:336] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
I20260709 12:24:14.234993 28292 leader_election.cc:304] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:14.237149  9154 raft_consensus.cc:2749] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:24:14.235783 28295 consensus_peers.cc:597] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:24:14.245592  9154 raft_consensus.cc:493] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:14.246279  9154 raft_consensus.cc:515] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
W20260709 12:24:14.254580 28295 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:14.259795  9159 raft_consensus.cc:493] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.260442  9159 raft_consensus.cc:515] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.263163  9159 leader_election.cc:290] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:14.265053  9154 leader_election.cc:290] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.279155 28295 leader_election.cc:336] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:14.286583 28292 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.287137 28292 leader_election.cc:304] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:14.287946  9154 raft_consensus.cc:2749] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:24:14.301949 28295 consensus_peers.cc:597] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:14.303304 28292 leader_election.cc:336] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.306293 28292 leader_election.cc:304] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:14.307147  9154 raft_consensus.cc:2749] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:24:14.314368 28292 leader_election.cc:336] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:14.315385 28292 leader_election.cc:336] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
I20260709 12:24:14.448647  9159 raft_consensus.cc:493] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:14.449282  9159 raft_consensus.cc:515] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.466519  9159 leader_election.cc:290] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051)
W20260709 12:24:14.469115 28295 consensus_peers.cc:597] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:24:14.469933 28295 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:14.482128 28292 leader_election.cc:336] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.482700 28292 leader_election.cc:304] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:14.483553  9159 raft_consensus.cc:2749] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:24:14.511971 28295 consensus_peers.cc:597] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:24:14.551605 28295 consensus_peers.cc:597] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a -> Peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Couldn't send request to peer 3f90188458da46f6ba138d5f82c55b69. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:24:14.563971  9159 raft_consensus.cc:493] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.564821  9159 raft_consensus.cc:515] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.573344  9159 leader_election.cc:290] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.581995 28292 leader_election.cc:336] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:14.586926 28295 leader_election.cc:336] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:14.590576  9159 raft_consensus.cc:493] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.591252  9159 raft_consensus.cc:515] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.595664  9159 leader_election.cc:290] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:14.598073 28295 leader_election.cc:304] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:14.602841  9154 raft_consensus.cc:2749] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:24:14.606868  9159 raft_consensus.cc:493] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9c09719959a54e8eb66d2041dd9c3113)
I20260709 12:24:14.609215  9159 raft_consensus.cc:515] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.615947  9154 raft_consensus.cc:493] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.616541  9154 raft_consensus.cc:515] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } attrs { replace: true } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.618937  9154 leader_election.cc:290] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.625078 28292 leader_election.cc:336] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:14.627090 28295 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:14.629961 28295 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:14.614070  9159 leader_election.cc:290] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.634187 28292 leader_election.cc:336] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.634728 28292 leader_election.cc:304] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:14.636621  9159 raft_consensus.cc:2749] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:24:14.642580 28292 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.643165 28292 leader_election.cc:304] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:14.644061  9159 raft_consensus.cc:2749] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:24:14.645166 28292 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.645666 28292 leader_election.cc:304] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, 9c09719959a54e8eb66d2041dd9c3113
I20260709 12:24:14.648275  9154 raft_consensus.cc:493] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.648847  9154 raft_consensus.cc:515] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.650848  9154 leader_election.cc:290] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
I20260709 12:24:14.647683  9159 raft_consensus.cc:2749] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:24:14.657137 28295 leader_election.cc:336] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
I20260709 12:24:14.659181  9159 raft_consensus.cc:493] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.659920  9159 raft_consensus.cc:515] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.662294  9159 leader_election.cc:290] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.671007 28292 leader_election.cc:336] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:14.671986 28292 leader_election.cc:336] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:14.678345 28292 leader_election.cc:336] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
I20260709 12:24:14.678862 28292 leader_election.cc:304] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 3f90188458da46f6ba138d5f82c55b69, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:14.679702  9159 raft_consensus.cc:2749] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:24:14.680812 28292 leader_election.cc:336] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:14.681666 28292 leader_election.cc:336] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:24:14.682107 28292 leader_election.cc:304] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:14.682917  9159 raft_consensus.cc:2749] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:24:14.777858  9159 raft_consensus.cc:493] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ce69a3f9261a47a7a22f02f9c1f07964)
I20260709 12:24:14.778435  9159 raft_consensus.cc:515] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: false } }
I20260709 12:24:14.780591  9159 leader_election.cc:290] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051), ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211)
W20260709 12:24:14.791534 28292 leader_election.cc:336] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:14.792315 28292 leader_election.cc:336] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
I20260709 12:24:14.792762 28292 leader_election.cc:304] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aa1523b9af84c3db99ba394e2b9aa9a; no voters: 9c09719959a54e8eb66d2041dd9c3113, ce69a3f9261a47a7a22f02f9c1f07964
I20260709 12:24:14.793563  9159 raft_consensus.cc:2749] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:24:14.888482 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
W20260709 12:24:14.979009 28292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111) [suppressed 600 similar messages]
I20260709 12:24:15.022758 26408 tablet_server.cc:179] TabletServer@127.25.202.4:0 shutting down...
I20260709 12:24:15.101524 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:24:15.102474 26408 tablet_replica.cc:333] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.103722 26408 raft_consensus.cc:2243] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.104621 26408 raft_consensus.cc:2272] T a6d40162b2bb4b61adc69c9fa3f03326 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.108111 26408 tablet_replica.cc:333] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.108830 26408 raft_consensus.cc:2243] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.109911 26408 raft_consensus.cc:2272] T 8355fe4a548a43f8b5215ea9ae7ba8b2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.112524 26408 tablet_replica.cc:333] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.113222 26408 raft_consensus.cc:2243] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.113790 26408 raft_consensus.cc:2272] T 6b423a3b3ee1452fb419df4c9c0a8d67 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.116653 26408 tablet_replica.cc:333] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.117511 26408 raft_consensus.cc:2243] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEARNER]: Raft consensus shutting down.
I20260709 12:24:15.118058 26408 pending_rounds.cc:70] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Trying to abort 1 pending ops.
I20260709 12:24:15.118362 26408 pending_rounds.cc:77] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a: Aborting op as it isn't in flight: id { term: 4 index: 4 } timestamp: 7305623656419205120 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c596ddd50b894549b567a5b66ac1a31e" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } } new_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3f90188458da46f6ba138d5f82c55b69" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 41657 } } peers { permanent_uuid: "9c09719959a54e8eb66d2041dd9c3113" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 33051 } attrs { replace: true } } peers { permanent_uuid: "ce69a3f9261a47a7a22f02f9c1f07964" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 41211 } } peers { permanent_uuid: "4aa1523b9af84c3db99ba394e2b9aa9a" member_type: NON_VOTER last_known_addr { host: "127.25.202.4" port: 39321 } attrs { promote: true } } } }
I20260709 12:24:15.119624 26408 raft_consensus.cc:2917] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 LEARNER]: Aborting config change with OpId 4.4: Aborted: Op aborted
I20260709 12:24:15.120114 26408 raft_consensus.cc:2889] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 12:24:15.120512 26408 raft_consensus.cc:2272] T c596ddd50b894549b567a5b66ac1a31e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 4 NON_PARTICIPANT]: Raft consensus is shut down!
I20260709 12:24:15.123621 26408 tablet_replica.cc:333] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.124226 26408 raft_consensus.cc:2243] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.124886 26408 raft_consensus.cc:2272] T 81e6b2dabf8f4e6cb549ca250430e568 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.127228 26408 tablet_replica.cc:333] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.127852 26408 raft_consensus.cc:2243] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.128829 26408 raft_consensus.cc:2272] T e634fbff5fb24e2fa133b95599b0db1f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.130968 26408 tablet_replica.cc:333] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.131652 26408 raft_consensus.cc:2243] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.132174 26408 raft_consensus.cc:2272] T 310b1b37298244c0bd5bbe2b0c7328d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.134334 26408 tablet_replica.cc:333] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.134991 26408 raft_consensus.cc:2243] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.135617 26408 raft_consensus.cc:2272] T 1c3943cf144748fca8d2892751757c1a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.137848 26408 tablet_replica.cc:333] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.138499 26408 raft_consensus.cc:2243] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.139420 26408 raft_consensus.cc:2272] T 22597fcae204491f84839e680c083ecc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.142021 26408 tablet_replica.cc:333] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.142627 26408 raft_consensus.cc:2243] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.143188 26408 raft_consensus.cc:2272] T 83433b8ed26448aca1d62a3b80ba4198 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.145651 26408 tablet_replica.cc:333] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.146304 26408 raft_consensus.cc:2243] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.147449 26408 raft_consensus.cc:2272] T 5865d310fddd49789e9de0a6a7bb260a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.149947 26408 tablet_replica.cc:333] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.150552 26408 raft_consensus.cc:2243] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.151614 26408 raft_consensus.cc:2272] T dbdcf17dc981450f989b93d262e9a5d0 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.153890 26408 tablet_replica.cc:333] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.154487 26408 raft_consensus.cc:2243] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.155570 26408 raft_consensus.cc:2272] T 5357eb349b4b4934a5b0d36f90884fc6 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.158145 26408 tablet_replica.cc:333] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.159085 26408 raft_consensus.cc:2243] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.159873 26408 raft_consensus.cc:2272] T 31e76c3cbbc042dcbbbfe8b1ab6948e1 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.162793 26408 tablet_replica.cc:333] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.163648 26408 raft_consensus.cc:2243] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.164935 26408 raft_consensus.cc:2272] T 7efb19c4fe844c2ea043bd3638faae4e P 4aa1523b9af84c3db99ba394e2b9aa9a [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.167995 26408 tablet_replica.cc:333] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.168844 26408 raft_consensus.cc:2243] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.169566 26408 raft_consensus.cc:2272] T e1ffc143bb914f039c5b8bb615692aa3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.171917 26408 tablet_replica.cc:333] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.172622 26408 raft_consensus.cc:2243] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.173604 26408 raft_consensus.cc:2272] T 7274fd8a2115405c9b78c61214bff736 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.176052 26408 tablet_replica.cc:333] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.176661 26408 raft_consensus.cc:2243] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.177884 26408 raft_consensus.cc:2272] T 42e7d195f3f545b98a8dedb26c479c9c P 4aa1523b9af84c3db99ba394e2b9aa9a [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.180131 26408 tablet_replica.cc:333] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.180703 26408 raft_consensus.cc:2243] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.181236 26408 raft_consensus.cc:2272] T 3b881b4e55d543f9ade316642ea1b7e3 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.183467 26408 tablet_replica.cc:333] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.184139 26408 raft_consensus.cc:2243] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.184628 26408 raft_consensus.cc:2272] T 4f392cf87198499d9c064d1ec232d3d7 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.186721 26408 tablet_replica.cc:333] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.187296 26408 raft_consensus.cc:2243] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.188376 26408 raft_consensus.cc:2272] T eb215cc72c384ec8b9421c81d953589b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.190486 26408 tablet_replica.cc:333] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.191066 26408 raft_consensus.cc:2243] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.191967 26408 raft_consensus.cc:2272] T 85b7862879764c33b1db83d8596e3e87 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.194476 26408 tablet_replica.cc:333] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.195271 26408 raft_consensus.cc:2243] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.196496 26408 raft_consensus.cc:2272] T 73b2d9da2f874f958d24ff32c2097c0a P 4aa1523b9af84c3db99ba394e2b9aa9a [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.199268 26408 tablet_replica.cc:333] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.200129 26408 raft_consensus.cc:2243] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.201109 26408 raft_consensus.cc:2272] T 62a7e798fd5e41e0bb2d9f5c7dab1adc P 4aa1523b9af84c3db99ba394e2b9aa9a [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.203387 26408 tablet_replica.cc:333] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.203960 26408 raft_consensus.cc:2243] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.204939 26408 raft_consensus.cc:2272] T 723b2024e4414796ba50ece654596156 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.207072 26408 tablet_replica.cc:333] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.207671 26408 raft_consensus.cc:2243] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.208745 26408 raft_consensus.cc:2272] T 4811ee81866e469abca31f212c08ace2 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.211024 26408 tablet_replica.cc:333] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.211771 26408 raft_consensus.cc:2243] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.212318 26408 raft_consensus.cc:2272] T fd03dfdd894243c3af75d463b6913332 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.214401 26408 tablet_replica.cc:333] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.214958 26408 raft_consensus.cc:2243] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.215999 26408 raft_consensus.cc:2272] T 6491d84db64f4d48b3b3d5be8aec385b P 4aa1523b9af84c3db99ba394e2b9aa9a [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.218310 26408 tablet_replica.cc:333] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.218873 26408 raft_consensus.cc:2243] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.219893 26408 raft_consensus.cc:2272] T 74b289dab6a94832ad11d8ac1e30d802 P 4aa1523b9af84c3db99ba394e2b9aa9a [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.222106 26408 tablet_replica.cc:333] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.222679 26408 raft_consensus.cc:2243] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:24:15.223830 26408 raft_consensus.cc:2272] T 073b12eba02f4990a4fbe7080d8b8aff P 4aa1523b9af84c3db99ba394e2b9aa9a [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.226222 26408 tablet_replica.cc:333] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.226761 26408 raft_consensus.cc:2243] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.227492 26408 raft_consensus.cc:2272] T 456e2eac35d647a3bf9eeb45a3d7596f P 4aa1523b9af84c3db99ba394e2b9aa9a [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.229712 26408 tablet_replica.cc:333] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a: stopping tablet replica
I20260709 12:24:15.230201 26408 raft_consensus.cc:2243] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:15.230685 26408 raft_consensus.cc:2272] T 27d7343bc8174a5f89f4652b4401c3ec P 4aa1523b9af84c3db99ba394e2b9aa9a [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:15.935530 26408 tablet_server.cc:196] TabletServer@127.25.202.4:0 shutdown complete.
I20260709 12:24:15.998569 26408 master.cc:562] Master@127.25.202.62:43715 shutting down...
W20260709 12:24:16.035375 27535 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:24:16.035714 27535 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:24:16.035950 27535 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:24:18.106650 27474 catalog_manager.cc:4755] TS 3f90188458da46f6ba138d5f82c55b69 (127.25.202.1:41657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 073b12eba02f4990a4fbe7080d8b8aff: Network error: Client connection negotiation failed: client connection to 127.25.202.1:41657: connect: Connection refused (error 111)
W20260709 12:24:22.384584 27471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260709 12:24:22.387836 27471 catalog_manager.cc:4755] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): ChangeConfig:REMOVE_PEER RPC failed for tablet 83433b8ed26448aca1d62a3b80ba4198: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
W20260709 12:24:55.286068 27471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:55.289865 27471 catalog_manager.cc:4755] TS ce69a3f9261a47a7a22f02f9c1f07964 (127.25.202.3:41211): ChangeConfig:REMOVE_PEER RPC failed for tablet 6b423a3b3ee1452fb419df4c9c0a8d67: Network error: Client connection negotiation failed: client connection to 127.25.202.3:41211: connect: Connection refused (error 111)
W20260709 12:24:59.249800 27471 catalog_manager.cc:4755] TS 9c09719959a54e8eb66d2041dd9c3113 (127.25.202.2:33051): ChangeConfig:REMOVE_PEER RPC failed for tablet a6d40162b2bb4b61adc69c9fa3f03326: Network error: Client connection negotiation failed: client connection to 127.25.202.2:33051: connect: Connection refused (error 111)
I20260709 12:25:07.497556 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:07.499797 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:07.500306 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P 04e0ca717d6b41608e4d35acaee1cac8: stopping tablet replica
I20260709 12:25:07.731238 26408 master.cc:584] Master@127.25.202.62:43715 shutdown complete.
I20260709 12:25:07.786727 26408 test_util.cc:182] -----------------------------------------------
I20260709 12:25:07.786950 26408 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599387827064-26408-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (458265 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:25:07.790568 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:39287
I20260709 12:25:07.791823 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:07.798727  9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:07.798797  9180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:07.799731  9183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:07.800138 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:07.805115 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:07.805364 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:07.805524 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599907805509 us; error 0 us; skew 500 ppm
I20260709 12:25:07.806077 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:07.808773 26408 webserver.cc:533] Webserver started at http://127.25.202.62:44913/ using document root <none> and password file <none>
I20260709 12:25:07.809303 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:07.809504 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:07.809801 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:07.811187 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "3de727ba387e472ab4c4baca2c3bd906"
format_stamp: "Formatted at 2026-07-09 12:25:07 on dist-test-slave-m57n"
I20260709 12:25:07.815939 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:25:07.819406  9188 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:07.820245 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20260709 12:25:07.820591 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "3de727ba387e472ab4c4baca2c3bd906"
format_stamp: "Formatted at 2026-07-09 12:25:07 on dist-test-slave-m57n"
I20260709 12:25:07.820899 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:07.840369 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:07.841671 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:07.884850 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:39287
I20260709 12:25:07.884972  9239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:39287 every 8 connection(s)
I20260709 12:25:07.889173  9240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:07.901011  9240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906: Bootstrap starting.
I20260709 12:25:07.906056  9240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:07.910840  9240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906: No bootstrap required, opened a new log
I20260709 12:25:07.913323  9240 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de727ba387e472ab4c4baca2c3bd906" member_type: VOTER }
I20260709 12:25:07.913794  9240 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:07.914109  9240 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de727ba387e472ab4c4baca2c3bd906, State: Initialized, Role: FOLLOWER
I20260709 12:25:07.914736  9240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de727ba387e472ab4c4baca2c3bd906" member_type: VOTER }
I20260709 12:25:07.915217  9240 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:25:07.915472  9240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:25:07.915787  9240 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:07.921010  9240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de727ba387e472ab4c4baca2c3bd906" member_type: VOTER }
I20260709 12:25:07.921586  9240 leader_election.cc:304] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de727ba387e472ab4c4baca2c3bd906; no voters: 
I20260709 12:25:07.922896  9240 leader_election.cc:290] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:25:07.923259  9243 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:07.924641  9243 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 1 LEADER]: Becoming Leader. State: Replica: 3de727ba387e472ab4c4baca2c3bd906, State: Running, Role: LEADER
I20260709 12:25:07.925359  9243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de727ba387e472ab4c4baca2c3bd906" member_type: VOTER }
I20260709 12:25:07.926069  9240 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:25:07.933311  9245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de727ba387e472ab4c4baca2c3bd906. Latest consensus state: current_term: 1 leader_uuid: "3de727ba387e472ab4c4baca2c3bd906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de727ba387e472ab4c4baca2c3bd906" member_type: VOTER } }
I20260709 12:25:07.933516  9244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de727ba387e472ab4c4baca2c3bd906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de727ba387e472ab4c4baca2c3bd906" member_type: VOTER } }
I20260709 12:25:07.934193  9245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:07.934289  9244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:07.936514  9253 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:25:07.944510  9253 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:25:07.945556 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:25:07.954751  9253 catalog_manager.cc:1383] Generated new cluster ID: 7a03048dcde340f68bdb45c93882d8f7
I20260709 12:25:07.955067  9253 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:25:07.981653  9253 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:25:07.983017  9253 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:25:08.009542  9253 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906: Generated new TSK 0
I20260709 12:25:08.010334  9253 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:25:08.076651 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:08.083395  9261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:08.084419  9262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:08.085752  9264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:08.087445 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:08.090482 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:08.090691 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:08.090816 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599908090803 us; error 0 us; skew 500 ppm
I20260709 12:25:08.091388 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:08.094470 26408 webserver.cc:533] Webserver started at http://127.25.202.1:44039/ using document root <none> and password file <none>
I20260709 12:25:08.095013 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:08.095213 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:08.095706 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:08.097178 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "4e58af1026634b45bafbbfde02a6ecd0"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.102706 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 12:25:08.106954  9269 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.107859 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:25:08.108348 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "4e58af1026634b45bafbbfde02a6ecd0"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.108667 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:08.120848 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:08.122102 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:08.123733 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:08.126086 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:08.126283 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.126539 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:08.126719 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.169734 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:33407
I20260709 12:25:08.169889  9331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:33407 every 8 connection(s)
I20260709 12:25:08.175232 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:08.183243  9336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:08.183995  9337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:08.187988 26408 server_base.cc:1061] running on GCE node
W20260709 12:25:08.189088  9340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:08.191084  9332 heartbeater.cc:344] Connected to a master server at 127.25.202.62:39287
I20260709 12:25:08.191560  9332 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:08.192363  9332 heartbeater.cc:507] Master 127.25.202.62:39287 requested a full tablet report, sending...
I20260709 12:25:08.192514 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:08.192871 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:08.193023 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599908193010 us; error 0 us; skew 500 ppm
I20260709 12:25:08.193874 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:08.195461  9205 ts_manager.cc:194] Registered new tserver with Master: 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:08.196718 26408 webserver.cc:533] Webserver started at http://127.25.202.2:45945/ using document root <none> and password file <none>
I20260709 12:25:08.197296 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:08.197561 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:08.197855 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:08.198307  9205 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50992
I20260709 12:25:08.199246 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.204722 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:25:08.208603  9344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.209486 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:25:08.209796 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.210139 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:08.221189 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:08.222470 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:08.224119 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:08.226500 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:08.226691 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.227001 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:08.227139 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.275523 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:42279
I20260709 12:25:08.275732  9406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:42279 every 8 connection(s)
I20260709 12:25:08.280521 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:08.289188  9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:08.290696  9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:08.294621  9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:08.295639  9407 heartbeater.cc:344] Connected to a master server at 127.25.202.62:39287
I20260709 12:25:08.295876 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:08.296226  9407 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:08.297127 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:08.297483 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:08.297470  9407 heartbeater.cc:507] Master 127.25.202.62:39287 requested a full tablet report, sending...
I20260709 12:25:08.297979 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599908297955 us; error 0 us; skew 500 ppm
I20260709 12:25:08.298909 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:08.300568  9205 ts_manager.cc:194] Registered new tserver with Master: e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:08.302057 26408 webserver.cc:533] Webserver started at http://127.25.202.3:36339/ using document root <none> and password file <none>
I20260709 12:25:08.302561 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:08.302560  9205 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51004
I20260709 12:25:08.303129 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:08.303469 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:08.304776 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "074a6ac642e4430ba8045181f0118ee3"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.309584 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:25:08.313736  9418 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.314610 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:25:08.314908 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "074a6ac642e4430ba8045181f0118ee3"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.315161 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:08.359558 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:08.361001 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:08.362646 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:08.367012 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:08.367233 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.367625 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:08.367836 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.417586 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:39369
I20260709 12:25:08.417678  9480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:39369 every 8 connection(s)
I20260709 12:25:08.424069 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:08.432402  9484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:08.433360  9485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:08.437291 26408 server_base.cc:1061] running on GCE node
W20260709 12:25:08.437094  9487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:08.438145  9481 heartbeater.cc:344] Connected to a master server at 127.25.202.62:39287
I20260709 12:25:08.438649  9481 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:08.439131 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:08.439419 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:08.439623 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599908439593 us; error 0 us; skew 500 ppm
I20260709 12:25:08.439630  9481 heartbeater.cc:507] Master 127.25.202.62:39287 requested a full tablet report, sending...
I20260709 12:25:08.440322 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:08.442246  9205 ts_manager.cc:194] Registered new tserver with Master: 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:08.443132 26408 webserver.cc:533] Webserver started at http://127.25.202.4:42687/ using document root <none> and password file <none>
I20260709 12:25:08.443815 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:08.444044 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:08.444201  9205 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51020
I20260709 12:25:08.444360 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:08.445930 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-3-root/instance:
uuid: "a5f89c82eda14a9ca425fae398bae986"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.450959 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.004s
I20260709 12:25:08.454615  9492 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.455561 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:25:08.455860 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-3-root
uuid: "a5f89c82eda14a9ca425fae398bae986"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:08.456174 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:08.470917 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:08.472244 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:08.473645 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:08.476135 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:08.476397 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.476670 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:08.476861 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:08.522239 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.4:39855
I20260709 12:25:08.522346  9554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.4:39855 every 8 connection(s)
I20260709 12:25:08.537899  9555 heartbeater.cc:344] Connected to a master server at 127.25.202.62:39287
I20260709 12:25:08.538431  9555 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:08.539237  9555 heartbeater.cc:507] Master 127.25.202.62:39287 requested a full tablet report, sending...
I20260709 12:25:08.541342  9205 ts_manager.cc:194] Registered new tserver with Master: a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:08.541810 26408 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.016205943s
I20260709 12:25:08.542790  9205 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51036
I20260709 12:25:08.570636  9205 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51038:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:25:08.843390  9297 tablet_service.cc:1511] Processing CreateTablet for tablet 8c302268ead84f94a04993f710b731e6 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:25:08.845016  9297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c302268ead84f94a04993f710b731e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.844776  9296 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7e7ca3cc2243559667954882dfa118 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:25:08.847476  9296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7e7ca3cc2243559667954882dfa118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.848480  9293 tablet_service.cc:1511] Processing CreateTablet for tablet 8a891dab57c74222928537e93876be86 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:25:08.850670  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a891dab57c74222928537e93876be86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.853358  9294 tablet_service.cc:1511] Processing CreateTablet for tablet a53d66736d1046bea06b3935c396942d (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:25:08.853471  9295 tablet_service.cc:1511] Processing CreateTablet for tablet 160804417e4443b9b25105c759111c8b (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:25:08.856118  9294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53d66736d1046bea06b3935c396942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.857110  9520 tablet_service.cc:1511] Processing CreateTablet for tablet 8c302268ead84f94a04993f710b731e6 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:25:08.858208  9295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160804417e4443b9b25105c759111c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.858566  9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c302268ead84f94a04993f710b731e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.860699  9519 tablet_service.cc:1511] Processing CreateTablet for tablet 160804417e4443b9b25105c759111c8b (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:25:08.862592  9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160804417e4443b9b25105c759111c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.865095  9518 tablet_service.cc:1511] Processing CreateTablet for tablet c26afa4025e24ea192d424c506f40c56 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:25:08.867295  9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c26afa4025e24ea192d424c506f40c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.882409  9446 tablet_service.cc:1511] Processing CreateTablet for tablet 8c302268ead84f94a04993f710b731e6 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:25:08.873227  9292 tablet_service.cc:1511] Processing CreateTablet for tablet ffb83e944aed43718284cb48f994efcc (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:25:08.884061  9446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c302268ead84f94a04993f710b731e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.884996  9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffb83e944aed43718284cb48f994efcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.886458  9445 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7e7ca3cc2243559667954882dfa118 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:25:08.869308  9517 tablet_service.cc:1511] Processing CreateTablet for tablet 8a891dab57c74222928537e93876be86 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:25:08.888933  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a891dab57c74222928537e93876be86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.889348  9444 tablet_service.cc:1511] Processing CreateTablet for tablet c26afa4025e24ea192d424c506f40c56 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:25:08.874966  9516 tablet_service.cc:1511] Processing CreateTablet for tablet ffb83e944aed43718284cb48f994efcc (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:25:08.891846  9443 tablet_service.cc:1511] Processing CreateTablet for tablet a53d66736d1046bea06b3935c396942d (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:25:08.893071  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffb83e944aed43718284cb48f994efcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.894316  9442 tablet_service.cc:1511] Processing CreateTablet for tablet ffb83e944aed43718284cb48f994efcc (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:25:08.877846  9515 tablet_service.cc:1511] Processing CreateTablet for tablet 84b0698af7d34bc9a3f26a3b75435b16 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:25:08.896567  9441 tablet_service.cc:1511] Processing CreateTablet for tablet 84b0698af7d34bc9a3f26a3b75435b16 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:25:08.887920  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7e7ca3cc2243559667954882dfa118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.898042  9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b0698af7d34bc9a3f26a3b75435b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.900322  9514 tablet_service.cc:1511] Processing CreateTablet for tablet 890c806dd6f94382ba4e14a3d3bc6f85 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:25:08.903501  9513 tablet_service.cc:1511] Processing CreateTablet for tablet c9515a897d574915ba4d8a478406aa1a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:25:08.906123  9291 tablet_service.cc:1511] Processing CreateTablet for tablet 5a5f846566064f6f932c156f0f9c9d7a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:25:08.907480  9371 tablet_service.cc:1511] Processing CreateTablet for tablet 160804417e4443b9b25105c759111c8b (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:25:08.896992  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b0698af7d34bc9a3f26a3b75435b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.908727  9290 tablet_service.cc:1511] Processing CreateTablet for tablet 890c806dd6f94382ba4e14a3d3bc6f85 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:25:08.909984  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5f846566064f6f932c156f0f9c9d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.916746  9440 tablet_service.cc:1511] Processing CreateTablet for tablet 5a5f846566064f6f932c156f0f9c9d7a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:25:08.910836  9370 tablet_service.cc:1511] Processing CreateTablet for tablet c26afa4025e24ea192d424c506f40c56 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:25:08.911051  9288 tablet_service.cc:1511] Processing CreateTablet for tablet b494d68a603a4ba5a28ee8514cfa3f41 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:25:08.920035  9439 tablet_service.cc:1511] Processing CreateTablet for tablet c9515a897d574915ba4d8a478406aa1a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:25:08.920855  9288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b494d68a603a4ba5a28ee8514cfa3f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.920775  9372 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7e7ca3cc2243559667954882dfa118 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:25:08.913553  9369 tablet_service.cc:1511] Processing CreateTablet for tablet a53d66736d1046bea06b3935c396942d (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:25:08.922770  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 890c806dd6f94382ba4e14a3d3bc6f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.914274  9289 tablet_service.cc:1511] Processing CreateTablet for tablet c9515a897d574915ba4d8a478406aa1a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:25:08.927711  9512 tablet_service.cc:1511] Processing CreateTablet for tablet 19cf5127045948269927c8b47835bb6e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:25:08.928908  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9515a897d574915ba4d8a478406aa1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.914935  9444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c26afa4025e24ea192d424c506f40c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.908911  9371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160804417e4443b9b25105c759111c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.932237  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffb83e944aed43718284cb48f994efcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.932648  9369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53d66736d1046bea06b3935c396942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.933521  9368 tablet_service.cc:1511] Processing CreateTablet for tablet 8a891dab57c74222928537e93876be86 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:25:08.935153  9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9515a897d574915ba4d8a478406aa1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.937294  9367 tablet_service.cc:1511] Processing CreateTablet for tablet 84b0698af7d34bc9a3f26a3b75435b16 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
W20260709 12:25:08.938759  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:08.935102  9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a891dab57c74222928537e93876be86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.942219  9366 tablet_service.cc:1511] Processing CreateTablet for tablet 5a5f846566064f6f932c156f0f9c9d7a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:25:08.942380  9438 tablet_service.cc:1511] Processing CreateTablet for tablet 19cf5127045948269927c8b47835bb6e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:25:08.942766  9365 tablet_service.cc:1511] Processing CreateTablet for tablet 890c806dd6f94382ba4e14a3d3bc6f85 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:25:08.943965  9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5f846566064f6f932c156f0f9c9d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.946157  9364 tablet_service.cc:1511] Processing CreateTablet for tablet 19cf5127045948269927c8b47835bb6e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:25:08.947746  9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53d66736d1046bea06b3935c396942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.952430  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7e7ca3cc2243559667954882dfa118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.953533  9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9515a897d574915ba4d8a478406aa1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.955586  9366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5f846566064f6f932c156f0f9c9d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.959646  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19cf5127045948269927c8b47835bb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.966624  9370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c26afa4025e24ea192d424c506f40c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.967799  9511 tablet_service.cc:1511] Processing CreateTablet for tablet 3cefb8e130fd43a1aba9f4773da6f272 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:25:08.968974  9437 tablet_service.cc:1511] Processing CreateTablet for tablet b494d68a603a4ba5a28ee8514cfa3f41 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:25:08.969463  9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cefb8e130fd43a1aba9f4773da6f272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.970415  9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b494d68a603a4ba5a28ee8514cfa3f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.971736  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b0698af7d34bc9a3f26a3b75435b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.978317  9365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 890c806dd6f94382ba4e14a3d3bc6f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.978499  9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 890c806dd6f94382ba4e14a3d3bc6f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.983786  9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19cf5127045948269927c8b47835bb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.985220  9364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19cf5127045948269927c8b47835bb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:08.987253  9363 tablet_service.cc:1511] Processing CreateTablet for tablet b494d68a603a4ba5a28ee8514cfa3f41 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:25:08.988801  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b494d68a603a4ba5a28ee8514cfa3f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.088654  9513 tablet_service.cc:1511] Processing CreateTablet for tablet 1976a69f7f174bce837050f43e2dd6fe (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:25:09.091158  9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1976a69f7f174bce837050f43e2dd6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.114926  9440 tablet_service.cc:1511] Processing CreateTablet for tablet fc204ad755be48768bf67ef7e84254bb (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:25:09.116585  9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc204ad755be48768bf67ef7e84254bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.122198  9294 tablet_service.cc:1511] Processing CreateTablet for tablet 3cefb8e130fd43a1aba9f4773da6f272 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:25:09.101055  9518 tablet_service.cc:1511] Processing CreateTablet for tablet 30315f436f8c4ef4b1ac5381d6a34811 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:25:09.124449  9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30315f436f8c4ef4b1ac5381d6a34811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.142072  9297 tablet_service.cc:1511] Processing CreateTablet for tablet 1976a69f7f174bce837050f43e2dd6fe (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:25:09.144160  9575 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:09.150533  9575 tablet_bootstrap.cc:654] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.152263  9520 tablet_service.cc:1511] Processing CreateTablet for tablet fc204ad755be48768bf67ef7e84254bb (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:25:09.153644  9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc204ad755be48768bf67ef7e84254bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.155920  9574 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:09.157124  9293 tablet_service.cc:1511] Processing CreateTablet for tablet 30315f436f8c4ef4b1ac5381d6a34811 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:25:09.166277  9437 tablet_service.cc:1511] Processing CreateTablet for tablet 30315f436f8c4ef4b1ac5381d6a34811 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:25:09.172318  9576 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:09.173669  9516 tablet_service.cc:1511] Processing CreateTablet for tablet a97a9ba3c90f464eacdc4e78d6a91f5e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:25:09.175412  9294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cefb8e130fd43a1aba9f4773da6f272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.175822  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97a9ba3c90f464eacdc4e78d6a91f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.179164  9577 tablet_bootstrap.cc:492] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:09.177642  9439 tablet_service.cc:1511] Processing CreateTablet for tablet 68ff98ec6813440cae57fcf054f25732 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:25:09.184798  9369 tablet_service.cc:1511] Processing CreateTablet for tablet 3cefb8e130fd43a1aba9f4773da6f272 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:25:09.185814  9372 tablet_service.cc:1511] Processing CreateTablet for tablet fc204ad755be48768bf67ef7e84254bb (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:25:09.187727  9368 tablet_service.cc:1511] Processing CreateTablet for tablet 68ff98ec6813440cae57fcf054f25732 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:25:09.210219  9292 tablet_service.cc:1511] Processing CreateTablet for tablet f950306b3d994137989bf2f2f5512441 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:25:09.188372  9289 tablet_service.cc:1511] Processing CreateTablet for tablet 68ff98ec6813440cae57fcf054f25732 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:25:09.189520  9297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1976a69f7f174bce837050f43e2dd6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.188746  9515 tablet_service.cc:1511] Processing CreateTablet for tablet 47c1f0a1e81a4ff0a3652fafff1288b4 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:25:09.215842  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30315f436f8c4ef4b1ac5381d6a34811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.219570  9371 tablet_service.cc:1511] Processing CreateTablet for tablet bf66e0c0d4ec4b10989c1f0abbbb8eb8 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:25:09.220765  9576 tablet_bootstrap.cc:654] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.201699  9442 tablet_service.cc:1511] Processing CreateTablet for tablet bf66e0c0d4ec4b10989c1f0abbbb8eb8 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:25:09.198325  9441 tablet_service.cc:1511] Processing CreateTablet for tablet 47c1f0a1e81a4ff0a3652fafff1288b4 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:25:09.201699  9332 heartbeater.cc:499] Master 127.25.202.62:39287 was elected leader, sending a full tablet report...
I20260709 12:25:09.201897  9370 tablet_service.cc:1511] Processing CreateTablet for tablet 47c1f0a1e81a4ff0a3652fafff1288b4 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:25:09.202288  9517 tablet_service.cc:1511] Processing CreateTablet for tablet 77b8b272744248da9eb46aef40c57d9a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:25:09.203979  9512 tablet_service.cc:1511] Processing CreateTablet for tablet f950306b3d994137989bf2f2f5512441 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:25:09.204968  9363 tablet_service.cc:1511] Processing CreateTablet for tablet 77b8b272744248da9eb46aef40c57d9a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:25:09.205178  9445 tablet_service.cc:1511] Processing CreateTablet for tablet 1976a69f7f174bce837050f43e2dd6fe (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:25:09.202288  9444 tablet_service.cc:1511] Processing CreateTablet for tablet f950306b3d994137989bf2f2f5512441 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:25:09.233314  9574 tablet_bootstrap.cc:654] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.234786  9514 tablet_service.cc:1511] Processing CreateTablet for tablet 6dbb3c7d37504398aab89200647dab24 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:25:09.208601  9290 tablet_service.cc:1511] Processing CreateTablet for tablet 77b8b272744248da9eb46aef40c57d9a (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:25:09.208806  9364 tablet_service.cc:1511] Processing CreateTablet for tablet 6dbb3c7d37504398aab89200647dab24 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:25:09.211124  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc204ad755be48768bf67ef7e84254bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.216161  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c1f0a1e81a4ff0a3652fafff1288b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.194285  9296 tablet_service.cc:1511] Processing CreateTablet for tablet a97a9ba3c90f464eacdc4e78d6a91f5e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:25:09.221470  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ff98ec6813440cae57fcf054f25732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.192219  9367 tablet_service.cc:1511] Processing CreateTablet for tablet a97a9ba3c90f464eacdc4e78d6a91f5e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:25:09.243906  9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f950306b3d994137989bf2f2f5512441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.199582  9291 tablet_service.cc:1511] Processing CreateTablet for tablet bf66e0c0d4ec4b10989c1f0abbbb8eb8 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:25:09.200065  9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30315f436f8c4ef4b1ac5381d6a34811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.245779  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1976a69f7f174bce837050f43e2dd6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.240069  9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f950306b3d994137989bf2f2f5512441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.239179  9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ff98ec6813440cae57fcf054f25732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.247671  9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ff98ec6813440cae57fcf054f25732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.251701  9444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f950306b3d994137989bf2f2f5512441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.255048  9577 tablet_bootstrap.cc:654] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.257058  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b8b272744248da9eb46aef40c57d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.257586  9370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c1f0a1e81a4ff0a3652fafff1288b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.286573  9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbb3c7d37504398aab89200647dab24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.258489  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf66e0c0d4ec4b10989c1f0abbbb8eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.287896  9364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbb3c7d37504398aab89200647dab24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.261015  9440 tablet_service.cc:1511] Processing CreateTablet for tablet 6dbb3c7d37504398aab89200647dab24 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:25:09.289831  9371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf66e0c0d4ec4b10989c1f0abbbb8eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.336956  9369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cefb8e130fd43a1aba9f4773da6f272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.270368  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b8b272744248da9eb46aef40c57d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.290376  9440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbb3c7d37504398aab89200647dab24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.312007  9407 heartbeater.cc:499] Master 127.25.202.62:39287 was elected leader, sending a full tablet report...
I20260709 12:25:09.345435  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b8b272744248da9eb46aef40c57d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.352159  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97a9ba3c90f464eacdc4e78d6a91f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.362417  9296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97a9ba3c90f464eacdc4e78d6a91f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.365952  9441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c1f0a1e81a4ff0a3652fafff1288b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.262435  9288 tablet_service.cc:1511] Processing CreateTablet for tablet 4ae03511a1cc4b369544d0a2b61c2515 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:25:09.388267  9575 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:09.388870  9575 ts_tablet_manager.cc:1403] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.245s	user 0.004s	sys 0.015s
W20260709 12:25:09.380330  9190 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 114267us
I20260709 12:25:09.402508  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf66e0c0d4ec4b10989c1f0abbbb8eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.421485  9366 tablet_service.cc:1511] Processing CreateTablet for tablet 4ae03511a1cc4b369544d0a2b61c2515 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:25:09.422952  9366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae03511a1cc4b369544d0a2b61c2515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.427203  9438 tablet_service.cc:1511] Processing CreateTablet for tablet 4ae03511a1cc4b369544d0a2b61c2515 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:25:09.428615  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae03511a1cc4b369544d0a2b61c2515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.447259  9481 heartbeater.cc:499] Master 127.25.202.62:39287 was elected leader, sending a full tablet report...
I20260709 12:25:09.463568  9288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae03511a1cc4b369544d0a2b61c2515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.416585  9575 raft_consensus.cc:359] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:09.476181  9575 raft_consensus.cc:385] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:09.476575  9575 raft_consensus.cc:740] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:09.477624  9575 consensus_queue.cc:260] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:09.481393  9577 tablet_bootstrap.cc:492] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:09.481954  9577 ts_tablet_manager.cc:1403] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.303s	user 0.018s	sys 0.010s
I20260709 12:25:09.485237  9576 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:09.485064  9577 raft_consensus.cc:359] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:09.485805  9576 ts_tablet_manager.cc:1403] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.314s	user 0.020s	sys 0.011s
I20260709 12:25:09.485953  9577 raft_consensus.cc:385] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:09.486445  9577 raft_consensus.cc:740] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:09.487560  9577 consensus_queue.cc:260] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:09.489009  9574 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:09.489236  9576 raft_consensus.cc:359] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:09.490793  9576 raft_consensus.cc:385] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:09.491092  9576 raft_consensus.cc:740] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:09.491911  9576 consensus_queue.cc:260] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:09.499351  9555 heartbeater.cc:499] Master 127.25.202.62:39287 was elected leader, sending a full tablet report...
I20260709 12:25:09.501390  9575 ts_tablet_manager.cc:1434] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.112s	user 0.008s	sys 0.007s
I20260709 12:25:09.502823  9575 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:09.512421  9575 tablet_bootstrap.cc:654] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.493924  9574 ts_tablet_manager.cc:1403] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.338s	user 0.011s	sys 0.033s
W20260709 12:25:09.531687  9556 tablet.cc:2406] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:25:09.535573  9516 tablet_service.cc:1511] Processing CreateTablet for tablet 109883acf19d4de4b90dfea6f147d368 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:25:09.536986  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109883acf19d4de4b90dfea6f147d368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.570792  9289 tablet_service.cc:1511] Processing CreateTablet for tablet 109883acf19d4de4b90dfea6f147d368 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:25:09.572377  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109883acf19d4de4b90dfea6f147d368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.584322  9438 tablet_service.cc:1511] Processing CreateTablet for tablet 4e722577ca9c4859ac9d6f335a1edfc9 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:25:09.586244  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e722577ca9c4859ac9d6f335a1edfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.591738  9372 tablet_service.cc:1511] Processing CreateTablet for tablet 109883acf19d4de4b90dfea6f147d368 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:25:09.593137  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109883acf19d4de4b90dfea6f147d368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.592983  9520 tablet_service.cc:1511] Processing CreateTablet for tablet 4e722577ca9c4859ac9d6f335a1edfc9 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:25:09.594938  9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e722577ca9c4859ac9d6f335a1edfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.598356  9574 raft_consensus.cc:359] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:09.613727  9576 ts_tablet_manager.cc:1434] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.128s	user 0.008s	sys 0.023s
I20260709 12:25:09.614928  9576 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:09.599148  9574 raft_consensus.cc:385] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:09.626025  9574 raft_consensus.cc:740] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:09.643673  9574 consensus_queue.cc:260] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:09.647256  9577 ts_tablet_manager.cc:1434] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.165s	user 0.041s	sys 0.099s
I20260709 12:25:09.648561  9577 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:09.662132  9445 tablet_service.cc:1511] Processing CreateTablet for tablet 11e5772eda0648d9a96d2b5755c23569 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:25:09.663652  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11e5772eda0648d9a96d2b5755c23569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.656443  9292 tablet_service.cc:1511] Processing CreateTablet for tablet 4e722577ca9c4859ac9d6f335a1edfc9 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:25:09.664693  9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e722577ca9c4859ac9d6f335a1edfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.674510  9367 tablet_service.cc:1511] Processing CreateTablet for tablet 11e5772eda0648d9a96d2b5755c23569 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:25:09.675989  9576 tablet_bootstrap.cc:654] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.676867  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11e5772eda0648d9a96d2b5755c23569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.680893  9575 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:09.681492  9575 ts_tablet_manager.cc:1403] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.179s	user 0.014s	sys 0.020s
I20260709 12:25:09.684262  9517 tablet_service.cc:1511] Processing CreateTablet for tablet 11e5772eda0648d9a96d2b5755c23569 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:25:09.692973  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11e5772eda0648d9a96d2b5755c23569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.700676  9289 tablet_service.cc:1511] Processing CreateTablet for tablet 2bb9dcd4bb4647b08e76fcc074ad8e40 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:25:09.702088  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bb9dcd4bb4647b08e76fcc074ad8e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.701920  9575 raft_consensus.cc:359] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:09.703070  9575 raft_consensus.cc:385] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:09.702888  9577 tablet_bootstrap.cc:654] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.707669  9574 ts_tablet_manager.cc:1434] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.112s	user 0.022s	sys 0.020s
I20260709 12:25:09.709460  9370 tablet_service.cc:1511] Processing CreateTablet for tablet 2bb9dcd4bb4647b08e76fcc074ad8e40 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:25:09.712898  9370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bb9dcd4bb4647b08e76fcc074ad8e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.705252  9575 raft_consensus.cc:740] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:09.713856  9575 consensus_queue.cc:260] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:09.716410  9575 ts_tablet_manager.cc:1434] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.000s
I20260709 12:25:09.717523  9575 tablet_bootstrap.cc:492] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:09.719916  9584 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.720495  9584 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:09.723925  9438 tablet_service.cc:1511] Processing CreateTablet for tablet 2bb9dcd4bb4647b08e76fcc074ad8e40 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:25:09.734157  9574 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:09.734453  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bb9dcd4bb4647b08e76fcc074ad8e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.738727  9290 tablet_service.cc:1511] Processing CreateTablet for tablet 51d19f01907340eb8dc56ed742606710 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:25:09.740232  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d19f01907340eb8dc56ed742606710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.724318  9584 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:09.741693  9574 tablet_bootstrap.cc:654] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.746568  9582 raft_consensus.cc:493] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.747228  9582 raft_consensus.cc:515] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:09.750449  9582 leader_election.cc:290] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:09.780736  9520 tablet_service.cc:1511] Processing CreateTablet for tablet 51d19f01907340eb8dc56ed742606710 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:25:09.781103  9516 tablet_service.cc:1511] Processing CreateTablet for tablet 9acfb739951443068b4ff6d65ebb3691 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:25:09.782284  9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d19f01907340eb8dc56ed742606710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.788045  9291 tablet_service.cc:1511] Processing CreateTablet for tablet 9acfb739951443068b4ff6d65ebb3691 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:25:09.789880  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9acfb739951443068b4ff6d65ebb3691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.791599  9512 tablet_service.cc:1511] Processing CreateTablet for tablet ac21f3e309014f859e6995b20e5414bb (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:25:09.792738  9293 tablet_service.cc:1511] Processing CreateTablet for tablet f9fe6f29f3144d21b12db35664f1f69e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:25:09.794095  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9fe6f29f3144d21b12db35664f1f69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.797703  9576 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:09.798357  9576 ts_tablet_manager.cc:1403] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.184s	user 0.020s	sys 0.019s
I20260709 12:25:09.801175  9367 tablet_service.cc:1511] Processing CreateTablet for tablet 51d19f01907340eb8dc56ed742606710 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:25:09.802805  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d19f01907340eb8dc56ed742606710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.816284  9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac21f3e309014f859e6995b20e5414bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.820338  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:09.823132  9347 leader_election.cc:343] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:09.824563  9586 raft_consensus.cc:493] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.825129  9586 raft_consensus.cc:515] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:09.828018  9586 leader_election.cc:290] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:09.839987  9372 tablet_service.cc:1511] Processing CreateTablet for tablet ac21f3e309014f859e6995b20e5414bb (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:25:09.841128  9583 raft_consensus.cc:493] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.841737  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac21f3e309014f859e6995b20e5414bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.841704  9583 raft_consensus.cc:515] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:09.843467  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9acfb739951443068b4ff6d65ebb3691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.849555  9575 tablet_bootstrap.cc:654] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:09.855436  9514 tablet_service.cc:1511] Processing CreateTablet for tablet 059f94ed1c1a42a5bc94cff7da97c777 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:25:09.872833  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:09.870951  9445 tablet_service.cc:1511] Processing CreateTablet for tablet 9acfb739951443068b4ff6d65ebb3691 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:25:09.875459  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9acfb739951443068b4ff6d65ebb3691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.860914  9583 leader_election.cc:290] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:09.855931  9292 tablet_service.cc:1511] Processing CreateTablet for tablet 059f94ed1c1a42a5bc94cff7da97c777 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260709 12:25:09.889261  9346 leader_election.cc:343] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:09.892145  9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059f94ed1c1a42a5bc94cff7da97c777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.892006  9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059f94ed1c1a42a5bc94cff7da97c777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.901067  9346 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:09.902209  9584 raft_consensus.cc:2749] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:09.903678  9442 tablet_service.cc:1511] Processing CreateTablet for tablet ac21f3e309014f859e6995b20e5414bb (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:25:09.905532  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac21f3e309014f859e6995b20e5414bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.906559  9574 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:09.907140  9574 ts_tablet_manager.cc:1403] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.180s	user 0.016s	sys 0.001s
I20260709 12:25:09.911664  9515 tablet_service.cc:1511] Processing CreateTablet for tablet 642ee9123c0945a7b10a89d4ff3a01c6 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:25:09.913113  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 642ee9123c0945a7b10a89d4ff3a01c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.913512  9577 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:09.914268  9577 ts_tablet_manager.cc:1403] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.266s	user 0.015s	sys 0.001s
I20260709 12:25:09.900068  9439 tablet_service.cc:1511] Processing CreateTablet for tablet f9fe6f29f3144d21b12db35664f1f69e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:25:09.916972  9293 tablet_service.cc:1511] Processing CreateTablet for tablet 642ee9123c0945a7b10a89d4ff3a01c6 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:25:09.917402  9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9fe6f29f3144d21b12db35664f1f69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.919219  9363 tablet_service.cc:1511] Processing CreateTablet for tablet 059f94ed1c1a42a5bc94cff7da97c777 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:25:09.920718  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059f94ed1c1a42a5bc94cff7da97c777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.923692  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 642ee9123c0945a7b10a89d4ff3a01c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.896908  9370 tablet_service.cc:1511] Processing CreateTablet for tablet f9fe6f29f3144d21b12db35664f1f69e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:25:09.928193  9370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9fe6f29f3144d21b12db35664f1f69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.929595  9574 raft_consensus.cc:359] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:09.941573  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:09.946982  9438 tablet_service.cc:1511] Processing CreateTablet for tablet 642ee9123c0945a7b10a89d4ff3a01c6 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:25:09.930430  9574 raft_consensus.cc:385] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:09.975993  9574 raft_consensus.cc:740] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:09.977257  9574 consensus_queue.cc:260] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:09.961215  9369 tablet_service.cc:1511] Processing CreateTablet for tablet 30d357fc75ee4a55a448f070c32e6499 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:25:09.980399  9369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d357fc75ee4a55a448f070c32e6499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.980854  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffb83e944aed43718284cb48f994efcc" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:09.984220  9494 leader_election.cc:343] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:09.974882  9576 raft_consensus.cc:359] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:09.956292  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 642ee9123c0945a7b10a89d4ff3a01c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.988868  9576 raft_consensus.cc:385] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.002159  9575 tablet_bootstrap.cc:492] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:10.002229  9437 tablet_service.cc:1511] Processing CreateTablet for tablet 30d357fc75ee4a55a448f070c32e6499 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:25:10.003008  9575 ts_tablet_manager.cc:1403] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.286s	user 0.014s	sys 0.011s
I20260709 12:25:10.003960  9437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d357fc75ee4a55a448f070c32e6499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.006740  9575 raft_consensus.cc:359] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.007716  9575 raft_consensus.cc:385] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.008103  9575 raft_consensus.cc:740] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.009196  9575 consensus_queue.cc:260] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.018071  9575 ts_tablet_manager.cc:1434] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.002s
I20260709 12:25:10.019461  9575 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:10.020470  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:10.022318  9271 leader_election.cc:343] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.029806  9517 tablet_service.cc:1511] Processing CreateTablet for tablet 30d357fc75ee4a55a448f070c32e6499 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:25:10.031365  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d357fc75ee4a55a448f070c32e6499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.032243  9574 ts_tablet_manager.cc:1434] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.125s	user 0.007s	sys 0.002s
I20260709 12:25:10.033813  9363 tablet_service.cc:1511] Processing CreateTablet for tablet e8fabd519bb34fbaae2738d3cfead584 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:25:10.035348  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fabd519bb34fbaae2738d3cfead584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:09.997246  9576 raft_consensus.cc:740] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.031906  9293 tablet_service.cc:1511] Processing CreateTablet for tablet e8fabd519bb34fbaae2738d3cfead584 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:25:10.036580  9576 consensus_queue.cc:260] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.037148  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fabd519bb34fbaae2738d3cfead584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.035580  9574 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:10.050602  9575 tablet_bootstrap.cc:654] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.045186  9576 ts_tablet_manager.cc:1434] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.246s	user 0.012s	sys 0.000s
I20260709 12:25:10.052459  9576 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:10.002056  9577 raft_consensus.cc:359] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.064203  9577 raft_consensus.cc:385] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.064569  9577 raft_consensus.cc:740] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.065500  9577 consensus_queue.cc:260] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.068373  9577 ts_tablet_manager.cc:1434] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.154s	user 0.009s	sys 0.001s
I20260709 12:25:10.069521  9577 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:10.071729  9586 raft_consensus.cc:493] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.072500  9586 raft_consensus.cc:515] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.080432  9586 leader_election.cc:290] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:10.083683  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffb83e944aed43718284cb48f994efcc" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:10.085309  9574 tablet_bootstrap.cc:654] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:10.087028  9495 leader_election.cc:343] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.087682  9495 leader_election.cc:304] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:10.088675  9582 raft_consensus.cc:2749] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:10.090858  9576 tablet_bootstrap.cc:654] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.094950  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:10.097132  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
W20260709 12:25:10.098552  9271 leader_election.cc:343] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.099056  9271 leader_election.cc:304] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:10.100581  9586 raft_consensus.cc:2749] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:25:10.106600  9271 leader_election.cc:343] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.111507  9582 raft_consensus.cc:493] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.113409  9582 raft_consensus.cc:515] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.116338  9582 leader_election.cc:290] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:10.117360  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.118554  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:10.121683  9494 leader_election.cc:343] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:10.122932  9495 leader_election.cc:343] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.123471  9495 leader_election.cc:304] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:10.124682  9582 raft_consensus.cc:2749] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:10.126205  9516 tablet_service.cc:1511] Processing CreateTablet for tablet 1b2fa83e4cf5494ba63becbde8d026b2 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:25:10.127791  9291 tablet_service.cc:1511] Processing CreateTablet for tablet 1b2fa83e4cf5494ba63becbde8d026b2 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:25:10.130623  9445 tablet_service.cc:1511] Processing CreateTablet for tablet e8fabd519bb34fbaae2738d3cfead584 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:25:10.136441  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2fa83e4cf5494ba63becbde8d026b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.136633  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fabd519bb34fbaae2738d3cfead584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.131745  9372 tablet_service.cc:1511] Processing CreateTablet for tablet 1b2fa83e4cf5494ba63becbde8d026b2 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:25:10.144809  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2fa83e4cf5494ba63becbde8d026b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.145812  9442 tablet_service.cc:1511] Processing CreateTablet for tablet 36a1d25b3256456ebdf25d5b805c8086 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:25:10.148037  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a1d25b3256456ebdf25d5b805c8086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.147018  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2fa83e4cf5494ba63becbde8d026b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.151310  9293 tablet_service.cc:1511] Processing CreateTablet for tablet 36a1d25b3256456ebdf25d5b805c8086 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:25:10.151789  9514 tablet_service.cc:1511] Processing CreateTablet for tablet 36a1d25b3256456ebdf25d5b805c8086 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:25:10.152768  9576 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:10.153110  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a1d25b3256456ebdf25d5b805c8086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.153321  9438 tablet_service.cc:1511] Processing CreateTablet for tablet b9508782ff1d4df7867468f207693333 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:25:10.154821  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9508782ff1d4df7867468f207693333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.156687  9517 tablet_service.cc:1511] Processing CreateTablet for tablet b9508782ff1d4df7867468f207693333 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:25:10.153348  9576 ts_tablet_manager.cc:1403] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.101s	user 0.013s	sys 0.002s
I20260709 12:25:10.164286  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9508782ff1d4df7867468f207693333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.164469  9575 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:10.164376  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:10.165251  9575 ts_tablet_manager.cc:1403] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.146s	user 0.011s	sys 0.007s
W20260709 12:25:10.166216  9420 leader_election.cc:343] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.165299  9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a1d25b3256456ebdf25d5b805c8086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.169067  9575 raft_consensus.cc:359] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.169924  9575 raft_consensus.cc:385] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.170265  9575 raft_consensus.cc:740] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.170920  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:10.174708  9421 leader_election.cc:343] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.175221  9421 leader_election.cc:304] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:10.176590  9583 raft_consensus.cc:2749] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:10.166276  9363 tablet_service.cc:1511] Processing CreateTablet for tablet b9508782ff1d4df7867468f207693333 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:25:10.180940  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9508782ff1d4df7867468f207693333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.182330  9577 tablet_bootstrap.cc:654] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.182366  9292 tablet_service.cc:1511] Processing CreateTablet for tablet 6272a586489844f8a507c2bea1c89107 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:25:10.184751  9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6272a586489844f8a507c2bea1c89107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.190717  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:10.191437  9382 raft_consensus.cc:2468] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:10.192634  9576 raft_consensus.cc:359] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.193044  9272 leader_election.cc:304] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:10.193460  9576 raft_consensus.cc:385] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.193940  9586 raft_consensus.cc:2804] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.194329  9586 raft_consensus.cc:493] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.194669  9586 raft_consensus.cc:3060] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.171159  9575 consensus_queue.cc:260] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.204133  9367 tablet_service.cc:1511] Processing CreateTablet for tablet 6272a586489844f8a507c2bea1c89107 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:25:10.205669  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6272a586489844f8a507c2bea1c89107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.206120  9575 ts_tablet_manager.cc:1434] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.004s
I20260709 12:25:10.207785  9515 tablet_service.cc:1511] Processing CreateTablet for tablet eb1d8de1bb30430eb7ae1bbbf66232df (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:25:10.208441  9439 tablet_service.cc:1511] Processing CreateTablet for tablet 6272a586489844f8a507c2bea1c89107 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:25:10.209246  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1d8de1bb30430eb7ae1bbbf66232df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.209860  9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6272a586489844f8a507c2bea1c89107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.202837  9586 raft_consensus.cc:515] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.217990  9586 leader_election.cc:290] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:10.218267  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:10.219264  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:10.219885  9382 raft_consensus.cc:3060] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.220698  9576 raft_consensus.cc:740] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.221607  9576 consensus_queue.cc:260] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:10.222286  9271 leader_election.cc:343] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.231704  9576 ts_tablet_manager.cc:1434] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.068s	user 0.006s	sys 0.002s
I20260709 12:25:10.232817  9576 tablet_bootstrap.cc:492] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:10.207283  9575 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:10.238125  9368 tablet_service.cc:1511] Processing CreateTablet for tablet eb1d8de1bb30430eb7ae1bbbf66232df (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:25:10.240468  9293 tablet_service.cc:1511] Processing CreateTablet for tablet eb1d8de1bb30430eb7ae1bbbf66232df (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:25:10.243007  9445 tablet_service.cc:1511] Processing CreateTablet for tablet fd0b217a2c7841638bd1d47ad5ea6783 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:25:10.247087  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1d8de1bb30430eb7ae1bbbf66232df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.243989  9515 tablet_service.cc:1511] Processing CreateTablet for tablet fd0b217a2c7841638bd1d47ad5ea6783 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:25:10.250866  9371 tablet_service.cc:1511] Processing CreateTablet for tablet f907c1045137407887423995a153fea0 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:25:10.247388  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0b217a2c7841638bd1d47ad5ea6783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.251973  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0b217a2c7841638bd1d47ad5ea6783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.255462  9574 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:10.256075  9574 ts_tablet_manager.cc:1403] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.221s	user 0.012s	sys 0.002s
I20260709 12:25:10.244772  9291 tablet_service.cc:1511] Processing CreateTablet for tablet fd0b217a2c7841638bd1d47ad5ea6783 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:25:10.258373  9289 tablet_service.cc:1511] Processing CreateTablet for tablet c9863504e96043ba8fafaa2396daa5f4 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:25:10.259698  9574 raft_consensus.cc:359] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.258845  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0b217a2c7841638bd1d47ad5ea6783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.249981  9382 raft_consensus.cc:2468] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:10.261024  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9863504e96043ba8fafaa2396daa5f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.260519  9574 raft_consensus.cc:385] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.266402  9574 raft_consensus.cc:740] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.266963  9272 leader_election.cc:304] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:10.267611  9574 consensus_queue.cc:260] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.258373  9442 tablet_service.cc:1511] Processing CreateTablet for tablet f907c1045137407887423995a153fea0 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:25:10.280435  9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1d8de1bb30430eb7ae1bbbf66232df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.282691  9577 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:10.283313  9577 ts_tablet_manager.cc:1403] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.214s	user 0.003s	sys 0.011s
I20260709 12:25:10.278231  9586 raft_consensus.cc:2804] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:10.287724  9371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f907c1045137407887423995a153fea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.289582  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f907c1045137407887423995a153fea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.262494  9516 tablet_service.cc:1511] Processing CreateTablet for tablet f907c1045137407887423995a153fea0 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:25:10.302457  9575 tablet_bootstrap.cc:654] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.302181  9372 tablet_service.cc:1511] Processing CreateTablet for tablet c9863504e96043ba8fafaa2396daa5f4 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:25:10.303577  9576 tablet_bootstrap.cc:654] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.303414  9577 raft_consensus.cc:359] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.307391  9438 tablet_service.cc:1511] Processing CreateTablet for tablet c9863504e96043ba8fafaa2396daa5f4 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:25:10.311409  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f907c1045137407887423995a153fea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.311344  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9863504e96043ba8fafaa2396daa5f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.321173  9582 raft_consensus.cc:493] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.321756  9582 raft_consensus.cc:515] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.323310  9363 tablet_service.cc:1511] Processing CreateTablet for tablet 58056f74cd464091a7f28a4a999ce936 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:25:10.324021  9582 leader_election.cc:290] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.325831  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:10.326546  9307 raft_consensus.cc:2468] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:10.328244  9584 raft_consensus.cc:493] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.328595  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.328836  9584 raft_consensus.cc:515] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.330505  9495 leader_election.cc:304] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
W20260709 12:25:10.331643  9494 leader_election.cc:343] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.332800  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7e7ca3cc2243559667954882dfa118" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.333334  9584 leader_election.cc:290] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.333514  9456 raft_consensus.cc:2468] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:10.334942  9346 leader_election.cc:304] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:10.337150  9445 tablet_service.cc:1511] Processing CreateTablet for tablet b303f27117804b4c8dc4ef4e31e0384e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:25:10.338476  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b303f27117804b4c8dc4ef4e31e0384e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.314885  9514 tablet_service.cc:1511] Processing CreateTablet for tablet 58056f74cd464091a7f28a4a999ce936 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:25:10.316762  9290 tablet_service.cc:1511] Processing CreateTablet for tablet 58056f74cd464091a7f28a4a999ce936 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:25:10.310012  9577 raft_consensus.cc:385] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.318604  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9863504e96043ba8fafaa2396daa5f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.342762  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58056f74cd464091a7f28a4a999ce936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.342984  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7e7ca3cc2243559667954882dfa118" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:10.343674  9574 ts_tablet_manager.cc:1434] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.087s	user 0.009s	sys 0.008s
W20260709 12:25:10.344650  9347 leader_election.cc:343] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.344982  9574 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:10.341431  9577 raft_consensus.cc:740] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.346369  9577 consensus_queue.cc:260] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.341576  9584 raft_consensus.cc:2804] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.352754  9584 raft_consensus.cc:493] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.353160  9584 raft_consensus.cc:3060] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.356351  9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58056f74cd464091a7f28a4a999ce936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.360761  9577 ts_tablet_manager.cc:1434] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.077s	user 0.006s	sys 0.001s
I20260709 12:25:10.361971  9577 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:10.341948  9582 raft_consensus.cc:2804] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.363662  9582 raft_consensus.cc:493] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.364059  9582 raft_consensus.cc:3060] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.368695  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58056f74cd464091a7f28a4a999ce936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.368510  9367 tablet_service.cc:1511] Processing CreateTablet for tablet 1a6ceb4648564df0b7e690a8e3343b5b (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:25:10.370177  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6ceb4648564df0b7e690a8e3343b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.368510  9293 tablet_service.cc:1511] Processing CreateTablet for tablet b303f27117804b4c8dc4ef4e31e0384e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:25:10.376075  9439 tablet_service.cc:1511] Processing CreateTablet for tablet 1a6ceb4648564df0b7e690a8e3343b5b (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:25:10.379825  9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6ceb4648564df0b7e690a8e3343b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.342265  9586 raft_consensus.cc:697] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:10.378235  9517 tablet_service.cc:1511] Processing CreateTablet for tablet 1a6ceb4648564df0b7e690a8e3343b5b (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:25:10.385083  9516 tablet_service.cc:1511] Processing CreateTablet for tablet b303f27117804b4c8dc4ef4e31e0384e (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:25:10.388087  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6ceb4648564df0b7e690a8e3343b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.392920  9584 raft_consensus.cc:515] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.395668  9574 tablet_bootstrap.cc:654] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.396039  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7e7ca3cc2243559667954882dfa118" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
W20260709 12:25:10.397549  9347 leader_election.cc:343] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.399730  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7e7ca3cc2243559667954882dfa118" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:10.400326  9456 raft_consensus.cc:3060] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.388845  9586 consensus_queue.cc:237] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.405556  9584 leader_election.cc:290] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.392040  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b303f27117804b4c8dc4ef4e31e0384e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.407243  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b303f27117804b4c8dc4ef4e31e0384e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.414789  9577 tablet_bootstrap.cc:654] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.415449  9456 raft_consensus.cc:2468] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:10.416981  9346 leader_election.cc:304] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:10.418334  9584 raft_consensus.cc:2804] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:10.424540  9290 tablet_service.cc:1511] Processing CreateTablet for tablet 6a7c46b071ed4f03bb81dc8a4f55555c (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:25:10.426004  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7c46b071ed4f03bb81dc8a4f55555c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.443805  9575 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:10.444506  9575 ts_tablet_manager.cc:1403] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.237s	user 0.016s	sys 0.019s
I20260709 12:25:10.451478  9575 raft_consensus.cc:359] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.452283  9575 raft_consensus.cc:385] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.452572  9575 raft_consensus.cc:740] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.453087  9368 tablet_service.cc:1511] Processing CreateTablet for tablet 6a7c46b071ed4f03bb81dc8a4f55555c (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:25:10.453325  9575 consensus_queue.cc:260] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.463995  9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7c46b071ed4f03bb81dc8a4f55555c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.447458  9582 raft_consensus.cc:515] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.478348  9442 tablet_service.cc:1511] Processing CreateTablet for tablet 6a7c46b071ed4f03bb81dc8a4f55555c (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:25:10.479792  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7c46b071ed4f03bb81dc8a4f55555c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.480820  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:10.481465  9307 raft_consensus.cc:3060] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.492723  9307 raft_consensus.cc:2468] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:10.496171  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:10.498677  9290 tablet_service.cc:1511] Processing CreateTablet for tablet 9eed29862fa545d7b5bf1232e33f0798 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:25:10.499958  9584 raft_consensus.cc:697] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:10.501413  9582 leader_election.cc:290] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.512285  9583 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.517033  9583 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.517764  9368 tablet_service.cc:1511] Processing CreateTablet for tablet 9eed29862fa545d7b5bf1232e33f0798 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:25:10.519156  9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eed29862fa545d7b5bf1232e33f0798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.519074  9495 leader_election.cc:304] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:10.522082  9445 tablet_service.cc:1511] Processing CreateTablet for tablet 85baaa21d67f43da9f328b0b51a71215 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:25:10.520709  9515 tablet_service.cc:1511] Processing CreateTablet for tablet 9eed29862fa545d7b5bf1232e33f0798 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:25:10.527679  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eed29862fa545d7b5bf1232e33f0798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.527906  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eed29862fa545d7b5bf1232e33f0798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.524211  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85baaa21d67f43da9f328b0b51a71215. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:25:10.523412  9494 leader_election.cc:343] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.534101  9582 raft_consensus.cc:493] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.531993  9584 consensus_queue.cc:237] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.546507  9574 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:10.546710  9289 tablet_service.cc:1511] Processing CreateTablet for tablet 85baaa21d67f43da9f328b0b51a71215 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:25:10.548141  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85baaa21d67f43da9f328b0b51a71215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.549497  9583 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:10.551004  9618 raft_consensus.cc:2804] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:10.551110  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:10.551640  9618 raft_consensus.cc:697] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
W20260709 12:25:10.552723  9421 leader_election.cc:343] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.552573  9618 consensus_queue.cc:237] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.557899  9619 raft_consensus.cc:493] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.558425  9619 raft_consensus.cc:515] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:10.561655  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:10.561367  9205 catalog_manager.cc:5697] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:10.563256  9495 leader_election.cc:343] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.566372  9619 leader_election.cc:290] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:10.543674  9582 raft_consensus.cc:515] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.570358  9582 leader_election.cc:290] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.547102  9574 ts_tablet_manager.cc:1403] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.202s	user 0.032s	sys 0.031s
I20260709 12:25:10.572660  9576 tablet_bootstrap.cc:492] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:10.573407  9576 ts_tablet_manager.cc:1403] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.341s	user 0.011s	sys 0.008s
I20260709 12:25:10.575568  9575 ts_tablet_manager.cc:1434] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.131s	user 0.017s	sys 0.006s
I20260709 12:25:10.577171  9575 tablet_bootstrap.cc:492] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:10.579923  9574 raft_consensus.cc:359] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.580714  9574 raft_consensus.cc:385] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.581019  9363 tablet_service.cc:1511] Processing CreateTablet for tablet f4ba22da1d594bf19f8b83581343da38 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:25:10.581271  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c26afa4025e24ea192d424c506f40c56" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.582392  9363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4ba22da1d594bf19f8b83581343da38. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:25:10.582903  9494 leader_election.cc:343] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.581027  9574 raft_consensus.cc:740] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.584182  9574 consensus_queue.cc:260] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.593617  9438 tablet_service.cc:1511] Processing CreateTablet for tablet f4ba22da1d594bf19f8b83581343da38 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:25:10.619853  9586 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.627952  9517 tablet_service.cc:1511] Processing CreateTablet for tablet f4ba22da1d594bf19f8b83581343da38 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:25:10.629312  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4ba22da1d594bf19f8b83581343da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.629750  9445 tablet_service.cc:1511] Processing CreateTablet for tablet 05a47d242f094b218e731992a142e472 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:25:10.630914  9514 tablet_service.cc:1511] Processing CreateTablet for tablet 85baaa21d67f43da9f328b0b51a71215 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:25:10.632212  9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85baaa21d67f43da9f328b0b51a71215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.632401  9586 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.632799  9367 tablet_service.cc:1511] Processing CreateTablet for tablet 05a47d242f094b218e731992a142e472 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:25:10.634634  9586 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.634460  9576 raft_consensus.cc:359] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.638237  9293 tablet_service.cc:1511] Processing CreateTablet for tablet 05a47d242f094b218e731992a142e472 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:25:10.639150  9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05a47d242f094b218e731992a142e472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.636880  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:10.640220  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.642704  9574 ts_tablet_manager.cc:1434] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.002s
I20260709 12:25:10.642858  9382 raft_consensus.cc:2468] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
W20260709 12:25:10.644145  9271 leader_election.cc:343] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.645174  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4ba22da1d594bf19f8b83581343da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.645856  9512 tablet_service.cc:1511] Processing CreateTablet for tablet 4cecdb3fe70f46d8af803d3d7db78976 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:25:10.647185  9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cecdb3fe70f46d8af803d3d7db78976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.647827  9202 catalog_manager.cc:5697] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:10.653170  9272 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:10.653491  9575 tablet_bootstrap.cc:654] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.654560  9586 raft_consensus.cc:2804] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.654948  9586 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.655324  9586 raft_consensus.cc:3060] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.658480  9574 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:10.662940  9367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05a47d242f094b218e731992a142e472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.635269  9576 raft_consensus.cc:385] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.663650  9576 raft_consensus.cc:740] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.664450  9576 consensus_queue.cc:260] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.666375  9576 ts_tablet_manager.cc:1434] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.093s	user 0.007s	sys 0.000s
I20260709 12:25:10.674153  9586 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.679083  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:10.675413  9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05a47d242f094b218e731992a142e472. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:25:10.680712  9421 leader_election.cc:343] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.681191  9421 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:10.682426  9583 raft_consensus.cc:2749] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:10.686820  9292 tablet_service.cc:1511] Processing CreateTablet for tablet 4cecdb3fe70f46d8af803d3d7db78976 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:25:10.687937  9372 tablet_service.cc:1511] Processing CreateTablet for tablet 4cecdb3fe70f46d8af803d3d7db78976 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:25:10.688395  9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cecdb3fe70f46d8af803d3d7db78976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.688902  9438 tablet_service.cc:1511] Processing CreateTablet for tablet aad56d798d414d5eb49051f39b8ae755 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:25:10.689391  9372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cecdb3fe70f46d8af803d3d7db78976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.690117  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad56d798d414d5eb49051f39b8ae755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.693460  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:10.695199  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c26afa4025e24ea192d424c506f40c56" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:10.696130  9381 raft_consensus.cc:2468] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:10.667232  9576 tablet_bootstrap.cc:492] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
W20260709 12:25:10.697877  9495 leader_election.cc:343] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.698365  9495 leader_election.cc:304] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:10.699398  9618 raft_consensus.cc:2749] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:10.699853  9495 leader_election.cc:304] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:10.700759  9618 raft_consensus.cc:2804] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.701143  9618 raft_consensus.cc:493] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.701473  9618 raft_consensus.cc:3060] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.704267  9577 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:10.704865  9577 ts_tablet_manager.cc:1403] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.343s	user 0.017s	sys 0.019s
I20260709 12:25:10.712507  9515 tablet_service.cc:1511] Processing CreateTablet for tablet aad56d798d414d5eb49051f39b8ae755 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:25:10.714576  9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad56d798d414d5eb49051f39b8ae755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.720377  9618 raft_consensus.cc:515] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.723285  9576 tablet_bootstrap.cc:654] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.723843  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c26afa4025e24ea192d424c506f40c56" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:10.736661  9291 tablet_service.cc:1511] Processing CreateTablet for tablet d7076b0f22f54a05838aec7b64bf5575 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:25:10.736948  9577 raft_consensus.cc:359] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.734107  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c26afa4025e24ea192d424c506f40c56" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:10.737936  9577 raft_consensus.cc:385] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.737114  9381 raft_consensus.cc:3060] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.738317  9439 tablet_service.cc:1511] Processing CreateTablet for tablet ccf9f64ab9404078902886a07eef6a66 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
W20260709 12:25:10.739063  9494 leader_election.cc:343] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.740224  9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf9f64ab9404078902886a07eef6a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.740885  9575 tablet_bootstrap.cc:492] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:10.741452  9575 ts_tablet_manager.cc:1403] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.165s	user 0.010s	sys 0.004s
I20260709 12:25:10.738560  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7076b0f22f54a05838aec7b64bf5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.745296  9381 raft_consensus.cc:2468] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:10.736677  9516 tablet_service.cc:1511] Processing CreateTablet for tablet ccf9f64ab9404078902886a07eef6a66 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:25:10.746701  9495 leader_election.cc:304] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:10.729516  9290 tablet_service.cc:1511] Processing CreateTablet for tablet aad56d798d414d5eb49051f39b8ae755 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:25:10.747684  9582 raft_consensus.cc:2804] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:10.748188  9582 raft_consensus.cc:697] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:10.749015  9582 consensus_queue.cc:237] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.751811  9438 tablet_service.cc:1511] Processing CreateTablet for tablet d7076b0f22f54a05838aec7b64bf5575 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:25:10.753106  9438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7076b0f22f54a05838aec7b64bf5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.754454  9371 tablet_service.cc:1511] Processing CreateTablet for tablet d7076b0f22f54a05838aec7b64bf5575 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:25:10.754877  9290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad56d798d414d5eb49051f39b8ae755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.755897  9371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7076b0f22f54a05838aec7b64bf5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.750638  9203 catalog_manager.cc:5697] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:10.733182  9368 tablet_service.cc:1511] Processing CreateTablet for tablet ccf9f64ab9404078902886a07eef6a66 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:25:10.761289  9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf9f64ab9404078902886a07eef6a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.761626  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:10.761775  9586 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.762270  9382 raft_consensus.cc:3060] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.762976  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf9f64ab9404078902886a07eef6a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.767848  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
W20260709 12:25:10.769503  9271 leader_election.cc:343] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:10.737396  9618 leader_election.cc:290] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:10.738328  9577 raft_consensus.cc:740] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.759642  9517 tablet_service.cc:1511] Processing CreateTablet for tablet 46e96ef5e406430f9972ca4831f609fe (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:25:10.774569  9577 consensus_queue.cc:260] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.775228  9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e96ef5e406430f9972ca4831f609fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.777230  9577 ts_tablet_manager.cc:1434] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.072s	user 0.008s	sys 0.000s
I20260709 12:25:10.769593  9382 raft_consensus.cc:2468] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:10.782500  9272 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:10.783864  9575 raft_consensus.cc:359] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.784798  9575 raft_consensus.cc:385] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.785208  9575 raft_consensus.cc:740] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.786155  9575 consensus_queue.cc:260] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.786252  9370 tablet_service.cc:1511] Processing CreateTablet for tablet 46e96ef5e406430f9972ca4831f609fe (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:25:10.787519  9586 raft_consensus.cc:2804] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:10.787925  9370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e96ef5e406430f9972ca4831f609fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.788108  9586 raft_consensus.cc:697] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:10.789091  9586 consensus_queue.cc:237] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.798398  9612 raft_consensus.cc:493] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.799032  9612 raft_consensus.cc:515] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.786401  9289 tablet_service.cc:1511] Processing CreateTablet for tablet 46e96ef5e406430f9972ca4831f609fe (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:25:10.801600  9612 leader_election.cc:290] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:10.802663  9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e96ef5e406430f9972ca4831f609fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.804638  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:10.805342  9530 raft_consensus.cc:2468] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:10.734596  9574 tablet_bootstrap.cc:654] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.807077  9271 leader_election.cc:304] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:10.809830  9612 raft_consensus.cc:2804] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.810673  9612 raft_consensus.cc:493] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.811918  9612 raft_consensus.cc:3060] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.814210  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.814826  9577 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:10.814972  9456 raft_consensus.cc:2468] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:10.829540  9612 raft_consensus.cc:515] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.833629  9442 tablet_service.cc:1511] Processing CreateTablet for tablet 76060914d3af47e6ad7cb572ad9cc073 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:25:10.836261  9576 tablet_bootstrap.cc:492] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:10.838307  9612 leader_election.cc:290] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:10.839032  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:10.839735  9456 raft_consensus.cc:3060] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.844094  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:10.844801  9530 raft_consensus.cc:3060] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:10.847806  9203 catalog_manager.cc:5697] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:10.853137  9575 ts_tablet_manager.cc:1434] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.111s	user 0.003s	sys 0.004s
I20260709 12:25:10.854216  9442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76060914d3af47e6ad7cb572ad9cc073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.854372  9575 tablet_bootstrap.cc:492] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:10.837634  9576 ts_tablet_manager.cc:1403] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.171s	user 0.012s	sys 0.002s
I20260709 12:25:10.869057  9530 raft_consensus.cc:2468] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:10.870543  9271 leader_election.cc:304] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:10.871097  9577 tablet_bootstrap.cc:654] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.871407  9612 raft_consensus.cc:2804] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:10.862450  9456 raft_consensus.cc:2468] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:10.876772  9612 raft_consensus.cc:697] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:10.877748  9612 consensus_queue.cc:237] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.866953  9576 raft_consensus.cc:359] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.889855  9516 tablet_service.cc:1511] Processing CreateTablet for tablet 76060914d3af47e6ad7cb572ad9cc073 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:25:10.891592  9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76060914d3af47e6ad7cb572ad9cc073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.892247  9574 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:10.893215  9574 ts_tablet_manager.cc:1403] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.235s	user 0.007s	sys 0.008s
I20260709 12:25:10.887372  9576 raft_consensus.cc:385] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.894820  9576 raft_consensus.cc:740] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.895674  9576 consensus_queue.cc:260] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.897893  9202 catalog_manager.cc:5697] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:10.903514  9574 raft_consensus.cc:359] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.904332  9574 raft_consensus.cc:385] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.904665  9574 raft_consensus.cc:740] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.905478  9574 consensus_queue.cc:260] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:10.909998  9291 tablet_service.cc:1511] Processing CreateTablet for tablet 76060914d3af47e6ad7cb572ad9cc073 (DEFAULT_TABLE table=test-workload [id=bf0a1b0194b54b2395bbcd0a355c8e84]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:25:10.911394  9291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76060914d3af47e6ad7cb572ad9cc073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:10.920042  9577 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:10.920609  9577 ts_tablet_manager.cc:1403] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.106s	user 0.013s	sys 0.000s
I20260709 12:25:10.923445  9575 tablet_bootstrap.cc:654] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:10.924019  9576 ts_tablet_manager.cc:1434] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.060s	user 0.005s	sys 0.002s
I20260709 12:25:10.923791  9577 raft_consensus.cc:359] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.924563  9577 raft_consensus.cc:385] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:10.924880  9577 raft_consensus.cc:740] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:10.925067  9576 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:10.925639  9577 consensus_queue.cc:260] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:10.928390  9574 ts_tablet_manager.cc:1434] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20260709 12:25:10.929468  9574 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:10.931650  9612 consensus_queue.cc:1048] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:10.939407  9584 raft_consensus.cc:493] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.940133  9584 raft_consensus.cc:515] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:10.942474  9584 leader_election.cc:290] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:10.942265  9202 catalog_manager.cc:5697] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:10.944096  9347 consensus_peers.cc:597] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:10.944602  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:10.945295  9456 raft_consensus.cc:2468] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:10.959779  9582 consensus_queue.cc:1048] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:10.982316  9574 tablet_bootstrap.cc:654] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:10.983696  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:10.984615  9577 ts_tablet_manager.cc:1434] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.064s	user 0.006s	sys 0.002s
I20260709 12:25:10.985795  9577 tablet_bootstrap.cc:492] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:10.995486  9346 leader_election.cc:304] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:10.996785  9584 raft_consensus.cc:2804] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:10.998029  9584 raft_consensus.cc:493] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.998395  9584 raft_consensus.cc:3060] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:11.001940  9271 consensus_peers.cc:597] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 -> Peer a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855): Couldn't send request to peer a5f89c82eda14a9ca425fae398bae986. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:11.005630  9576 tablet_bootstrap.cc:654] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.020625  9584 raft_consensus.cc:515] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.023083  9584 leader_election.cc:290] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:11.023983  9574 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
W20260709 12:25:11.023875  9494 consensus_peers.cc:597] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:11.024598  9574 ts_tablet_manager.cc:1403] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.095s	user 0.015s	sys 0.004s
I20260709 12:25:11.027393  9574 raft_consensus.cc:359] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.028113  9574 raft_consensus.cc:385] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.028376  9574 raft_consensus.cc:740] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.030711  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:11.031365  9456 raft_consensus.cc:3060] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.031292  9574 consensus_queue.cc:260] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.033844  9574 ts_tablet_manager.cc:1434] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:25:11.035923  9575 tablet_bootstrap.cc:492] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.036475  9575 ts_tablet_manager.cc:1403] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.182s	user 0.013s	sys 0.005s
I20260709 12:25:11.039116  9456 raft_consensus.cc:2468] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:11.039464  9574 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.039768  9575 raft_consensus.cc:359] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.040585  9575 raft_consensus.cc:385] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.040530  9346 leader_election.cc:304] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.041141  9575 raft_consensus.cc:740] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.042091  9575 consensus_queue.cc:260] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.044404  9575 ts_tablet_manager.cc:1434] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:25:11.044863  9584 raft_consensus.cc:2804] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.045616  9584 raft_consensus.cc:697] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:11.046975  9574 tablet_bootstrap.cc:654] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.050987  9576 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.051162  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:11.051550  9575 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.051699  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:11.051868  9530 raft_consensus.cc:2468] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:11.051697  9576 ts_tablet_manager.cc:1403] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.127s	user 0.006s	sys 0.007s
I20260709 12:25:11.047480  9584 consensus_queue.cc:237] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.055632  9577 tablet_bootstrap.cc:654] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.056310  9576 raft_consensus.cc:359] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.057135  9576 raft_consensus.cc:385] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.057449  9576 raft_consensus.cc:740] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.058243  9576 consensus_queue.cc:260] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.064693  9575 tablet_bootstrap.cc:654] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.068257  9616 consensus_queue.cc:1048] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:11.084537  9576 ts_tablet_manager.cc:1434] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.001s
I20260709 12:25:11.086122  9576 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.083731  9203 catalog_manager.cc:5697] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:11.090943  9577 tablet_bootstrap.cc:492] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.091236  9574 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.091564  9577 ts_tablet_manager.cc:1403] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.106s	user 0.018s	sys 0.024s
I20260709 12:25:11.091764  9574 ts_tablet_manager.cc:1403] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.053s	user 0.013s	sys 0.005s
I20260709 12:25:11.095005  9574 raft_consensus.cc:359] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.095345  9577 raft_consensus.cc:359] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.095781  9574 raft_consensus.cc:385] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.096045  9577 raft_consensus.cc:385] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.096099  9574 raft_consensus.cc:740] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.096509  9577 raft_consensus.cc:740] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.097420  9577 consensus_queue.cc:260] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.099632  9574 consensus_queue.cc:260] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.100155  9577 ts_tablet_manager.cc:1434] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:25:11.101331  9577 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.104995  9576 tablet_bootstrap.cc:654] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.108846  9577 tablet_bootstrap.cc:654] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.109212  9574 ts_tablet_manager.cc:1434] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:25:11.110721  9574 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.117300  9575 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.117834  9575 ts_tablet_manager.cc:1403] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.067s	user 0.029s	sys 0.015s
I20260709 12:25:11.120394  9574 tablet_bootstrap.cc:654] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.120735  9575 raft_consensus.cc:359] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.121450  9575 raft_consensus.cc:385] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.121702  9575 raft_consensus.cc:740] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.122467  9575 consensus_queue.cc:260] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.124863  9575 ts_tablet_manager.cc:1434] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:25:11.126039  9575 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.129976  9576 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.131493  9576 ts_tablet_manager.cc:1403] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.046s	user 0.012s	sys 0.006s
I20260709 12:25:11.133443  9577 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.133996  9577 ts_tablet_manager.cc:1403] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.004s
I20260709 12:25:11.137234  9577 raft_consensus.cc:359] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.137476  9576 raft_consensus.cc:359] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.137969  9577 raft_consensus.cc:385] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.138175  9576 raft_consensus.cc:385] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.138327  9577 raft_consensus.cc:740] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.138520  9576 raft_consensus.cc:740] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.139371  9577 consensus_queue.cc:260] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.140988  9576 consensus_queue.cc:260] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.142270  9575 tablet_bootstrap.cc:654] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.147971  9576 ts_tablet_manager.cc:1434] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260709 12:25:11.149135  9576 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.152781  9577 ts_tablet_manager.cc:1434] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.000s
I20260709 12:25:11.159653  9574 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.160621  9574 ts_tablet_manager.cc:1403] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.050s	user 0.016s	sys 0.000s
I20260709 12:25:11.161952  9576 tablet_bootstrap.cc:654] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.162756  9618 raft_consensus.cc:493] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.163287  9618 raft_consensus.cc:515] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.164469  9574 raft_consensus.cc:359] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.165182  9577 tablet_bootstrap.cc:492] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.165512  9618 leader_election.cc:290] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.168753  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.169835  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.165200  9574 raft_consensus.cc:385] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.171500  9586 raft_consensus.cc:493] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.171761  9574 raft_consensus.cc:740] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.171998  9586 raft_consensus.cc:515] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:11.171629  9494 leader_election.cc:343] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.172561  9574 consensus_queue.cc:260] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:11.173000  9495 leader_election.cc:343] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.173453  9495 leader_election.cc:304] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.174088  9586 leader_election.cc:290] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.174877  9574 ts_tablet_manager.cc:1434] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260709 12:25:11.176080  9618 raft_consensus.cc:2749] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.176964  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7e7ca3cc2243559667954882dfa118" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.178232  9577 tablet_bootstrap.cc:654] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.179144  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7e7ca3cc2243559667954882dfa118" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.180114  9574 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.181181  9271 leader_election.cc:304] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.182708  9612 raft_consensus.cc:2749] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.184923  9575 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.185441  9575 ts_tablet_manager.cc:1403] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.060s	user 0.019s	sys 0.011s
I20260709 12:25:11.186563  9574 tablet_bootstrap.cc:654] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.188299  9575 raft_consensus.cc:359] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.189024  9575 raft_consensus.cc:385] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.189293  9575 raft_consensus.cc:740] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.190074  9576 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.190084  9575 consensus_queue.cc:260] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.190758  9576 ts_tablet_manager.cc:1403] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.042s	user 0.012s	sys 0.010s
I20260709 12:25:11.192678  9575 ts_tablet_manager.cc:1434] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:25:11.193596  9575 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.194530  9576 raft_consensus.cc:359] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.195500  9576 raft_consensus.cc:385] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.200870  9575 tablet_bootstrap.cc:654] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.195886  9576 raft_consensus.cc:740] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.203776  9583 raft_consensus.cc:493] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.204504  9583 raft_consensus.cc:515] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.204515  9576 consensus_queue.cc:260] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.207960  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc204ad755be48768bf67ef7e84254bb" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.208846  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc204ad755be48768bf67ef7e84254bb" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
W20260709 12:25:11.209483  9421 leader_election.cc:343] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:11.210389  9420 leader_election.cc:343] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.210834  9420 leader_election.cc:304] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.211509  9574 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.211585  9576 ts_tablet_manager.cc:1434] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.001s
I20260709 12:25:11.212212  9574 ts_tablet_manager.cc:1403] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.032s	user 0.009s	sys 0.009s
I20260709 12:25:11.213542  9576 tablet_bootstrap.cc:492] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.214622  9577 tablet_bootstrap.cc:492] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.215147  9577 ts_tablet_manager.cc:1403] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.050s	user 0.018s	sys 0.001s
I20260709 12:25:11.220919  9576 tablet_bootstrap.cc:654] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.208201  9583 leader_election.cc:290] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:11.223069  9583 raft_consensus.cc:2749] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.224066  9583 raft_consensus.cc:493] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.224730  9583 raft_consensus.cc:515] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.227439  9583 leader_election.cc:290] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:11.228096  9575 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.222255  9577 raft_consensus.cc:359] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.229766  9575 ts_tablet_manager.cc:1403] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.007s
I20260709 12:25:11.229818  9577 raft_consensus.cc:385] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.231696  9577 raft_consensus.cc:740] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.232621  9575 raft_consensus.cc:359] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.233191  9576 tablet_bootstrap.cc:492] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.232715  9574 raft_consensus.cc:359] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.233386  9575 raft_consensus.cc:385] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.233696  9574 raft_consensus.cc:385] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.233906  9576 ts_tablet_manager.cc:1403] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.003s
I20260709 12:25:11.233945  9575 raft_consensus.cc:740] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.234216  9574 raft_consensus.cc:740] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.234965  9575 consensus_queue.cc:260] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.235167  9574 consensus_queue.cc:260] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.237316  9575 ts_tablet_manager.cc:1434] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:25:11.237514  9574 ts_tablet_manager.cc:1434] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260709 12:25:11.238382  9575 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.238801  9574 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.243376  9576 raft_consensus.cc:359] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.244148  9576 raft_consensus.cc:385] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.244426  9576 raft_consensus.cc:740] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.244612  9575 tablet_bootstrap.cc:654] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.245728  9574 tablet_bootstrap.cc:654] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.245277  9576 consensus_queue.cc:260] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.259632  9576 ts_tablet_manager.cc:1434] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 12:25:11.260778  9584 raft_consensus.cc:493] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.261586  9575 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.261385  9584 raft_consensus.cc:515] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.262085  9575 ts_tablet_manager.cc:1403] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.024s	user 0.017s	sys 0.004s
I20260709 12:25:11.235055  9577 consensus_queue.cc:260] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.265035  9575 raft_consensus.cc:359] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.265408  9577 ts_tablet_manager.cc:1434] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.050s	user 0.006s	sys 0.000s
W20260709 12:25:11.265430  9271 consensus_peers.cc:597] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:11.266587  9577 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.268294  9575 raft_consensus.cc:385] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.268646  9575 raft_consensus.cc:740] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.269510  9575 consensus_queue.cc:260] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.269834  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:11.270569  9529 raft_consensus.cc:2468] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:11.268513  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.271762  9346 leader_election.cc:304] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.272199  9307 raft_consensus.cc:2468] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:11.272724  9616 raft_consensus.cc:2804] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.273119  9616 raft_consensus.cc:493] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.273451  9616 raft_consensus.cc:3060] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.274492  9574 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.274995  9574 ts_tablet_manager.cc:1403] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.036s	user 0.015s	sys 0.006s
I20260709 12:25:11.274878  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b494d68a603a4ba5a28ee8514cfa3f41" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.275671  9307 raft_consensus.cc:2468] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:11.277740  9421 leader_election.cc:304] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:11.277971  9574 raft_consensus.cc:359] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.278591  9583 raft_consensus.cc:493] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.279181  9583 raft_consensus.cc:515] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.279583  9575 ts_tablet_manager.cc:1434] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:25:11.280134  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b494d68a603a4ba5a28ee8514cfa3f41" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.280751  9575 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.280844  9382 raft_consensus.cc:2468] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:11.281430  9583 leader_election.cc:290] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.281980  9583 raft_consensus.cc:2804] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.282305  9583 raft_consensus.cc:493] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.282629  9583 raft_consensus.cc:3060] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.283360  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffb83e944aed43718284cb48f994efcc" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:11.284044  9529 raft_consensus.cc:2468] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:11.285342  9420 leader_election.cc:304] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:11.263684  9584 leader_election.cc:290] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.287467  9575 tablet_bootstrap.cc:654] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.287361  9661 raft_consensus.cc:2804] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.287930  9661 raft_consensus.cc:493] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.288281  9661 raft_consensus.cc:3060] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.291574  9583 raft_consensus.cc:515] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.293190  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffb83e944aed43718284cb48f994efcc" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.298620  9307 raft_consensus.cc:2468] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:11.278726  9574 raft_consensus.cc:385] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.300508  9574 raft_consensus.cc:740] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.301538  9574 consensus_queue.cc:260] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.305727  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b494d68a603a4ba5a28ee8514cfa3f41" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:11.301970  9616 raft_consensus.cc:515] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.306326  9307 raft_consensus.cc:3060] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.295859  9661 raft_consensus.cc:515] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.308059  9616 leader_election.cc:290] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.260947  9576 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.309226  9661 leader_election.cc:290] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.309264  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b494d68a603a4ba5a28ee8514cfa3f41" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:11.310009  9382 raft_consensus.cc:3060] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.312471  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:11.312942  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffb83e944aed43718284cb48f994efcc" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:11.313108  9529 raft_consensus.cc:3060] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.313552  9530 raft_consensus.cc:3060] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.316870  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffb83e944aed43718284cb48f994efcc" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:11.316910  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:11.317536  9305 raft_consensus.cc:3060] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.317739  9306 raft_consensus.cc:3060] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.317797  9618 consensus_queue.cc:1048] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:25:11.325042  9575 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.325614  9575 ts_tablet_manager.cc:1403] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.045s	user 0.008s	sys 0.003s
I20260709 12:25:11.326257  9382 raft_consensus.cc:2468] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:11.327832  9421 leader_election.cc:304] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.328729  9575 raft_consensus.cc:359] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.329438  9575 raft_consensus.cc:385] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.329617  9576 tablet_bootstrap.cc:654] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.329728  9575 raft_consensus.cc:740] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.330631  9575 consensus_queue.cc:260] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.330962  9305 raft_consensus.cc:2468] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:11.331599  9661 raft_consensus.cc:2804] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.323025  9307 raft_consensus.cc:2468] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:11.334067  9529 raft_consensus.cc:2468] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:11.332828  9306 raft_consensus.cc:2468] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:11.336900  9346 leader_election.cc:304] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.336598  9421 leader_election.cc:304] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:11.338099  9616 raft_consensus.cc:2804] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.338480  9662 raft_consensus.cc:2804] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.335091  9530 raft_consensus.cc:2468] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:11.306387  9574 ts_tablet_manager.cc:1434] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.001s
I20260709 12:25:11.335057  9661 raft_consensus.cc:697] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:11.341015  9574 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.341125  9661 consensus_queue.cc:237] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.298062  9577 tablet_bootstrap.cc:654] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.335609  9575 ts_tablet_manager.cc:1434] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:25:11.338676  9583 leader_election.cc:290] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:11.343972  9575 tablet_bootstrap.cc:492] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.350594  9575 tablet_bootstrap.cc:654] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.350994  9612 consensus_queue.cc:1048] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:11.351969  9662 raft_consensus.cc:697] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:11.352854  9616 raft_consensus.cc:697] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:11.353780  9576 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.357426  9574 tablet_bootstrap.cc:654] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.368965  9576 ts_tablet_manager.cc:1403] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.108s	user 0.014s	sys 0.005s
I20260709 12:25:11.371604  9662 consensus_queue.cc:237] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.373344  9456 raft_consensus.cc:3060] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.381036  9577 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.381500  9577 ts_tablet_manager.cc:1403] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.115s	user 0.018s	sys 0.000s
I20260709 12:25:11.375754  9582 raft_consensus.cc:493] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.382325  9616 consensus_queue.cc:237] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.383424  9582 raft_consensus.cc:515] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.387343  9618 raft_consensus.cc:493] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.387907  9618 raft_consensus.cc:515] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.392472  9618 leader_election.cc:290] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:11.392674  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160804417e4443b9b25105c759111c8b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.393659  9380 raft_consensus.cc:2468] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:11.394727  9576 raft_consensus.cc:359] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.395430  9576 raft_consensus.cc:385] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.395744  9576 raft_consensus.cc:740] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.396322  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.396486  9576 consensus_queue.cc:260] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.396974  9305 raft_consensus.cc:2468] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:11.398658  9576 ts_tablet_manager.cc:1434] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.001s
I20260709 12:25:11.399688  9576 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.400305  9582 leader_election.cc:290] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.403396  9612 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.404053  9612 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.407966  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.408638  9379 raft_consensus.cc:2468] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:11.409054  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.409710  9455 raft_consensus.cc:2468] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:11.409525  9582 consensus_queue.cc:1048] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:11.413137  9612 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.420706  9576 tablet_bootstrap.cc:654] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.422202  9495 leader_election.cc:304] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.423194  9582 raft_consensus.cc:2804] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.423717  9582 raft_consensus.cc:493] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.424147  9582 raft_consensus.cc:3060] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.424341  9495 leader_election.cc:304] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:11.418152  9272 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.415611  9577 raft_consensus.cc:359] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.425594  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160804417e4443b9b25105c759111c8b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.433252  9577 raft_consensus.cc:385] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.433938  9577 raft_consensus.cc:740] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.435408  9577 consensus_queue.cc:260] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.452651  9577 ts_tablet_manager.cc:1434] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.071s	user 0.009s	sys 0.000s
I20260709 12:25:11.454025  9577 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.447676  9618 raft_consensus.cc:2804] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.460906  9612 consensus_queue.cc:1048] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.449050  9586 raft_consensus.cc:2804] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.465093  9618 raft_consensus.cc:493] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.473961  9618 raft_consensus.cc:3060] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.471446  9586 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.475113  9575 tablet_bootstrap.cc:492] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.475383  9586 raft_consensus.cc:3060] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.475696  9575 ts_tablet_manager.cc:1403] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.132s	user 0.012s	sys 0.012s
I20260709 12:25:11.478487  9575 raft_consensus.cc:359] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.479172  9575 raft_consensus.cc:385] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.480288  9575 raft_consensus.cc:740] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
W20260709 12:25:11.480804  9495 leader_election.cc:343] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.481032  9575 consensus_queue.cc:260] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.483143  9575 ts_tablet_manager.cc:1434] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:11.465374  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.464716  9582 raft_consensus.cc:515] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.490139  9575 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.491667  9576 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.492311  9576 ts_tablet_manager.cc:1403] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.093s	user 0.011s	sys 0.001s
I20260709 12:25:11.493191  9203 catalog_manager.cc:5697] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:11.488888  9618 raft_consensus.cc:515] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.496069  9576 raft_consensus.cc:359] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.496575  9577 tablet_bootstrap.cc:654] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.498090  9574 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.499809  9575 tablet_bootstrap.cc:654] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.517024  9574 ts_tablet_manager.cc:1403] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.176s	user 0.025s	sys 0.014s
I20260709 12:25:11.516119  9576 raft_consensus.cc:385] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.513978  9582 leader_election.cc:290] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.521590  9678 consensus_queue.cc:1048] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:11.493202  9619 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.522385  9305 raft_consensus.cc:3060] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.522622  9619 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.520685  9576 raft_consensus.cc:740] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.524479  9576 consensus_queue.cc:260] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.525213  9619 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.525005  9574 raft_consensus.cc:359] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.525739  9574 raft_consensus.cc:385] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.526024  9574 raft_consensus.cc:740] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.526701  9574 consensus_queue.cc:260] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.529117  9574 ts_tablet_manager.cc:1434] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:25:11.503310  9618 leader_election.cc:290] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:11.530161  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.529841  9618 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.533854  9574 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.530841  9618 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:11.516755  9272 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:42279, user_credentials={real_user=slave}} blocked reactor thread for 46410.4us
I20260709 12:25:11.488287  9586 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.535535  9619 raft_consensus.cc:493] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.539054  9616 consensus_queue.cc:1048] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260709 12:25:11.536288  9494 leader_election.cc:343] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.544364  9530 raft_consensus.cc:3060] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.510475  9201 catalog_manager.cc:5697] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:11.561789  9576 ts_tablet_manager.cc:1434] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.069s	user 0.007s	sys 0.004s
I20260709 12:25:11.562995  9576 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.568684  9616 consensus_queue.cc:1048] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.541376  9619 raft_consensus.cc:515] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.587551  9586 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.588608  9574 tablet_bootstrap.cc:654] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.589326  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:11.590073  9380 raft_consensus.cc:3060] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.598034  9577 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.598786  9577 ts_tablet_manager.cc:1403] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.145s	user 0.010s	sys 0.021s
I20260709 12:25:11.600899  9576 tablet_bootstrap.cc:654] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:11.577490  9271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 81500.9us
W20260709 12:25:11.604741  9495 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 89450.7us
I20260709 12:25:11.613950  9618 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:11.614895  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:11.618184  9495 leader_election.cc:343] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.619626  9619 leader_election.cc:290] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:11.619767  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:11.621181  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160804417e4443b9b25105c759111c8b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:11.621804  9381 raft_consensus.cc:3060] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.624588  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:11.625257  9306 raft_consensus.cc:3060] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.626664  9380 raft_consensus.cc:2468] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:11.628060  9272 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.629189  9682 raft_consensus.cc:2804] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.629663  9682 raft_consensus.cc:697] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:11.630299  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "160804417e4443b9b25105c759111c8b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:11.630456  9682 consensus_queue.cc:237] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.631724  9304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.632418  9688 raft_consensus.cc:493] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.633129  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.632998  9688 raft_consensus.cc:515] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
W20260709 12:25:11.635221  9494 leader_election.cc:343] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.636293  9577 raft_consensus.cc:359] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.637737  9577 raft_consensus.cc:385] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.638417  9577 raft_consensus.cc:740] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.639358  9456 raft_consensus.cc:3060] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.640082  9577 consensus_queue.cc:260] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.630961  9307 raft_consensus.cc:3060] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.631865  9575 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.642444  9575 ts_tablet_manager.cc:1403] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.153s	user 0.018s	sys 0.003s
I20260709 12:25:11.643669  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc204ad755be48768bf67ef7e84254bb" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.648473  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.649046  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc204ad755be48768bf67ef7e84254bb" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.649807  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:11.649978  9453 raft_consensus.cc:2468] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:11.650406  9454 raft_consensus.cc:3060] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.651765  9494 leader_election.cc:304] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:11.654732  9619 raft_consensus.cc:2804] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.655190  9619 raft_consensus.cc:493] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.655617  9619 raft_consensus.cc:3060] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.656484  9528 raft_consensus.cc:3060] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.659502  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:11.679641  9495 leader_election.cc:343] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.680408  9495 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:11.681479  9618 raft_consensus.cc:2749] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.682296  9688 leader_election.cc:290] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.696669  9575 raft_consensus.cc:359] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.697597  9575 raft_consensus.cc:385] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.697978  9575 raft_consensus.cc:740] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.698894  9575 consensus_queue.cc:260] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.701609  9307 raft_consensus.cc:2468] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:11.702987  9698 raft_consensus.cc:493] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:11.703953  9495 leader_election.cc:343] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:11.708844  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 51898us
I20260709 12:25:11.708910  9495 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
W20260709 12:25:11.704479  9347 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 71116.7us
I20260709 12:25:11.700224  9577 ts_tablet_manager.cc:1434] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.101s	user 0.012s	sys 0.000s
I20260709 12:25:11.711807  9688 raft_consensus.cc:2749] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.712831  9495 leader_election.cc:304] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
W20260709 12:25:11.713912  9495 leader_election.cc:343] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.708658  9698 raft_consensus.cc:515] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.706673  9204 catalog_manager.cc:5697] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:11.715660  9688 raft_consensus.cc:2804] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.716159  9575 ts_tablet_manager.cc:1434] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.073s	user 0.008s	sys 0.000s
I20260709 12:25:11.716233  9688 raft_consensus.cc:697] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:11.711795  9577 tablet_bootstrap.cc:492] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.717377  9688 consensus_queue.cc:237] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.707109  9576 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.718833  9576 ts_tablet_manager.cc:1403] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.156s	user 0.015s	sys 0.007s
I20260709 12:25:11.719700  9698 leader_election.cc:290] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
W20260709 12:25:11.714665  9495 leader_election.cc:343] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.722293  9619 raft_consensus.cc:515] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.723431  9495 leader_election.cc:304] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.717547  9381 raft_consensus.cc:2468] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:11.717377  9575 tablet_bootstrap.cc:492] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.707149  9574 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.726011  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc204ad755be48768bf67ef7e84254bb" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:11.726272  9574 ts_tablet_manager.cc:1403] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.193s	user 0.007s	sys 0.022s
I20260709 12:25:11.721968  9576 raft_consensus.cc:359] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.727043  9576 raft_consensus.cc:385] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.727517  9576 raft_consensus.cc:740] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.726683  9453 raft_consensus.cc:3060] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.732273  9574 raft_consensus.cc:359] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.733131  9574 raft_consensus.cc:385] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.734051  9576 consensus_queue.cc:260] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.734798  9574 raft_consensus.cc:740] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.740657  9453 raft_consensus.cc:2468] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:11.742197  9494 leader_election.cc:304] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:11.728307  9619 leader_election.cc:290] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.744596  9619 raft_consensus.cc:2804] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.745280  9576 ts_tablet_manager.cc:1434] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:25:11.745421  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:11.746173  9530 raft_consensus.cc:2468] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:11.745307  9619 raft_consensus.cc:697] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:11.747938  9574 consensus_queue.cc:260] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.751668  9619 consensus_queue.cc:237] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.752981  9574 ts_tablet_manager.cc:1434] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.001s
I20260709 12:25:11.759758  9694 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.760749  9694 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.757858  9576 tablet_bootstrap.cc:492] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.763654  9694 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.764356  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.765635  9453 raft_consensus.cc:3060] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:11.766000  9347 leader_election.cc:343] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.748553  9577 tablet_bootstrap.cc:654] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.755719  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.769974  9689 raft_consensus.cc:2749] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.771581  9574 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.756152  9454 raft_consensus.cc:2468] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:11.732191  9306 raft_consensus.cc:2468] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:11.756480  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc204ad755be48768bf67ef7e84254bb" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:11.775209  9495 leader_election.cc:304] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.777005  9662 raft_consensus.cc:493] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5f89c82eda14a9ca425fae398bae986)
I20260709 12:25:11.756927  9346 leader_election.cc:304] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.758761  9205 catalog_manager.cc:5697] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:11.777549  9662 raft_consensus.cc:515] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:11.779510  9346 leader_election.cc:343] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.779611  9586 raft_consensus.cc:493] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.780215  9586 raft_consensus.cc:515] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.780894  9616 raft_consensus.cc:2804] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.781296  9616 raft_consensus.cc:493] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.781628  9616 raft_consensus.cc:3060] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.781785  9662 leader_election.cc:290] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:11.782220  9586 leader_election.cc:290] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.784227  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:11.785670  9271 leader_election.cc:343] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.786563  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:11.787758  9495 leader_election.cc:343] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:11.788400  9346 leader_election.cc:343] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.788797  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:11.790820  9688 raft_consensus.cc:2804] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.791141  9577 tablet_bootstrap.cc:492] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.788926  9346 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:11.791658  9577 ts_tablet_manager.cc:1403] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.080s	user 0.013s	sys 0.000s
I20260709 12:25:11.791860  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:11.754084  9575 tablet_bootstrap.cc:654] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.792963  9688 raft_consensus.cc:697] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:11.793022  9584 raft_consensus.cc:2749] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.793646  9698 consensus_queue.cc:1048] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.794382  9688 consensus_queue.cc:237] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:11.795894  9577 raft_consensus.cc:359] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.796599  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c302268ead84f94a04993f710b731e6" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:11.798065  9421 leader_election.cc:304] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:11.799242  9662 raft_consensus.cc:2749] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:11.801903  9272 leader_election.cc:343] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.800747  9574 tablet_bootstrap.cc:654] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.802402  9272 leader_election.cc:304] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.803484  9586 raft_consensus.cc:2749] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.802007  9576 tablet_bootstrap.cc:654] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.804970  9616 raft_consensus.cc:515] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.807911  9204 catalog_manager.cc:5697] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:11.809660  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:11.810186  9616 leader_election.cc:290] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:11.810472  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:11.811208  9530 raft_consensus.cc:3060] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.796615  9577 raft_consensus.cc:385] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.816097  9577 raft_consensus.cc:740] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.816886  9577 consensus_queue.cc:260] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.822166  9454 raft_consensus.cc:3060] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.822922  9661 consensus_queue.cc:1048] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.823802  9577 ts_tablet_manager.cc:1434] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 12:25:11.824870  9577 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.820657  9530 raft_consensus.cc:2468] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:11.829766  9454 raft_consensus.cc:2468] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:11.830610  9346 leader_election.cc:304] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.832358  9616 raft_consensus.cc:2804] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:11.835364  9616 raft_consensus.cc:697] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:11.836226  9616 consensus_queue.cc:237] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.836846  9577 tablet_bootstrap.cc:654] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.837971  9661 consensus_queue.cc:1048] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.840833  9575 tablet_bootstrap.cc:492] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.841410  9575 ts_tablet_manager.cc:1403] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.124s	user 0.010s	sys 0.001s
I20260709 12:25:11.856164  9575 raft_consensus.cc:359] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.857173  9575 raft_consensus.cc:385] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.857515  9575 raft_consensus.cc:740] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.860127  9575 consensus_queue.cc:260] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.853106  9661 consensus_queue.cc:1048] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:25:11.863173  9575 ts_tablet_manager.cc:1434] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.001s
I20260709 12:25:11.864790  9575 tablet_bootstrap.cc:492] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:11.868036  9689 raft_consensus.cc:493] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.863015  9204 catalog_manager.cc:5697] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:11.868578  9689 raft_consensus.cc:515] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.869583  9204 catalog_manager.cc:5697] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:11.874454  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47c1f0a1e81a4ff0a3652fafff1288b4" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:11.877987  9495 leader_election.cc:343] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.881753  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47c1f0a1e81a4ff0a3652fafff1288b4" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:11.883448  9494 leader_election.cc:343] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.883912  9575 tablet_bootstrap.cc:654] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.886029  9574 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.883942  9494 leader_election.cc:304] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:11.886737  9574 ts_tablet_manager.cc:1403] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.115s	user 0.006s	sys 0.014s
I20260709 12:25:11.887118  9688 raft_consensus.cc:2749] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:11.875275  9689 leader_election.cc:290] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.897043  9574 raft_consensus.cc:359] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.898891  9574 raft_consensus.cc:385] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.899446  9574 raft_consensus.cc:740] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.900626  9574 consensus_queue.cc:260] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.903405  9574 ts_tablet_manager.cc:1434] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.003s
I20260709 12:25:11.904604  9574 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.910804  9574 tablet_bootstrap.cc:654] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.918291  9662 consensus_queue.cc:1048] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.918987  9205 catalog_manager.cc:5697] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:11.927736  9576 tablet_bootstrap.cc:492] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:11.928392  9576 ts_tablet_manager.cc:1403] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.182s	user 0.012s	sys 0.004s
I20260709 12:25:11.931766  9576 raft_consensus.cc:359] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.934155  9576 raft_consensus.cc:385] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.937347  9576 raft_consensus.cc:740] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.940668  9577 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:11.944681  9577 ts_tablet_manager.cc:1403] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.120s	user 0.019s	sys 0.003s
I20260709 12:25:11.942142  9616 consensus_queue.cc:1048] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:11.948220  9577 raft_consensus.cc:359] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.941313  9576 consensus_queue.cc:260] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.948968  9577 raft_consensus.cc:385] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.949457  9577 raft_consensus.cc:740] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.950311  9577 consensus_queue.cc:260] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.951190  9576 ts_tablet_manager.cc:1434] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:25:11.952227  9576 tablet_bootstrap.cc:492] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:11.952589  9577 ts_tablet_manager.cc:1434] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:11.953096  9616 raft_consensus.cc:493] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.953574  9616 raft_consensus.cc:515] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.955916  9616 leader_election.cc:290] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:11.953568  9577 tablet_bootstrap.cc:492] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:11.960381  9576 tablet_bootstrap.cc:654] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.965364  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:11.966137  9575 tablet_bootstrap.cc:492] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:11.966101  9453 raft_consensus.cc:2468] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:11.966747  9575 ts_tablet_manager.cc:1403] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.102s	user 0.011s	sys 0.015s
I20260709 12:25:11.967789  9346 leader_election.cc:304] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:11.969551  9698 raft_consensus.cc:2804] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:11.969928  9698 raft_consensus.cc:493] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.970283  9698 raft_consensus.cc:3060] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.970335  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:11.971897  9347 leader_election.cc:343] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.977872  9698 raft_consensus.cc:515] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:11.980381  9574 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:11.980931  9574 ts_tablet_manager.cc:1403] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.077s	user 0.009s	sys 0.016s
I20260709 12:25:11.982347  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:11.982620  9698 leader_election.cc:290] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
W20260709 12:25:11.983994  9347 leader_election.cc:343] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:11.984313  9574 raft_consensus.cc:359] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.985060  9574 raft_consensus.cc:385] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.986454  9574 raft_consensus.cc:740] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.987056  9575 raft_consensus.cc:359] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.987365  9574 consensus_queue.cc:260] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:11.990618  9575 raft_consensus.cc:385] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:11.991001  9575 raft_consensus.cc:740] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:11.992015  9575 consensus_queue.cc:260] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.992452  9574 ts_tablet_manager.cc:1434] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:25:11.993580  9574 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:11.994369  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:11.994995  9453 raft_consensus.cc:3060] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:11.997102  9689 raft_consensus.cc:493] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.997628  9689 raft_consensus.cc:515] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:11.990165  9577 tablet_bootstrap.cc:654] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:11.999874  9689 leader_election.cc:290] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.001096  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dbb3c7d37504398aab89200647dab24" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.001588  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dbb3c7d37504398aab89200647dab24" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.002236  9381 raft_consensus.cc:2468] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
W20260709 12:25:12.002777  9494 leader_election.cc:343] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.003894  9574 tablet_bootstrap.cc:654] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.003795  9495 leader_election.cc:304] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:12.004611  9689 raft_consensus.cc:2804] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.005048  9689 raft_consensus.cc:493] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.005450  9689 raft_consensus.cc:3060] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.002913  9453 raft_consensus.cc:2468] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:12.009897  9346 leader_election.cc:304] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:12.011143  9616 raft_consensus.cc:2804] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.011648  9616 raft_consensus.cc:697] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:12.012550  9616 consensus_queue.cc:237] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.020658  9575 ts_tablet_manager.cc:1434] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.054s	user 0.006s	sys 0.003s
I20260709 12:25:12.021848  9575 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.023607  9689 raft_consensus.cc:515] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.028311  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dbb3c7d37504398aab89200647dab24" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:12.029116  9381 raft_consensus.cc:3060] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:12.037546  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:12.037838  9586 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.042634  9576 tablet_bootstrap.cc:492] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.043020  9576 ts_tablet_manager.cc:1403] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.091s	user 0.015s	sys 0.024s
I20260709 12:25:12.038578  9586 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.046154  9586 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:12.048035  9577 tablet_bootstrap.cc:492] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:12.048578  9577 ts_tablet_manager.cc:1403] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.095s	user 0.015s	sys 0.008s
I20260709 12:25:12.049367  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.050009  9528 raft_consensus.cc:2468] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:12.051276  9271 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:12.045827  9576 raft_consensus.cc:359] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.052372  9586 raft_consensus.cc:2804] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.052964  9586 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.053383  9586 raft_consensus.cc:3060] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.055042  9574 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:12.052572  9576 raft_consensus.cc:385] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.055480  9576 raft_consensus.cc:740] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.055583  9574 ts_tablet_manager.cc:1403] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.062s	user 0.007s	sys 0.012s
I20260709 12:25:12.056107  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.056353  9576 consensus_queue.cc:260] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
W20260709 12:25:12.057623  9271 leader_election.cc:343] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.058853  9689 leader_election.cc:290] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.060039  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dbb3c7d37504398aab89200647dab24" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
W20260709 12:25:12.061331  9494 leader_election.cc:343] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.061439  9201 catalog_manager.cc:5697] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:12.061333  9576 ts_tablet_manager.cc:1434] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.001s
I20260709 12:25:12.063242  9574 raft_consensus.cc:359] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.064035  9574 raft_consensus.cc:385] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.064292  9574 raft_consensus.cc:740] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.064400  9576 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.064982  9574 consensus_queue.cc:260] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.065403  9381 raft_consensus.cc:2468] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:12.066629  9612 consensus_queue.cc:1048] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.066828  9495 leader_election.cc:304] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:12.068055  9689 raft_consensus.cc:2804] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.068575  9689 raft_consensus.cc:697] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:12.069398  9689 consensus_queue.cc:237] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.070197  9574 ts_tablet_manager.cc:1434] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260709 12:25:12.071342  9574 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:12.077256  9574 tablet_bootstrap.cc:654] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.064420  9575 tablet_bootstrap.cc:654] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.092792  9680 consensus_queue.cc:1048] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.096442  9576 tablet_bootstrap.cc:654] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.097683  9680 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.098212  9680 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.102550  9680 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:12.103541  9575 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.104081  9575 ts_tablet_manager.cc:1403] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.082s	user 0.010s	sys 0.003s
I20260709 12:25:12.104460  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.105189  9528 raft_consensus.cc:2468] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:12.106823  9271 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:12.087896  9586 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.110649  9576 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.051183  9577 raft_consensus.cc:359] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.108130  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.111178  9576 ts_tablet_manager.cc:1403] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.047s	user 0.006s	sys 0.005s
I20260709 12:25:12.112479  9577 raft_consensus.cc:385] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.112879  9577 raft_consensus.cc:740] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.114061  9576 raft_consensus.cc:359] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.114763  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
W20260709 12:25:12.115378  9271 leader_election.cc:343] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.115988  9678 raft_consensus.cc:2804] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.116375  9678 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.116758  9678 raft_consensus.cc:3060] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.116923  9586 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:12.118446  9575 raft_consensus.cc:359] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.119136  9575 raft_consensus.cc:385] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.119427  9575 raft_consensus.cc:740] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.120123  9575 consensus_queue.cc:260] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.120639  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:12.114784  9576 raft_consensus.cc:385] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.121359  9528 raft_consensus.cc:3060] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.121590  9576 raft_consensus.cc:740] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.122398  9576 consensus_queue.cc:260] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.125725  9575 ts_tablet_manager.cc:1434] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:25:12.126742  9575 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.128360  9576 ts_tablet_manager.cc:1434] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
W20260709 12:25:12.128659  9271 leader_election.cc:343] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.129463  9576 tablet_bootstrap.cc:492] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.131596  9205 catalog_manager.cc:5697] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:12.136422  9575 tablet_bootstrap.cc:654] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.113852  9577 consensus_queue.cc:260] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.144423  9576 tablet_bootstrap.cc:654] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.145062  9577 ts_tablet_manager.cc:1434] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.096s	user 0.008s	sys 0.000s
I20260709 12:25:12.145170  9574 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:12.145855  9574 ts_tablet_manager.cc:1403] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.075s	user 0.008s	sys 0.025s
I20260709 12:25:12.145649  9678 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.148754  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:12.149142  9574 raft_consensus.cc:359] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.150039  9574 raft_consensus.cc:385] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.151281  9574 raft_consensus.cc:740] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.152202  9574 consensus_queue.cc:260] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.153927  9577 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
W20260709 12:25:12.153791  9271 leader_election.cc:343] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.154631  9574 ts_tablet_manager.cc:1434] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:25:12.154891  9689 raft_consensus.cc:493] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.155670  9574 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:12.155395  9689 raft_consensus.cc:515] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.156527  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:12.157138  9530 raft_consensus.cc:3060] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.157423  9689 leader_election.cc:290] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:12.158982  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a97a9ba3c90f464eacdc4e78d6a91f5e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.159690  9306 raft_consensus.cc:2468] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:12.160818  9575 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.161295  9575 ts_tablet_manager.cc:1403] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.035s	user 0.011s	sys 0.000s
I20260709 12:25:12.163677  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a97a9ba3c90f464eacdc4e78d6a91f5e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.164100  9575 raft_consensus.cc:359] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.164774  9575 raft_consensus.cc:385] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.165033  9575 raft_consensus.cc:740] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
W20260709 12:25:12.165380  9495 leader_election.cc:343] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.165738  9575 consensus_queue.cc:260] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.154579  9576 tablet_bootstrap.cc:492] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.167126  9576 ts_tablet_manager.cc:1403] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.009s
I20260709 12:25:12.167024  9495 leader_election.cc:304] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:12.170298  9576 raft_consensus.cc:359] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.170967  9576 raft_consensus.cc:385] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.171226  9576 raft_consensus.cc:740] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.172040  9574 tablet_bootstrap.cc:654] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.172297  9689 raft_consensus.cc:2804] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.172703  9689 raft_consensus.cc:493] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.173079  9689 raft_consensus.cc:3060] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.175525  9530 raft_consensus.cc:2468] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:12.176561  9678 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:12.178740  9577 tablet_bootstrap.cc:654] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.143189  9528 raft_consensus.cc:2468] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:12.181026  9271 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:12.182080  9586 raft_consensus.cc:2804] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.182567  9586 raft_consensus.cc:697] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:12.183533  9586 consensus_queue.cc:237] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.183923  9689 raft_consensus.cc:515] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.182487  9271 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:12.186666  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a97a9ba3c90f464eacdc4e78d6a91f5e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:12.186837  9678 raft_consensus.cc:2804] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.187244  9306 raft_consensus.cc:3060] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.187408  9678 raft_consensus.cc:697] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:12.189072  9575 ts_tablet_manager.cc:1434] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.001s
I20260709 12:25:12.190243  9575 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.189752  9678 consensus_queue.cc:237] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.195910  9689 leader_election.cc:290] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:12.196707  9576 consensus_queue.cc:260] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.198544  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a97a9ba3c90f464eacdc4e78d6a91f5e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:12.198830  9576 ts_tablet_manager.cc:1434] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.000s
W20260709 12:25:12.200588  9495 leader_election.cc:343] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.200897  9576 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.202736  9688 consensus_queue.cc:1048] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:12.204545  9688 consensus_queue.cc:1048] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.208576  9575 tablet_bootstrap.cc:654] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.211423  9576 tablet_bootstrap.cc:654] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.218712  9577 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:12.219779  9577 ts_tablet_manager.cc:1403] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.066s	user 0.015s	sys 0.006s
I20260709 12:25:12.221463  9576 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.221972  9576 ts_tablet_manager.cc:1403] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.004s
I20260709 12:25:12.222675  9306 raft_consensus.cc:2468] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:12.224234  9381 raft_consensus.cc:3060] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.232146  9576 raft_consensus.cc:359] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.232906  9576 raft_consensus.cc:385] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.233209  9576 raft_consensus.cc:740] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.233903  9576 consensus_queue.cc:260] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.236146  9576 ts_tablet_manager.cc:1434] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.003s
I20260709 12:25:12.237187  9576 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.224958  9577 raft_consensus.cc:359] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.227373  9575 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.239930  9575 ts_tablet_manager.cc:1403] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.050s	user 0.015s	sys 0.000s
I20260709 12:25:12.242053  9688 consensus_queue.cc:1048] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.243408  9576 tablet_bootstrap.cc:654] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.242951  9575 raft_consensus.cc:359] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.243817  9575 raft_consensus.cc:385] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.244098  9575 raft_consensus.cc:740] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.244760  9575 consensus_queue.cc:260] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.246064  9577 raft_consensus.cc:385] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.246449  9577 raft_consensus.cc:740] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.248543  9577 consensus_queue.cc:260] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.253990  9495 leader_election.cc:304] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:12.256484  9618 raft_consensus.cc:2804] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.257009  9618 raft_consensus.cc:697] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:12.257848  9618 consensus_queue.cc:237] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.258967  9577 ts_tablet_manager.cc:1434] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.001s
I20260709 12:25:12.259217  9575 ts_tablet_manager.cc:1434] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:25:12.260051  9577 tablet_bootstrap.cc:492] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:12.268795  9575 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.271369  9205 catalog_manager.cc:5697] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:12.278156  9577 tablet_bootstrap.cc:654] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.280145  9661 raft_consensus.cc:493] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.283391  9616 raft_consensus.cc:493] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.283478  9688 consensus_queue.cc:1048] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:12.284081  9616 raft_consensus.cc:515] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.286212  9616 leader_election.cc:290] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:12.286698  9616 raft_consensus.cc:493] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.280689  9661 raft_consensus.cc:515] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.287473  9616 raft_consensus.cc:515] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.295521  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47c1f0a1e81a4ff0a3652fafff1288b4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.295521  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47c1f0a1e81a4ff0a3652fafff1288b4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.296519  9528 raft_consensus.cc:2468] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:12.296396  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.298089  9530 raft_consensus.cc:2468] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:12.299407  9420 leader_election.cc:304] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:12.300450  9662 raft_consensus.cc:2804] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.300806  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.300848  9662 raft_consensus.cc:493] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.301512  9307 raft_consensus.cc:2468] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:12.301751  9662 raft_consensus.cc:3060] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.303009  9347 leader_election.cc:304] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:12.300477  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.303927  9694 raft_consensus.cc:2804] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.304446  9694 raft_consensus.cc:493] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.304559  9612 raft_consensus.cc:493] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.304963  9694 raft_consensus.cc:3060] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.305414  9346 leader_election.cc:304] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:12.306442  9584 raft_consensus.cc:2804] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.306818  9584 raft_consensus.cc:493] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.307175  9584 raft_consensus.cc:3060] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.308316  9689 raft_consensus.cc:493] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.308902  9689 raft_consensus.cc:515] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:12.310448  9421 leader_election.cc:343] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.310969  9689 leader_election.cc:290] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:12.296528  9454 raft_consensus.cc:2468] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:12.305152  9612 raft_consensus.cc:515] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.319137  9576 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.290279  9661 leader_election.cc:290] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.319916  9576 ts_tablet_manager.cc:1403] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.083s	user 0.013s	sys 0.000s
I20260709 12:25:12.312090  9381 raft_consensus.cc:3060] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.321033  9612 leader_election.cc:290] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.312201  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.323184  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.316790  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.316527  9694 raft_consensus.cc:515] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.324218  9576 raft_consensus.cc:359] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:12.325445  9495 leader_election.cc:343] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.325525  9576 raft_consensus.cc:385] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.326057  9576 raft_consensus.cc:740] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.326181  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.326663  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:12.327025  9456 raft_consensus.cc:2393] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e58af1026634b45bafbbfde02a6ecd0 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:12.327157  9576 consensus_queue.cc:260] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.327291  9307 raft_consensus.cc:3060] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.347555  9271 leader_election.cc:304] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:12.348541  9612 raft_consensus.cc:3060] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.296168  9616 leader_election.cc:290] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.351614  9575 tablet_bootstrap.cc:654] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.357299  9698 consensus_queue.cc:1048] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:12.358366  9662 raft_consensus.cc:515] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.359642  9688 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.360167  9688 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.360824  9574 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:12.361341  9574 ts_tablet_manager.cc:1403] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.206s	user 0.009s	sys 0.022s
I20260709 12:25:12.362214  9688 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:12.368153  9612 raft_consensus.cc:2749] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.363807  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.373602  9662 leader_election.cc:290] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:12.374320  9495 leader_election.cc:343] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.374891  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:12.375739  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:12.369207  9694 leader_election.cc:290] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
W20260709 12:25:12.376322  9421 leader_election.cc:343] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.377139  9574 raft_consensus.cc:359] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.333906  9689 consensus_queue.cc:1048] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260709 12:25:12.380192  9201 catalog_manager.cc:5697] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:12.376385  9528 raft_consensus.cc:3060] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.377857  9574 raft_consensus.cc:385] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.384207  9574 raft_consensus.cc:740] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.384943  9574 consensus_queue.cc:260] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.401463  9574 ts_tablet_manager.cc:1434] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.001s
I20260709 12:25:12.407640  9574 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:12.407835  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.414222  9576 ts_tablet_manager.cc:1434] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.094s	user 0.026s	sys 0.017s
I20260709 12:25:12.415462  9576 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.420080  9307 raft_consensus.cc:2468] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:12.420831  9584 raft_consensus.cc:515] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.423504  9584 leader_election.cc:290] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:12.423802  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:12.425551  9347 leader_election.cc:304] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:12.426301  9528 raft_consensus.cc:2468] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:12.429816  9205 catalog_manager.cc:5697] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:12.433385  9420 leader_election.cc:304] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:12.434553  9661 raft_consensus.cc:2804] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.436219  9575 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.436918  9575 ts_tablet_manager.cc:1403] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.175s	user 0.021s	sys 0.015s
I20260709 12:25:12.440763  9575 raft_consensus.cc:359] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.431476  9694 raft_consensus.cc:2804] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.441697  9575 raft_consensus.cc:385] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.442392  9575 raft_consensus.cc:740] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.442420  9694 raft_consensus.cc:697] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:12.443607  9575 consensus_queue.cc:260] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.443898  9694 consensus_queue.cc:237] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.445596  9744 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.446081  9744 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.446846  9575 ts_tablet_manager.cc:1434] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 12:25:12.448069  9575 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.448235  9744 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.454689  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:12.456204  9347 leader_election.cc:343] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.465919  9575 tablet_bootstrap.cc:654] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:12.469542  9346 leader_election.cc:343] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:12.479331  9346 leader_election.cc:343] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.496634  9577 tablet_bootstrap.cc:492] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:12.497337  9577 ts_tablet_manager.cc:1403] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.237s	user 0.037s	sys 0.021s
I20260709 12:25:12.499624  9744 consensus_queue.cc:1048] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:12.503496  9574 tablet_bootstrap.cc:654] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.505648  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.506332  9380 raft_consensus.cc:2468] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:12.512185  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47c1f0a1e81a4ff0a3652fafff1288b4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:12.513010  9453 raft_consensus.cc:3060] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.513818  9576 tablet_bootstrap.cc:654] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:12.524772  9495 leader_election.cc:343] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.525343  9495 leader_election.cc:304] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:12.519207  9661 raft_consensus.cc:697] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:12.526232  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47c1f0a1e81a4ff0a3652fafff1288b4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:12.527716  9619 raft_consensus.cc:2749] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.529922  9530 raft_consensus.cc:3060] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.528441  9618 consensus_queue.cc:1048] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:25:12.528268  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.533378  9528 raft_consensus.cc:2468] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:12.539548  9575 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.540149  9575 ts_tablet_manager.cc:1403] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.092s	user 0.016s	sys 0.004s
I20260709 12:25:12.533282  9574 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:12.541412  9574 ts_tablet_manager.cc:1403] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.139s	user 0.015s	sys 0.011s
I20260709 12:25:12.540035  9495 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:12.545457  9577 raft_consensus.cc:359] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.547597  9577 raft_consensus.cc:385] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.548063  9577 raft_consensus.cc:740] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.544958  9618 raft_consensus.cc:2804] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.549688  9577 consensus_queue.cc:260] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.551136  9618 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.551124  9612 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.560437  9577 ts_tablet_manager.cc:1434] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.063s	user 0.008s	sys 0.000s
I20260709 12:25:12.553471  9618 raft_consensus.cc:3060] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.562510  9577 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:12.561879  9454 raft_consensus.cc:3060] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.552263  9661 consensus_queue.cc:237] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.553745  9612 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.571482  9530 raft_consensus.cc:2468] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:12.565287  9698 raft_consensus.cc:493] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.574024  9698 raft_consensus.cc:515] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.576103  9577 tablet_bootstrap.cc:654] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.576258  9698 leader_election.cc:290] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.578126  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.553334  9743 consensus_queue.cc:1048] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 12:25:12.593976  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.594303  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.594930  9380 raft_consensus.cc:2468] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
W20260709 12:25:12.595906  9271 leader_election.cc:343] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.596935  9272 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:12.598589  9576 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.599110  9576 ts_tablet_manager.cc:1403] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.184s	user 0.011s	sys 0.014s
I20260709 12:25:12.587663  9612 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.592967  9575 raft_consensus.cc:359] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.600231  9575 raft_consensus.cc:385] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.600502  9575 raft_consensus.cc:740] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.601094  9575 consensus_queue.cc:260] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:12.604429  9271 consensus_peers.cc:597] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:12.604794  9678 raft_consensus.cc:2804] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.608721  9678 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.583917  9346 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:12.609493  9575 ts_tablet_manager.cc:1434] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.069s	user 0.005s	sys 0.002s
I20260709 12:25:12.610419  9575 tablet_bootstrap.cc:492] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.611088  9678 raft_consensus.cc:3060] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.604434  9618 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.602852  9453 raft_consensus.cc:2468] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:12.615653  9743 raft_consensus.cc:2804] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:12.616092  9743 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.616495  9743 raft_consensus.cc:3060] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:12.620106  9751 log.cc:927] Time spent T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:25:12.624084  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 56607.8us
I20260709 12:25:12.625317  9574 raft_consensus.cc:359] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.626040  9574 raft_consensus.cc:385] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.626286  9574 raft_consensus.cc:740] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.626919  9577 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:12.630985  9577 ts_tablet_manager.cc:1403] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.069s	user 0.016s	sys 0.008s
I20260709 12:25:12.630049  9576 raft_consensus.cc:359] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.631520  9576 raft_consensus.cc:385] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.632251  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:12.634678  9205 catalog_manager.cc:5697] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:12.635216  9586 consensus_queue.cc:1048] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.638914  9346 leader_election.cc:304] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:12.636947  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.640499  9577 raft_consensus.cc:359] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.641193  9577 raft_consensus.cc:385] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.641469  9577 raft_consensus.cc:740] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.642191  9698 raft_consensus.cc:2804] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.626937  9574 consensus_queue.cc:260] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.631826  9576 raft_consensus.cc:740] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.642838  9575 tablet_bootstrap.cc:654] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.642787  9698 raft_consensus.cc:697] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:12.643618  9576 consensus_queue.cc:260] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.651178  9619 consensus_queue.cc:1048] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.654148  9574 ts_tablet_manager.cc:1434] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.112s	user 0.005s	sys 0.003s
I20260709 12:25:12.656551  9577 consensus_queue.cc:260] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.657439  9694 consensus_queue.cc:1048] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.647593  9307 raft_consensus.cc:3060] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.630442  9618 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:12.680632  9574 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:12.685631  9576 ts_tablet_manager.cc:1434] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.086s	user 0.007s	sys 0.001s
I20260709 12:25:12.687053  9576 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.652280  9698 consensus_queue.cc:237] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.697363  9577 ts_tablet_manager.cc:1434] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.066s	user 0.005s	sys 0.001s
I20260709 12:25:12.698326  9577 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:12.673363  9678 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.701807  9678 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.702298  9618 raft_consensus.cc:493] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.705997  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:12.707659  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:12.709335  9743 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.709915  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:12.712157  9619 consensus_queue.cc:1048] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.714349  9346 leader_election.cc:304] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:12.719286  9574 tablet_bootstrap.cc:654] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:12.723722  9271 leader_election.cc:343] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.745698  9694 raft_consensus.cc:2749] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:12.750602  9661 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.751658  9661 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.729089  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:12.721757  9205 catalog_manager.cc:5697] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:12.753928  9661 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.727871  9618 raft_consensus.cc:515] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:12.751917  9307 raft_consensus.cc:2468] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:12.720772  9576 tablet_bootstrap.cc:654] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.724524  9379 raft_consensus.cc:2393] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a5f89c82eda14a9ca425fae398bae986 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:12.727624  9382 raft_consensus.cc:3060] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.773340  9577 tablet_bootstrap.cc:654] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.729089  9743 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.776011  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e722577ca9c4859ac9d6f335a1edfc9" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:12.794006  9618 leader_election.cc:290] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.773675  9689 consensus_queue.cc:1048] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:25:12.785787  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:12.787783  9743 raft_consensus.cc:493] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.789605  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.815518  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e722577ca9c4859ac9d6f335a1edfc9" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.814872  9529 raft_consensus.cc:2393] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e07a645c9ee14f11a5d64fa7b460acf9 in current term 1: Already voted for candidate a5f89c82eda14a9ca425fae398bae986 in this term.
I20260709 12:25:12.805030  9743 raft_consensus.cc:515] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.813115  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.835294  9530 raft_consensus.cc:2393] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074a6ac642e4430ba8045181f0118ee3 in current term 1: Already voted for candidate 4e58af1026634b45bafbbfde02a6ecd0 in this term.
W20260709 12:25:12.837500  9271 consensus_peers.cc:597] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:12.828192  9494 leader_election.cc:343] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.839462  9346 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:12.842512  9575 tablet_bootstrap.cc:492] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.843003  9575 ts_tablet_manager.cc:1403] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.233s	user 0.030s	sys 0.014s
I20260709 12:25:12.843586  9698 raft_consensus.cc:2749] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.844921  9743 leader_election.cc:290] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:12.844858  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a97a9ba3c90f464eacdc4e78d6a91f5e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:12.846887  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a97a9ba3c90f464eacdc4e78d6a91f5e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.847448  9575 raft_consensus.cc:359] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.849234  9575 raft_consensus.cc:385] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.849546  9575 raft_consensus.cc:740] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.850368  9575 consensus_queue.cc:260] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
W20260709 12:25:12.857483  9495 leader_election.cc:343] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.858009  9495 leader_election.cc:304] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:12.859617  9382 raft_consensus.cc:2468] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:12.859889  9420 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:12.861207  9272 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:12.861888  9346 leader_election.cc:304] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:12.862778  9381 raft_consensus.cc:3060] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.863592  9688 raft_consensus.cc:2749] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.865589  9612 raft_consensus.cc:2804] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.865718  9495 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:12.866181  9612 raft_consensus.cc:697] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:12.866724  9688 raft_consensus.cc:2804] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:12.867252  9688 raft_consensus.cc:697] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:12.867096  9612 consensus_queue.cc:237] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:12.868153  9688 consensus_queue.cc:237] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.868991  9661 raft_consensus.cc:3060] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:12.875897  9575 ts_tablet_manager.cc:1434] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.000s
I20260709 12:25:12.876969  9575 tablet_bootstrap.cc:492] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.877343  9765 consensus_queue.cc:1048] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.881587  9618 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.882155  9618 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.884600  9618 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:12.887814  9575 tablet_bootstrap.cc:654] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.888437  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:12.889818  9618 raft_consensus.cc:493] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.890319  9618 raft_consensus.cc:515] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.884275  9201 catalog_manager.cc:5697] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:12.903486  9743 raft_consensus.cc:2749] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:25:12.904882  9495 leader_election.cc:343] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.909948  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:12.910324  9619 consensus_queue.cc:1048] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.911262  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:12.911538  9495 leader_election.cc:343] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.912390  9576 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:12.912860  9576 ts_tablet_manager.cc:1403] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.226s	user 0.020s	sys 0.023s
I20260709 12:25:12.904882  9575 tablet_bootstrap.cc:492] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:12.914721  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:12.916322  9494 leader_election.cc:343] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.916782  9494 leader_election.cc:304] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:12.917160  9576 raft_consensus.cc:359] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:12.917683  9619 raft_consensus.cc:2749] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.917919  9576 raft_consensus.cc:385] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.918293  9576 raft_consensus.cc:740] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.906684  9618 leader_election.cc:290] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:12.919126  9576 consensus_queue.cc:260] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:12.924078  9495 leader_election.cc:343] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:12.926234  9575 ts_tablet_manager.cc:1403] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.049s	user 0.006s	sys 0.008s
I20260709 12:25:12.930008  9661 raft_consensus.cc:2749] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.931070  9575 raft_consensus.cc:359] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.924718  9495 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:12.932854  9618 raft_consensus.cc:2749] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:12.931962  9575 raft_consensus.cc:385] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:12.936254  9576 ts_tablet_manager.cc:1434] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.001s
I20260709 12:25:12.936178  9575 raft_consensus.cc:740] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:12.937695  9576 tablet_bootstrap.cc:492] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:12.938109  9575 consensus_queue.cc:260] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:12.948138  9575 ts_tablet_manager.cc:1434] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:25:12.949126  9575 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:12.955049  9575 tablet_bootstrap.cc:654] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:12.968376  9205 catalog_manager.cc:5697] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:12.984543  9662 consensus_queue.cc:1048] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:12.998777  9577 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:12.999585  9577 ts_tablet_manager.cc:1403] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.301s	user 0.027s	sys 0.033s
I20260709 12:25:12.989943  9201 catalog_manager.cc:5697] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:13.003399  9577 raft_consensus.cc:359] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.004357  9577 raft_consensus.cc:385] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.004709  9577 raft_consensus.cc:740] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.005626  9577 consensus_queue.cc:260] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.006649  9618 raft_consensus.cc:493] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.007184  9618 raft_consensus.cc:515] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.008934  9577 ts_tablet_manager.cc:1434] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:25:13.010456  9577 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.010746  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:13.012475  9494 leader_election.cc:343] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:13.013803  9346 consensus_peers.cc:597] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:13.019069  9577 tablet_bootstrap.cc:654] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.009253  9618 leader_election.cc:290] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:13.020659  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.022222  9576 tablet_bootstrap.cc:654] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:13.026715  9421 consensus_peers.cc:597] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:13.035884  9495 leader_election.cc:343] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.036391  9495 leader_election.cc:304] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.037994  9618 raft_consensus.cc:2749] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.049410  9574 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:13.049988  9574 ts_tablet_manager.cc:1403] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.370s	user 0.021s	sys 0.005s
I20260709 12:25:13.080657  9575 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.081278  9575 ts_tablet_manager.cc:1403] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.132s	user 0.034s	sys 0.017s
I20260709 12:25:13.090621  9577 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.091152  9577 ts_tablet_manager.cc:1403] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.081s	user 0.023s	sys 0.013s
I20260709 12:25:13.094092  9577 raft_consensus.cc:359] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.094790  9577 raft_consensus.cc:385] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.095079  9577 raft_consensus.cc:740] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.096768  9698 consensus_queue.cc:1048] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:25:13.099088  9574 raft_consensus.cc:359] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.108157  9574 raft_consensus.cc:385] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.108485  9574 raft_consensus.cc:740] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.109314  9574 consensus_queue.cc:260] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.112003  9574 ts_tablet_manager.cc:1434] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.062s	user 0.008s	sys 0.010s
I20260709 12:25:13.112537  9577 consensus_queue.cc:260] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.116750  9574 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:13.119145  9577 ts_tablet_manager.cc:1434] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.002s
I20260709 12:25:13.120514  9577 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.124432  9575 raft_consensus.cc:359] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.125413  9575 raft_consensus.cc:385] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.125763  9575 raft_consensus.cc:740] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.135043  9574 tablet_bootstrap.cc:654] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.128348  9575 consensus_queue.cc:260] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.148149  9575 ts_tablet_manager.cc:1434] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.067s	user 0.006s	sys 0.003s
I20260709 12:25:13.153403  9575 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.161582  9577 tablet_bootstrap.cc:654] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.162151  9661 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.164158  9661 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.166317  9661 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.167761  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.168702  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.170411  9420 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:13.171456  9576 tablet_bootstrap.cc:492] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:13.171963  9576 ts_tablet_manager.cc:1403] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.234s	user 0.024s	sys 0.005s
I20260709 12:25:13.174978  9661 raft_consensus.cc:2749] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.175976  9576 raft_consensus.cc:359] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.176687  9576 raft_consensus.cc:385] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.176961  9576 raft_consensus.cc:740] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.177654  9576 consensus_queue.cc:260] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.181779  9575 tablet_bootstrap.cc:654] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.181951  9576 ts_tablet_manager.cc:1434] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:25:13.183192  9576 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:13.189095  9576 tablet_bootstrap.cc:654] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:13.195209  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:13.199218  9784 consensus_queue.cc:1048] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:25:13.202955  9584 consensus_queue.cc:1048] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:13.205068  9577 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.205102  9575 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.205768  9577 ts_tablet_manager.cc:1403] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.086s	user 0.010s	sys 0.004s
I20260709 12:25:13.205777  9575 ts_tablet_manager.cc:1403] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.001s
I20260709 12:25:13.208657  9577 raft_consensus.cc:359] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.209347  9577 raft_consensus.cc:385] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.209609  9577 raft_consensus.cc:740] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.210803  9575 raft_consensus.cc:359] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.211627  9575 raft_consensus.cc:385] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.219555  9575 raft_consensus.cc:740] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.216037  9577 consensus_queue.cc:260] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.220810  9575 consensus_queue.cc:260] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.222558  9577 ts_tablet_manager.cc:1434] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260709 12:25:13.223760  9577 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.226431  9575 ts_tablet_manager.cc:1434] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 12:25:13.225112  9574 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:13.227035  9574 ts_tablet_manager.cc:1403] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.114s	user 0.015s	sys 0.002s
I20260709 12:25:13.232025  9574 raft_consensus.cc:359] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.236622  9574 raft_consensus.cc:385] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.237006  9574 raft_consensus.cc:740] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.237859  9574 consensus_queue.cc:260] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.242039  9575 tablet_bootstrap.cc:492] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.244307  9577 tablet_bootstrap.cc:654] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.246845  9574 ts_tablet_manager.cc:1434] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
I20260709 12:25:13.247974  9574 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:13.254755  9574 tablet_bootstrap.cc:654] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.261581  9575 tablet_bootstrap.cc:654] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.262001  9577 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.262600  9577 ts_tablet_manager.cc:1403] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.002s
I20260709 12:25:13.265666  9577 raft_consensus.cc:359] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.268316  9577 raft_consensus.cc:385] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.268610  9577 raft_consensus.cc:740] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.268842  9576 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:13.269327  9576 ts_tablet_manager.cc:1403] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.086s	user 0.042s	sys 0.021s
I20260709 12:25:13.270272  9577 consensus_queue.cc:260] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.271411  9576 raft_consensus.cc:359] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.271975  9576 raft_consensus.cc:385] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.272198  9576 raft_consensus.cc:740] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.272490  9577 ts_tablet_manager.cc:1434] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.004s
I20260709 12:25:13.272804  9576 consensus_queue.cc:260] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.273445  9577 tablet_bootstrap.cc:492] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.275074  9576 ts_tablet_manager.cc:1434] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:13.276005  9576 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:13.278853  9784 raft_consensus.cc:493] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.279428  9784 raft_consensus.cc:515] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.281679  9784 leader_election.cc:290] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:13.286649  9576 tablet_bootstrap.cc:654] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.288436  9577 tablet_bootstrap.cc:654] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.290863  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.291858  9529 raft_consensus.cc:2468] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:13.295605  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:13.304657  9688 consensus_queue.cc:1048] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:13.307837  9346 leader_election.cc:304] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:13.308614  9574 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:13.309134  9574 ts_tablet_manager.cc:1403] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.061s	user 0.013s	sys 0.015s
W20260709 12:25:13.309310  9346 leader_election.cc:343] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.310680  9584 raft_consensus.cc:2804] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.312023  9584 raft_consensus.cc:493] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.312435  9584 raft_consensus.cc:3060] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.312776  9574 raft_consensus.cc:359] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.313702  9574 raft_consensus.cc:385] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.314067  9574 raft_consensus.cc:740] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.317408  9577 tablet_bootstrap.cc:492] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.317895  9577 ts_tablet_manager.cc:1403] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.045s	user 0.012s	sys 0.001s
I20260709 12:25:13.320779  9577 raft_consensus.cc:359] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.321450  9577 raft_consensus.cc:385] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.321725  9577 raft_consensus.cc:740] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.322386  9577 consensus_queue.cc:260] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.325222  9575 tablet_bootstrap.cc:492] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.325732  9575 ts_tablet_manager.cc:1403] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.090s	user 0.009s	sys 0.000s
I20260709 12:25:13.314999  9574 consensus_queue.cc:260] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.328022  9698 raft_consensus.cc:493] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.328531  9698 raft_consensus.cc:515] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.329031  9574 ts_tablet_manager.cc:1434] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:25:13.329988  9574 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:13.331794  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.332415  9575 raft_consensus.cc:359] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.334239  9575 raft_consensus.cc:385] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.334589  9575 raft_consensus.cc:740] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.335405  9575 consensus_queue.cc:260] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.336841  9453 raft_consensus.cc:3060] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.332468  9307 raft_consensus.cc:2468] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:13.337625  9574 tablet_bootstrap.cc:654] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.337741  9575 ts_tablet_manager.cc:1434] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:25:13.340377  9575 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.339118  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.332867  9577 ts_tablet_manager.cc:1434] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260709 12:25:13.344198  9530 raft_consensus.cc:2468] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:13.344872  9577 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.345832  9347 leader_election.cc:304] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:13.346794  9694 raft_consensus.cc:2804] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.347168  9694 raft_consensus.cc:493] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.347594  9694 raft_consensus.cc:3060] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.338724  9661 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.353140  9577 tablet_bootstrap.cc:654] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.353281  9661 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.357275  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.360442  9575 tablet_bootstrap.cc:654] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.364684  9661 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.367058  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.367974  9381 raft_consensus.cc:2393] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074a6ac642e4430ba8045181f0118ee3 in current term 1: Already voted for candidate 4e58af1026634b45bafbbfde02a6ecd0 in this term.
I20260709 12:25:13.369400  9421 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.372113  9662 raft_consensus.cc:3060] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.376123  9612 consensus_queue.cc:1048] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:13.379493  9575 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.380029  9575 ts_tablet_manager.cc:1403] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.040s	user 0.008s	sys 0.005s
I20260709 12:25:13.382388  9682 raft_consensus.cc:493] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.382701  9575 raft_consensus.cc:359] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.383455  9575 raft_consensus.cc:385] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.383740  9575 raft_consensus.cc:740] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.382921  9682 raft_consensus.cc:515] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.384416  9575 consensus_queue.cc:260] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.385730  9682 leader_election.cc:290] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:13.386838  9662 raft_consensus.cc:2749] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.390404  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.390522  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:13.393512  9575 ts_tablet_manager.cc:1434] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:25:13.394533  9575 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.396378  9698 leader_election.cc:290] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.392437  9271 leader_election.cc:304] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:13.376735  9577 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.404189  9577 ts_tablet_manager.cc:1403] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.060s	user 0.012s	sys 0.001s
I20260709 12:25:13.412590  9694 raft_consensus.cc:515] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.425058  9584 raft_consensus.cc:515] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.403425  9688 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.428280  9688 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.425590  9694 leader_election.cc:290] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.430948  9688 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.408298  9612 raft_consensus.cc:2749] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.435496  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.434304  9574 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
W20260709 12:25:13.438273  9495 leader_election.cc:343] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.440132  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:13.440737  9307 raft_consensus.cc:3060] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.442119  9577 raft_consensus.cc:359] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.442911  9577 raft_consensus.cc:385] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.443918  9306 raft_consensus.cc:3060] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.433068  9618 consensus_queue.cc:1048] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:25:13.448148  9584 leader_election.cc:290] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:13.454471  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:13.457508  9494 leader_election.cc:343] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.443220  9577 raft_consensus.cc:740] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.460153  9662 raft_consensus.cc:493] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.436868  9574 ts_tablet_manager.cc:1403] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.107s	user 0.038s	sys 0.019s
I20260709 12:25:13.463549  9662 raft_consensus.cc:515] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.464566  9577 consensus_queue.cc:260] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.466550  9662 leader_election.cc:290] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.467733  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:13.473078  9454 raft_consensus.cc:3060] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.470758  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.476130  9307 raft_consensus.cc:2468] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:13.469327  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.476332  9576 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:13.458069  9494 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.480015  9576 ts_tablet_manager.cc:1403] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.204s	user 0.026s	sys 0.061s
I20260709 12:25:13.480222  9577 ts_tablet_manager.cc:1434] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.076s	user 0.008s	sys 0.000s
I20260709 12:25:13.480443  9347 leader_election.cc:304] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:13.481438  9577 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.481572  9698 raft_consensus.cc:2804] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.482154  9698 raft_consensus.cc:697] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:13.483006  9698 consensus_queue.cc:237] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.484115  9618 raft_consensus.cc:2749] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.483124  9576 raft_consensus.cc:359] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.475395  9574 raft_consensus.cc:359] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.485438  9576 raft_consensus.cc:385] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.485540  9574 raft_consensus.cc:385] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.485942  9574 raft_consensus.cc:740] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.485858  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:13.486608  9530 raft_consensus.cc:3060] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.486908  9574 consensus_queue.cc:260] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.489171  9574 ts_tablet_manager.cc:1434] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 12:25:13.489943  9574 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:13.495645  9421 leader_election.cc:304] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.496728  9661 raft_consensus.cc:2749] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.485859  9576 raft_consensus.cc:740] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.500265  9612 consensus_queue.cc:1048] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:25:13.506317  9576 consensus_queue.cc:260] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.524379  9576 ts_tablet_manager.cc:1434] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.001s
I20260709 12:25:13.525849  9576 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:13.524260  9661 raft_consensus.cc:493] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.528843  9661 raft_consensus.cc:515] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.525665  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:13.532752  9529 raft_consensus.cc:3060] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.533970  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e722577ca9c4859ac9d6f335a1edfc9" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.534670  9528 raft_consensus.cc:2468] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:13.536883  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e722577ca9c4859ac9d6f335a1edfc9" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.537576  9305 raft_consensus.cc:2468] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:13.538224  9575 tablet_bootstrap.cc:654] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.531086  9661 leader_election.cc:290] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.526217  9530 raft_consensus.cc:2468] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:13.538704  9421 leader_election.cc:304] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:13.538596  9454 raft_consensus.cc:2468] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:13.550220  9688 raft_consensus.cc:493] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.562325  9662 raft_consensus.cc:493] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.569439  9662 raft_consensus.cc:515] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.570523  9618 raft_consensus.cc:493] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.571062  9618 raft_consensus.cc:515] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.573155  9618 leader_election.cc:290] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.574034  9661 raft_consensus.cc:2804] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.574476  9661 raft_consensus.cc:493] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.574834  9661 raft_consensus.cc:3060] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.574752  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.576547  9346 leader_election.cc:304] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:13.545099  9698 raft_consensus.cc:493] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.585848  9662 leader_election.cc:290] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:13.586743  9574 tablet_bootstrap.cc:654] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.586091  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.580605  9784 raft_consensus.cc:2804] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.594730  9576 tablet_bootstrap.cc:654] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.567637  9688 raft_consensus.cc:515] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:13.591821  9495 leader_election.cc:343] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.606191  9661 raft_consensus.cc:515] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.609324  9661 leader_election.cc:290] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.609428  9688 leader_election.cc:290] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.610404  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e722577ca9c4859ac9d6f335a1edfc9" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:13.611114  9305 raft_consensus.cc:3060] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.616663  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "642ee9123c0945a7b10a89d4ff3a01c6" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:13.619768  9494 leader_election.cc:343] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.594866  9784 raft_consensus.cc:697] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:13.621188  9784 consensus_queue.cc:237] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.584231  9698 raft_consensus.cc:515] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.626235  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.628895  9382 raft_consensus.cc:2468] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 0.
I20260709 12:25:13.630173  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "642ee9123c0945a7b10a89d4ff3a01c6" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.631256  9201 catalog_manager.cc:5697] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:13.626725  9420 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 81485.6us
I20260709 12:25:13.628203  9529 raft_consensus.cc:2468] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:13.635666  9577 tablet_bootstrap.cc:654] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:13.642983  9495 leader_election.cc:343] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.643522  9495 leader_election.cc:304] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.644639  9776 raft_consensus.cc:2749] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.645612  9495 leader_election.cc:304] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.647246  9575 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.647760  9575 ts_tablet_manager.cc:1403] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.253s	user 0.024s	sys 0.019s
I20260709 12:25:13.647503  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bb9dcd4bb4647b08e76fcc074ad8e40" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:13.649650  9347 leader_election.cc:343] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.650377  9689 raft_consensus.cc:2804] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.650846  9689 raft_consensus.cc:493] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.651199  9689 raft_consensus.cc:3060] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.652070  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.652307  9575 raft_consensus.cc:359] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.652757  9529 raft_consensus.cc:2468] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:13.652958  9575 raft_consensus.cc:385] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.653272  9575 raft_consensus.cc:740] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.653936  9575 consensus_queue.cc:260] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.655124  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e722577ca9c4859ac9d6f335a1edfc9" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:13.655786  9530 raft_consensus.cc:3060] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.656369  9575 ts_tablet_manager.cc:1434] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:25:13.658751  9575 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.659879  9420 leader_election.cc:304] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.663798  9661 raft_consensus.cc:2804] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:13.664179  9661 raft_consensus.cc:493] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.664338  9575 tablet_bootstrap.cc:654] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.664590  9661 raft_consensus.cc:3060] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:13.676397  9455 raft_consensus.cc:3060] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.677245  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bb9dcd4bb4647b08e76fcc074ad8e40" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:13.677176  9661 raft_consensus.cc:515] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.678058  9454 raft_consensus.cc:2468] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:13.655784  9698 leader_election.cc:290] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:13.681741  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:13.682768  9661 leader_election.cc:290] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:13.684078  9346 leader_election.cc:304] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.684806  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11e5772eda0648d9a96d2b5755c23569" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:13.691560  9528 raft_consensus.cc:3060] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:13.692039  9694 raft_consensus.cc:2804] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.703979  9694 raft_consensus.cc:493] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.704413  9694 raft_consensus.cc:3060] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.686305  9575 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.691429  9305 raft_consensus.cc:2468] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:13.706102  9575 ts_tablet_manager.cc:1403] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.048s	user 0.014s	sys 0.004s
I20260709 12:25:13.708226  9528 raft_consensus.cc:2468] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:13.709666  9421 leader_election.cc:304] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:13.697908  9689 raft_consensus.cc:515] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.701442  9530 raft_consensus.cc:2468] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:13.721012  9661 raft_consensus.cc:2804] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.721585  9661 raft_consensus.cc:697] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:13.722389  9420 leader_election.cc:304] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.722512  9661 consensus_queue.cc:237] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.723224  9689 leader_election.cc:290] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.716261  9694 raft_consensus.cc:515] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.724229  9749 raft_consensus.cc:2804] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:13.724727  9749 raft_consensus.cc:697] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:13.725517  9749 consensus_queue.cc:237] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.725575  9694 leader_election.cc:290] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:13.727262  9689 raft_consensus.cc:493] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.727778  9689 raft_consensus.cc:515] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.728818  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bb9dcd4bb4647b08e76fcc074ad8e40" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:13.729853  9689 leader_election.cc:290] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:13.730353  9347 leader_election.cc:343] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.731268  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:13.731419  9618 raft_consensus.cc:493] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.732563  9618 raft_consensus.cc:515] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.733426  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:13.733984  9379 raft_consensus.cc:3060] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.734238  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb1d8de1bb30430eb7ae1bbbf66232df" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.734856  9618 leader_election.cc:290] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:13.734681  9575 raft_consensus.cc:359] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.735414  9575 raft_consensus.cc:385] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.735800  9575 raft_consensus.cc:740] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.738976  9577 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.739574  9577 ts_tablet_manager.cc:1403] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.258s	user 0.009s	sys 0.007s
I20260709 12:25:13.741899  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bb9dcd4bb4647b08e76fcc074ad8e40" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:13.742456  9577 raft_consensus.cc:359] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.741497  9574 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:13.735482  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb1d8de1bb30430eb7ae1bbbf66232df" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.743476  9577 raft_consensus.cc:385] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.743832  9577 raft_consensus.cc:740] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.744067  9574 ts_tablet_manager.cc:1403] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.254s	user 0.014s	sys 0.000s
I20260709 12:25:13.744243  9455 raft_consensus.cc:3060] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.744572  9577 consensus_queue.cc:260] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.745553  9576 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:13.736483  9575 consensus_queue.cc:260] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.746160  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:13.747722  9574 raft_consensus.cc:359] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.748306  9575 ts_tablet_manager.cc:1434] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.001s
I20260709 12:25:13.748404  9574 raft_consensus.cc:385] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:25:13.748821  9494 leader_election.cc:343] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.749315  9575 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.748826  9574 raft_consensus.cc:740] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.755466  9455 raft_consensus.cc:2468] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
W20260709 12:25:13.755959  9495 leader_election.cc:343] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.756793  9577 ts_tablet_manager.cc:1434] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
W20260709 12:25:13.756758  9495 leader_election.cc:343] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.746166  9576 ts_tablet_manager.cc:1403] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.221s	user 0.014s	sys 0.001s
I20260709 12:25:13.760684  9682 raft_consensus.cc:493] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.761838  9682 raft_consensus.cc:515] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.764426  9689 raft_consensus.cc:493] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.764657  9682 leader_election.cc:290] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:13.764999  9689 raft_consensus.cc:515] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.765450  9379 raft_consensus.cc:2468] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:13.767016  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.755275  9574 consensus_queue.cc:260] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:13.768018  9495 leader_election.cc:343] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.757964  9577 tablet_bootstrap.cc:492] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.768560  9495 leader_election.cc:304] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
W20260709 12:25:13.768904  9272 leader_election.cc:343] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.767166  9689 leader_election.cc:290] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.759044  9346 leader_election.cc:304] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.770411  9689 raft_consensus.cc:2749] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.770588  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b2fa83e4cf5494ba63becbde8d026b2" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.771551  9784 raft_consensus.cc:2804] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.769649  9574 ts_tablet_manager.cc:1434] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.001s
I20260709 12:25:13.772229  9784 raft_consensus.cc:697] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:13.773102  9574 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:13.773149  9784 consensus_queue.cc:237] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.771548  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:13.775696  9495 leader_election.cc:343] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.770651  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:13.776798  9453 raft_consensus.cc:2468] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:13.777989  9271 leader_election.cc:304] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.779292  9682 raft_consensus.cc:2804] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.779752  9682 raft_consensus.cc:493] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.780031  9574 tablet_bootstrap.cc:654] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.780166  9682 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:13.783949  9495 leader_election.cc:343] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.784400  9495 leader_election.cc:304] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.785995  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b2fa83e4cf5494ba63becbde8d026b2" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.787047  9618 raft_consensus.cc:2749] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:25:13.788245  9495 leader_election.cc:343] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.788727  9495 leader_election.cc:304] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.789841  9618 raft_consensus.cc:2749] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.760428  9575 tablet_bootstrap.cc:654] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.796957  9765 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.797497  9765 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.801321  9765 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.801518  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.802268  9530 raft_consensus.cc:2468] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:13.803470  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.804996  9271 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:13.805979  9765 raft_consensus.cc:2804] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.806313  9765 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.806598  9765 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.814036  9765 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.817327  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:13.817330  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:13.818722  9530 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:13.819031  9272 leader_election.cc:343] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:13.820601  9272 leader_election.cc:343] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.825558  9530 raft_consensus.cc:2468] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:13.826119  9765 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:13.827348  9271 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.828332  9765 raft_consensus.cc:2804] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.825806  9574 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:13.828994  9765 raft_consensus.cc:697] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:13.829255  9575 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.829881  9575 ts_tablet_manager.cc:1403] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.081s	user 0.008s	sys 0.005s
I20260709 12:25:13.829725  9495 leader_election.cc:304] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.831133  9618 raft_consensus.cc:2804] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.831501  9577 tablet_bootstrap.cc:654] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.831686  9618 raft_consensus.cc:697] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:13.832680  9618 consensus_queue.cc:237] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.832984  9575 raft_consensus.cc:359] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.833679  9575 raft_consensus.cc:385] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.833981  9575 raft_consensus.cc:740] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.829401  9574 ts_tablet_manager.cc:1403] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.057s	user 0.017s	sys 0.009s
I20260709 12:25:13.834959  9575 consensus_queue.cc:260] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.837919  9574 raft_consensus.cc:359] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.837852  9575 ts_tablet_manager.cc:1434] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:25:13.845717  9577 tablet_bootstrap.cc:492] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:13.839001  9765 consensus_queue.cc:237] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.838699  9574 raft_consensus.cc:385] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.846487  9577 ts_tablet_manager.cc:1403] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.089s	user 0.009s	sys 0.004s
I20260709 12:25:13.846426  9574 raft_consensus.cc:740] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.842120  9682 raft_consensus.cc:515] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.849494  9682 leader_election.cc:290] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:13.849802  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:13.850026  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:13.850629  9453 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:13.829010  9576 raft_consensus.cc:359] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.850955  9574 consensus_queue.cc:260] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:13.851583  9272 leader_election.cc:343] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.850998  9575 tablet_bootstrap.cc:492] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.851595  9576 raft_consensus.cc:385] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.854036  9576 raft_consensus.cc:740] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.853729  9577 raft_consensus.cc:359] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.854772  9577 raft_consensus.cc:385] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.855139  9577 raft_consensus.cc:740] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.855945  9577 consensus_queue.cc:260] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.858342  9577 ts_tablet_manager.cc:1434] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.004s
I20260709 12:25:13.859248  9576 consensus_queue.cc:260] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:13.860855  9577 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:13.861905  9576 ts_tablet_manager.cc:1434] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.101s	user 0.009s	sys 0.000s
I20260709 12:25:13.866389  9576 tablet_bootstrap.cc:492] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:13.868839  9575 tablet_bootstrap.cc:654] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.870200  9453 raft_consensus.cc:2468] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:13.871847  9271 leader_election.cc:304] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.872428  9576 tablet_bootstrap.cc:654] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.873034  9765 raft_consensus.cc:2804] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:13.873795  9765 raft_consensus.cc:697] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:13.861508  9202 catalog_manager.cc:5697] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:13.863963  9574 ts_tablet_manager.cc:1434] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 12:25:13.874639  9765 consensus_queue.cc:237] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:13.875963  9574 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:13.875180  9201 catalog_manager.cc:5697] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:13.889103  9694 consensus_queue.cc:1048] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:13.890316  9618 raft_consensus.cc:493] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.891780  9618 raft_consensus.cc:515] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.907792  9574 tablet_bootstrap.cc:654] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.910493  9577 tablet_bootstrap.cc:654] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.921820  9203 catalog_manager.cc:5697] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:13.931175  9200 catalog_manager.cc:5697] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 1 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:13.934527  9618 leader_election.cc:290] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:13.934993  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36a1d25b3256456ebdf25d5b805c8086" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:13.939429  9575 tablet_bootstrap.cc:492] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:13.939961  9575 ts_tablet_manager.cc:1403] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.094s	user 0.020s	sys 0.017s
I20260709 12:25:13.941121  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36a1d25b3256456ebdf25d5b805c8086" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:13.925593  9203 catalog_manager.cc:5697] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:13.943996  9494 leader_election.cc:343] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.947500  9576 tablet_bootstrap.cc:492] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:13.948918  9576 ts_tablet_manager.cc:1403] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.083s	user 0.015s	sys 0.007s
W20260709 12:25:13.950809  9495 leader_election.cc:343] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.951756  9576 raft_consensus.cc:359] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.952499  9576 raft_consensus.cc:385] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.952770  9576 raft_consensus.cc:740] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.953953  9576 consensus_queue.cc:260] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.956779  9662 raft_consensus.cc:493] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.957307  9662 raft_consensus.cc:515] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.961661  9575 raft_consensus.cc:359] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:13.962801  9575 raft_consensus.cc:385] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.963194  9575 raft_consensus.cc:740] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.964095  9662 leader_election.cc:290] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:13.964110  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:13.964596  9575 consensus_queue.cc:260] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
W20260709 12:25:13.969857  9421 leader_election.cc:343] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:13.970160  9574 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:13.970707  9574 ts_tablet_manager.cc:1403] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.095s	user 0.026s	sys 0.016s
I20260709 12:25:13.971096  9576 ts_tablet_manager.cc:1434] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:25:13.974566  9576 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:13.976209  9574 raft_consensus.cc:359] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.976464  9495 leader_election.cc:304] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:13.976939  9574 raft_consensus.cc:385] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:13.977308  9574 raft_consensus.cc:740] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:13.978067  9584 consensus_queue.cc:1048] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:13.978251  9574 consensus_queue.cc:260] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:13.980111  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:13.980859  9528 raft_consensus.cc:2468] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:13.978113  9618 raft_consensus.cc:2749] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:13.982370  9420 leader_election.cc:304] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:13.982929  9576 tablet_bootstrap.cc:654] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:13.984181  9574 ts_tablet_manager.cc:1434] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:25:13.986032  9575 ts_tablet_manager.cc:1434] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.000s
I20260709 12:25:14.002211  9574 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:14.002732  9575 tablet_bootstrap.cc:492] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:13.985836  9662 raft_consensus.cc:2804] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:13.994176  9577 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.003613  9662 raft_consensus.cc:493] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.003844  9577 ts_tablet_manager.cc:1403] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.144s	user 0.013s	sys 0.011s
I20260709 12:25:14.004041  9662 raft_consensus.cc:3060] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.004565  9201 catalog_manager.cc:5697] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:14.006783  9577 raft_consensus.cc:359] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.007440  9577 raft_consensus.cc:385] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.007699  9577 raft_consensus.cc:740] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.008339  9577 consensus_queue.cc:260] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.012916  9575 tablet_bootstrap.cc:654] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.014016  9577 ts_tablet_manager.cc:1434] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.005s
I20260709 12:25:14.015295  9577 tablet_bootstrap.cc:492] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:14.018913  9574 tablet_bootstrap.cc:654] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.028132  9202 catalog_manager.cc:5697] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:14.030449  9577 tablet_bootstrap.cc:654] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.034145  9574 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:14.034746  9574 ts_tablet_manager.cc:1403] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.000s
I20260709 12:25:14.035234  9576 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:14.036075  9576 ts_tablet_manager.cc:1403] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.062s	user 0.013s	sys 0.000s
I20260709 12:25:14.053591  9530 raft_consensus.cc:1240] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Rejecting Update request from peer e07a645c9ee14f11a5d64fa7b460acf9 for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:14.054689  9576 raft_consensus.cc:359] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.066074  9576 raft_consensus.cc:385] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.066416  9576 raft_consensus.cc:740] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.067220  9584 consensus_queue.cc:1059] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:14.067153  9576 consensus_queue.cc:260] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.068087  9584 raft_consensus.cc:3055] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:14.068437  9584 raft_consensus.cc:740] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:14.069458  9576 ts_tablet_manager.cc:1434] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.033s	user 0.009s	sys 0.005s
I20260709 12:25:14.069228  9584 consensus_queue.cc:260] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.070428  9584 raft_consensus.cc:3060] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:14.070431  9576 tablet_bootstrap.cc:492] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:14.076908  9576 tablet_bootstrap.cc:654] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.093271  9662 raft_consensus.cc:515] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.097661  9575 tablet_bootstrap.cc:492] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:14.103938  9618 raft_consensus.cc:493] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.112911  9618 raft_consensus.cc:515] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.115726  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:14.116382  9618 leader_election.cc:290] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:14.116842  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9508782ff1d4df7867468f207693333" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:14.117175  9575 ts_tablet_manager.cc:1403] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.115s	user 0.011s	sys 0.011s
W20260709 12:25:14.117973  9421 leader_election.cc:343] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:14.119069  9495 leader_election.cc:343] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.119241  9662 leader_election.cc:290] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.120307  9575 raft_consensus.cc:359] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.120765  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9508782ff1d4df7867468f207693333" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.121101  9575 raft_consensus.cc:385] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.121439  9575 raft_consensus.cc:740] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.121619  9577 tablet_bootstrap.cc:492] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.122294  9577 ts_tablet_manager.cc:1403] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.107s	user 0.009s	sys 0.035s
I20260709 12:25:14.105434  9576 tablet_bootstrap.cc:492] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:14.105438  9574 raft_consensus.cc:359] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:14.122856  9494 leader_election.cc:343] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.123112  9576 ts_tablet_manager.cc:1403] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.053s	user 0.017s	sys 0.002s
I20260709 12:25:14.123036  9574 raft_consensus.cc:385] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.123682  9574 raft_consensus.cc:740] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.124413  9574 consensus_queue.cc:260] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.125202  9577 raft_consensus.cc:359] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.125854  9577 raft_consensus.cc:385] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.126116  9577 raft_consensus.cc:740] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.126780  9574 ts_tablet_manager.cc:1434] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.091s	user 0.019s	sys 0.030s
I20260709 12:25:14.126952  9576 raft_consensus.cc:359] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.126866  9577 consensus_queue.cc:260] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.127898  9576 raft_consensus.cc:385] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.128113  9574 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:14.128286  9576 raft_consensus.cc:740] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.122210  9575 consensus_queue.cc:260] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.131999  9576 consensus_queue.cc:260] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.129385  9577 ts_tablet_manager.cc:1434] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:25:14.133874  9577 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:14.134693  9575 ts_tablet_manager.cc:1434] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:25:14.135725  9574 tablet_bootstrap.cc:654] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.137876  9576 ts_tablet_manager.cc:1434] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:25:14.138916  9576 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:14.142146  9577 tablet_bootstrap.cc:654] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.145118  9576 tablet_bootstrap.cc:654] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.135767  9575 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:14.153012  9575 tablet_bootstrap.cc:654] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.158695  9574 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:14.159492  9574 ts_tablet_manager.cc:1403] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.032s	user 0.018s	sys 0.004s
I20260709 12:25:14.163511  9494 leader_election.cc:304] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:14.163134  9574 raft_consensus.cc:359] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.164599  9618 raft_consensus.cc:2749] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.164588  9574 raft_consensus.cc:385] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.165428  9574 raft_consensus.cc:740] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.166183  9574 consensus_queue.cc:260] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.168322  9577 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.168541  9575 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:14.168540  9574 ts_tablet_manager.cc:1434] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:25:14.169189  9577 ts_tablet_manager.cc:1403] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.036s	user 0.008s	sys 0.013s
I20260709 12:25:14.169328  9575 ts_tablet_manager.cc:1403] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.001s
I20260709 12:25:14.169817  9574 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:14.170375  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.171087  9530 raft_consensus.cc:3060] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.172307  9577 raft_consensus.cc:359] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.173135  9577 raft_consensus.cc:385] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.173475  9577 raft_consensus.cc:740] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.174253  9576 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:14.174305  9577 consensus_queue.cc:260] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.174989  9576 ts_tablet_manager.cc:1403] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.036s	user 0.007s	sys 0.006s
I20260709 12:25:14.176156  9574 tablet_bootstrap.cc:654] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.178364  9575 raft_consensus.cc:359] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.179143  9575 raft_consensus.cc:385] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.179451  9575 raft_consensus.cc:740] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.180265  9575 consensus_queue.cc:260] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.183034  9577 ts_tablet_manager.cc:1434] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 12:25:14.183815  9303 raft_consensus.cc:3060] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.184096  9577 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:14.182621  9576 raft_consensus.cc:359] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.184581  9576 raft_consensus.cc:385] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.184900  9576 raft_consensus.cc:740] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.185664  9576 consensus_queue.cc:260] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.189059  9575 ts_tablet_manager.cc:1434] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:25:14.190127  9575 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:14.191121  9577 tablet_bootstrap.cc:654] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.192878  9584 consensus_queue.cc:1048] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:14.196396  9575 tablet_bootstrap.cc:654] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.200227  9576 ts_tablet_manager.cc:1434] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.002s
I20260709 12:25:14.201320  9576 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:14.202083  9577 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.202579  9577 ts_tablet_manager.cc:1403] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.003s
I20260709 12:25:14.209236  9576 tablet_bootstrap.cc:654] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.217525  9528 raft_consensus.cc:1275] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:25:14.225598  9784 consensus_queue.cc:1048] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:14.246187  9765 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.276135  9765 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.276564  9662 consensus_queue.cc:1048] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:14.256179  9661 consensus_queue.cc:1048] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:14.277797  9784 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.280747  9784 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.282826  9784 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.258543  9574 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:14.283627  9574 ts_tablet_manager.cc:1403] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.114s	user 0.014s	sys 0.000s
I20260709 12:25:14.284834  9765 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.287092  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.266396  9618 consensus_queue.cc:1048] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:14.289801  9574 raft_consensus.cc:359] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:14.273468  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:14.271890  9847 raft_consensus.cc:493] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.290570  9574 raft_consensus.cc:385] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.290019  9662 consensus_queue.cc:1048] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:14.290930  9574 raft_consensus.cc:740] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.290994  9847 raft_consensus.cc:515] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.205411  9577 raft_consensus.cc:359] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.271927  9530 raft_consensus.cc:2468] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:14.294831  9577 raft_consensus.cc:385] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.298142  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.298925  9530 raft_consensus.cc:2393] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e07a645c9ee14f11a5d64fa7b460acf9 in current term 1: Already voted for candidate 4e58af1026634b45bafbbfde02a6ecd0 in this term.
I20260709 12:25:14.298161  9577 raft_consensus.cc:740] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.300235  9577 consensus_queue.cc:260] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.296013  9574 consensus_queue.cc:260] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.299083  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.304199  9420 leader_election.cc:304] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:14.304826  9577 ts_tablet_manager.cc:1434] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.002s
W20260709 12:25:14.305469  9495 consensus_peers.cc:597] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:14.325826  9577 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:14.295490  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.327251  9784 raft_consensus.cc:493] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.313771  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.333202  9784 raft_consensus.cc:515] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.333736  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.294464  9847 leader_election.cc:290] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:14.334674  9530 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:14.334091  9454 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:14.338518  9379 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:14.328534  9346 leader_election.cc:343] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.340050  9346 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:14.340860  9574 ts_tablet_manager.cc:1434] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.000s
I20260709 12:25:14.305487  9749 raft_consensus.cc:2804] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.344002  9381 raft_consensus.cc:1275] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:25:14.344386  9381 pending_rounds.cc:85] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: Aborting all ops after (but not including) 0
I20260709 12:25:14.344624  9381 pending_rounds.cc:107] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:25:14.344884  9381 raft_consensus.cc:2889] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:14.345872  9749 raft_consensus.cc:697] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:14.348340  9749 consensus_queue.cc:237] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.353458  9847 raft_consensus.cc:2749] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.354434  9765 consensus_queue.cc:1048] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:14.333225  9306 raft_consensus.cc:2468] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:14.338828  9271 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:14.363286  9347 leader_election.cc:304] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:14.364063  9765 raft_consensus.cc:2804] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.364475  9765 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.364598  9694 raft_consensus.cc:2804] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.364878  9765 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.365151  9694 raft_consensus.cc:493] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.365638  9694 raft_consensus.cc:3060] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.336277  9784 leader_election.cc:290] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.341876  9574 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:14.377856  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.378929  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.379769  9530 raft_consensus.cc:2468] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:14.380671  9661 consensus_queue.cc:1048] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:14.381743  9346 leader_election.cc:304] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:14.384232  9694 raft_consensus.cc:515] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.387042  9584 raft_consensus.cc:2804] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.378903  9454 raft_consensus.cc:2468] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:14.387560  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:14.387662  9584 raft_consensus.cc:493] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.388214  9305 raft_consensus.cc:3060] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.388391  9584 raft_consensus.cc:3060] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.398475  9765 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.411280  9577 tablet_bootstrap.cc:654] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.413767  9765 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.418494  9574 tablet_bootstrap.cc:654] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.425487  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:14.425858  9575 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:14.426172  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.426316  9454 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.426502  9575 ts_tablet_manager.cc:1403] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.237s	user 0.034s	sys 0.032s
I20260709 12:25:14.426810  9530 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.430063  9575 raft_consensus.cc:359] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.430795  9575 raft_consensus.cc:385] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.433671  9530 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:14.431116  9575 raft_consensus.cc:740] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.447365  9575 consensus_queue.cc:260] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.449760  9575 ts_tablet_manager.cc:1434] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.003s
I20260709 12:25:14.434208  9694 leader_election.cc:290] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:14.452422  9575 tablet_bootstrap.cc:492] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:14.459681  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:14.464835  9847 raft_consensus.cc:493] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.465431  9847 raft_consensus.cc:515] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.466840  9855 raft_consensus.cc:493] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.467597  9847 leader_election.cc:290] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.467383  9855 raft_consensus.cc:515] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.465844  9765 consensus_queue.cc:1048] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:25:14.477566  9689 raft_consensus.cc:493] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.478379  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "642ee9123c0945a7b10a89d4ff3a01c6" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.479000  9530 raft_consensus.cc:2468] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:14.479051  9271 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:14.479063  9574 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:14.480346  9765 raft_consensus.cc:2804] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.481034  9420 leader_election.cc:304] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:14.481288  9584 raft_consensus.cc:515] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.483497  9584 leader_election.cc:290] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.481031  9765 raft_consensus.cc:697] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:14.484427  9855 leader_election.cc:290] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.486749  9855 raft_consensus.cc:2804] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.487232  9855 raft_consensus.cc:493] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.487650  9855 raft_consensus.cc:3060] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.480466  9574 ts_tablet_manager.cc:1403] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.139s	user 0.012s	sys 0.001s
I20260709 12:25:14.484853  9765 consensus_queue.cc:237] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.478178  9689 raft_consensus.cc:515] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.433671  9454 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:14.486009  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "642ee9123c0945a7b10a89d4ff3a01c6" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.494525  9574 raft_consensus.cc:359] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.505265  9577 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.508800  9574 raft_consensus.cc:385] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.509425  9306 raft_consensus.cc:2468] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:14.509007  9577 ts_tablet_manager.cc:1403] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.183s	user 0.013s	sys 0.030s
I20260709 12:25:14.509639  9574 raft_consensus.cc:740] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.511374  9689 leader_election.cc:290] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:14.512207  9574 consensus_queue.cc:260] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.512498  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.513240  9577 raft_consensus.cc:359] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.513906  9577 raft_consensus.cc:385] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.514192  9577 raft_consensus.cc:740] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
W20260709 12:25:14.514276  9494 leader_election.cc:343] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.514878  9577 consensus_queue.cc:260] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:14.515491  9272 consensus_peers.cc:597] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:14.515856  9574 ts_tablet_manager.cc:1434] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.002s
I20260709 12:25:14.517553  9574 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
W20260709 12:25:14.515969  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 57964us
I20260709 12:25:14.451083  9305 raft_consensus.cc:2468] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:14.524096  9855 raft_consensus.cc:515] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.527813  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.528975  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.531926  9530 raft_consensus.cc:2393] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e07a645c9ee14f11a5d64fa7b460acf9 in current term 1: Already voted for candidate 074a6ac642e4430ba8045181f0118ee3 in this term.
I20260709 12:25:14.531430  9347 leader_election.cc:304] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:14.532948  9584 raft_consensus.cc:2804] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.533418  9584 raft_consensus.cc:697] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:14.535578  9577 ts_tablet_manager.cc:1434] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260709 12:25:14.536926  9577 tablet_bootstrap.cc:492] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
W20260709 12:25:14.540431  9346 leader_election.cc:343] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.544839  9855 leader_election.cc:290] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.541890  9584 consensus_queue.cc:237] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.540086  9694 raft_consensus.cc:493] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.546744  9694 raft_consensus.cc:515] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.548907  9694 leader_election.cc:290] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.572893  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.573743  9530 raft_consensus.cc:3060] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.568158  9689 raft_consensus.cc:493] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.568661  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "642ee9123c0945a7b10a89d4ff3a01c6" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:14.571769  9586 consensus_queue.cc:1048] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:25:14.591467  9530 raft_consensus.cc:2468] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:14.592571  9574 tablet_bootstrap.cc:654] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.585343  9305 raft_consensus.cc:3060] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.583410  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:14.582043  9689 raft_consensus.cc:515] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.596758  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b2fa83e4cf5494ba63becbde8d026b2" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.598321  9689 leader_election.cc:290] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:14.598821  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.599193  9576 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:14.599737  9576 ts_tablet_manager.cc:1403] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.399s	user 0.038s	sys 0.041s
W20260709 12:25:14.600412  9495 leader_election.cc:343] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.599650  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "642ee9123c0945a7b10a89d4ff3a01c6" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.602955  9346 leader_election.cc:304] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:14.607992  9527 raft_consensus.cc:3060] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.609192  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:14.610164  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b2fa83e4cf5494ba63becbde8d026b2" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.610822  9529 raft_consensus.cc:2468] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:14.611531  9682 raft_consensus.cc:493] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.612032  9682 raft_consensus.cc:515] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.601202  9455 raft_consensus.cc:3060] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.615032  9682 leader_election.cc:290] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:14.618463  9346 leader_election.cc:304] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:14.619292  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.624917  9305 raft_consensus.cc:2468] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:14.626416  9421 leader_election.cc:304] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:14.622018  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:14.627645  9862 raft_consensus.cc:2804] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.627657  9527 raft_consensus.cc:2468] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:14.628389  9862 raft_consensus.cc:697] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
W20260709 12:25:14.629230  9852 log.cc:927] Time spent T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: Append to log took a long time: real 0.103s	user 0.000s	sys 0.005s
I20260709 12:25:14.629597  9862 consensus_queue.cc:237] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.631845  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:14.633430  9855 raft_consensus.cc:493] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.633971  9855 raft_consensus.cc:515] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.637104  9855 leader_election.cc:290] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.626430  9846 raft_consensus.cc:2804] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.638509  9846 raft_consensus.cc:697] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:14.639367  9846 consensus_queue.cc:237] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.642786  9455 raft_consensus.cc:2468] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
W20260709 12:25:14.619997  9347 leader_election.cc:343] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.648629  9575 tablet_bootstrap.cc:654] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.619310  9694 raft_consensus.cc:2804] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.651535  9694 raft_consensus.cc:493] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.651889  9694 raft_consensus.cc:3060] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.655385  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36a1d25b3256456ebdf25d5b805c8086" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.656659  9346 leader_election.cc:304] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
W20260709 12:25:14.663782  9495 leader_election.cc:343] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.664377  9495 leader_election.cc:304] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:14.666563  9577 tablet_bootstrap.cc:654] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:14.667764  9495 leader_election.cc:343] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.668567  9495 leader_election.cc:304] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:14.665587  9698 raft_consensus.cc:3060] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.667245  9618 raft_consensus.cc:2749] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.671151  9776 raft_consensus.cc:2749] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.666213  9305 raft_consensus.cc:2468] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:14.682338  9421 leader_election.cc:304] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:14.682612  9576 raft_consensus.cc:359] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.683728  9856 raft_consensus.cc:2804] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.685334  9272 leader_election.cc:304] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:14.700459  9202 catalog_manager.cc:5697] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.711431  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36a1d25b3256456ebdf25d5b805c8086" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.710390  9856 raft_consensus.cc:493] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.713366  9856 raft_consensus.cc:3060] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.715634  9306 raft_consensus.cc:3060] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.719054  9694 raft_consensus.cc:515] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.723663  9694 leader_election.cc:290] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.709374  9576 raft_consensus.cc:385] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.727663  9576 raft_consensus.cc:740] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.728442  9576 consensus_queue.cc:260] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.730834  9576 ts_tablet_manager.cc:1434] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.131s	user 0.007s	sys 0.000s
I20260709 12:25:14.735258  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b2fa83e4cf5494ba63becbde8d026b2" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.741014  9526 raft_consensus.cc:3060] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.741678  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b2fa83e4cf5494ba63becbde8d026b2" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:14.743042  9698 raft_consensus.cc:2749] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:25:14.743247  9347 leader_election.cc:343] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.730448  9527 raft_consensus.cc:2468] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:14.747689  9576 tablet_bootstrap.cc:492] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:14.750260  9575 tablet_bootstrap.cc:492] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:14.750731  9575 ts_tablet_manager.cc:1403] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.299s	user 0.021s	sys 0.016s
I20260709 12:25:14.745920  9856 raft_consensus.cc:515] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.753630  9856 leader_election.cc:290] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.760196  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36a1d25b3256456ebdf25d5b805c8086" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:14.760895  9305 raft_consensus.cc:3060] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.764194  9526 raft_consensus.cc:2468] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:14.765779  9346 leader_election.cc:304] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:14.770462  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36a1d25b3256456ebdf25d5b805c8086" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.770258  9200 catalog_manager.cc:5697] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:14.771152  9526 raft_consensus.cc:3060] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.771665  9577 tablet_bootstrap.cc:492] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.772238  9577 ts_tablet_manager.cc:1403] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.236s	user 0.012s	sys 0.001s
I20260709 12:25:14.772660  9200 catalog_manager.cc:5697] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:14.775780  9575 raft_consensus.cc:359] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.777191  9575 raft_consensus.cc:385] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.777433  9575 raft_consensus.cc:740] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.778117  9575 consensus_queue.cc:260] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.784663  9305 raft_consensus.cc:2468] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:14.788201  9698 raft_consensus.cc:2804] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.788829  9698 raft_consensus.cc:697] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:14.789748  9698 consensus_queue.cc:237] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.793325  9421 leader_election.cc:304] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:14.794785  9855 consensus_queue.cc:1048] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:14.794790  9662 raft_consensus.cc:2804] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.796767  9662 raft_consensus.cc:697] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:14.797664  9662 consensus_queue.cc:237] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.802415  9574 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:14.803131  9574 ts_tablet_manager.cc:1403] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.286s	user 0.010s	sys 0.004s
I20260709 12:25:14.807516  9577 raft_consensus.cc:359] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.808382  9577 raft_consensus.cc:385] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.808755  9577 raft_consensus.cc:740] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.809649  9577 consensus_queue.cc:260] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.812165  9577 ts_tablet_manager.cc:1434] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.040s	user 0.008s	sys 0.000s
I20260709 12:25:14.813030  9577 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:14.817934  9575 ts_tablet_manager.cc:1434] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.067s	user 0.005s	sys 0.003s
I20260709 12:25:14.823769  9575 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:14.828889  9574 raft_consensus.cc:359] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.829581  9574 raft_consensus.cc:385] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.829881  9574 raft_consensus.cc:740] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.835362  9576 tablet_bootstrap.cc:654] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.838032  9574 consensus_queue.cc:260] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.815840  9526 raft_consensus.cc:2468] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:14.842481  9574 ts_tablet_manager.cc:1434] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.002s
I20260709 12:25:14.843431  9574 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:14.847033  9862 consensus_queue.cc:1048] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:14.855965  9575 tablet_bootstrap.cc:654] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.850772  9662 raft_consensus.cc:493] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.857710  9662 raft_consensus.cc:515] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.862004  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:14.862934  9305 raft_consensus.cc:2393] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074a6ac642e4430ba8045181f0118ee3 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:14.867816  9576 tablet_bootstrap.cc:492] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:14.867695  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:14.868317  9576 ts_tablet_manager.cc:1403] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.121s	user 0.014s	sys 0.000s
I20260709 12:25:14.869374  9612 raft_consensus.cc:2749] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.871132  9576 raft_consensus.cc:359] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.871608  9688 raft_consensus.cc:493] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.871842  9576 raft_consensus.cc:385] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.872290  9576 raft_consensus.cc:740] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.872228  9688 raft_consensus.cc:515] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.873119  9576 consensus_queue.cc:260] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.873481  9421 leader_election.cc:304] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:14.874353  9856 raft_consensus.cc:3060] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.874655  9688 leader_election.cc:290] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:14.876333  9576 ts_tablet_manager.cc:1434] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:25:14.877923  9576 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:14.877959  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0b217a2c7841638bd1d47ad5ea6783" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.878443  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0b217a2c7841638bd1d47ad5ea6783" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:14.879634  9494 leader_election.cc:343] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:14.880141  9495 leader_election.cc:343] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:14.880568  9495 leader_election.cc:304] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:14.884833  9574 tablet_bootstrap.cc:654] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.885448  9576 tablet_bootstrap.cc:654] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.867233  9662 leader_election.cc:290] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:14.888760  9688 raft_consensus.cc:2749] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.893054  9856 raft_consensus.cc:2749] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:14.903625  9575 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:14.904181  9575 ts_tablet_manager.cc:1403] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.085s	user 0.014s	sys 0.002s
I20260709 12:25:14.906409  9577 tablet_bootstrap.cc:654] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.907795  9575 raft_consensus.cc:359] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.908721  9575 raft_consensus.cc:385] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.909010  9575 raft_consensus.cc:740] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.909644  9575 consensus_queue.cc:260] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:14.922423  9847 raft_consensus.cc:493] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.924850  9575 ts_tablet_manager.cc:1434] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.004s
I20260709 12:25:14.925880  9575 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:14.926798  9201 catalog_manager.cc:5697] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:14.923142  9847 raft_consensus.cc:515] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.929072  9201 catalog_manager.cc:5697] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:14.940055  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9508782ff1d4df7867468f207693333" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:14.940798  9455 raft_consensus.cc:2468] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:14.942219  9346 leader_election.cc:304] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:14.944579  9698 raft_consensus.cc:2804] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:14.944983  9698 raft_consensus.cc:493] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.945341  9698 raft_consensus.cc:3060] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.945566  9575 tablet_bootstrap.cc:654] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.955461  9847 leader_election.cc:290] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.957048  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9508782ff1d4df7867468f207693333" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:14.957286  9698 raft_consensus.cc:515] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.957857  9526 raft_consensus.cc:2468] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:14.959937  9698 leader_election.cc:290] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:14.960630  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9508782ff1d4df7867468f207693333" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:14.961275  9455 raft_consensus.cc:3060] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.962711  9576 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:14.963253  9576 ts_tablet_manager.cc:1403] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.086s	user 0.028s	sys 0.025s
I20260709 12:25:14.964174  9577 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:14.964694  9577 ts_tablet_manager.cc:1403] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.152s	user 0.014s	sys 0.004s
I20260709 12:25:14.966003  9576 raft_consensus.cc:359] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.966702  9576 raft_consensus.cc:385] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.966962  9576 raft_consensus.cc:740] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.967887  9577 raft_consensus.cc:359] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.967803  9576 consensus_queue.cc:260] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.968606  9577 raft_consensus.cc:385] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.968673  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9508782ff1d4df7867468f207693333" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:14.969017  9577 raft_consensus.cc:740] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.969360  9526 raft_consensus.cc:3060] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.970371  9576 ts_tablet_manager.cc:1434] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:14.970046  9577 consensus_queue.cc:260] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:14.971958  9577 ts_tablet_manager.cc:1434] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 12:25:14.971972  9574 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:14.973060  9577 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:14.973256  9574 ts_tablet_manager.cc:1403] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.130s	user 0.017s	sys 0.007s
I20260709 12:25:14.976269  9574 raft_consensus.cc:359] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.976991  9574 raft_consensus.cc:385] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:14.977276  9574 raft_consensus.cc:740] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:14.978699  9526 raft_consensus.cc:2468] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:14.980125  9346 leader_election.cc:304] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:14.980468  9577 tablet_bootstrap.cc:654] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:14.981179  9698 raft_consensus.cc:2804] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:14.981745  9698 raft_consensus.cc:697] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:14.982630  9698 consensus_queue.cc:237] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:14.985025  9576 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:14.985941  9200 catalog_manager.cc:5697] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.991211  9381 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:14.993759  9574 consensus_queue.cc:260] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:14.996104  9574 ts_tablet_manager.cc:1434] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:25:14.997073  9574 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:15.001214  9682 consensus_queue.cc:1048] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.003304  9576 tablet_bootstrap.cc:654] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.006513  9455 raft_consensus.cc:2468] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.008520  9574 tablet_bootstrap.cc:654] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.016259  9576 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:15.016757  9576 ts_tablet_manager.cc:1403] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.032s	user 0.014s	sys 0.000s
I20260709 12:25:15.020560  9576 raft_consensus.cc:359] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.021231  9576 raft_consensus.cc:385] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.021493  9576 raft_consensus.cc:740] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.022246  9576 consensus_queue.cc:260] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.024876  9576 ts_tablet_manager.cc:1434] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:25:15.025774  9576 tablet_bootstrap.cc:492] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.027637  9574 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:15.028323  9574 ts_tablet_manager.cc:1403] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.031s	user 0.011s	sys 0.000s
I20260709 12:25:15.031641  9576 tablet_bootstrap.cc:654] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.035754  9612 consensus_queue.cc:1048] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:15.039921  9698 consensus_queue.cc:1048] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:15.047909  9574 raft_consensus.cc:359] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.086761  9574 raft_consensus.cc:385] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.080541  9784 consensus_queue.cc:1048] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.087234  9574 raft_consensus.cc:740] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.084760  9698 consensus_queue.cc:1048] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:15.088368  9574 consensus_queue.cc:260] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.114658  9698 raft_consensus.cc:493] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.115581  9698 raft_consensus.cc:515] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.120965  9698 leader_election.cc:290] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.121556  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6272a586489844f8a507c2bea1c89107" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:15.123039  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6272a586489844f8a507c2bea1c89107" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:15.124215  9347 leader_election.cc:343] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.124387  9455 raft_consensus.cc:2468] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:15.126510  9346 leader_election.cc:304] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:15.129066  9574 ts_tablet_manager.cc:1434] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.100s	user 0.011s	sys 0.000s
I20260709 12:25:15.130223  9574 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:15.130638  9688 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.131471  9688 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.135475  9688 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.136276  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.136273  9698 raft_consensus.cc:2804] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.137684  9576 tablet_bootstrap.cc:492] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
W20260709 12:25:15.137962  9494 leader_election.cc:343] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.138357  9576 ts_tablet_manager.cc:1403] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.113s	user 0.031s	sys 0.028s
I20260709 12:25:15.138466  9201 catalog_manager.cc:5697] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:15.139392  9749 consensus_queue.cc:1048] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.141448  9576 raft_consensus.cc:359] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.142174  9576 raft_consensus.cc:385] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.142460  9576 raft_consensus.cc:740] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.137738  9698 raft_consensus.cc:493] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.148772  9698 raft_consensus.cc:3060] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.152086  9575 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:15.152717  9575 ts_tablet_manager.cc:1403] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.227s	user 0.045s	sys 0.077s
I20260709 12:25:15.153215  9574 tablet_bootstrap.cc:654] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.167982  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:15.168663  9576 consensus_queue.cc:260] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.172701  9576 ts_tablet_manager.cc:1434] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.001s
I20260709 12:25:15.173695  9576 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.180094  9575 raft_consensus.cc:359] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.180799  9575 raft_consensus.cc:385] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.181068  9575 raft_consensus.cc:740] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.181785  9575 consensus_queue.cc:260] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.182878  9576 tablet_bootstrap.cc:654] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.183872  9575 ts_tablet_manager.cc:1434] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 12:25:15.184849  9575 tablet_bootstrap.cc:492] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:15.186009  9306 raft_consensus.cc:3060] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.188647  9576 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:15.189148  9576 ts_tablet_manager.cc:1403] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.004s
I20260709 12:25:15.190747  9575 tablet_bootstrap.cc:654] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.192027  9576 raft_consensus.cc:359] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.192735  9576 raft_consensus.cc:385] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.193013  9576 raft_consensus.cc:740] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.193830  9576 consensus_queue.cc:260] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.196349  9576 ts_tablet_manager.cc:1434] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:25:15.197250  9576 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.197517  9847 consensus_queue.cc:1048] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.204252  9612 raft_consensus.cc:493] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.204507  9576 tablet_bootstrap.cc:654] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.204900  9612 raft_consensus.cc:515] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.206321  9575 tablet_bootstrap.cc:492] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:15.206830  9575 ts_tablet_manager.cc:1403] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.022s	user 0.008s	sys 0.011s
I20260709 12:25:15.209091  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:15.210330  9575 raft_consensus.cc:359] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
W20260709 12:25:15.210777  9272 leader_election.cc:343] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.211000  9575 raft_consensus.cc:385] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.211575  9575 raft_consensus.cc:740] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.211675  9612 leader_election.cc:290] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:15.212431  9575 consensus_queue.cc:260] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.217540  9574 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:15.219822  9574 ts_tablet_manager.cc:1403] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.090s	user 0.012s	sys 0.000s
I20260709 12:25:15.221125  9749 consensus_queue.cc:1048] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.222782  9574 raft_consensus.cc:359] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.223510  9574 raft_consensus.cc:385] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.223767  9574 raft_consensus.cc:740] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.224397  9574 consensus_queue.cc:260] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.224994  9576 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:15.225507  9576 ts_tablet_manager.cc:1403] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.028s	user 0.013s	sys 0.003s
I20260709 12:25:15.226697  9574 ts_tablet_manager.cc:1434] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:15.228734  9575 ts_tablet_manager.cc:1434] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.004s
I20260709 12:25:15.229686  9575 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:15.229003  9576 raft_consensus.cc:359] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.230393  9576 raft_consensus.cc:385] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.230788  9576 raft_consensus.cc:740] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
W20260709 12:25:15.219132  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 50797.2us
I20260709 12:25:15.232128  9574 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:15.231747  9576 consensus_queue.cc:260] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.149148  9577 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:15.234292  9577 ts_tablet_manager.cc:1403] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.261s	user 0.013s	sys 0.002s
I20260709 12:25:15.234823  9576 ts_tablet_manager.cc:1434] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:25:15.236130  9576 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.239883  9698 raft_consensus.cc:515] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.242614  9576 tablet_bootstrap.cc:654] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.244973  9846 consensus_queue.cc:1048] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:25:15.238948  9495 leader_election.cc:343] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.259642  9495 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:15.263010  9618 raft_consensus.cc:2749] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:15.264453  9846 consensus_queue.cc:1048] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:15.296818  9749 consensus_queue.cc:1048] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.297559  9577 raft_consensus.cc:359] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.298271  9577 raft_consensus.cc:385] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.298573  9577 raft_consensus.cc:740] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.301011  9574 tablet_bootstrap.cc:654] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.303964  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.299217  9577 consensus_queue.cc:260] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.304855  9528 raft_consensus.cc:2468] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:15.305806  9577 ts_tablet_manager.cc:1434] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.071s	user 0.006s	sys 0.000s
I20260709 12:25:15.306134  9688 raft_consensus.cc:493] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.306749  9577 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:15.306666  9688 raft_consensus.cc:515] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.331908  9846 raft_consensus.cc:493] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.332728  9846 raft_consensus.cc:515] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.340395  9846 leader_election.cc:290] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.342535  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.343747  9574 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:15.344522  9574 ts_tablet_manager.cc:1403] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.113s	user 0.010s	sys 0.007s
I20260709 12:25:15.344828  9618 raft_consensus.cc:493] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.345336  9618 raft_consensus.cc:515] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.346292  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb1d8de1bb30430eb7ae1bbbf66232df" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.346985  9528 raft_consensus.cc:2468] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:15.348145  9346 leader_election.cc:304] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:15.340857  9851 raft_consensus.cc:493] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.349581  9851 raft_consensus.cc:515] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:15.351449  9494 leader_election.cc:343] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.351745  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:15.353188  9495 leader_election.cc:343] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.353941  9618 leader_election.cc:290] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.355062  9575 tablet_bootstrap.cc:654] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.354990  9851 leader_election.cc:290] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:15.316846  9688 leader_election.cc:290] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.337395  9201 catalog_manager.cc:5697] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:15.363402  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.365687  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:15.366715  9494 leader_election.cc:343] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.367007  9577 tablet_bootstrap.cc:654] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:15.370363  9495 leader_election.cc:343] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.371023  9495 leader_election.cc:304] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:15.368475  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:15.367712  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:15.374224  9851 raft_consensus.cc:2749] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:25:15.380203  9495 leader_election.cc:343] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:15.382105  9495 leader_election.cc:343] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.382534  9495 leader_election.cc:304] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:15.384720  9618 raft_consensus.cc:2749] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:15.387041  9694 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.387606  9694 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.389665  9694 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.367259  9494 leader_election.cc:304] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:15.394610  9455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.360059  9846 raft_consensus.cc:2804] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.399418  9574 raft_consensus.cc:359] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.405759  9574 raft_consensus.cc:385] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.406136  9574 raft_consensus.cc:740] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.406914  9574 consensus_queue.cc:260] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.390941  9303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb1d8de1bb30430eb7ae1bbbf66232df" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:15.409777  9346 leader_election.cc:343] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.410965  9574 ts_tablet_manager.cc:1434] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.066s	user 0.004s	sys 0.004s
I20260709 12:25:15.404109  9851 raft_consensus.cc:2749] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:15.403939  9846 raft_consensus.cc:493] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.416332  9846 raft_consensus.cc:3060] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.413792  9271 leader_election.cc:304] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:15.421284  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:15.421641  9682 raft_consensus.cc:2804] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.422130  9682 raft_consensus.cc:493] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.422575  9682 raft_consensus.cc:3060] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:15.430088  9347 leader_election.cc:343] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.430658  9347 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:15.431412  9847 raft_consensus.cc:2749] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:15.415004  9303 raft_consensus.cc:2468] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:15.449518  9698 leader_election.cc:290] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.450448  9576 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:15.451244  9576 ts_tablet_manager.cc:1403] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.215s	user 0.024s	sys 0.084s
I20260709 12:25:15.455140  9612 raft_consensus.cc:493] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.455955  9612 raft_consensus.cc:515] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.458379  9612 leader_election.cc:290] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.461278  9574 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:15.462790  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6272a586489844f8a507c2bea1c89107" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:15.463550  9453 raft_consensus.cc:3060] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.469139  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6272a586489844f8a507c2bea1c89107" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.470291  9574 tablet_bootstrap.cc:654] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.473533  9576 raft_consensus.cc:359] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.474556  9576 raft_consensus.cc:385] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.474874  9576 raft_consensus.cc:740] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.481737  9576 consensus_queue.cc:260] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:15.480844  9908 log.cc:927] Time spent T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:25:15.502290  9576 ts_tablet_manager.cc:1434] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.001s
I20260709 12:25:15.503880  9576 tablet_bootstrap.cc:492] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.513985  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6272a586489844f8a507c2bea1c89107" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:15.514679  9307 raft_consensus.cc:3060] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.525362  9575 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:15.526170  9575 ts_tablet_manager.cc:1403] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.297s	user 0.025s	sys 0.016s
I20260709 12:25:15.514381  9856 consensus_queue.cc:1048] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:25:15.533926  9682 raft_consensus.cc:515] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:15.507264  9272 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:42279, user_credentials={real_user=slave}} blocked reactor thread for 78133us
I20260709 12:25:15.538552  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:15.539300  9529 raft_consensus.cc:3060] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.539597  9682 leader_election.cc:290] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:15.548529  9307 raft_consensus.cc:2468] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.555589  9576 tablet_bootstrap.cc:654] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.550735  9529 raft_consensus.cc:2468] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:15.555816  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6272a586489844f8a507c2bea1c89107" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:15.572530  9271 leader_election.cc:304] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:15.576828  9347 leader_election.cc:304] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:15.576890  9846 raft_consensus.cc:515] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.578070  9694 consensus_queue.cc:1048] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.587422  9912 raft_consensus.cc:2804] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.589958  9912 raft_consensus.cc:697] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:15.590848  9912 consensus_queue.cc:237] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.591502  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb1d8de1bb30430eb7ae1bbbf66232df" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:15.588152  9577 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:15.592851  9577 ts_tablet_manager.cc:1403] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.286s	user 0.009s	sys 0.010s
I20260709 12:25:15.593117  9527 raft_consensus.cc:3060] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.595736  9378 raft_consensus.cc:2393] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e58af1026634b45bafbbfde02a6ecd0 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:15.598400  9698 raft_consensus.cc:2804] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.600261  9574 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:15.600857  9574 ts_tablet_manager.cc:1403] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.140s	user 0.019s	sys 0.023s
I20260709 12:25:15.603511  9698 raft_consensus.cc:697] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:15.604589  9698 consensus_queue.cc:237] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.605263  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:15.605979  9381 raft_consensus.cc:3060] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.605831  9272 leader_election.cc:304] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:15.607077  9846 leader_election.cc:290] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.607818  9612 raft_consensus.cc:2749] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:15.611189  9575 raft_consensus.cc:359] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.611905  9575 raft_consensus.cc:385] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.612157  9575 raft_consensus.cc:740] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.612802  9575 consensus_queue.cc:260] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.625288  9577 raft_consensus.cc:359] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.626757  9577 raft_consensus.cc:385] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.627233  9577 raft_consensus.cc:740] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
W20260709 12:25:15.630980  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:15.631968  9575 ts_tablet_manager.cc:1434] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.105s	user 0.009s	sys 0.000s
I20260709 12:25:15.635004  9575 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:15.631004  9577 consensus_queue.cc:260] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.638561  9574 raft_consensus.cc:359] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.643137  9577 ts_tablet_manager.cc:1434] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.000s
I20260709 12:25:15.643102  9574 raft_consensus.cc:385] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.643910  9574 raft_consensus.cc:740] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.644681  9577 tablet_bootstrap.cc:492] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:15.644729  9574 consensus_queue.cc:260] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.629838  9453 raft_consensus.cc:2468] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.651006  9574 ts_tablet_manager.cc:1434] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.000s
I20260709 12:25:15.653223  9574 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
W20260709 12:25:15.637430  9347 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 63345.9us
I20260709 12:25:15.658895  9381 raft_consensus.cc:2468] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:15.667721  9916 raft_consensus.cc:493] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.668294  9916 raft_consensus.cc:515] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.670063  9527 raft_consensus.cc:2468] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.670823  9916 leader_election.cc:290] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:15.671285  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb1d8de1bb30430eb7ae1bbbf66232df" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:15.671933  9305 raft_consensus.cc:3060] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.675637  9346 leader_election.cc:304] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:15.682399  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.680217  9846 raft_consensus.cc:2804] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.685374  9846 raft_consensus.cc:697] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:15.686224  9846 consensus_queue.cc:237] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:15.688436  9346 leader_election.cc:343] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.693440  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.694159  9529 raft_consensus.cc:2468] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:15.695832  9346 leader_election.cc:304] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:15.696834  9847 raft_consensus.cc:2804] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.697196  9847 raft_consensus.cc:493] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.697539  9847 raft_consensus.cc:3060] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.707734  9575 tablet_bootstrap.cc:654] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.709494  9694 consensus_queue.cc:1048] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:15.726079  9305 raft_consensus.cc:2468] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.727874  9576 tablet_bootstrap.cc:492] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:15.731817  9576 ts_tablet_manager.cc:1403] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.228s	user 0.033s	sys 0.029s
I20260709 12:25:15.756227  9201 catalog_manager.cc:5697] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:15.758775  9847 raft_consensus.cc:515] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.760952  9847 leader_election.cc:290] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:15.764371  9576 raft_consensus.cc:359] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.767887  9576 raft_consensus.cc:385] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.768189  9576 raft_consensus.cc:740] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.768815  9576 consensus_queue.cc:260] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.771237  9576 ts_tablet_manager.cc:1434] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.039s	user 0.005s	sys 0.000s
I20260709 12:25:15.776162  9576 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.779793  9574 tablet_bootstrap.cc:654] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.783098  9205 catalog_manager.cc:5697] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:15.779112  9347 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 60046.1us
I20260709 12:25:15.799168  9856 raft_consensus.cc:493] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.799760  9856 raft_consensus.cc:515] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.801889  9856 leader_election.cc:290] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:15.812495  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 78012.7us
I20260709 12:25:15.814484  9576 tablet_bootstrap.cc:654] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.816987  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:15.821035  9529 raft_consensus.cc:3060] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.821676  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:15.827767  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0b217a2c7841638bd1d47ad5ea6783" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:15.818483  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0b217a2c7841638bd1d47ad5ea6783" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.830274  9527 raft_consensus.cc:2468] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:15.831676  9420 leader_election.cc:304] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:15.832441  9863 raft_consensus.cc:2804] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.832856  9863 raft_consensus.cc:493] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.833166  9863 raft_consensus.cc:3060] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.828491  9306 raft_consensus.cc:2468] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
W20260709 12:25:15.845992  9920 log.cc:927] Time spent T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:25:15.848187  9346 leader_election.cc:343] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.848539  9577 tablet_bootstrap.cc:654] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:15.856566  9205 catalog_manager.cc:5697] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:15.864413  9749 raft_consensus.cc:493] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.869947  9749 raft_consensus.cc:515] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.881505  9576 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:15.882298  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.882616  9576 ts_tablet_manager.cc:1403] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.107s	user 0.021s	sys 0.003s
I20260709 12:25:15.883028  9528 raft_consensus.cc:2468] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 0.
I20260709 12:25:15.887240  9576 raft_consensus.cc:359] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.888016  9576 raft_consensus.cc:385] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.888280  9576 raft_consensus.cc:740] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.889046  9576 consensus_queue.cc:260] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.892046  9420 leader_election.cc:304] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:15.893563  9856 raft_consensus.cc:2804] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.893934  9856 raft_consensus.cc:493] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.894279  9856 raft_consensus.cc:3060] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.895017  9575 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:15.895574  9575 ts_tablet_manager.cc:1403] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.261s	user 0.011s	sys 0.012s
I20260709 12:25:15.895030  9576 ts_tablet_manager.cc:1434] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.003s
I20260709 12:25:15.897369  9576 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:15.900890  9749 leader_election.cc:290] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.902783  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:15.905735  9421 leader_election.cc:343] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.908777  9856 raft_consensus.cc:515] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.910048  9529 raft_consensus.cc:2468] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.911245  9856 leader_election.cc:290] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.911342  9346 leader_election.cc:304] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:15.912331  9847 raft_consensus.cc:2804] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.912789  9847 raft_consensus.cc:697] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:15.912838  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:15.913455  9528 raft_consensus.cc:3060] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.913903  9847 consensus_queue.cc:237] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.912811  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:15.917729  9927 raft_consensus.cc:493] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.918244  9927 raft_consensus.cc:515] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.916985  9863 raft_consensus.cc:515] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.920500  9577 tablet_bootstrap.cc:492] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:15.920996  9577 ts_tablet_manager.cc:1403] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.277s	user 0.013s	sys 0.012s
I20260709 12:25:15.921159  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.921836  9526 raft_consensus.cc:2468] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:15.922127  9927 leader_election.cc:290] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.923380  9346 leader_election.cc:304] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:15.924454  9698 raft_consensus.cc:2804] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.924821  9698 raft_consensus.cc:493] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.925156  9698 raft_consensus.cc:3060] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:15.925552  9421 leader_election.cc:343] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.926898  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0b217a2c7841638bd1d47ad5ea6783" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:15.927603  9307 raft_consensus.cc:3060] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.928393  9577 raft_consensus.cc:359] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.929102  9577 raft_consensus.cc:385] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.929381  9577 raft_consensus.cc:740] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.928778  9574 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:15.920696  9863 leader_election.cc:290] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.932010  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0b217a2c7841638bd1d47ad5ea6783" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:15.932611  9530 raft_consensus.cc:3060] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.933991  9851 raft_consensus.cc:493] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.935648  9912 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.936489  9574 ts_tablet_manager.cc:1403] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.284s	user 0.021s	sys 0.004s
I20260709 12:25:15.936213  9912 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.939251  9381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:15.939932  9381 raft_consensus.cc:2468] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:15.941268  9272 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:15.941519  9698 raft_consensus.cc:515] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.942103  9612 raft_consensus.cc:2804] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:15.942574  9612 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.943013  9612 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.945118  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:15.945860  9529 raft_consensus.cc:3060] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.935923  9851 raft_consensus.cc:515] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.948693  9851 leader_election.cc:290] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.949928  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.950771  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:15.951496  9494 leader_election.cc:343] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:15.952390  9495 leader_election.cc:343] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.952796  9495 leader_election.cc:304] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:15.953977  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:15.955653  9698 leader_election.cc:290] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:15.956105  9912 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:15.956280  9529 raft_consensus.cc:2468] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:15.957149  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:15.957700  9346 leader_election.cc:304] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:15.957250  9307 raft_consensus.cc:2468] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
W20260709 12:25:15.958716  9271 leader_election.cc:343] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.930068  9577 consensus_queue.cc:260] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.959299  9698 raft_consensus.cc:2804] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.959867  9618 raft_consensus.cc:2749] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:15.960129  9698 raft_consensus.cc:697] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:15.959887  9421 leader_election.cc:304] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:15.961382  9577 ts_tablet_manager.cc:1434] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.000s
I20260709 12:25:15.961243  9698 consensus_queue.cc:237] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.962487  9577 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:15.925359  9575 raft_consensus.cc:359] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.963085  9575 raft_consensus.cc:385] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.963395  9575 raft_consensus.cc:740] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.964179  9575 consensus_queue.cc:260] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:15.966563  9575 ts_tablet_manager.cc:1434] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.071s	user 0.006s	sys 0.000s
I20260709 12:25:15.962512  9662 raft_consensus.cc:493] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.967871  9662 raft_consensus.cc:515] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:15.968384  9530 raft_consensus.cc:2468] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
W20260709 12:25:15.957041  9347 leader_election.cc:343] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.970193  9662 leader_election.cc:290] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:15.961966  9749 raft_consensus.cc:2804] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.972324  9749 raft_consensus.cc:697] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:15.973269  9749 consensus_queue.cc:237] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.974658  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:15.975379  9530 raft_consensus.cc:2393] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074a6ac642e4430ba8045181f0118ee3 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:15.974315  9575 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:15.979220  9528 raft_consensus.cc:2468] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:15.981940  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:15.983489  9420 leader_election.cc:304] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:15.973253  9574 raft_consensus.cc:359] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.985363  9856 raft_consensus.cc:2804] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:15.985761  9574 raft_consensus.cc:385] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:15.980288  9612 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:15.985992  9856 raft_consensus.cc:697] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:15.987680  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:15.986094  9574 raft_consensus.cc:740] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:15.988374  9856 consensus_queue.cc:237] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:15.992786  9575 tablet_bootstrap.cc:654] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:15.989025  9347 leader_election.cc:343] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:15.990069  9421 leader_election.cc:304] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:15.990489  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:15.997553  9378 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:15.995719  9577 tablet_bootstrap.cc:654] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.007771  9662 raft_consensus.cc:3060] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.013806  9576 tablet_bootstrap.cc:654] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.017479  9612 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:16.013109  9574 consensus_queue.cc:260] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.018748  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
W20260709 12:25:16.020761  9271 leader_election.cc:343] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.006947  9378 raft_consensus.cc:2468] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:16.020787  9574 ts_tablet_manager.cc:1434] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.084s	user 0.005s	sys 0.002s
I20260709 12:25:16.024082  9272 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:16.025812  9682 raft_consensus.cc:2804] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:16.026448  9682 raft_consensus.cc:697] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:16.027541  9682 consensus_queue.cc:237] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.031474  9205 catalog_manager.cc:5697] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:16.051107  9574 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:16.045663  9662 raft_consensus.cc:2749] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:16.065642  9200 catalog_manager.cc:5697] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:16.074589  9574 tablet_bootstrap.cc:654] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.077836  9847 raft_consensus.cc:493] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.078553  9847 raft_consensus.cc:515] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.089747  9927 consensus_queue.cc:1048] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:16.101248  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:16.103142  9201 catalog_manager.cc:5697] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:16.104233  9307 raft_consensus.cc:2468] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:16.108765  9347 leader_election.cc:304] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:16.115025  9846 raft_consensus.cc:2804] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:16.110905  9847 leader_election.cc:290] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:16.111122  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:16.122924  9846 raft_consensus.cc:493] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.118458  9698 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.131394  9205 catalog_manager.cc:5697] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:16.144393  9346 leader_election.cc:343] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.136682  9846 raft_consensus.cc:3060] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.126155  9927 consensus_queue.cc:1048] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 12:25:16.151135  9577 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:16.151624  9577 ts_tablet_manager.cc:1403] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.189s	user 0.019s	sys 0.006s
I20260709 12:25:16.154790  9201 catalog_manager.cc:5697] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:16.164620  9575 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:16.165138  9575 ts_tablet_manager.cc:1403] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.191s	user 0.018s	sys 0.010s
I20260709 12:25:16.174460  9575 raft_consensus.cc:359] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.177091  9575 raft_consensus.cc:385] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.177320  9575 raft_consensus.cc:740] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.177872  9575 consensus_queue.cc:260] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.139775  9698 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.183857  9912 consensus_queue.cc:1048] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:16.185113  9576 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:16.185628  9576 ts_tablet_manager.cc:1403] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.289s	user 0.011s	sys 0.008s
I20260709 12:25:16.183182  9577 raft_consensus.cc:359] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.190927  9577 raft_consensus.cc:385] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.191358  9577 raft_consensus.cc:740] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.192943  9577 consensus_queue.cc:260] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.197383  9574 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:16.197979  9574 ts_tablet_manager.cc:1403] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.147s	user 0.026s	sys 0.023s
I20260709 12:25:16.202446  9574 raft_consensus.cc:359] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.203207  9574 raft_consensus.cc:385] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.203539  9574 raft_consensus.cc:740] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.204327  9574 consensus_queue.cc:260] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.206049  9574 ts_tablet_manager.cc:1434] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:25:16.207062  9574 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:16.217819  9575 ts_tablet_manager.cc:1434] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.000s
I20260709 12:25:16.220918  9575 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:16.223691  9698 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:16.223181  9577 ts_tablet_manager.cc:1434] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.071s	user 0.006s	sys 0.000s
I20260709 12:25:16.227481  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:16.228420  9530 raft_consensus.cc:2468] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:16.238090  9346 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:16.239089  9698 raft_consensus.cc:2804] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:16.232921  9576 raft_consensus.cc:359] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.239564  9698 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.240029  9576 raft_consensus.cc:385] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.240115  9698 raft_consensus.cc:3060] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.241154  9576 raft_consensus.cc:740] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.245782  9576 consensus_queue.cc:260] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.226393  9682 consensus_queue.cc:1048] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 12:25:16.252729  9576 ts_tablet_manager.cc:1434] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.067s	user 0.009s	sys 0.000s
I20260709 12:25:16.254529  9576 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:16.269560  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:16.273710  9927 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.274286  9927 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.276096  9851 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.276479  9927 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:16.276615  9851 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.277637  9577 tablet_bootstrap.cc:492] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:16.278632  9851 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:16.296664  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:16.297140  9847 consensus_queue.cc:1048] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:16.297523  9574 tablet_bootstrap.cc:654] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:16.298281  9495 leader_election.cc:343] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.300551  9698 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.303153  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:16.304258  9846 raft_consensus.cc:515] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.304975  9618 raft_consensus.cc:493] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.305454  9618 raft_consensus.cc:515] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.307065  9575 tablet_bootstrap.cc:654] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.307530  9618 leader_election.cc:290] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:16.317251  9577 tablet_bootstrap.cc:654] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.308638  9698 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:16.309268  9304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:16.319120  9304 raft_consensus.cc:3060] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.310333  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:16.320353  9530 raft_consensus.cc:3060] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.308679  9306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:16.322460  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:16.314260  9456 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:16.314949  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
W20260709 12:25:16.336855  9495 leader_election.cc:343] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.337395  9495 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
W20260709 12:25:16.338711  9346 leader_election.cc:343] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.348371  9530 raft_consensus.cc:2468] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:16.349246  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46e96ef5e406430f9972ca4831f609fe" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:16.333935  9846 leader_election.cc:290] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:16.363725  9305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46e96ef5e406430f9972ca4831f609fe" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
W20260709 12:25:16.366210  9347 leader_election.cc:343] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.340344  9851 raft_consensus.cc:2749] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:16.367417  9846 consensus_queue.cc:1048] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:25:16.373314  9346 leader_election.cc:343] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.375572  9346 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
W20260709 12:25:16.376145  9495 leader_election.cc:343] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:16.406292  9495 leader_election.cc:343] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.407037  9495 leader_election.cc:304] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:16.403203  9847 raft_consensus.cc:2804] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:16.410717  9304 raft_consensus.cc:2468] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:16.413388  9847 raft_consensus.cc:697] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:16.414669  9576 tablet_bootstrap.cc:654] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.414716  9847 consensus_queue.cc:237] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.424592  9851 raft_consensus.cc:2749] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:16.439831  9347 leader_election.cc:304] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:16.444279  9694 raft_consensus.cc:2804] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:16.444983  9694 raft_consensus.cc:697] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
W20260709 12:25:16.479912  9346 leader_election.cc:343] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.480726  9346 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
W20260709 12:25:16.490531  9346 leader_election.cc:343] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:16.497110  9271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 71412.7us
I20260709 12:25:16.446133  9694 consensus_queue.cc:237] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:16.470187  9272 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:42279, user_credentials={real_user=slave}} blocked reactor thread for 50910.7us
I20260709 12:25:16.483553  9927 raft_consensus.cc:2749] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:16.558080  9574 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:16.558748  9574 ts_tablet_manager.cc:1403] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.352s	user 0.050s	sys 0.059s
I20260709 12:25:16.562467  9574 raft_consensus.cc:359] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.568958  9574 raft_consensus.cc:385] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.569456  9574 raft_consensus.cc:740] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.570279  9574 consensus_queue.cc:260] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.575965  9574 ts_tablet_manager.cc:1434] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:25:16.578904  9574 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:16.596992  9574 tablet_bootstrap.cc:654] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.618811  9612 raft_consensus.cc:493] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.619477  9612 raft_consensus.cc:515] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.621470  9612 leader_election.cc:290] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:16.622602  9612 consensus_queue.cc:1048] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:16.627434  9574 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:16.627826  9577 tablet_bootstrap.cc:492] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:16.627992  9574 ts_tablet_manager.cc:1403] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.051s	user 0.013s	sys 0.002s
I20260709 12:25:16.628446  9577 ts_tablet_manager.cc:1403] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.351s	user 0.039s	sys 0.015s
I20260709 12:25:16.631011  9574 raft_consensus.cc:359] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.631800  9574 raft_consensus.cc:385] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.632153  9574 raft_consensus.cc:740] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.632826  9574 consensus_queue.cc:260] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.636049  9574 ts_tablet_manager.cc:1434] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:25:16.637043  9574 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:16.642473  9576 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:16.643114  9576 ts_tablet_manager.cc:1403] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.389s	user 0.018s	sys 0.006s
I20260709 12:25:16.643607  9574 tablet_bootstrap.cc:654] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:16.654126  9420 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 117795us
I20260709 12:25:16.654670  9576 raft_consensus.cc:359] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.655452  9576 raft_consensus.cc:385] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.655786  9576 raft_consensus.cc:740] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.655107  9577 raft_consensus.cc:359] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.656987  9577 raft_consensus.cc:385] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.657347  9577 raft_consensus.cc:740] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.656610  9576 consensus_queue.cc:260] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:16.658375  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 152223us
I20260709 12:25:16.658221  9577 consensus_queue.cc:260] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.659049  9576 ts_tablet_manager.cc:1434] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:16.660218  9577 ts_tablet_manager.cc:1434] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 12:25:16.660645  9576 tablet_bootstrap.cc:492] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:16.662024  9577 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:16.672878  9576 tablet_bootstrap.cc:654] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.684216  9529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:16.685173  9529 raft_consensus.cc:2393] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e58af1026634b45bafbbfde02a6ecd0 in current term 1: Already voted for candidate 074a6ac642e4430ba8045181f0118ee3 in this term.
I20260709 12:25:16.685642  9451 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.686278  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b303f27117804b4c8dc4ef4e31e0384e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:16.688266  9574 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:16.688803  9574 ts_tablet_manager.cc:1403] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.052s	user 0.007s	sys 0.005s
I20260709 12:25:16.697247  9574 raft_consensus.cc:359] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.697935  9574 raft_consensus.cc:385] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.698204  9574 raft_consensus.cc:740] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.700793  9271 leader_election.cc:304] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:16.701916  9956 raft_consensus.cc:3060] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.701970  9574 consensus_queue.cc:260] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:16.708129  9749 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.714097  9574 ts_tablet_manager.cc:1434] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.001s
I20260709 12:25:16.711313  9942 consensus_queue.cc:1048] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:25:16.711299  9749 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.717641  9749 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:16.719959  9577 tablet_bootstrap.cc:654] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.724370  9856 consensus_queue.cc:1048] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:16.721669  9574 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:16.726223  9863 consensus_queue.cc:1048] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:16.726169  9956 raft_consensus.cc:2749] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:16.741838  9698 consensus_queue.cc:1048] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:16.750218  9576 tablet_bootstrap.cc:492] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:16.750828  9576 ts_tablet_manager.cc:1403] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.090s	user 0.029s	sys 0.003s
I20260709 12:25:16.783723  9575 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:16.784305  9575 ts_tablet_manager.cc:1403] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.564s	user 0.013s	sys 0.007s
I20260709 12:25:16.791009  9577 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:16.791831  9577 ts_tablet_manager.cc:1403] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.131s	user 0.012s	sys 0.009s
W20260709 12:25:16.797562  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:16.811523  9307 raft_consensus.cc:3060] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:16.816849  9575 raft_consensus.cc:359] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.817647  9575 raft_consensus.cc:385] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.817903  9575 raft_consensus.cc:740] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.818600  9575 consensus_queue.cc:260] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:16.822626  9574 tablet_bootstrap.cc:654] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.825152  9575 ts_tablet_manager.cc:1434] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.040s	user 0.006s	sys 0.000s
I20260709 12:25:16.793556  9847 consensus_queue.cc:1048] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260709 12:25:16.832981  9577 raft_consensus.cc:359] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.833940  9577 raft_consensus.cc:385] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.834250  9577 raft_consensus.cc:740] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.835825  9576 raft_consensus.cc:359] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.836828  9576 raft_consensus.cc:385] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:16.834995  9577 consensus_queue.cc:260] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.837199  9576 raft_consensus.cc:740] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:16.834765  9201 catalog_manager.cc:5697] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:16.842060  9201 catalog_manager.cc:5697] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:16.844208  9576 consensus_queue.cc:260] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
W20260709 12:25:16.866776  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 65924.6us
I20260709 12:25:16.875627  9576 ts_tablet_manager.cc:1434] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.124s	user 0.005s	sys 0.006s
I20260709 12:25:16.876616  9576 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:16.877573  9575 tablet_bootstrap.cc:492] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:16.877699  9577 ts_tablet_manager.cc:1434] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.086s	user 0.008s	sys 0.000s
I20260709 12:25:16.878610  9577 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:16.887727  9302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:16.891037  9662 consensus_queue.cc:1048] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 12:25:16.920660  9618 raft_consensus.cc:493] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.921487  9618 raft_consensus.cc:515] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:16.942162  9575 tablet_bootstrap.cc:654] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.943681  9618 leader_election.cc:290] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:16.935628  9977 consensus_queue.cc:1048] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:25:16.954124  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:16.958648  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:16.960279  9494 leader_election.cc:343] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.975869  9962 consensus_queue.cc:1048] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:25:16.965436  9856 raft_consensus.cc:493] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.977696  9856 raft_consensus.cc:515] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:16.980510  9856 leader_election.cc:290] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:16.981717  9495 leader_election.cc:343] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:16.991901  9495 leader_election.cc:304] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
W20260709 12:25:16.942762  9347 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 51917.3us
I20260709 12:25:16.998448  9576 tablet_bootstrap.cc:654] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:16.999603  9577 tablet_bootstrap.cc:654] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.013569  9682 raft_consensus.cc:493] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.017120  9959 raft_consensus.cc:493] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.017640  9682 raft_consensus.cc:515] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.017728  9959 raft_consensus.cc:515] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.011055  9851 raft_consensus.cc:2749] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:17.027144  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:17.030907  9454 raft_consensus.cc:3060] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.037016  9452 raft_consensus.cc:3060] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.039531  9927 consensus_queue.cc:1048] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:25:17.014065  9421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 54862.4us
I20260709 12:25:17.069804  9682 leader_election.cc:290] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:17.062266  9958 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.070816  9977 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.070887  9958 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.071458  9977 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.074004  9958 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:17.076560  9959 leader_election.cc:290] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.077047  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.077741  9528 raft_consensus.cc:2468] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:17.081478  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:17.083428  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:17.084388  9378 raft_consensus.cc:2468] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:17.087258  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.091171  9846 consensus_queue.cc:1048] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:17.093039  9574 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:17.093575  9574 ts_tablet_manager.cc:1403] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.372s	user 0.033s	sys 0.013s
I20260709 12:25:17.094823  9346 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:17.100751  9574 raft_consensus.cc:359] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.101894  9574 raft_consensus.cc:385] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.102243  9574 raft_consensus.cc:740] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.110100  9574 consensus_queue.cc:260] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:17.118109  9271 leader_election.cc:343] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.116108  9847 raft_consensus.cc:2804] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:17.140074  9272 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:17.142697  9574 ts_tablet_manager.cc:1434] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.002s
I20260709 12:25:17.145416  9574 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:17.150779  9959 raft_consensus.cc:2804] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:17.151250  9959 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.151733  9959 raft_consensus.cc:3060] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.157030  9662 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.157636  9662 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.160985  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9eed29862fa545d7b5bf1232e33f0798" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.116991  9916 consensus_queue.cc:1048] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 12:25:17.165383  9977 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:17.166363  9688 raft_consensus.cc:493] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.166867  9688 raft_consensus.cc:515] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.168212  9421 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:17.168910  9688 leader_election.cc:290] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:17.169121  9984 raft_consensus.cc:2749] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:17.165297  9301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:17.170197  9301 raft_consensus.cc:2393] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074a6ac642e4430ba8045181f0118ee3 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:17.143908  9847 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.179171  9847 raft_consensus.cc:3060] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.180858  9662 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.199904  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.210351  9575 tablet_bootstrap.cc:492] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:17.210911  9575 ts_tablet_manager.cc:1403] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.333s	user 0.017s	sys 0.007s
I20260709 12:25:17.211467  9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.212164  9523 raft_consensus.cc:2468] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
W20260709 12:25:17.214221  9494 leader_election.cc:343] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.214557  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:17.203890  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.215926  9524 raft_consensus.cc:2468] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:17.220574  9271 leader_election.cc:304] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:17.224174  9576 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:17.224695  9576 ts_tablet_manager.cc:1403] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.348s	user 0.016s	sys 0.006s
I20260709 12:25:17.225564  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:17.233237  9958 raft_consensus.cc:2749] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:17.238050  9302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:17.239609  9421 leader_election.cc:304] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:17.240083  9576 raft_consensus.cc:359] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.240442  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.241034  9575 raft_consensus.cc:359] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.234014  9271 leader_election.cc:304] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:17.243018  9576 raft_consensus.cc:385] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.262213  9576 raft_consensus.cc:740] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.269843  9575 raft_consensus.cc:385] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.270471  9575 raft_consensus.cc:740] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.229468  9959 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.273514  9960 raft_consensus.cc:2804] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:17.274168  9960 raft_consensus.cc:493] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.274592  9959 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:17.275595  9307 raft_consensus.cc:2393] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e07a645c9ee14f11a5d64fa7b460acf9 in current term 1: Already voted for candidate 4e58af1026634b45bafbbfde02a6ecd0 in this term.
I20260709 12:25:17.277422  9574 tablet_bootstrap.cc:654] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.261123  9527 raft_consensus.cc:2468] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:17.280436  9749 raft_consensus.cc:2749] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:17.263126  9576 consensus_queue.cc:260] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.284463  9576 ts_tablet_manager.cc:1434] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.000s
I20260709 12:25:17.285671  9576 tablet_bootstrap.cc:492] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:17.290789  9575 consensus_queue.cc:260] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.274622  9960 raft_consensus.cc:3060] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:17.294122  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 174507us
W20260709 12:25:17.299223  9495 leader_election.cc:343] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.310038  9982 consensus_queue.cc:1048] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:25:17.305069  9575 ts_tablet_manager.cc:1434] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.094s	user 0.014s	sys 0.000s
I20260709 12:25:17.310992  9495 leader_election.cc:304] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:17.312382  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:17.296595  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:17.314332  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:17.314656  9618 raft_consensus.cc:2749] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:17.315045  9527 raft_consensus.cc:3060] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.322193  9846 consensus_queue.cc:1048] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:17.324592  9527 raft_consensus.cc:2468] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:17.328384  9420 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:17.331067  9856 raft_consensus.cc:2804] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:17.331542  9856 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.331929  9856 raft_consensus.cc:3060] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:17.333446  9999 raft_consensus.cc:493] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.334033  9999 raft_consensus.cc:515] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.335389  9575 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: Bootstrap starting.
I20260709 12:25:17.338104  9847 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.340157  9377 raft_consensus.cc:2393] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e58af1026634b45bafbbfde02a6ecd0 in current term 1: Already voted for candidate e07a645c9ee14f11a5d64fa7b460acf9 in this term.
I20260709 12:25:17.348922  9577 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:17.349639  9577 ts_tablet_manager.cc:1403] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.471s	user 0.004s	sys 0.028s
I20260709 12:25:17.358803  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.359109  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.359596  9524 raft_consensus.cc:2468] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
W20260709 12:25:17.367027  9271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 59855.3us
I20260709 12:25:17.368333  9575 tablet_bootstrap.cc:654] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.375499  9271 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:17.376549  9576 tablet_bootstrap.cc:654] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.377372  9577 raft_consensus.cc:359] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.390148  9577 raft_consensus.cc:385] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.390578  9577 raft_consensus.cc:740] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.391577  9577 consensus_queue.cc:260] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.394228  9577 ts_tablet_manager.cc:1434] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.044s	user 0.006s	sys 0.001s
I20260709 12:25:17.336521  9999 leader_election.cc:290] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.396682  9577 tablet_bootstrap.cc:492] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:17.400811  9942 raft_consensus.cc:2804] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:17.401496  9942 raft_consensus.cc:697] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:17.377300  9960 raft_consensus.cc:515] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.377985  9856 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.382297  9847 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:17.402436  9942 consensus_queue.cc:237] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.403384  9451 raft_consensus.cc:1240] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Rejecting Update request from peer e07a645c9ee14f11a5d64fa7b460acf9 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:25:17.384785  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:17.409870  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:17.410372  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:17.414153  9960 leader_election.cc:290] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.414858  9421 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:17.415958  9749 raft_consensus.cc:2749] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:25:17.420356  9346 leader_election.cc:343] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.423102  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:17.423113  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:17.424094  9451 raft_consensus.cc:3060] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.430622  9346 leader_election.cc:304] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:17.424176  9525 raft_consensus.cc:3060] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.427359  9856 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.433893  9925 raft_consensus.cc:2804] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:17.444762  9925 raft_consensus.cc:493] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.445353  9925 raft_consensus.cc:3060] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.428560  9999 consensus_queue.cc:1059] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:25:17.437649  9574 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:17.447832  9574 ts_tablet_manager.cc:1403] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.303s	user 0.018s	sys 0.021s
I20260709 12:25:17.447907  9980 raft_consensus.cc:3055] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:17.448364  9980 raft_consensus.cc:740] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:17.449188  9980 consensus_queue.cc:260] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.450475  9980 raft_consensus.cc:3060] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:17.450685  9574 raft_consensus.cc:359] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:17.451381  9698 consensus_queue.cc:1175] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a5f89c82eda14a9ca425fae398bae986" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:25:17.430296  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:17.454173  9527 raft_consensus.cc:2393] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e07a645c9ee14f11a5d64fa7b460acf9 in current term 1: Already voted for candidate 4e58af1026634b45bafbbfde02a6ecd0 in this term.
I20260709 12:25:17.451429  9574 raft_consensus.cc:385] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.456359  9574 raft_consensus.cc:740] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.464627  9574 consensus_queue.cc:260] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.466238  9574 ts_tablet_manager.cc:1434] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.000s
I20260709 12:25:17.475772  9574 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:17.463244  9200 catalog_manager.cc:5697] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:17.489315  9346 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:17.494395  9980 raft_consensus.cc:2749] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:25:17.502638  9576 tablet_bootstrap.cc:492] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:17.503368  9576 ts_tablet_manager.cc:1403] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.218s	user 0.011s	sys 0.000s
I20260709 12:25:17.506435  9577 tablet_bootstrap.cc:654] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.506857  9576 raft_consensus.cc:359] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.507906  9576 raft_consensus.cc:385] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.508239  9576 raft_consensus.cc:740] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.509033  9576 consensus_queue.cc:260] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.515055  9575 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: No bootstrap required, opened a new log
I20260709 12:25:17.515592  9575 ts_tablet_manager.cc:1403] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: Time spent bootstrapping tablet: real 0.180s	user 0.013s	sys 0.003s
I20260709 12:25:17.522125  9574 tablet_bootstrap.cc:654] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.526381  9451 raft_consensus.cc:2468] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:17.528260  9271 leader_election.cc:304] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:17.528975  9942 raft_consensus.cc:2804] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:17.529467  9942 raft_consensus.cc:697] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:17.530303  9942 consensus_queue.cc:237] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.540691  9925 raft_consensus.cc:515] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.509740  9525 raft_consensus.cc:2468] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:17.544343  9576 ts_tablet_manager.cc:1434] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.000s
I20260709 12:25:17.545498  9576 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9: Bootstrap starting.
I20260709 12:25:17.549160  9575 raft_consensus.cc:359] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.549803  9575 raft_consensus.cc:385] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.550066  9575 raft_consensus.cc:740] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.550931  9925 leader_election.cc:290] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.551939  9575 consensus_queue.cc:260] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.554445  9575 ts_tablet_manager.cc:1434] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.002s
I20260709 12:25:17.576825  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
W20260709 12:25:17.578626  9346 leader_election.cc:343] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.579454  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:17.580134  9524 raft_consensus.cc:3060] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.596873  9524 raft_consensus.cc:2468] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:17.598049  9346 leader_election.cc:304] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:17.598853  9698 raft_consensus.cc:2804] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:17.599431  9698 raft_consensus.cc:697] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:17.600453  9698 consensus_queue.cc:237] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.604959  9576 tablet_bootstrap.cc:654] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.610293  9574 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:17.611516  9574 ts_tablet_manager.cc:1403] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.136s	user 0.010s	sys 0.014s
I20260709 12:25:17.614192  9577 tablet_bootstrap.cc:492] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:17.608671  9942 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.614338  9574 raft_consensus.cc:359] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.618227  9574 raft_consensus.cc:385] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.617463  9942 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.620313  9577 ts_tablet_manager.cc:1403] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.224s	user 0.017s	sys 0.010s
I20260709 12:25:17.621181  9942 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:17.621757  9574 raft_consensus.cc:740] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.622469  9574 consensus_queue.cc:260] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.622974  9577 raft_consensus.cc:359] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.623718  9577 raft_consensus.cc:385] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.623987  9577 raft_consensus.cc:740] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.624598  9577 consensus_queue.cc:260] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.626721  9577 ts_tablet_manager.cc:1434] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:25:17.627602  9577 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:17.632279  9698 raft_consensus.cc:493] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.632820  9698 raft_consensus.cc:515] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.634011  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:17.634582  9577 tablet_bootstrap.cc:654] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.634665  9382 raft_consensus.cc:2468] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:17.634724  9698 leader_election.cc:290] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:17.638744  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.639420  9451 raft_consensus.cc:2468] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:17.640780  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:17.641409  9524 raft_consensus.cc:2468] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:17.644563  9346 leader_election.cc:304] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:17.645332  9698 raft_consensus.cc:2804] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:17.645795  9698 raft_consensus.cc:493] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.646163  9698 raft_consensus.cc:3060] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.646829  9576 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9: No bootstrap required, opened a new log
I20260709 12:25:17.647351  9576 ts_tablet_manager.cc:1403] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9: Time spent bootstrapping tablet: real 0.102s	user 0.021s	sys 0.010s
I20260709 12:25:17.653818  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.655660  9272 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:17.656687 10002 raft_consensus.cc:2804] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:17.657105 10002 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.657514 10002 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:17.661149  9271 leader_election.cc:343] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.678272 10002 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.681128  9200 catalog_manager.cc:5697] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:17.655055  9202 catalog_manager.cc:5697] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:17.694972  9574 ts_tablet_manager.cc:1434] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.083s	user 0.005s	sys 0.000s
I20260709 12:25:17.700600  9574 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:17.702598  9698 raft_consensus.cc:515] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.706766  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:17.707444  9450 raft_consensus.cc:3060] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.730281  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:17.731401  9379 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.753660 10002 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:17.757969  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:17.758589  9698 leader_election.cc:290] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
W20260709 12:25:17.762239  9271 leader_election.cc:343] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:17.767622  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4ba22da1d594bf19f8b83581343da38" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:17.768689  9524 raft_consensus.cc:3060] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:17.772624  9450 raft_consensus.cc:2468] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:17.783951  9698 raft_consensus.cc:493] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.784523  9698 raft_consensus.cc:515] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:17.793670  9346 leader_election.cc:304] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:17.797369  9576 raft_consensus.cc:359] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.798764  9576 raft_consensus.cc:385] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.799131  9576 raft_consensus.cc:740] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Initialized, Role: FOLLOWER
I20260709 12:25:17.796401  9925 raft_consensus.cc:2804] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:17.810518  9698 leader_election.cc:290] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:17.820904  9576 consensus_queue.cc:260] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:17.818778  9925 raft_consensus.cc:697] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:17.826102  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.892503  9618 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.893280  9618 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.897091  9574 tablet_bootstrap.cc:654] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:17.889331  9925 consensus_queue.cc:237] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:17.905313  9618 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:17.908574  9576 ts_tablet_manager.cc:1434] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9: Time spent starting tablet: real 0.261s	user 0.008s	sys 0.000s
I20260709 12:25:17.910068  9577 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:17.910642  9577 ts_tablet_manager.cc:1403] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.283s	user 0.065s	sys 0.074s
I20260709 12:25:17.831105  9304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:17.966320  9304 raft_consensus.cc:2468] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:17.986235  9347 leader_election.cc:304] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:17.991811  9577 raft_consensus.cc:359] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:17.993790  9577 raft_consensus.cc:385] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:17.995913  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:17.996836  9379 raft_consensus.cc:2468] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
W20260709 12:25:18.012749  9494 leader_election.cc:343] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:18.014086  9957 raft_consensus.cc:2804] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:18.000481  9577 raft_consensus.cc:740] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.020601  9698 raft_consensus.cc:493] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.029546  9957 raft_consensus.cc:493] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.059707  9574 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:18.060779  9574 ts_tablet_manager.cc:1403] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.360s	user 0.020s	sys 0.004s
I20260709 12:25:18.068006  9574 raft_consensus.cc:359] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:18.068975  9574 raft_consensus.cc:385] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:18.069269  9574 raft_consensus.cc:740] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.070200  9574 consensus_queue.cc:260] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:18.079859  9912 consensus_queue.cc:1048] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.085357  9577 consensus_queue.cc:260] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.087337  9272 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:18.088802  9577 ts_tablet_manager.cc:1434] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.178s	user 0.007s	sys 0.008s
I20260709 12:25:18.092049  9574 ts_tablet_manager.cc:1434] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.004s
I20260709 12:25:18.093861  9574 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:18.046048  9957 raft_consensus.cc:3060] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.100682  9959 raft_consensus.cc:2804] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:18.101347  9959 raft_consensus.cc:697] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:18.103078  9959 consensus_queue.cc:237] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.113737  9524 raft_consensus.cc:2468] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:18.118755  9698 raft_consensus.cc:515] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:18.147714 10002 raft_consensus.cc:493] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.148602 10002 raft_consensus.cc:515] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.170436  9577 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:18.151284 10002 leader_election.cc:290] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:18.196285  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:18.197207  9450 raft_consensus.cc:2468] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:18.210755  9574 tablet_bootstrap.cc:654] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:18.212419  9957 raft_consensus.cc:515] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.215796  9698 leader_election.cc:290] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:18.225862  9346 leader_election.cc:343] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:18.227598  9301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:18.228351  9301 raft_consensus.cc:3060] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.244411  9304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46e96ef5e406430f9972ca4831f609fe" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:18.245448  9304 raft_consensus.cc:2468] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:18.253228  9202 catalog_manager.cc:5697] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:18.260478  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:18.260921  9347 leader_election.cc:304] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:18.261314  9301 raft_consensus.cc:2468] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:18.267203  9528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46e96ef5e406430f9972ca4831f609fe" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:18.268296  9528 raft_consensus.cc:2468] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 0.
I20260709 12:25:18.268567  9271 leader_election.cc:304] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:18.262161  9957 leader_election.cc:290] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:18.269953 10018 raft_consensus.cc:2804] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:18.270586 10018 raft_consensus.cc:493] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.272032 10018 raft_consensus.cc:3060] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.278292 10025 consensus_queue.cc:1048] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.271229 10020 consensus_queue.cc:1048] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.279204  9200 catalog_manager.cc:5697] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:18.304338  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:18.305619  9524 raft_consensus.cc:2468] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:18.309574 10020 consensus_queue.cc:1048] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.316257  9577 tablet_bootstrap.cc:654] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:18.320399  9347 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 59564.9us
I20260709 12:25:18.322034  9957 consensus_queue.cc:1048] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.328728  9925 raft_consensus.cc:2804] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:18.329357  9925 raft_consensus.cc:493] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.329813  9925 raft_consensus.cc:3060] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.333923  9347 leader_election.cc:304] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:18.334939  9847 raft_consensus.cc:2804] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:18.335664  9847 raft_consensus.cc:697] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:18.322324 10018 raft_consensus.cc:515] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.336987  9847 consensus_queue.cc:237] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.339164 10018 leader_election.cc:290] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:18.360209  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:18.360383  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:18.361359  9454 raft_consensus.cc:3060] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.404892  9454 raft_consensus.cc:2468] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:18.426314  9271 leader_election.cc:304] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:18.430402  9574 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: No bootstrap required, opened a new log
I20260709 12:25:18.430881  9574 ts_tablet_manager.cc:1403] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.338s	user 0.013s	sys 0.005s
I20260709 12:25:18.430305 10025 raft_consensus.cc:2804] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:18.441499  9523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:18.441534 10025 raft_consensus.cc:697] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:18.442242  9523 raft_consensus.cc:3060] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.442986 10025 consensus_queue.cc:237] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.449044  9925 raft_consensus.cc:515] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:18.458146  9346 consensus_peers.cc:597] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:18.460213  9577 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:18.460768  9577 ts_tablet_manager.cc:1403] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.333s	user 0.011s	sys 0.012s
W20260709 12:25:18.463827  9495 leader_election.cc:343] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:18.464284  9495 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:18.464363  9301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46e96ef5e406430f9972ca4831f609fe" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
I20260709 12:25:18.465209  9301 raft_consensus.cc:3060] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.465898  9618 raft_consensus.cc:2749] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:25:18.462190  9346 leader_election.cc:343] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:18.473630  9577 raft_consensus.cc:359] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.474330  9577 raft_consensus.cc:385] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:18.474601  9577 raft_consensus.cc:740] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.475301  9577 consensus_queue.cc:260] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.460630  9925 leader_election.cc:290] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:18.486820  9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46e96ef5e406430f9972ca4831f609fe" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:18.497268  9530 raft_consensus.cc:3060] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:25:18.511510 10033 log.cc:927] Time spent T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:25:18.529871  9252 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:18.532564  9301 raft_consensus.cc:2468] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:18.533849  9574 raft_consensus.cc:359] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.539218  9574 raft_consensus.cc:385] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:18.539901  9574 raft_consensus.cc:740] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.558910  9574 consensus_queue.cc:260] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.533865  9523 raft_consensus.cc:2468] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:18.566071  9530 raft_consensus.cc:2468] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:18.579114  9574 ts_tablet_manager.cc:1434] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.148s	user 0.004s	sys 0.004s
I20260709 12:25:18.580813 10023 consensus_queue.cc:1048] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260709 12:25:18.582803  9980 consensus_queue.cc:1048] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.592051  9347 leader_election.cc:304] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:18.592700  9577 ts_tablet_manager.cc:1434] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.132s	user 0.007s	sys 0.002s
I20260709 12:25:18.598589  9577 tablet_bootstrap.cc:492] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:18.598918  9925 raft_consensus.cc:2804] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:18.599546  9925 raft_consensus.cc:697] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:18.600677  9925 consensus_queue.cc:237] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:18.722572  9271 consensus_peers.cc:597] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:18.729395  9577 tablet_bootstrap.cc:654] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:18.734715  9420 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 48295.4us
I20260709 12:25:18.731770  9912 consensus_queue.cc:1048] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:25:18.772573  9200 catalog_manager.cc:5697] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:18.785020 10023 consensus_queue.cc:1048] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:18.793674  9577 tablet_bootstrap.cc:492] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:18.794253  9577 ts_tablet_manager.cc:1403] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.196s	user 0.016s	sys 0.001s
I20260709 12:25:18.801620 10051 raft_consensus.cc:493] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.802196 10051 raft_consensus.cc:515] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.804598 10051 leader_election.cc:290] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:18.805248  9577 raft_consensus.cc:359] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:18.805944  9577 raft_consensus.cc:385] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:18.806248  9577 raft_consensus.cc:740] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.806893  9577 consensus_queue.cc:260] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:18.808207  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:18.819692  9851 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07a645c9ee14f11a5d64fa7b460acf9)
I20260709 12:25:18.820269  9851 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:18.821908  9307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0" is_pre_election: true
I20260709 12:25:18.823313  9851 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:18.823705  9577 ts_tablet_manager.cc:1434] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 12:25:18.823766  9421 leader_election.cc:304] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:18.824939  9577 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:18.825246  9749 raft_consensus.cc:2749] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:18.830245  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:18.831157  9450 raft_consensus.cc:2393] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5f89c82eda14a9ca425fae398bae986 in current term 2: Already voted for candidate 074a6ac642e4430ba8045181f0118ee3 in this term.
I20260709 12:25:18.832531  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:18.833292  9382 raft_consensus.cc:2468] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:18.842828  9495 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:18.837594  9577 tablet_bootstrap.cc:654] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:18.846168  9618 raft_consensus.cc:2804] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:18.846531  9618 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e07a645c9ee14f11a5d64fa7b460acf9)
I20260709 12:25:18.846859  9618 raft_consensus.cc:3060] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:18.854581  9577 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:18.855248  9577 ts_tablet_manager.cc:1403] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.003s
I20260709 12:25:18.858688  9577 raft_consensus.cc:359] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.859448  9577 raft_consensus.cc:385] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:18.859797  9577 raft_consensus.cc:740] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.861289  9577 consensus_queue.cc:260] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:18.863517  9618 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:18.866322  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:18.867237  9452 raft_consensus.cc:2393] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a5f89c82eda14a9ca425fae398bae986 in current term 2: Already voted for candidate 074a6ac642e4430ba8045181f0118ee3 in this term.
I20260709 12:25:18.867489  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:18.870745  9577 ts_tablet_manager.cc:1434] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:25:18.871951  9577 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3: Bootstrap starting.
I20260709 12:25:18.881599  9452 raft_consensus.cc:3060] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:18.886178  9577 tablet_bootstrap.cc:654] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:18.888023  9959 consensus_queue.cc:1048] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:18.888656  9618 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:18.878576  9378 raft_consensus.cc:2468] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:18.893390  9495 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:18.903033 10015 raft_consensus.cc:2804] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:18.904639 10015 raft_consensus.cc:697] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:18.906556 10015 consensus_queue.cc:237] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:18.927913 10025 consensus_queue.cc:1048] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:18.929791  9577 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3: No bootstrap required, opened a new log
I20260709 12:25:18.930539  9577 ts_tablet_manager.cc:1403] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3: Time spent bootstrapping tablet: real 0.059s	user 0.012s	sys 0.022s
I20260709 12:25:18.933537  9577 raft_consensus.cc:359] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.934306  9577 raft_consensus.cc:385] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:18.934604  9577 raft_consensus.cc:740] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Initialized, Role: FOLLOWER
I20260709 12:25:18.935528  9577 consensus_queue.cc:260] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:18.936115  9202 catalog_manager.cc:5697] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } }
I20260709 12:25:18.938090  9577 ts_tablet_manager.cc:1434] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:18.953372  9847 consensus_queue.cc:1048] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:18.985613  9200 catalog_manager.cc:5697] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:18.990370  9201 catalog_manager.cc:5697] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:18.993292  9847 consensus_queue.cc:1048] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:19.016609  9698 consensus_queue.cc:1048] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:19.025193 10051 raft_consensus.cc:493] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.025787 10051 raft_consensus.cc:515] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:19.029594  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:19.030211 10051 leader_election.cc:290] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:19.032951  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:19.036839  9420 leader_election.cc:304] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:19.037958 10051 raft_consensus.cc:2749] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:19.050585 10025 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.051160 10025 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:19.053308 10025 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:19.053893  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:19.055229  9452 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:19.056517  9271 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:19.056646  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:19.057314 10025 raft_consensus.cc:2804] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:19.057739 10025 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.058369 10025 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:19.057734  9526 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 0.
I20260709 12:25:19.068312 10025 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:19.070302 10025 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:19.070967  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:19.071273  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:19.071695  9452 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:19.071923  9526 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:19.081771  9452 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:19.082947  9271 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:19.083815  9959 raft_consensus.cc:2804] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:19.084303  9959 raft_consensus.cc:697] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:19.085300  9526 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 1.
I20260709 12:25:19.085150  9959 consensus_queue.cc:237] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:19.096415  9200 catalog_manager.cc:5697] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:19.110466  9452 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:19.210707  9452 raft_consensus.cc:3060] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:19.219190 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
W20260709 12:25:19.231658  9347 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20260709 12:25:19.245078  9495 consensus_peers.cc:597] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.268879  9347 consensus_peers.cc:597] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.274927  9347 consensus_peers.cc:597] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.275632  9347 consensus_peers.cc:597] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.278687  9421 consensus_peers.cc:597] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.281450  9347 consensus_peers.cc:597] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.283406  9347 consensus_peers.cc:597] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.309646 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:19.311179 26408 tablet_replica.cc:333] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.312297 26408 raft_consensus.cc:2243] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.312755 26408 raft_consensus.cc:2272] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.316248 26408 tablet_replica.cc:333] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.317174 26408 raft_consensus.cc:2243] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.318305 26408 raft_consensus.cc:2272] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.321290 26408 tablet_replica.cc:333] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.321882 26408 raft_consensus.cc:2243] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.322602 26408 raft_consensus.cc:2272] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.324966 26408 tablet_replica.cc:333] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.325577 26408 raft_consensus.cc:2243] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.326012 26408 raft_consensus.cc:2272] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.329311 26408 tablet_replica.cc:333] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.330071 26408 raft_consensus.cc:2243] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.330566 26408 raft_consensus.cc:2272] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.333261 26408 tablet_replica.cc:333] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.333905 26408 raft_consensus.cc:2243] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.334791 26408 raft_consensus.cc:2272] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.336865 26408 tablet_replica.cc:333] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.337406 26408 raft_consensus.cc:2243] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.337834 26408 raft_consensus.cc:2272] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.340530 26408 tablet_replica.cc:333] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
W20260709 12:25:19.341367  9495 consensus_peers.cc:597] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.341420 26408 raft_consensus.cc:2243] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.342218 26408 raft_consensus.cc:2272] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.345104 26408 tablet_replica.cc:333] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.345623 26408 raft_consensus.cc:2243] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.346067 26408 raft_consensus.cc:2272] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.348358 26408 tablet_replica.cc:333] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.348969 26408 raft_consensus.cc:2243] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.349736 26408 raft_consensus.cc:2272] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.351846 26408 tablet_replica.cc:333] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.352382 26408 raft_consensus.cc:2243] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.352772 26408 raft_consensus.cc:2272] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.355003 26408 tablet_replica.cc:333] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.355644 26408 raft_consensus.cc:2243] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.356125 26408 raft_consensus.cc:2272] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.358356 26408 tablet_replica.cc:333] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.358944 26408 raft_consensus.cc:2243] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.359931 26408 raft_consensus.cc:2272] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.361989 26408 tablet_replica.cc:333] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.362587 26408 raft_consensus.cc:2243] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.363154 26408 raft_consensus.cc:2272] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.365399 26408 tablet_replica.cc:333] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.365974 26408 raft_consensus.cc:2243] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.366671 26408 pending_rounds.cc:70] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Trying to abort 1 pending ops.
I20260709 12:25:19.366930 26408 pending_rounds.cc:77] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305625268583964672 op_type: NO_OP noop_request { }
I20260709 12:25:19.367345 26408 raft_consensus.cc:2889] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:19.367692 26408 raft_consensus.cc:2272] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.370388 26408 tablet_replica.cc:333] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.371137 26408 raft_consensus.cc:2243] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.372053 26408 raft_consensus.cc:2272] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.374094 26408 tablet_replica.cc:333] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.374671 26408 raft_consensus.cc:2243] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.375231 26408 raft_consensus.cc:2272] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.377993 26408 tablet_replica.cc:333] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.378649 26408 raft_consensus.cc:2243] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:19.379129  9347 consensus_peers.cc:597] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.379127 26408 raft_consensus.cc:2272] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.382419 26408 tablet_replica.cc:333] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
W20260709 12:25:19.382932  9347 consensus_peers.cc:597] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.383225 26408 raft_consensus.cc:2243] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.384229 26408 raft_consensus.cc:2272] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:19.385547  9421 consensus_peers.cc:597] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.387147 26408 tablet_replica.cc:333] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.387893 26408 raft_consensus.cc:2243] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.388783 26408 raft_consensus.cc:2272] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.391095 26408 tablet_replica.cc:333] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.391741 26408 raft_consensus.cc:2243] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.392211 26408 raft_consensus.cc:2272] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.394351 26408 tablet_replica.cc:333] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.394930 26408 raft_consensus.cc:2243] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.395807 26408 raft_consensus.cc:2272] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.398505 26408 tablet_replica.cc:333] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.399143 26408 raft_consensus.cc:2243] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.399787 26408 raft_consensus.cc:2272] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.402276 26408 tablet_replica.cc:333] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.402915 26408 raft_consensus.cc:2243] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.403501 26408 raft_consensus.cc:2272] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:19.404953  9347 consensus_peers.cc:597] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.406008 26408 tablet_replica.cc:333] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.406723 26408 raft_consensus.cc:2243] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.407810 26408 raft_consensus.cc:2272] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.410233 26408 tablet_replica.cc:333] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.410773 26408 raft_consensus.cc:2243] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.411288 26408 raft_consensus.cc:2272] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.413816 26408 tablet_replica.cc:333] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.414418 26408 raft_consensus.cc:2243] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.414887 26408 raft_consensus.cc:2272] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.417361 26408 tablet_replica.cc:333] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.418094 26408 raft_consensus.cc:2243] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.419287 26408 raft_consensus.cc:2272] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:19.421629  9347 consensus_peers.cc:597] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.422758 26408 tablet_replica.cc:333] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.423635 26408 raft_consensus.cc:2243] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.424585 26408 raft_consensus.cc:2272] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:19.425561  9495 consensus_peers.cc:597] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.427508 26408 tablet_replica.cc:333] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.428234 26408 raft_consensus.cc:2243] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.428715 26408 raft_consensus.cc:2272] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.431216 26408 tablet_replica.cc:333] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.431857 26408 raft_consensus.cc:2243] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.432348 26408 raft_consensus.cc:2272] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.435284 26408 tablet_replica.cc:333] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.436075 26408 raft_consensus.cc:2243] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.437250 26408 raft_consensus.cc:2272] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.440248 26408 tablet_replica.cc:333] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.441010 26408 raft_consensus.cc:2243] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.441570 26408 raft_consensus.cc:2272] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.443701 10051 raft_consensus.cc:493] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.444454 26408 tablet_replica.cc:333] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.444235 10051 raft_consensus.cc:515] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:19.445284 26408 raft_consensus.cc:2243] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.445917 26408 raft_consensus.cc:2272] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.446342 10051 leader_election.cc:290] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:19.447376  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:19.448002  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a6ceb4648564df0b7e690a8e3343b5b" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:19.448377  9382 raft_consensus.cc:2410] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074a6ac642e4430ba8045181f0118ee3 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 12:25:19.449081 26408 tablet_replica.cc:333] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.449785  9420 leader_election.cc:304] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:19.449924 26408 raft_consensus.cc:2243] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.450615 10051 raft_consensus.cc:2749] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:19.451308 26408 raft_consensus.cc:2272] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.454770 26408 tablet_replica.cc:333] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.455636 26408 raft_consensus.cc:2243] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.456043 26408 raft_consensus.cc:2272] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.458313 26408 tablet_replica.cc:333] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.458845 26408 raft_consensus.cc:2243] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.459277 26408 raft_consensus.cc:2272] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.461645 26408 tablet_replica.cc:333] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.462144 26408 raft_consensus.cc:2243] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.462548 26408 raft_consensus.cc:2272] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.464748 26408 tablet_replica.cc:333] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.465242 26408 raft_consensus.cc:2243] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.465643 26408 raft_consensus.cc:2272] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.468478 26408 tablet_replica.cc:333] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.469235 26408 raft_consensus.cc:2243] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.469750 26408 raft_consensus.cc:2272] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.472059 26408 tablet_replica.cc:333] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.472635 26408 raft_consensus.cc:2243] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.473074 26408 raft_consensus.cc:2272] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.475800 26408 tablet_replica.cc:333] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.476464 26408 raft_consensus.cc:2243] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.476994 26408 raft_consensus.cc:2272] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:19.478204  9347 consensus_peers.cc:597] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.480307 26408 tablet_replica.cc:333] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.481145 26408 raft_consensus.cc:2243] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.482203 26408 raft_consensus.cc:2272] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:19.483978  9421 consensus_peers.cc:597] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.485162 26408 tablet_replica.cc:333] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.485769 26408 raft_consensus.cc:2243] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:19.486227 26408 raft_consensus.cc:2272] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.488793 26408 tablet_replica.cc:333] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: stopping tablet replica
I20260709 12:25:19.489336 26408 raft_consensus.cc:2243] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:19.489894  9452 raft_consensus.cc:1275] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:19.490284 26408 raft_consensus.cc:2272] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:19.491395 10015 consensus_queue.cc:1048] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:25:19.526571  9421 consensus_peers.cc:597] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.531317  9382 raft_consensus.cc:1275] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:19.533957 10073 consensus_queue.cc:1048] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:25:19.577538  9421 consensus_peers.cc:597] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.633328  9495 consensus_peers.cc:597] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.633878  9495 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 53138us
W20260709 12:25:19.639436  9421 consensus_peers.cc:597] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.641669  9347 consensus_peers.cc:597] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.644605  9347 connection.cc:573] client connection to 127.25.202.1:33407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260709 12:25:19.664544  9347 consensus_peers.cc:597] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.670567 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
W20260709 12:25:19.682370  9347 consensus_peers.cc:597] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.684311  9495 consensus_peers.cc:597] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.690402  9347 consensus_peers.cc:597] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.703858  9421 consensus_peers.cc:597] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.767503  9495 consensus_peers.cc:597] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:19.788847  9421 consensus_peers.cc:597] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:19.802261 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:19.815186 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:19.815685 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:19.821502 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:19.821759 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:19.825444 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:19.825729 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:19.825973 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599919825950 us; error 0 us; skew 500 ppm
I20260709 12:25:19.826864 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:19.836722 26408 webserver.cc:533] Webserver started at http://127.25.202.1:44039/ using document root <none> and password file <none>
I20260709 12:25:19.837467 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:19.837726 26408 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:25:19.848651 26408 fs_manager.cc:714] Time spent opening directory manager: real 0.009s	user 0.000s	sys 0.008s
I20260709 12:25:19.857254 10088 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:19.858281 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.001s
I20260709 12:25:19.858618 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "4e58af1026634b45bafbbfde02a6ecd0"
format_stamp: "Formatted at 2026-07-09 12:25:08 on dist-test-slave-m57n"
I20260709 12:25:19.858955 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:19.889235 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:19.890961 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:19.899498 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:19.907776 10096 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:25:20.162264 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:25:20.162673 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.257s	user 0.004s	sys 0.005s
I20260709 12:25:20.163017 26408 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:25:20.168054 10096 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.183559 10096 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.184512 10096 tablet_bootstrap.cc:492] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.185114 10096 ts_tablet_manager.cc:1403] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.004s
I20260709 12:25:20.187667 10096 raft_consensus.cc:359] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.188342 10096 raft_consensus.cc:740] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.189016 10096 consensus_queue.cc:260] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.209192 10096 ts_tablet_manager.cc:1434] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.024s	user 0.013s	sys 0.012s
I20260709 12:25:20.210060 10096 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.225283 10096 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.226274 10096 tablet_bootstrap.cc:492] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.226976 10096 ts_tablet_manager.cc:1403] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 12:25:20.229679 10096 raft_consensus.cc:359] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.230461 10096 raft_consensus.cc:740] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.231212 10096 consensus_queue.cc:260] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.233299 10096 ts_tablet_manager.cc:1434] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:25:20.234154 10096 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.256711 10096 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.257851 10096 tablet_bootstrap.cc:492] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.258574 10096 ts_tablet_manager.cc:1403] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.010s
I20260709 12:25:20.261433 10096 raft_consensus.cc:359] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.262200 10096 raft_consensus.cc:740] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.263432 10096 consensus_queue.cc:260] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.265691 10096 ts_tablet_manager.cc:1434] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:25:20.266536 10096 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.288946 10096 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.289965 10096 tablet_bootstrap.cc:492] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.290638 10096 ts_tablet_manager.cc:1403] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.024s	user 0.016s	sys 0.004s
I20260709 12:25:20.293254 10096 raft_consensus.cc:359] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.293980 10096 raft_consensus.cc:740] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.294895 10096 consensus_queue.cc:260] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.297228 10096 ts_tablet_manager.cc:1434] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20260709 12:25:20.298074 10096 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.326596 10096 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.327544 10096 tablet_bootstrap.cc:492] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.328135 10096 ts_tablet_manager.cc:1403] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.030s	user 0.023s	sys 0.003s
I20260709 12:25:20.330000 10096 raft_consensus.cc:359] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.330507 10096 raft_consensus.cc:740] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.331049 10096 consensus_queue.cc:260] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.332423 10096 ts_tablet_manager.cc:1434] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.001s
I20260709 12:25:20.333029 10096 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.345618 10096 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.346768 10096 tablet_bootstrap.cc:492] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.347461 10096 ts_tablet_manager.cc:1403] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.015s	user 0.013s	sys 0.000s
I20260709 12:25:20.349237 10096 raft_consensus.cc:359] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.349761 10096 raft_consensus.cc:740] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.350343 10096 consensus_queue.cc:260] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.352147 10096 ts_tablet_manager.cc:1434] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.004s
I20260709 12:25:20.352936 10096 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.374951 10096 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.375907 10096 tablet_bootstrap.cc:492] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.376561 10096 ts_tablet_manager.cc:1403] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.008s
I20260709 12:25:20.378944 10096 raft_consensus.cc:359] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.379608 10096 raft_consensus.cc:740] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.380308 10096 consensus_queue.cc:260] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.382269 10096 ts_tablet_manager.cc:1434] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:20.383020 10096 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.397636 10096 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.398595 10096 tablet_bootstrap.cc:492] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.399200 10096 ts_tablet_manager.cc:1403] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.000s
I20260709 12:25:20.401722 10096 raft_consensus.cc:359] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.402372 10096 raft_consensus.cc:740] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.403044 10096 consensus_queue.cc:260] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.405077 10096 ts_tablet_manager.cc:1434] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:20.405834 10096 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.420483 10096 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.421465 10096 tablet_bootstrap.cc:492] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.422106 10096 ts_tablet_manager.cc:1403] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.006s
I20260709 12:25:20.424576 10096 raft_consensus.cc:359] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.425206 10096 raft_consensus.cc:740] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.425832 10096 consensus_queue.cc:260] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.427229 10093 raft_consensus.cc:493] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.427706 10096 ts_tablet_manager.cc:1434] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.002s
I20260709 12:25:20.427788 10093 raft_consensus.cc:515] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.428635 10096 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.430550  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
W20260709 12:25:20.434226  9421 leader_election.cc:336] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.434341 10093 leader_election.cc:290] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.434689  9421 leader_election.cc:304] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.435906 10093 raft_consensus.cc:2749] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.443728 10096 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.444692 10096 tablet_bootstrap.cc:492] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.445349 10096 ts_tablet_manager.cc:1403] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.015s	sys 0.000s
I20260709 12:25:20.447942 10096 raft_consensus.cc:359] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.448581 10096 raft_consensus.cc:740] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.449237 10096 consensus_queue.cc:260] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.451109 10096 ts_tablet_manager.cc:1434] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:20.451936 10096 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.454497 10015 raft_consensus.cc:493] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.454993 10015 raft_consensus.cc:515] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.456782 10015 leader_election.cc:290] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.457513  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.458132  9454 raft_consensus.cc:2468] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.459719  9494 leader_election.cc:304] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.460501 10073 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.460979 10073 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:20.462469  9495 leader_election.cc:336] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.464835 10015 raft_consensus.cc:2804] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.465237 10015 raft_consensus.cc:493] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.463996 10073 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.465754  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.465596 10015 raft_consensus.cc:3060] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:20.470208  9495 leader_election.cc:336] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.470633  9495 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:20.471540 10073 raft_consensus.cc:2749] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.474874 10015 raft_consensus.cc:515] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.476707 10096 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.477700 10096 tablet_bootstrap.cc:492] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.477757  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aad56d798d414d5eb49051f39b8ae755" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:20.478389 10096 ts_tablet_manager.cc:1403] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.008s
I20260709 12:25:20.478461  9454 raft_consensus.cc:3060] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.481128 10096 raft_consensus.cc:359] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.481765 10096 raft_consensus.cc:740] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.482548 10096 consensus_queue.cc:260] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:20.484336  9495 leader_election.cc:336] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.484606 10096 ts_tablet_manager.cc:1434] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:25:20.485036 10015 leader_election.cc:290] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.485375 10096 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.489084 26408 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 12:25:20.489432 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.326s	user 0.277s	sys 0.000s
I20260709 12:25:20.489913  9454 raft_consensus.cc:2468] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:20.491256 10093 raft_consensus.cc:493] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.491565  9494 leader_election.cc:304] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.491777 10093 raft_consensus.cc:515] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.493052 10015 raft_consensus.cc:2804] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.494572  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.494624 10015 raft_consensus.cc:697] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:20.495999 10015 consensus_queue.cc:237] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.501266  9698 raft_consensus.cc:493] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.501945  9698 raft_consensus.cc:515] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.504366  9698 leader_election.cc:290] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.507830  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:20.513096  9421 leader_election.cc:336] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.513765  9421 leader_election.cc:304] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.514685 10079 raft_consensus.cc:2749] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.494025 10093 leader_election.cc:290] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:20.528925  9347 leader_election.cc:336] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.529536  9347 leader_election.cc:304] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.534811  9698 raft_consensus.cc:2749] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.581434 10093 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.582348 10093 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.586786 10079 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.587931 10093 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.587610 10079 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.590562 10079 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.591161  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.595180 10073 raft_consensus.cc:493] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.595690 10073 raft_consensus.cc:515] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.597605 10073 leader_election.cc:290] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.600517  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.604331  9200 catalog_manager.cc:5697] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:20.611778  9421 leader_election.cc:336] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.612275  9421 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:20.614228 10079 raft_consensus.cc:2749] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:20.615454  9421 leader_election.cc:336] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.615870  9421 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.616029  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.618341 10079 raft_consensus.cc:2749] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.619082 10079 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.619087 10096 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.619570 10079 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.620389 10096 tablet_bootstrap.cc:492] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.621289 10096 ts_tablet_manager.cc:1403] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.136s	user 0.051s	sys 0.078s
I20260709 12:25:20.621809 10079 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:20.622332  9495 leader_election.cc:336] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.624102 10096 raft_consensus.cc:359] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.624823 10096 raft_consensus.cc:740] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.625010  9980 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.625536  9980 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.625651 10096 consensus_queue.cc:260] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.626317  9494 leader_election.cc:304] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.627360 10116 raft_consensus.cc:2749] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.628494 10116 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.628924  9698 raft_consensus.cc:493] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.629055 10116 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.629444  9698 raft_consensus.cc:515] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.631367  9698 leader_election.cc:290] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.631629 10116 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.632136  9980 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.634110  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.634326 10093 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.635903 10093 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.636487  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.637758 10113 raft_consensus.cc:493] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.639072 10093 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:20.640028  9347 leader_election.cc:336] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.638659 10113 raft_consensus.cc:515] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.640794  9749 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.641511  9749 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.628589 10096 ts_tablet_manager.cc:1434] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:20.643278 10096 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.644847  9749 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:20.644918  9347 leader_election.cc:336] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.648412 10093 raft_consensus.cc:493] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.649241  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.649605  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.648942 10093 raft_consensus.cc:515] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.652194  9346 leader_election.cc:304] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.653553  9346 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.654641  9980 raft_consensus.cc:2749] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.654881  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.642364 10113 leader_election.cc:290] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.655680  9980 raft_consensus.cc:2749] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:20.657186  9347 leader_election.cc:336] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:20.659898  9421 leader_election.cc:336] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.661224  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.662026  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.661996  9526 raft_consensus.cc:2468] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:20.665783 10093 leader_election.cc:290] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.669183  9346 leader_election.cc:304] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.669106  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.670147 10120 raft_consensus.cc:2749] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.660358  9421 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:20.670907  9420 leader_election.cc:304] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
W20260709 12:25:20.671422  9421 leader_election.cc:336] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.671921  9421 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:20.672044  9749 raft_consensus.cc:2749] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.672370 10079 raft_consensus.cc:2804] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.672811 10079 raft_consensus.cc:493] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.672967  9749 raft_consensus.cc:2749] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:20.682793  9421 leader_election.cc:336] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.673189 10079 raft_consensus.cc:3060] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.683291  9421 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.687211 10116 raft_consensus.cc:493] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.687462  9749 raft_consensus.cc:2749] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.687817 10116 raft_consensus.cc:515] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.690030 10116 leader_election.cc:290] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:20.691761  9421 leader_election.cc:336] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.696823  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.697654  9378 raft_consensus.cc:2468] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.703881 10116 raft_consensus.cc:493] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.704728 10116 raft_consensus.cc:515] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.707280 10116 leader_election.cc:290] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.710003 10093 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.710618 10093 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.715597  9495 leader_election.cc:304] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:20.718791 10093 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.719846 10116 raft_consensus.cc:2804] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.720497 10116 raft_consensus.cc:493] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.720979 10116 raft_consensus.cc:3060] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:20.723872  9421 leader_election.cc:336] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.724501 10096 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.724710  9749 raft_consensus.cc:493] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.725519 10096 tablet_bootstrap.cc:492] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.725231  9749 raft_consensus.cc:515] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.726255 10096 ts_tablet_manager.cc:1403] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.083s	user 0.036s	sys 0.000s
I20260709 12:25:20.727429  9749 leader_election.cc:290] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.727625  9847 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.728184  9847 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.729725 10116 raft_consensus.cc:515] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.729878 10096 raft_consensus.cc:359] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.732337 10096 raft_consensus.cc:740] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
W20260709 12:25:20.731786  9495 leader_election.cc:336] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.732971  9495 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
W20260709 12:25:20.733207  9421 leader_election.cc:336] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.733062 10096 consensus_queue.cc:260] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.734362 10115 raft_consensus.cc:2749] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.734658  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.735373  9454 raft_consensus.cc:2468] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:20.736320 10079 raft_consensus.cc:515] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.740752  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.736617 10096 ts_tablet_manager.cc:1434] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:25:20.736721  9346 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:20.739092 10116 leader_election.cc:290] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.739447  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.742609 10096 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.742851  9378 raft_consensus.cc:2468] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:20.744748  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.745383  9378 raft_consensus.cc:2468] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
W20260709 12:25:20.747052  9495 leader_election.cc:336] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.748368 10120 raft_consensus.cc:2804] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.747681 10079 leader_election.cc:290] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.751212  9980 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.751873  9421 leader_election.cc:304] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.751771  9980 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.753324  9749 raft_consensus.cc:2804] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.753719  9749 raft_consensus.cc:493] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.754091  9749 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.754295  9980 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.751912  9420 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:20.755388 10113 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.756346 10113 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.758440 10113 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.760059  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58056f74cd464091a7f28a4a999ce936" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:20.760845  9378 raft_consensus.cc:3060] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.761252  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:20.762522  9421 leader_election.cc:336] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.762845  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.763610  9452 raft_consensus.cc:2468] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:20.748775 10120 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.766635 10120 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.767369  9749 raft_consensus.cc:515] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.770243  9378 raft_consensus.cc:2468] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:20.756328 10079 raft_consensus.cc:2749] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:20.766723  9847 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.773217 10116 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.774139 10116 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:20.774683  9421 leader_election.cc:336] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:20.774828  9347 leader_election.cc:336] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.774518 10120 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.773192  9346 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:20.766561 10096 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:20.776441  9749 leader_election.cc:290] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.776648 10116 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.779762  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9515a897d574915ba4d8a478406aa1a" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:20.776880 10096 tablet_bootstrap.cc:492] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:20.780301  9526 raft_consensus.cc:3060] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.781127 10096 ts_tablet_manager.cc:1403] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.008s
I20260709 12:25:20.777380  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.783381  9454 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.777899  9980 raft_consensus.cc:2804] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.784024  9980 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.784477  9980 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.784520  9494 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.785485 10116 raft_consensus.cc:2804] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.786309 10116 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.786711 10116 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:20.778662  9495 leader_election.cc:336] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:20.787644  9495 leader_election.cc:336] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.792668  9526 raft_consensus.cc:2468] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:20.793185 10120 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.794646 10116 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.796592  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:20.797174  9454 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.797277 10116 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.799049  9749 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.799687  9749 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.800987  9980 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.801950  9749 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.802620  9495 leader_election.cc:304] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.805372  9847 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.805923  9847 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.806980 10113 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.807556 10113 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:20.807857  9421 leader_election.cc:336] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.805650  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:20.810266  9452 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.784483 10096 raft_consensus.cc:359] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.812448 10096 raft_consensus.cc:740] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:20.813706 10096 consensus_queue.cc:260] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.816057  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:20.816687  9378 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.818295 10096 ts_tablet_manager.cc:1434] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.001s
I20260709 12:25:20.819357 10096 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:20.821964 10115 raft_consensus.cc:493] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.822496 10115 raft_consensus.cc:515] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.823020  9749 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.823586  9749 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.824817 10115 leader_election.cc:290] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.825773  9749 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.829305  9980 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.831214  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.831614  9378 raft_consensus.cc:2468] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:20.831997  9450 raft_consensus.cc:2468] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.833630  9494 leader_election.cc:304] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.834229  9698 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.834766  9698 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.836158 10115 raft_consensus.cc:2804] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.836956  9618 raft_consensus.cc:2804] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.837572  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:20.837745 10115 raft_consensus.cc:493] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.838341  9450 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.838435 10115 raft_consensus.cc:3060] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.839861  9847 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:20.837592  9618 raft_consensus.cc:493] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.840862  9618 raft_consensus.cc:3060] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.837031  9698 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.845494 10073 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
W20260709 12:25:20.846625  9421 leader_election.cc:336] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.846287 10073 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.848049 10113 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.849637 10073 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.849785 10115 raft_consensus.cc:515] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:20.850747  9495 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.2:42279, user_credentials={real_user=slave}} blocked reactor thread for 48208.3us
I20260709 12:25:20.851590  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.853644  9421 leader_election.cc:304] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.856969  9453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.864085  9453 raft_consensus.cc:2468] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.864513  9495 leader_election.cc:304] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.857198  9452 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:20.858876  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.865823 10116 raft_consensus.cc:2804] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.866005  9525 raft_consensus.cc:2468] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:20.866396 10116 raft_consensus.cc:697] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:20.867489 10116 consensus_queue.cc:237] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:20.860368  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:20.869937  9526 raft_consensus.cc:2468] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
I20260709 12:25:20.875396  9420 leader_election.cc:304] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.876534 10079 raft_consensus.cc:2804] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.876273  9618 raft_consensus.cc:515] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.877101 10079 raft_consensus.cc:697] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:20.877952 10079 consensus_queue.cc:237] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.881019  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.882047  9452 raft_consensus.cc:2468] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 1.
W20260709 12:25:20.862555  9347 leader_election.cc:336] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.864092 10093 raft_consensus.cc:2804] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.885370 10093 raft_consensus.cc:697] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:20.886571 10093 consensus_queue.cc:237] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.864153  9382 raft_consensus.cc:2468] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:20.896944  9200 catalog_manager.cc:5697] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:20.899312 10073 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.900120 10073 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.903083 10073 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.858637  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:20.859896  9454 raft_consensus.cc:2468] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 2.
I20260709 12:25:20.868947  9494 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.916574 10115 leader_election.cc:290] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:20.917229 10015 raft_consensus.cc:2804] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:25:20.885504  9347 leader_election.cc:336] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.917729 10015 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.918079  9347 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07a645c9ee14f11a5d64fa7b460acf9; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.918215 10015 raft_consensus.cc:3060] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.918926 10120 raft_consensus.cc:2749] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:20.926606  9495 leader_election.cc:336] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.932075  9346 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
W20260709 12:25:20.933249  9494 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 64421.3us
I20260709 12:25:20.933724 10120 raft_consensus.cc:2804] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.935642 10120 raft_consensus.cc:493] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.936075 10120 raft_consensus.cc:3060] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.937469  9421 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:20.938628 10093 raft_consensus.cc:2804] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.939013 10093 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.939407 10093 raft_consensus.cc:3060] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.939771  9494 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.934800  9346 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:20.942304 10073 raft_consensus.cc:2804] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.942852  9450 raft_consensus.cc:2468] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 2.
I20260709 12:25:20.943778  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85baaa21d67f43da9f328b0b51a71215" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:20.944497  9454 raft_consensus.cc:3060] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.879076  9618 leader_election.cc:290] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.942871 10073 raft_consensus.cc:697] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:20.949245 10073 consensus_queue.cc:237] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.952790  9454 raft_consensus.cc:2468] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:20.955078 10015 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.957891 10015 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.958081 10093 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.946257  9346 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:20.946429  9980 raft_consensus.cc:2804] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.946602  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a891dab57c74222928537e93876be86" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:20.961870  9380 raft_consensus.cc:3060] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.963297  9980 raft_consensus.cc:697] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
W20260709 12:25:20.944901  9347 leader_election.cc:336] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.964427  9378 raft_consensus.cc:2468] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 1.
I20260709 12:25:20.961998 10136 raft_consensus.cc:2804] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:25:20.965231  9347 leader_election.cc:336] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.965605 10136 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
W20260709 12:25:20.966161  9347 leader_election.cc:336] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.966192 10136 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.967875 10015 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.968287  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:20.968487 10015 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.968954  9452 raft_consensus.cc:3060] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.969831  9494 leader_election.cc:304] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.970501 10093 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:20.975822  9421 leader_election.cc:336] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:20.977689  9200 catalog_manager.cc:5697] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:20.979125  9980 consensus_queue.cc:237] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:20.983415  9452 raft_consensus.cc:2468] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:20.971702  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.985471  9454 raft_consensus.cc:2468] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.987386  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:20.988158  9450 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 1.
I20260709 12:25:20.984079 10015 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:20.971040 10116 raft_consensus.cc:2804] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.990389 10116 raft_consensus.cc:697] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:20.985383  9346 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:20.991878  9698 raft_consensus.cc:2804] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:20.992282  9698 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:20.992635  9698 raft_consensus.cc:3060] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.991456 10116 consensus_queue.cc:237] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.987120  9494 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:20.996138  9618 raft_consensus.cc:2804] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:20.996680  9618 raft_consensus.cc:697] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:20.996912  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:20.997670  9377 raft_consensus.cc:3060] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:20.997776  9618 consensus_queue.cc:237] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:20.999269 10120 raft_consensus.cc:515] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
W20260709 12:25:21.001938  9495 leader_election.cc:336] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.007313  9380 raft_consensus.cc:2468] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:21.021145  9421 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.022100 10079 raft_consensus.cc:2804] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:21.022480 10079 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:21.023065 10079 raft_consensus.cc:3060] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.017795  9377 raft_consensus.cc:2468] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
W20260709 12:25:21.030634  9347 leader_election.cc:336] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.031692  9347 leader_election.cc:336] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.035622  9421 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.036520  9749 raft_consensus.cc:2804] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.037088  9749 raft_consensus.cc:697] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:21.036944 10079 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.038070  9749 consensus_queue.cc:237] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.032859  9494 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
W20260709 12:25:21.048938  9347 leader_election.cc:336] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.012650 10120 leader_election.cc:290] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:21.049220 10116 raft_consensus.cc:2804] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:21.050495 10116 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:21.050979 10116 raft_consensus.cc:3060] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:21.055505  9495 leader_election.cc:336] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.049258  9494 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:21.057581 10073 raft_consensus.cc:2804] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:21.058055 10073 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.058416 10073 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.010392  9698 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.061940  9200 catalog_manager.cc:5697] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.063409  9698 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:21.065559  9346 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.067001  9847 raft_consensus.cc:2804] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.067554  9847 raft_consensus.cc:697] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:21.068421  9847 consensus_queue.cc:237] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.071578  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf66e0c0d4ec4b10989c1f0abbbb8eb8" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:21.072322  9450 raft_consensus.cc:3060] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.072378 10073 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.075579 10073 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:21.080657  9203 catalog_manager.cc:5697] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:21.088361 10136 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.096699 10096 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:21.098233 10096 tablet_bootstrap.cc:492] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
W20260709 12:25:21.098820  9495 leader_election.cc:336] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.100023  9347 leader_election.cc:336] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.101857  9450 raft_consensus.cc:2468] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 2.
W20260709 12:25:21.100147  9495 leader_election.cc:336] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.114859  9495 leader_election.cc:336] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.118554  9495 leader_election.cc:336] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.122098  9347 leader_election.cc:336] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.112986 10136 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:21.113565 10096 ts_tablet_manager.cc:1403] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.294s	user 0.018s	sys 0.000s
W20260709 12:25:21.113755  9421 leader_election.cc:336] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.121913  9346 leader_election.cc:304] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.104373  9200 catalog_manager.cc:5697] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.127383 10096 raft_consensus.cc:359] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.128140 10096 raft_consensus.cc:740] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:21.128859 10113 raft_consensus.cc:2804] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.131594 10113 raft_consensus.cc:697] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:21.131184 10116 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.132560 10113 consensus_queue.cc:237] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.133553 10116 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:21.149184  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:21.151070  9526 raft_consensus.cc:3060] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.138000 10096 consensus_queue.cc:260] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.150472  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:21.152638  9527 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.153390  9452 raft_consensus.cc:1275] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.142139 10079 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:21.150804  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:21.155716  9450 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.144685  9495 leader_election.cc:304] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.157176 10116 raft_consensus.cc:2804] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.159962 10116 raft_consensus.cc:697] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:21.161396 10116 consensus_queue.cc:237] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.169082  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:21.169801  9380 raft_consensus.cc:3060] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:21.171881  9452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:21.172734  9452 raft_consensus.cc:3060] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:21.176067  9495 leader_election.cc:336] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.177065  9495 leader_election.cc:336] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.183344  9452 raft_consensus.cc:2468] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:21.187103  9450 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:21.188570  9527 raft_consensus.cc:2468] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 2.
I20260709 12:25:21.189327 10096 ts_tablet_manager.cc:1434] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.000s
I20260709 12:25:21.190145  9380 raft_consensus.cc:2468] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:21.192500  9526 raft_consensus.cc:2468] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07a645c9ee14f11a5d64fa7b460acf9 in term 2.
I20260709 12:25:21.192505 10096 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:21.194792  9421 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.196201 10079 raft_consensus.cc:2804] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.196733 10079 raft_consensus.cc:697] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:21.197132 10015 consensus_queue.cc:1048] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.197712 10079 consensus_queue.cc:237] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.200476  9494 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
W20260709 12:25:21.207636  9495 leader_election.cc:336] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:21.210569  9495 consensus_peers.cc:597] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.211050  9495 leader_election.cc:336] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:21.211913 10015 raft_consensus.cc:2804] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.211189  9200 catalog_manager.cc:5697] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.212639 10015 raft_consensus.cc:697] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:21.212610  9494 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.213863 10015 consensus_queue.cc:237] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.215128  9200 catalog_manager.cc:5697] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.220844 10116 raft_consensus.cc:2804] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.221377 10116 raft_consensus.cc:697] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:21.222424 10116 consensus_queue.cc:237] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.232908  9203 catalog_manager.cc:5697] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:21.242626 10096 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:21.245882 10096 tablet_bootstrap.cc:492] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:21.251341  9346 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.252679  9346 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:21.254076 10137 raft_consensus.cc:2804] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.254617 10137 raft_consensus.cc:697] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:21.255577  9980 raft_consensus.cc:2804] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:21.255575 10137 consensus_queue.cc:237] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.260028  9980 raft_consensus.cc:697] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Leader. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:21.246831 10096 ts_tablet_manager.cc:1403] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.055s	user 0.016s	sys 0.004s
I20260709 12:25:21.267805  9980 consensus_queue.cc:237] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.271451 10096 raft_consensus.cc:359] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.274597 10096 raft_consensus.cc:740] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:21.275539 10096 consensus_queue.cc:260] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.277920 10096 ts_tablet_manager.cc:1434] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:25:21.279047 10096 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:21.276823  9201 catalog_manager.cc:5697] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.281155  9380 raft_consensus.cc:1275] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.285363 10093 consensus_queue.cc:1048] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.289412  9201 catalog_manager.cc:5697] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.295928  9201 catalog_manager.cc:5697] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.337023  9524 raft_consensus.cc:1275] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.345175  9201 catalog_manager.cc:5697] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.385680  9201 catalog_manager.cc:5697] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.387840  9201 catalog_manager.cc:5697] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:21.392275 10096 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:21.394560 10096 tablet_bootstrap.cc:492] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:21.395289 10096 ts_tablet_manager.cc:1403] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.117s	user 0.030s	sys 0.028s
I20260709 12:25:21.398775 10096 raft_consensus.cc:359] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.399578 10096 raft_consensus.cc:740] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:21.399708  9200 catalog_manager.cc:5697] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:21.400530 10096 consensus_queue.cc:260] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.409858  9379 raft_consensus.cc:1275] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.408100 10093 consensus_queue.cc:1048] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 12:25:21.401796  9200 catalog_manager.cc:5697] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 reported cstate change: term changed from 1 to 2, leader changed from 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1) to e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:21.428522 10116 consensus_queue.cc:1048] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:25:21.431631  9495 consensus_peers.cc:597] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.433467  9421 consensus_peers.cc:597] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.436900  9495 consensus_peers.cc:597] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:21.403312 10096 ts_tablet_manager.cc:1434] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.001s
I20260709 12:25:21.445729 10096 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
W20260709 12:25:21.467057  9495 consensus_peers.cc:597] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.478438  9421 consensus_peers.cc:597] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:21.492511  9452 raft_consensus.cc:1275] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:25:21.494710  9495 consensus_peers.cc:597] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:21.495934 10116 consensus_queue.cc:1048] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.551551  9379 raft_consensus.cc:1275] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.557158 10098 raft_consensus.cc:493] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.558066 10098 raft_consensus.cc:515] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.560120 10015 consensus_queue.cc:1048] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:21.574116 10193 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:21.576721  9495 consensus_peers.cc:597] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:21.574822 10193 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.579027 10098 leader_election.cc:290] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
W20260709 12:25:21.589629 10186 log.cc:927] Time spent T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:25:21.597035 10193 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:21.604753  9450 raft_consensus.cc:1275] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.609103 10073 consensus_queue.cc:1048] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:21.609665  9454 raft_consensus.cc:1275] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:25:21.611228  9453 raft_consensus.cc:1275] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.618984 10015 consensus_queue.cc:1048] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:21.621389 10015 consensus_queue.cc:1048] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.675189  9450 raft_consensus.cc:1275] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.690557 10116 consensus_queue.cc:1048] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:21.707396  9347 consensus_peers.cc:597] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.723687  9346 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:39369, user_credentials={real_user=slave}} blocked reactor thread for 131796us
I20260709 12:25:21.726639  9453 raft_consensus.cc:1275] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer e07a645c9ee14f11a5d64fa7b460acf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.733698  9379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:21.735780  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:21.751350 10096 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:21.752472 10096 tablet_bootstrap.cc:492] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:21.753178 10096 ts_tablet_manager.cc:1403] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.308s	user 0.098s	sys 0.136s
I20260709 12:25:21.759784 10096 raft_consensus.cc:359] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.760622 10096 raft_consensus.cc:740] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
W20260709 12:25:21.761245  9495 consensus_peers.cc:597] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:21.761534 10096 consensus_queue.cc:260] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.764344 10096 ts_tablet_manager.cc:1434] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:25:21.765535 10193 raft_consensus.cc:493] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.767150  9453 raft_consensus.cc:1275] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer e07a645c9ee14f11a5d64fa7b460acf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.766902 10193 raft_consensus.cc:515] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.769937 10193 leader_election.cc:290] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:21.771634 10096 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:21.782618 10193 raft_consensus.cc:493] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.783270 10193 raft_consensus.cc:515] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.787185  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:21.795536 10193 leader_election.cc:290] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:21.811685 10096 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:21.812798 10096 tablet_bootstrap.cc:492] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:21.813628 10096 ts_tablet_manager.cc:1403] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.042s	user 0.025s	sys 0.000s
I20260709 12:25:21.817454 10096 raft_consensus.cc:359] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.818217 10096 raft_consensus.cc:740] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:21.818018  9524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:21.818717  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:21.819164 10096 consensus_queue.cc:260] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.820477  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
W20260709 12:25:21.821259  9495 consensus_peers.cc:597] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:21.821830 10096 ts_tablet_manager.cc:1434] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:25:21.822301 10090 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:21.822781 10096 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:21.824626 10193 raft_consensus.cc:2749] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:21.825999 10090 leader_election.cc:304] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:21.827389 10193 raft_consensus.cc:2749] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:21.833045  9527 raft_consensus.cc:1275] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer e07a645c9ee14f11a5d64fa7b460acf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:25:21.842350  9347 consensus_peers.cc:597] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.848907  9495 consensus_peers.cc:597] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:21.849757 10193 raft_consensus.cc:493] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.850497 10193 raft_consensus.cc:515] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:21.854568 10098 raft_consensus.cc:493] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.855720  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:21.855443 10098 raft_consensus.cc:515] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.862432 10193 leader_election.cc:290] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:21.862758  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b494d68a603a4ba5a28ee8514cfa3f41" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:21.864905 10098 leader_election.cc:290] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:21.868069  9527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "890c806dd6f94382ba4e14a3d3bc6f85" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:21.871728 10090 leader_election.cc:304] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:21.872713 10098 raft_consensus.cc:2749] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:21.875473 10098 raft_consensus.cc:493] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.876164 10098 raft_consensus.cc:515] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.880679 10098 leader_election.cc:290] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:21.881877  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:21.883886  9421 consensus_peers.cc:597] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.884536  9421 consensus_peers.cc:597] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:21.901763 10098 raft_consensus.cc:493] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.902628 10098 raft_consensus.cc:515] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:21.908520 10098 leader_election.cc:290] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:21.911646  9380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:21.915263  9378 raft_consensus.cc:1275] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.912599  9380 raft_consensus.cc:2393] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e58af1026634b45bafbbfde02a6ecd0 in current term 2: Already voted for candidate 074a6ac642e4430ba8045181f0118ee3 in this term.
I20260709 12:25:21.929497  9524 raft_consensus.cc:1275] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer e07a645c9ee14f11a5d64fa7b460acf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.943696  9382 raft_consensus.cc:1275] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:25:21.948009  9347 consensus_peers.cc:597] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:21.948549  9347 consensus_peers.cc:597] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.949005  9347 consensus_peers.cc:597] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:21.951875  9421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.1:33407, user_credentials={real_user=slave}} blocked reactor thread for 66951us
I20260709 12:25:21.955752 10136 consensus_queue.cc:1048] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.968566  9980 consensus_queue.cc:1048] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.971117  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1976a69f7f174bce837050f43e2dd6fe" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:21.973240  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fabd519bb34fbaae2738d3cfead584" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:21.975111 10120 consensus_queue.cc:1048] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:21.977526 10120 consensus_queue.cc:1048] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:21.981920  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b494d68a603a4ba5a28ee8514cfa3f41" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:21.984227  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:21.988945 10096 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:21.990121 10096 tablet_bootstrap.cc:492] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:21.990890 10096 ts_tablet_manager.cc:1403] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.168s	user 0.064s	sys 0.079s
I20260709 12:25:21.993585 10096 raft_consensus.cc:359] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:21.994297 10096 raft_consensus.cc:740] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:21.997786  9454 raft_consensus.cc:1275] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer e07a645c9ee14f11a5d64fa7b460acf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:21.995373 10096 consensus_queue.cc:260] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.003492 10096 ts_tablet_manager.cc:1434] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:25:22.004536 10096 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.005496  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a53d66736d1046bea06b3935c396942d" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
W20260709 12:25:22.007617  9421 consensus_peers.cc:597] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:22.010084 10098 raft_consensus.cc:493] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.010646 10098 raft_consensus.cc:515] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.013007 10098 leader_election.cc:290] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:22.014896  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
W20260709 12:25:22.020475  9495 consensus_peers.cc:597] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.026674  9347 consensus_peers.cc:597] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.028362 10090 leader_election.cc:304] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:22.029695 10090 leader_election.cc:304] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.030843 10090 leader_election.cc:304] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.031471 10193 raft_consensus.cc:2749] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.032183 10193 raft_consensus.cc:2749] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.033121 10079 consensus_queue.cc:1048] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:22.033479  9495 consensus_peers.cc:597] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.033874 10098 raft_consensus.cc:2749] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:22.034082  9347 consensus_peers.cc:597] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.034608  9347 consensus_peers.cc:597] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.035027  9347 consensus_peers.cc:597] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:22.034917 10090 leader_election.cc:304] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.035765 10098 raft_consensus.cc:2749] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.037271 10090 leader_election.cc:304] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.038247 10098 raft_consensus.cc:3060] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.039371 10079 consensus_queue.cc:1048] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:22.053488 10098 raft_consensus.cc:2749] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.056530  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68ff98ec6813440cae57fcf054f25732" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:22.057206 10137 consensus_queue.cc:1048] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:22.060109  9421 consensus_peers.cc:597] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.065598  9347 consensus_peers.cc:597] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.065743 10090 leader_election.cc:304] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.068081 10193 raft_consensus.cc:2749] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:22.066111  9347 consensus_peers.cc:597] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.070680  9347 consensus_peers.cc:597] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.071270  9347 consensus_peers.cc:597] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.071823  9347 consensus_peers.cc:597] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.072655  9347 consensus_peers.cc:597] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.073186  9347 consensus_peers.cc:597] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.088078 10098 raft_consensus.cc:493] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.088791 10098 raft_consensus.cc:515] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.091956 10096 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.092382  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:22.092979 10096 tablet_bootstrap.cc:492] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.093868 10096 ts_tablet_manager.cc:1403] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.090s	user 0.016s	sys 0.000s
I20260709 12:25:22.095432 10098 leader_election.cc:290] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:22.097882 10096 raft_consensus.cc:359] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.098618 10096 raft_consensus.cc:740] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.099377 10096 consensus_queue.cc:260] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.101653 10096 ts_tablet_manager.cc:1434] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:25:22.102466 10096 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.104339  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a5f846566064f6f932c156f0f9c9d7a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:22.111258 10090 leader_election.cc:304] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.112689 10098 raft_consensus.cc:2749] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:25:22.112884  9495 consensus_peers.cc:597] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:22.113929  9421 consensus_peers.cc:597] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.114714 10098 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.115267 10098 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
W20260709 12:25:22.116073  9495 consensus_peers.cc:597] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.119578 10098 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
W20260709 12:25:22.121709  9347 consensus_peers.cc:597] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.122143  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:22.129647  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:22.132174 10090 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:22.139636 10193 raft_consensus.cc:493] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.147789 10193 raft_consensus.cc:515] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.141351 10098 raft_consensus.cc:2749] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.161948 10193 leader_election.cc:290] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:22.164892  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:22.168133  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:22.169826 10091 leader_election.cc:304] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.171212 10098 raft_consensus.cc:2749] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.182905 10098 raft_consensus.cc:493] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.183635 10098 raft_consensus.cc:515] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.187784 10098 leader_election.cc:290] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:22.191215  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "074a6ac642e4430ba8045181f0118ee3" is_pre_election: true
I20260709 12:25:22.207798  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30315f436f8c4ef4b1ac5381d6a34811" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:22.214820 10090 leader_election.cc:304] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:22.216137 10098 raft_consensus.cc:2749] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.234218 10096 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.235466 10096 tablet_bootstrap.cc:492] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.236186 10096 ts_tablet_manager.cc:1403] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.134s	user 0.023s	sys 0.029s
I20260709 12:25:22.239771 10096 raft_consensus.cc:359] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.240597 10096 raft_consensus.cc:740] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
W20260709 12:25:22.241474  9421 consensus_peers.cc:597] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.241545 10096 consensus_queue.cc:260] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.243151 10234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:33407 every 8 connection(s)
I20260709 12:25:22.244105 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:33407
I20260709 12:25:22.244755 10096 ts_tablet_manager.cc:1434] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:25:22.245764 10096 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.333971 10236 heartbeater.cc:344] Connected to a master server at 127.25.202.62:39287
I20260709 12:25:22.334491 10236 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:22.335677 10096 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.335862 10236 heartbeater.cc:507] Master 127.25.202.62:39287 requested a full tablet report, sending...
I20260709 12:25:22.336862 10096 tablet_bootstrap.cc:492] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.337625 10096 ts_tablet_manager.cc:1403] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.092s	user 0.032s	sys 0.014s
I20260709 12:25:22.338531 10179 raft_consensus.cc:3060] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.340551 10096 raft_consensus.cc:359] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.341787 10096 raft_consensus.cc:740] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.343125 10096 consensus_queue.cc:260] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.344759 10096 ts_tablet_manager.cc:1434] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:22.345655 10096 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.351959 10176 raft_consensus.cc:3060] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:22.364094  9347 consensus_peers.cc:597] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.382313 10096 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.383530 10096 tablet_bootstrap.cc:492] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.384267 10096 ts_tablet_manager.cc:1403] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.039s	user 0.023s	sys 0.007s
I20260709 12:25:22.387027 10096 raft_consensus.cc:359] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.387780 10096 raft_consensus.cc:740] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.388526 10096 consensus_queue.cc:260] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.390913 10096 ts_tablet_manager.cc:1434] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:25:22.391856 10096 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.401187  9201 ts_manager.cc:194] Re-registered known tserver with Master: 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:22.409179  9421 consensus_peers.cc:597] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.417992 10096 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.419486 10096 tablet_bootstrap.cc:492] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.420207 10096 ts_tablet_manager.cc:1403] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.029s	user 0.022s	sys 0.004s
I20260709 12:25:22.422410 10171 raft_consensus.cc:3060] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.422935 10096 raft_consensus.cc:359] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.423731 10096 raft_consensus.cc:740] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.424448 10096 consensus_queue.cc:260] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.426900 10096 ts_tablet_manager.cc:1434] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:25:22.428087 10096 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.433995 10179 raft_consensus.cc:3060] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:22.442355  9347 consensus_peers.cc:597] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.446913 10096 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.448524 10096 tablet_bootstrap.cc:492] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.449870 10096 ts_tablet_manager.cc:1403] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.005s
I20260709 12:25:22.453692 10096 raft_consensus.cc:359] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.454758 10096 raft_consensus.cc:740] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.455581 10096 consensus_queue.cc:260] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.458364 10096 ts_tablet_manager.cc:1434] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:25:22.461241  9201 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60780
I20260709 12:25:22.465320 10096 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.469262 10236 heartbeater.cc:499] Master 127.25.202.62:39287 was elected leader, sending a full tablet report...
I20260709 12:25:22.479064 10096 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.480015 10096 tablet_bootstrap.cc:492] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.480666 10096 ts_tablet_manager.cc:1403] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.001s
I20260709 12:25:22.482990 10096 raft_consensus.cc:359] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.483659 10096 raft_consensus.cc:740] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.484476 10096 consensus_queue.cc:260] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
W20260709 12:25:22.486621  9495 consensus_peers.cc:597] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:22.486656 10096 ts_tablet_manager.cc:1434] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:25:22.487721 10096 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.503314 10096 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.504210 10096 tablet_bootstrap.cc:492] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.504817 10096 ts_tablet_manager.cc:1403] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.015s	sys 0.000s
I20260709 12:25:22.506825 10096 raft_consensus.cc:359] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.507546 10096 raft_consensus.cc:740] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.508241 10096 consensus_queue.cc:260] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.510170 10096 ts_tablet_manager.cc:1434] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:25:22.510948 10096 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.524955 10096 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.525918 10096 tablet_bootstrap.cc:492] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.526566 10096 ts_tablet_manager.cc:1403] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.000s
I20260709 12:25:22.529102 10096 raft_consensus.cc:359] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.529623 10096 raft_consensus.cc:740] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.530205 10096 consensus_queue.cc:260] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.531848 10096 ts_tablet_manager.cc:1434] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:25:22.532639 10096 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.546480 10096 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.547433 10096 tablet_bootstrap.cc:492] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.548015 10096 ts_tablet_manager.cc:1403] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.002s
I20260709 12:25:22.550204 10096 raft_consensus.cc:359] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.550740 10096 raft_consensus.cc:740] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.551354 10096 consensus_queue.cc:260] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.552656 10171 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.553350 10096 ts_tablet_manager.cc:1434] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20260709 12:25:22.554845 10096 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.577538 10242 raft_consensus.cc:493] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.578126 10242 raft_consensus.cc:515] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.581704  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" is_pre_election: true
I20260709 12:25:22.584050  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cecdb3fe70f46d8af803d3d7db78976" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
I20260709 12:25:22.586201 10090 leader_election.cc:304] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0; no voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:22.587572 10098 raft_consensus.cc:2749] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:22.589079 10096 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.590756 10096 tablet_bootstrap.cc:492] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.590782 10242 leader_election.cc:290] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:22.591490 10096 ts_tablet_manager.cc:1403] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.037s	user 0.018s	sys 0.003s
I20260709 12:25:22.594123 10171 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.594766 10096 raft_consensus.cc:359] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.595589 10096 raft_consensus.cc:740] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.596350 10096 consensus_queue.cc:260] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.604943 10096 ts_tablet_manager.cc:1434] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.003s
I20260709 12:25:22.605849 10096 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.629331 10096 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.630348 10096 tablet_bootstrap.cc:492] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.631018 10096 ts_tablet_manager.cc:1403] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.009s
I20260709 12:25:22.633447 10096 raft_consensus.cc:359] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.634130 10096 raft_consensus.cc:740] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.634856 10096 consensus_queue.cc:260] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.636957 10096 ts_tablet_manager.cc:1434] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:25:22.637996 10096 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.658535 10171 raft_consensus.cc:3060] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.664769 10179 raft_consensus.cc:3060] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.670943 10096 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.672041 10096 tablet_bootstrap.cc:492] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.672775 10096 ts_tablet_manager.cc:1403] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.035s	user 0.016s	sys 0.004s
I20260709 12:25:22.674978 10096 raft_consensus.cc:359] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.675683 10096 raft_consensus.cc:740] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.676409 10096 consensus_queue.cc:260] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.678053 10096 ts_tablet_manager.cc:1434] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:22.678843 10096 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.693938 10179 raft_consensus.cc:3060] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.694180 10096 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.695353 10096 tablet_bootstrap.cc:492] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.696039 10096 ts_tablet_manager.cc:1403] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.004s
I20260709 12:25:22.698577 10096 raft_consensus.cc:359] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.699249 10096 raft_consensus.cc:740] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.700136 10096 consensus_queue.cc:260] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.702358 10096 ts_tablet_manager.cc:1434] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:25:22.703261 10096 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.718508 10096 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.720263 10096 tablet_bootstrap.cc:492] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.721370 10096 ts_tablet_manager.cc:1403] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.000s
I20260709 12:25:22.722847 10179 raft_consensus.cc:3060] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.724306 10096 raft_consensus.cc:359] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.725154 10096 raft_consensus.cc:740] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.725909 10096 consensus_queue.cc:260] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.728238 10096 ts_tablet_manager.cc:1434] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:22.729097 10096 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.751668 10171 raft_consensus.cc:3060] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.774142 10096 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.775157 10096 tablet_bootstrap.cc:492] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.775979 10096 ts_tablet_manager.cc:1403] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.047s	user 0.023s	sys 0.009s
I20260709 12:25:22.778654 10096 raft_consensus.cc:359] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.779309 10096 raft_consensus.cc:740] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.779969 10096 consensus_queue.cc:260] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:22.782033 10096 ts_tablet_manager.cc:1434] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:25:22.782909 10096 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.802412 10096 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.803469 10096 tablet_bootstrap.cc:492] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.804157 10096 ts_tablet_manager.cc:1403] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 12:25:22.808414 10096 raft_consensus.cc:359] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.809139 10096 raft_consensus.cc:740] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.809794 10096 consensus_queue.cc:260] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.812634 10096 ts_tablet_manager.cc:1434] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:25:22.813723 10096 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.824949 10176 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.843019 10096 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:25:22.844259 10096 tablet_bootstrap.cc:492] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.845039 10096 ts_tablet_manager.cc:1403] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.032s	user 0.013s	sys 0.012s
I20260709 12:25:22.847721 10096 raft_consensus.cc:359] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.848555 10096 raft_consensus.cc:740] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.849261 10096 consensus_queue.cc:260] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.851402 10096 ts_tablet_manager.cc:1434] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:25:22.852177 10096 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.872159 10096 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.873175 10096 tablet_bootstrap.cc:492] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.873864 10096 ts_tablet_manager.cc:1403] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.004s
I20260709 12:25:22.876402 10096 raft_consensus.cc:359] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.877604 10096 raft_consensus.cc:740] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.878472 10096 consensus_queue.cc:260] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.880110 10096 ts_tablet_manager.cc:1434] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:25:22.880980 10096 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.910027 10179 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.912889 10096 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.914037 10096 tablet_bootstrap.cc:492] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.914716 10096 ts_tablet_manager.cc:1403] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.012s
I20260709 12:25:22.917551 10096 raft_consensus.cc:359] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.918221 10096 raft_consensus.cc:740] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.919036 10096 consensus_queue.cc:260] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:22.921150 10096 ts_tablet_manager.cc:1434] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:25:22.923020 10096 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:22.945201 10176 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:22.976526 10096 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:22.977401 10096 tablet_bootstrap.cc:492] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:22.978034 10096 ts_tablet_manager.cc:1403] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.055s	user 0.016s	sys 0.007s
I20260709 12:25:22.980558 10096 raft_consensus.cc:359] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:22.981107 10096 raft_consensus.cc:740] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:22.981756 10096 consensus_queue.cc:260] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:23.000023 10179 raft_consensus.cc:3060] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:23.000641 10096 ts_tablet_manager.cc:1434] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:25:23.001621 10096 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:23.016769 10096 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:23.018168 10096 tablet_bootstrap.cc:492] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:23.018816 10096 ts_tablet_manager.cc:1403] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.000s
I20260709 12:25:23.021763 10096 raft_consensus.cc:359] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:23.022294 10096 raft_consensus.cc:740] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:23.022949 10096 consensus_queue.cc:260] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:23.025053 10096 ts_tablet_manager.cc:1434] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:25:23.025805 10096 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:23.040629 10096 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:23.041575 10096 tablet_bootstrap.cc:492] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:23.042295 10096 ts_tablet_manager.cc:1403] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 12:25:23.044405 10096 raft_consensus.cc:359] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:23.044909 10096 raft_consensus.cc:740] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:23.045436 10096 consensus_queue.cc:260] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:23.047343 10096 ts_tablet_manager.cc:1434] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:23.048467 10096 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:23.064006 10096 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:23.065030 10096 tablet_bootstrap.cc:492] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:23.065814 10096 ts_tablet_manager.cc:1403] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.003s
I20260709 12:25:23.068511 10096 raft_consensus.cc:359] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:23.069007 10096 raft_consensus.cc:740] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:23.069525 10096 consensus_queue.cc:260] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:23.071995 10096 ts_tablet_manager.cc:1434] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:25:23.072819 10096 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap starting.
I20260709 12:25:23.087224 10096 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:23.088110 10096 tablet_bootstrap.cc:492] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Bootstrap complete.
I20260709 12:25:23.088683 10096 ts_tablet_manager.cc:1403] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.000s
I20260709 12:25:23.090953 10096 raft_consensus.cc:359] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:23.091631 10096 raft_consensus.cc:740] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Initialized, Role: FOLLOWER
I20260709 12:25:23.092157 10096 consensus_queue.cc:260] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:23.093817 10096 ts_tablet_manager.cc:1434] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:23.234964 10179 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:23.242621 10179 pending_rounds.cc:85] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Aborting all ops after (but not including) 0
I20260709 12:25:23.242885 10179 pending_rounds.cc:107] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:25:23.243156 10179 raft_consensus.cc:2889] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:32.300887 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:32.364290  9526 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:57784
I20260709 12:25:32.365067  9526 raft_consensus.cc:606] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:32.377465  9378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:32.378012  9378 raft_consensus.cc:606] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:32.380825  9526 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:57784
I20260709 12:25:32.381543  9526 raft_consensus.cc:606] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:32.384109  9526 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:57784
I20260709 12:25:32.384649  9526 raft_consensus.cc:606] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:32.387899  9378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:32.388466  9378 raft_consensus.cc:606] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:32.389658 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:32.391321 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:32.392189 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:32.404002 10303 raft_consensus.cc:993] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.404790 10302 raft_consensus.cc:1081] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.407006 10171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53636
I20260709 12:25:32.407996 10171 raft_consensus.cc:493] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:32.408638 10171 raft_consensus.cc:3060] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.419298 10171 raft_consensus.cc:515] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:32.421553 10171 leader_election.cc:290] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:32.422359  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:32.423029  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cefb8e130fd43a1aba9f4773da6f272" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:32.423769  9526 raft_consensus.cc:3055] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:32.423887  9378 raft_consensus.cc:3060] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.424369  9526 raft_consensus.cc:740] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:32.425429  9526 consensus_queue.cc:260] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:32.426788  9526 raft_consensus.cc:3060] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.434527  9378 raft_consensus.cc:2468] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:32.435014  9526 raft_consensus.cc:2468] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:32.436354 10091 leader_election.cc:304] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:32.438691 10310 raft_consensus.cc:2804] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:32.442386 10310 raft_consensus.cc:697] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:32.443706 10310 consensus_queue.cc:237] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:32.455464  9201 catalog_manager.cc:5697] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from a5f89c82eda14a9ca425fae398bae986 (127.25.202.4) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:32.501236 10309 raft_consensus.cc:993] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.501782 10303 raft_consensus.cc:1081] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.503589 10171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53636
I20260709 12:25:32.504098 10171 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:32.504472 10171 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:32.517261 10171 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:32.519176 10171 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:32.520614  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:32.521382  9450 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:32.521353  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:32.522205  9526 raft_consensus.cc:3055] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:32.522521  9526 raft_consensus.cc:740] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:32.523458  9526 consensus_queue.cc:260] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:32.524747  9526 raft_consensus.cc:3060] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:32.530218  9450 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:32.531708 10090 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:32.531781  9526 raft_consensus.cc:2468] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:32.532586 10310 raft_consensus.cc:2804] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:32.533102 10310 raft_consensus.cc:697] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:32.533968 10310 consensus_queue.cc:237] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:32.559294  9201 catalog_manager.cc:5697] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 2 to 3, leader changed from a5f89c82eda14a9ca425fae398bae986 (127.25.202.4) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:32.624485 10303 raft_consensus.cc:993] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.625061 10309 raft_consensus.cc:1081] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.626718 10171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53636
I20260709 12:25:32.627231 10171 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:32.627612 10171 raft_consensus.cc:3060] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.632727 10171 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:32.634003 10171 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:32.634691  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:32.634966  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b8b272744248da9eb46aef40c57d9a" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:32.635370  9526 raft_consensus.cc:3055] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:32.635525  9378 raft_consensus.cc:3060] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.635756  9526 raft_consensus.cc:740] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:32.636590  9526 consensus_queue.cc:260] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:32.637804  9526 raft_consensus.cc:3060] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.642681  9378 raft_consensus.cc:2468] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:32.645078 10091 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:32.645875 10310 raft_consensus.cc:2804] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:32.646301  9526 raft_consensus.cc:2468] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:32.646440 10310 raft_consensus.cc:697] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:32.647677 10310 consensus_queue.cc:237] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:32.658627  9201 catalog_manager.cc:5697] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from a5f89c82eda14a9ca425fae398bae986 (127.25.202.4) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:32.679643 10293 raft_consensus.cc:993] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:32.680317 10304 raft_consensus.cc:1081] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:32.682046  9451 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:44398
I20260709 12:25:32.682600  9451 raft_consensus.cc:493] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:32.682969  9451 raft_consensus.cc:3060] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.689260  9451 raft_consensus.cc:515] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:32.691509  9451 leader_election.cc:290] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:32.692245  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:32.692533  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccf9f64ab9404078902886a07eef6a66" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:32.692869  9378 raft_consensus.cc:3055] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:32.693202  9526 raft_consensus.cc:3060] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.693346  9378 raft_consensus.cc:740] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:32.694185  9378 consensus_queue.cc:260] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:32.695566  9378 raft_consensus.cc:3060] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.699879  9526 raft_consensus.cc:2468] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:32.702071  9378 raft_consensus.cc:2468] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:32.702266  9420 leader_election.cc:304] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:32.703819 10300 raft_consensus.cc:2804] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:32.708199 10300 raft_consensus.cc:697] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:32.709153 10300 consensus_queue.cc:237] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:32.720512  9201 catalog_manager.cc:5697] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:32.730245 10295 raft_consensus.cc:993] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.730785 10295 raft_consensus.cc:1081] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:32.734409 10171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:32.734931 10171 raft_consensus.cc:493] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:32.735265 10171 raft_consensus.cc:3060] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.742544 10171 raft_consensus.cc:515] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:32.745636 10171 leader_election.cc:290] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:32.746133  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:32.746790  9450 raft_consensus.cc:3060] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.746093  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ae03511a1cc4b369544d0a2b61c2515" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:32.752024  9378 raft_consensus.cc:3055] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:32.752367  9378 raft_consensus.cc:740] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:32.752161  9450 raft_consensus.cc:2468] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:32.753252  9378 consensus_queue.cc:260] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:32.753595 10090 leader_election.cc:304] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:32.754362 10310 raft_consensus.cc:2804] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:32.754417  9378 raft_consensus.cc:3060] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:32.754880 10310 raft_consensus.cc:697] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:32.755995 10310 consensus_queue.cc:237] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:32.762861  9378 raft_consensus.cc:2468] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:32.767601  9201 catalog_manager.cc:5697] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:32.959600  9378 raft_consensus.cc:1275] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:32.961267 10310 consensus_queue.cc:1048] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:32.979081  9526 raft_consensus.cc:1275] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:32.980546 10310 consensus_queue.cc:1048] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.031211  9526 raft_consensus.cc:1275] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.034164 10311 consensus_queue.cc:1048] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.043787  9382 raft_consensus.cc:1275] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.045559 10310 consensus_queue.cc:1048] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.090073  9451 raft_consensus.cc:1275] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:33.091262 10323 consensus_queue.cc:1048] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:33.101730  9525 raft_consensus.cc:1275] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:33.103258 10311 consensus_queue.cc:1048] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:33.187969  9451 raft_consensus.cc:1275] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.189416 10311 consensus_queue.cc:1048] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.198946  9378 raft_consensus.cc:1275] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.200558 10311 consensus_queue.cc:1048] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.256116  9378 raft_consensus.cc:1275] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.257467 10300 consensus_queue.cc:1048] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.270603  9525 raft_consensus.cc:1275] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.272054 10319 consensus_queue.cc:1048] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.438294 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:33.475431  9382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "05a47d242f094b218e731992a142e472"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:33.476042  9382 raft_consensus.cc:606] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:33.480872  9382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:33.481379  9382 raft_consensus.cc:606] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:33.483541  9382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:33.483969  9382 raft_consensus.cc:606] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:33.486239  9382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:33.486717  9382 raft_consensus.cc:606] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:33.489143  9382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f907c1045137407887423995a153fea0"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:33.489562  9382 raft_consensus.cc:606] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:33.490520 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:33.492182 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:33.492811 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:33.522122 10293 raft_consensus.cc:993] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.522724 10295 raft_consensus.cc:1081] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.524606 10171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "05a47d242f094b218e731992a142e472"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:33.525079 10295 raft_consensus.cc:993] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.525141 10171 raft_consensus.cc:493] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:33.525625 10295 raft_consensus.cc:1081] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.525673 10171 raft_consensus.cc:3060] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.527632 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:33.528146 10179 raft_consensus.cc:493] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:33.528519 10179 raft_consensus.cc:3060] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.532675 10171 raft_consensus.cc:515] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.533777 10179 raft_consensus.cc:515] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:33.534760 10171 leader_election.cc:290] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:33.535717  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:33.536293  9450 raft_consensus.cc:3060] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.536388  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:33.536775  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "109883acf19d4de4b90dfea6f147d368" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:33.537097  9526 raft_consensus.cc:3060] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.537384  9378 raft_consensus.cc:3055] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:33.537714  9378 raft_consensus.cc:740] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:33.538473  9378 consensus_queue.cc:260] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:33.539515  9378 raft_consensus.cc:3060] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.540174  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05a47d242f094b218e731992a142e472" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:33.540797  9382 raft_consensus.cc:3055] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:33.541075  9382 raft_consensus.cc:740] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:33.541839  9382 consensus_queue.cc:260] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.545675  9526 raft_consensus.cc:2468] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:33.546353  9378 raft_consensus.cc:2468] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:33.547281 10090 leader_election.cc:304] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:33.547461  9450 raft_consensus.cc:2468] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:33.546108 10179 leader_election.cc:290] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:33.548377 10329 raft_consensus.cc:2804] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:33.548777 10090 leader_election.cc:304] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:33.548911 10329 raft_consensus.cc:697] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:33.550442 10311 raft_consensus.cc:2804] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:33.550153 10329 consensus_queue.cc:237] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:33.550984 10311 raft_consensus.cc:697] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:33.551971 10311 consensus_queue.cc:237] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.553514  9382 raft_consensus.cc:3060] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.570057 10336 raft_consensus.cc:993] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.570604 10336 raft_consensus.cc:1081] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.575301 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:33.576241 10179 raft_consensus.cc:493] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:33.577526 10179 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:33.587890 10179 raft_consensus.cc:515] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.588632  9382 raft_consensus.cc:2468] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:33.590845 10179 leader_election.cc:290] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:33.591115  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:33.592109  9378 raft_consensus.cc:3055] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:33.592528  9378 raft_consensus.cc:740] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:33.593725  9378 consensus_queue.cc:260] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.599267  9378 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:33.601291  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7076b0f22f54a05838aec7b64bf5575" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:33.603178  9450 raft_consensus.cc:3060] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:33.607111  9378 raft_consensus.cc:2468] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:33.606895  9200 catalog_manager.cc:5697] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:33.608518 10091 leader_election.cc:304] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:33.609638 10329 raft_consensus.cc:2804] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:33.610481 10329 raft_consensus.cc:697] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:33.612155 10329 consensus_queue.cc:237] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.612409  9450 raft_consensus.cc:2468] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:33.634313  9200 catalog_manager.cc:5697] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:33.635865  9200 catalog_manager.cc:5697] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 2 to 3, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:33.665045 10295 raft_consensus.cc:993] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:33.665467 10293 raft_consensus.cc:1081] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:33.667145  9451 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f907c1045137407887423995a153fea0"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:44398
I20260709 12:25:33.667765  9451 raft_consensus.cc:493] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:33.668087  9451 raft_consensus.cc:3060] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.673156  9451 raft_consensus.cc:515] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:33.675148  9451 leader_election.cc:290] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:33.675779  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:33.676100  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f907c1045137407887423995a153fea0" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:33.676417  9378 raft_consensus.cc:3055] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:33.676638  9525 raft_consensus.cc:3060] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.676753  9378 raft_consensus.cc:740] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:33.677592  9378 consensus_queue.cc:260] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:33.678939  9378 raft_consensus.cc:3060] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.683943  9525 raft_consensus.cc:2468] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:33.685123  9420 leader_election.cc:304] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:33.685793  9378 raft_consensus.cc:2468] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:33.685915 10319 raft_consensus.cc:2804] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:33.686528 10319 raft_consensus.cc:697] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:33.687451 10319 consensus_queue.cc:237] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:33.700460  9200 catalog_manager.cc:5697] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:33.777755 10293 raft_consensus.cc:993] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.778230 10295 raft_consensus.cc:1081] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:33.780194 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:33.780879 10179 raft_consensus.cc:493] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:33.781272 10179 raft_consensus.cc:3060] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.792789 10179 raft_consensus.cc:515] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.794286 10179 leader_election.cc:290] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:33.795158  9451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:33.795960  9451 raft_consensus.cc:3060] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.795158  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9863504e96043ba8fafaa2396daa5f4" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:33.796628  9378 raft_consensus.cc:3055] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:33.797058  9378 raft_consensus.cc:740] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:33.797938  9378 consensus_queue.cc:260] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.799260  9378 raft_consensus.cc:3060] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:33.801465  9451 raft_consensus.cc:2468] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:33.802812 10090 leader_election.cc:304] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:33.803997 10329 raft_consensus.cc:2804] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:33.804509 10329 raft_consensus.cc:697] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:33.804688  9378 raft_consensus.cc:2468] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:33.805485 10329 consensus_queue.cc:237] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:33.815717  9200 catalog_manager.cc:5697] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:33.935912  9378 raft_consensus.cc:1275] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.937088 10329 consensus_queue.cc:1048] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:33.947180  9525 raft_consensus.cc:1275] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:33.949036 10329 consensus_queue.cc:1048] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.053402  9451 raft_consensus.cc:1275] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:34.054738 10323 consensus_queue.cc:1048] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.065994  9378 raft_consensus.cc:1275] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:34.068025 10310 consensus_queue.cc:1048] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.106020  9378 raft_consensus.cc:1275] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:34.107463 10310 consensus_queue.cc:1048] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:34.126497  9454 raft_consensus.cc:1275] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:34.128087 10311 consensus_queue.cc:1048] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:34.167543  9378 raft_consensus.cc:1275] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:34.168897 10300 consensus_queue.cc:1048] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.183532  9526 raft_consensus.cc:1275] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:34.185360 10300 consensus_queue.cc:1048] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.243456  9377 raft_consensus.cc:1275] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:34.244705 10310 consensus_queue.cc:1048] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.258014  9451 raft_consensus.cc:1275] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:34.259398 10323 consensus_queue.cc:1048] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:34.531908 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:34.572896  9377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19cf5127045948269927c8b47835bb6e"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:34.573669  9377 raft_consensus.cc:606] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:34.576519  9377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:34.577011  9377 raft_consensus.cc:606] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:34.579470  9377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "a5f89c82eda14a9ca425fae398bae986"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:34.580019  9377 raft_consensus.cc:606] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:34.582648  9377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:34.583159  9377 raft_consensus.cc:606] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Received request to transfer leadership to TS 074a6ac642e4430ba8045181f0118ee3
I20260709 12:25:34.599680  9454 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f950306b3d994137989bf2f2f5512441"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:44426
I20260709 12:25:34.600454  9454 raft_consensus.cc:606] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:34.601949 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:34.603464 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:34.604111 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:34.664651 10293 raft_consensus.cc:993] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:34.665241 10295 raft_consensus.cc:1081] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:34.666676  9454 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19cf5127045948269927c8b47835bb6e"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:44398
I20260709 12:25:34.667135  9454 raft_consensus.cc:493] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:34.667500  9454 raft_consensus.cc:3060] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.672585  9454 raft_consensus.cc:515] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:34.674154  9454 leader_election.cc:290] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:34.675001  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:34.675040  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19cf5127045948269927c8b47835bb6e" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:34.675736  9525 raft_consensus.cc:3060] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.676004  9377 raft_consensus.cc:3055] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:34.676355  9377 raft_consensus.cc:740] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:34.677062  9377 consensus_queue.cc:260] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:34.678112  9377 raft_consensus.cc:3060] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.681614  9525 raft_consensus.cc:2468] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:34.683820  9420 leader_election.cc:304] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:34.684644 10300 raft_consensus.cc:2804] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:34.684864  9377 raft_consensus.cc:2468] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:34.685087 10300 raft_consensus.cc:697] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:34.686336 10300 consensus_queue.cc:237] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:34.696226  9201 catalog_manager.cc:5697] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:34.927460 10295 raft_consensus.cc:993] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower a5f89c82eda14a9ca425fae398bae986 to start an election
I20260709 12:25:34.928077 10293 raft_consensus.cc:1081] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer a5f89c82eda14a9ca425fae398bae986 to start an election
I20260709 12:25:34.930040  9525 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
 from {username='slave'} at 127.0.0.1:57614
I20260709 12:25:34.930620  9525 raft_consensus.cc:493] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:34.930994  9525 raft_consensus.cc:3060] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.937841  9525 raft_consensus.cc:515] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:34.940011  9525 leader_election.cc:290] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:34.940721  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:34.941123  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b0698af7d34bc9a3f26a3b75435b16" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:34.941411  9377 raft_consensus.cc:3055] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:34.941764  9454 raft_consensus.cc:3060] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.941789  9377 raft_consensus.cc:740] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:34.942871  9377 consensus_queue.cc:260] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:34.944068  9377 raft_consensus.cc:3060] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.948660  9454 raft_consensus.cc:2468] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:34.950285  9494 leader_election.cc:304] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:34.951161 10303 raft_consensus.cc:2804] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:34.951637 10303 raft_consensus.cc:697] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:34.951797 10339 raft_consensus.cc:993] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:34.952462 10300 raft_consensus.cc:1081] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:34.952476 10303 consensus_queue.cc:237] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:34.954203 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f950306b3d994137989bf2f2f5512441"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53642
I20260709 12:25:34.954803 10179 raft_consensus.cc:493] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:34.955174 10179 raft_consensus.cc:3060] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.955260  9377 raft_consensus.cc:2468] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:34.962280 10179 raft_consensus.cc:515] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:34.966467  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:34.967103  9454 raft_consensus.cc:3055] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:34.967366 10179 leader_election.cc:290] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:34.967480  9454 raft_consensus.cc:740] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:34.968506  9454 consensus_queue.cc:260] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:34.968990  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f950306b3d994137989bf2f2f5512441" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:34.969575  9525 raft_consensus.cc:3060] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.969761  9454 raft_consensus.cc:3060] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:34.973737  9201 catalog_manager.cc:5697] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:34.977224  9525 raft_consensus.cc:2468] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:34.978674 10090 leader_election.cc:304] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:34.980034 10323 raft_consensus.cc:2804] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:34.980510 10323 raft_consensus.cc:697] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:34.981377 10323 consensus_queue.cc:237] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:34.990100  9454 raft_consensus.cc:2468] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:35.002264  9201 catalog_manager.cc:5697] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:35.036134 10295 raft_consensus.cc:993] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:35.036625 10295 raft_consensus.cc:1081] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:35.038223 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:35.038758 10179 raft_consensus.cc:493] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:35.039069 10179 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:35.044238 10179 raft_consensus.cc:515] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:35.045956 10179 leader_election.cc:290] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:35.046805  9378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:35.047183  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a7c46b071ed4f03bb81dc8a4f55555c" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:35.047485  9378 raft_consensus.cc:3055] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:35.047866  9454 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:35.047900  9378 raft_consensus.cc:740] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:35.048995  9378 consensus_queue.cc:260] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:35.050156  9378 raft_consensus.cc:3060] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:35.053499  9454 raft_consensus.cc:2468] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:35.054497 10090 leader_election.cc:304] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:35.055279 10310 raft_consensus.cc:2804] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:35.055506  9378 raft_consensus.cc:2468] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:35.055755 10310 raft_consensus.cc:697] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:35.057207 10310 consensus_queue.cc:237] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:35.068053  9201 catalog_manager.cc:5697] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 2 to 3, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:35.084378  9378 raft_consensus.cc:1275] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.085649 10339 consensus_queue.cc:1048] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:35.106179  9525 raft_consensus.cc:1275] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.107661 10300 consensus_queue.cc:1048] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:35.128597 10371 raft_consensus.cc:993] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:35.129160 10371 raft_consensus.cc:1081] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Signalling peer 074a6ac642e4430ba8045181f0118ee3 to start an election
I20260709 12:25:35.130887  9454 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
 from {username='slave'} at 127.0.0.1:44398
I20260709 12:25:35.131569  9454 raft_consensus.cc:493] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:35.131937  9454 raft_consensus.cc:3060] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:35.141103  9454 raft_consensus.cc:515] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:35.142848  9454 leader_election.cc:290] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:35.143533  9377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:35.144181  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac21f3e309014f859e6995b20e5414bb" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:35.144549  9377 raft_consensus.cc:3055] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:35.144959  9525 raft_consensus.cc:3060] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:35.145002  9377 raft_consensus.cc:740] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:35.146234  9377 consensus_queue.cc:260] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:35.147843  9377 raft_consensus.cc:3060] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:35.153072  9525 raft_consensus.cc:2468] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:35.154379  9420 leader_election.cc:304] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, a5f89c82eda14a9ca425fae398bae986; no voters: 
I20260709 12:25:35.155194 10300 raft_consensus.cc:2804] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:35.155714 10300 raft_consensus.cc:697] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Leader. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:35.156663  9377 raft_consensus.cc:2468] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074a6ac642e4430ba8045181f0118ee3 in term 2.
I20260709 12:25:35.156607 10300 consensus_queue.cc:237] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:35.169813  9201 catalog_manager.cc:5697] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 reported cstate change: term changed from 1 to 2, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "074a6ac642e4430ba8045181f0118ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:35.343206  9378 raft_consensus.cc:1275] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.344468 10303 consensus_queue.cc:1048] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:35.358762  9454 raft_consensus.cc:1275] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.360265 10309 consensus_queue.cc:1048] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:25:35.451794  9525 raft_consensus.cc:1275] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.454138 10310 consensus_queue.cc:1048] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:35.465260  9454 raft_consensus.cc:1275] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.468050 10323 consensus_queue.cc:1048] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:35.491818  9454 raft_consensus.cc:1275] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:35.493525 10323 consensus_queue.cc:1048] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:35.502911  9377 raft_consensus.cc:1275] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:35.504474 10385 consensus_queue.cc:1048] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:35.632220  9526 raft_consensus.cc:1275] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.633591 10300 consensus_queue.cc:1048] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:35.647159 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:35.647903  9378 raft_consensus.cc:1275] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 074a6ac642e4430ba8045181f0118ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:35.650054 10300 consensus_queue.cc:1048] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:25:35.720145  9525 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:57784
I20260709 12:25:35.720844  9525 raft_consensus.cc:606] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:35.723752  9450 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
mode: GRACEFUL
new_leader_uuid: "a5f89c82eda14a9ca425fae398bae986"
 from {username='slave'} at 127.0.0.1:44426
I20260709 12:25:35.724251  9450 raft_consensus.cc:606] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Received request to transfer leadership to TS a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:35.726986  9382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "51d19f01907340eb8dc56ed742606710"
dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:40900
I20260709 12:25:35.727680  9382 raft_consensus.cc:606] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:35.730304  9450 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e"
dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:44426
I20260709 12:25:35.730708  9450 raft_consensus.cc:606] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:35.731734 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:25:35.733136 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:35.733768 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:35.773279 10371 raft_consensus.cc:993] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:35.775141 10295 raft_consensus.cc:1081] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:35.776859 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "51d19f01907340eb8dc56ed742606710"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53652
I20260709 12:25:35.777472 10179 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:35.777881 10179 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:35.787679 10179 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:35.791427  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:35.792500  9382 raft_consensus.cc:3055] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:35.792847  9382 raft_consensus.cc:740] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e07a645c9ee14f11a5d64fa7b460acf9, State: Running, Role: LEADER
I20260709 12:25:35.792739  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d19f01907340eb8dc56ed742606710" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:35.794381  9525 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:35.794056  9382 consensus_queue.cc:260] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:35.795356  9382 raft_consensus.cc:3060] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:35.795780 10179 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:35.805599  9382 raft_consensus.cc:2468] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:35.807106 10091 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:35.808192 10310 raft_consensus.cc:2804] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:35.808593 10310 raft_consensus.cc:697] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:35.809513 10310 consensus_queue.cc:237] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:35.818054  9525 raft_consensus.cc:2468] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:35.820979  9201 catalog_manager.cc:5697] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 2 to 3, leader changed from e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:35.914666 10303 raft_consensus.cc:993] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:35.915165 10309 raft_consensus.cc:1081] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:35.916673 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53636
I20260709 12:25:35.917124 10179 raft_consensus.cc:493] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:35.917388 10179 raft_consensus.cc:3060] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:35.923309 10179 raft_consensus.cc:515] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:35.925376 10179 leader_election.cc:290] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Requested vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:35.925547  9525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:35.925951  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "059f94ed1c1a42a5bc94cff7da97c777" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:35.926154  9525 raft_consensus.cc:3055] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:35.926699  9525 raft_consensus.cc:740] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:35.926888  9382 raft_consensus.cc:3060] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:35.927500  9525 consensus_queue.cc:260] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:35.928534  9525 raft_consensus.cc:3060] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:35.932917  9382 raft_consensus.cc:2468] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:35.934111 10091 leader_election.cc:304] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:35.934785  9525 raft_consensus.cc:2468] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 2.
I20260709 12:25:35.935787 10310 raft_consensus.cc:2804] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:35.937071 10310 raft_consensus.cc:697] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:35.938046 10310 consensus_queue.cc:237] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:35.949875  9201 catalog_manager.cc:5697] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 1 to 2, leader changed from a5f89c82eda14a9ca425fae398bae986 (127.25.202.4) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:36.135154 10391 raft_consensus.cc:993] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:36.135732 10339 raft_consensus.cc:1081] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:36.138468 10179 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53642
I20260709 12:25:36.139009 10179 raft_consensus.cc:493] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:36.139369 10179 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:36.144259 10179 raft_consensus.cc:515] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:36.145638 10179 leader_election.cc:290] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:36.146677  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:36.146817  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9fe6f29f3144d21b12db35664f1f69e" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:36.147310  9450 raft_consensus.cc:3055] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:36.147418  9382 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:36.147716  9450 raft_consensus.cc:740] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:36.148595  9450 consensus_queue.cc:260] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:36.149889  9450 raft_consensus.cc:3060] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:36.155192  9382 raft_consensus.cc:2468] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:36.155561  9450 raft_consensus.cc:2468] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:36.156332 10091 leader_election.cc:304] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:36.157363 10310 raft_consensus.cc:2804] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:36.157848 10310 raft_consensus.cc:697] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:36.158716 10310 consensus_queue.cc:237] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:36.168151  9201 catalog_manager.cc:5697] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 2 to 3, leader changed from 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:36.253554 10339 raft_consensus.cc:993] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: : Instructing follower a5f89c82eda14a9ca425fae398bae986 to start an election
I20260709 12:25:36.254093 10300 raft_consensus.cc:1081] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Signalling peer a5f89c82eda14a9ca425fae398bae986 to start an election
I20260709 12:25:36.255620  9525 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
 from {username='slave'} at 127.0.0.1:57610
I20260709 12:25:36.256196  9525 raft_consensus.cc:493] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:36.256536  9525 raft_consensus.cc:3060] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:36.261930  9525 raft_consensus.cc:515] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:36.263909  9525 leader_election.cc:290] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Requested vote from peers e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279), 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369)
I20260709 12:25:36.264525  9382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e07a645c9ee14f11a5d64fa7b460acf9"
I20260709 12:25:36.264825  9454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30d357fc75ee4a55a448f070c32e6499" candidate_uuid: "a5f89c82eda14a9ca425fae398bae986" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:36.265244  9382 raft_consensus.cc:3060] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:36.265337  9454 raft_consensus.cc:3055] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:36.265708  9454 raft_consensus.cc:740] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 074a6ac642e4430ba8045181f0118ee3, State: Running, Role: LEADER
I20260709 12:25:36.266476  9454 consensus_queue.cc:260] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:36.267752  9454 raft_consensus.cc:3060] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:36.273638  9382 raft_consensus.cc:2468] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:36.274564  9454 raft_consensus.cc:2468] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f89c82eda14a9ca425fae398bae986 in term 2.
I20260709 12:25:36.274893  9495 leader_election.cc:304] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986, e07a645c9ee14f11a5d64fa7b460acf9; no voters: 
I20260709 12:25:36.275748 10392 raft_consensus.cc:2804] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:36.276196 10392 raft_consensus.cc:697] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Leader. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:36.276872 10392 consensus_queue.cc:237] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } }
I20260709 12:25:36.286197  9201 catalog_manager.cc:5697] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 reported cstate change: term changed from 1 to 2, leader changed from 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3) to a5f89c82eda14a9ca425fae398bae986 (127.25.202.4). New cstate: current_term: 2 leader_uuid: "a5f89c82eda14a9ca425fae398bae986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:36.335421  9382 raft_consensus.cc:1275] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:36.336880 10310 consensus_queue.cc:1048] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:36.352155  9525 raft_consensus.cc:1275] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:36.354133 10383 consensus_queue.cc:1048] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:36.407554  9525 raft_consensus.cc:1275] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:36.408782 10383 consensus_queue.cc:1048] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:36.423957  9377 raft_consensus.cc:1275] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:36.425549 10310 consensus_queue.cc:1048] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:36.651024  9454 raft_consensus.cc:1275] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:36.652443 10310 consensus_queue.cc:1048] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:36.661545  9382 raft_consensus.cc:1275] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:36.663023 10383 consensus_queue.cc:1048] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:36.710143  9382 raft_consensus.cc:1275] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:36.711303 10309 consensus_queue.cc:1048] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:36.723927  9454 raft_consensus.cc:1275] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Refusing update from remote peer a5f89c82eda14a9ca425fae398bae986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:36.725314 10309 consensus_queue.cc:1048] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:36.769820 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:36.809733  9526 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073"
dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
mode: GRACEFUL
new_leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:57784
I20260709 12:25:36.810369  9526 raft_consensus.cc:606] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Received request to transfer leadership to TS 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:36.811878 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:25:36.813354 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:36.813949 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:37.062181 10309 raft_consensus.cc:993] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: : Instructing follower 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:37.062733 10392 raft_consensus.cc:1081] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Signalling peer 4e58af1026634b45bafbbfde02a6ecd0 to start an election
I20260709 12:25:37.065092 10176 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073"
dest_uuid: "4e58af1026634b45bafbbfde02a6ecd0"
 from {username='slave'} at 127.0.0.1:53636
I20260709 12:25:37.065701 10176 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:37.066150 10176 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:37.073675 10176 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:37.076224 10176 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Requested vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855)
I20260709 12:25:37.077375  9450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "074a6ac642e4430ba8045181f0118ee3"
I20260709 12:25:37.078114  9450 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:37.078038  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76060914d3af47e6ad7cb572ad9cc073" candidate_uuid: "4e58af1026634b45bafbbfde02a6ecd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "a5f89c82eda14a9ca425fae398bae986"
I20260709 12:25:37.079005  9526 raft_consensus.cc:3055] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:37.079420  9526 raft_consensus.cc:740] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a5f89c82eda14a9ca425fae398bae986, State: Running, Role: LEADER
I20260709 12:25:37.080317  9526 consensus_queue.cc:260] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:37.086325  9526 raft_consensus.cc:3060] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:37.086849  9450 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:37.088307 10090 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0; no voters: 
I20260709 12:25:37.089254 10385 raft_consensus.cc:2804] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:37.089752 10385 raft_consensus.cc:697] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Becoming Leader. State: Replica: 4e58af1026634b45bafbbfde02a6ecd0, State: Running, Role: LEADER
I20260709 12:25:37.090703 10385 consensus_queue.cc:237] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:37.093588  9526 raft_consensus.cc:2468] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e58af1026634b45bafbbfde02a6ecd0 in term 3.
I20260709 12:25:37.109707  9201 catalog_manager.cc:5697] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 reported cstate change: term changed from 2 to 3, leader changed from a5f89c82eda14a9ca425fae398bae986 (127.25.202.4) to 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1). New cstate: current_term: 3 leader_uuid: "4e58af1026634b45bafbbfde02a6ecd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:37.535662  9450 raft_consensus.cc:1275] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:37.537184 10385 consensus_queue.cc:1048] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:37.552177  9526 raft_consensus.cc:1275] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Refusing update from remote peer 4e58af1026634b45bafbbfde02a6ecd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:37.553596 10423 consensus_queue.cc:1048] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:37.850929 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:37.893390 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:25:37.895104 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:37.895699 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:37.939105 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:37.979959 26408 tablet_server.cc:179] TabletServer@127.25.202.1:33407 shutting down...
I20260709 12:25:38.020498 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:38.023166 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.023998 26408 raft_consensus.cc:2243] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.024441 26408 raft_consensus.cc:2272] T eb1d8de1bb30430eb7ae1bbbf66232df P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.027189 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.028223 26408 raft_consensus.cc:2243] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.028769 26408 raft_consensus.cc:2272] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.032241 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.032924 26408 raft_consensus.cc:2243] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.033445 26408 raft_consensus.cc:2272] T 6e7e7ca3cc2243559667954882dfa118 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.036475 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.037324 26408 raft_consensus.cc:2243] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.037858 26408 raft_consensus.cc:2272] T 160804417e4443b9b25105c759111c8b P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.040055  9347 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 246 similar messages]
I20260709 12:25:38.041417 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.042471 26408 raft_consensus.cc:2243] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.043026 26408 raft_consensus.cc:2272] T 36a1d25b3256456ebdf25d5b805c8086 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.044427  9347 consensus_peers.cc:597] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.046381 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.047472 26408 raft_consensus.cc:2243] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.048270 26408 raft_consensus.cc:2272] T fd0b217a2c7841638bd1d47ad5ea6783 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.050370  9347 consensus_peers.cc:597] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.051723 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.052515 26408 raft_consensus.cc:2243] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:25:38.053294  9347 consensus_peers.cc:597] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.053570 26408 raft_consensus.cc:2272] T 3cefb8e130fd43a1aba9f4773da6f272 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.054566  9347 consensus_peers.cc:597] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.056413 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.057324 26408 raft_consensus.cc:2243] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.057825 26408 raft_consensus.cc:2272] T 8c302268ead84f94a04993f710b731e6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.060695 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.061450 26408 raft_consensus.cc:2243] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.062017 26408 raft_consensus.cc:2272] T aad56d798d414d5eb49051f39b8ae755 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.064949 26408 tablet_replica.cc:333] stopping tablet replica
W20260709 12:25:38.065574  9495 consensus_peers.cc:597] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.065893 26408 raft_consensus.cc:2243] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.066500 26408 raft_consensus.cc:2272] T 1b2fa83e4cf5494ba63becbde8d026b2 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.068955 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.069679 26408 raft_consensus.cc:2243] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.070545 26408 raft_consensus.cc:2272] T 51d19f01907340eb8dc56ed742606710 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.073089 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.073992 26408 raft_consensus.cc:2243] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.074462 26408 raft_consensus.cc:2272] T b303f27117804b4c8dc4ef4e31e0384e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.077489 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.078202 26408 raft_consensus.cc:2243] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.079105 26408 raft_consensus.cc:2272] T 76060914d3af47e6ad7cb572ad9cc073 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.082360 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.083875 26408 raft_consensus.cc:2243] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.084710 26408 raft_consensus.cc:2272] T c9515a897d574915ba4d8a478406aa1a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.088032 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.088754 26408 raft_consensus.cc:2243] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.089715 26408 raft_consensus.cc:2272] T 109883acf19d4de4b90dfea6f147d368 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.092572 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.093226 26408 raft_consensus.cc:2243] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.093706 26408 raft_consensus.cc:2272] T 85baaa21d67f43da9f328b0b51a71215 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.096215 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.096787 26408 raft_consensus.cc:2243] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.097352 26408 raft_consensus.cc:2272] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.100633 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.101348 26408 raft_consensus.cc:2243] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.102149 26408 raft_consensus.cc:2272] T f950306b3d994137989bf2f2f5512441 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.104269 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.104882 26408 raft_consensus.cc:2243] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.105662 26408 raft_consensus.cc:2272] T 9acfb739951443068b4ff6d65ebb3691 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.107784 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.108546 26408 raft_consensus.cc:2243] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.109071 26408 raft_consensus.cc:2272] T 642ee9123c0945a7b10a89d4ff3a01c6 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.112496 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.115118 26408 raft_consensus.cc:2243] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.115706 26408 raft_consensus.cc:2272] T a97a9ba3c90f464eacdc4e78d6a91f5e P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.145664 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.150060 26408 raft_consensus.cc:2243] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.151641 26408 raft_consensus.cc:2272] T 8a891dab57c74222928537e93876be86 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.183840  9347 consensus_peers.cc:597] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.184989  9347 consensus_peers.cc:597] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.191682 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.203799 26408 raft_consensus.cc:2243] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.204838 26408 raft_consensus.cc:2272] T e8fabd519bb34fbaae2738d3cfead584 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.204985  9495 consensus_peers.cc:597] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.206543  9421 consensus_peers.cc:597] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.210232 26408 tablet_replica.cc:333] stopping tablet replica
W20260709 12:25:38.212154  9421 consensus_peers.cc:597] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.213557  9495 consensus_peers.cc:597] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.218601 26408 raft_consensus.cc:2243] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.222355 26408 raft_consensus.cc:2272] T d7076b0f22f54a05838aec7b64bf5575 P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.229532 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.231251 26408 raft_consensus.cc:2243] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:25:38.232736  9347 consensus_peers.cc:597] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.234035 26408 raft_consensus.cc:2272] T 05a47d242f094b218e731992a142e472 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.242498 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.244695 26408 raft_consensus.cc:2243] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.245700 26408 raft_consensus.cc:2272] T 68ff98ec6813440cae57fcf054f25732 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.251873 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.253922 26408 raft_consensus.cc:2243] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.255124 26408 raft_consensus.cc:2272] T 30315f436f8c4ef4b1ac5381d6a34811 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.257997  9347 consensus_peers.cc:597] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.260537 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.262223 26408 raft_consensus.cc:2243] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.263163 26408 raft_consensus.cc:2272] T 9eed29862fa545d7b5bf1232e33f0798 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.271808 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.274133 26408 raft_consensus.cc:2243] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.280445 26408 raft_consensus.cc:2272] T 4e722577ca9c4859ac9d6f335a1edfc9 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.285785 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.293444 26408 raft_consensus.cc:2243] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.294490 26408 raft_consensus.cc:2272] T 1976a69f7f174bce837050f43e2dd6fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.303418 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.304754 26408 raft_consensus.cc:2243] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.305766 26408 raft_consensus.cc:2272] T ffb83e944aed43718284cb48f994efcc P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.314361  9495 consensus_peers.cc:597] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.321125 26408 tablet_replica.cc:333] stopping tablet replica
W20260709 12:25:38.296245  9421 consensus_peers.cc:597] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.324333  9347 consensus_peers.cc:597] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.322685 26408 raft_consensus.cc:2243] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.328547 26408 raft_consensus.cc:2272] T a53d66736d1046bea06b3935c396942d P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.331595  9347 consensus_peers.cc:597] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.341634  9421 consensus_peers.cc:597] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.345728  9495 consensus_peers.cc:597] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.347339  9421 consensus_peers.cc:597] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.348155  9420 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.4:39855, user_credentials={real_user=slave}} blocked reactor thread for 47772.7us
W20260709 12:25:38.353333  9421 consensus_peers.cc:597] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.364897  9347 consensus_peers.cc:597] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.367269  9421 consensus_peers.cc:597] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.374156 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.376303 26408 raft_consensus.cc:2243] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.377365 26408 raft_consensus.cc:2272] T 6272a586489844f8a507c2bea1c89107 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.383230 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.384392 26408 raft_consensus.cc:2243] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.386353 26408 raft_consensus.cc:2272] T 4ae03511a1cc4b369544d0a2b61c2515 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.390640 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.391918 26408 raft_consensus.cc:2243] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.394550 26408 raft_consensus.cc:2272] T f9fe6f29f3144d21b12db35664f1f69e P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.401788 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.403872 26408 raft_consensus.cc:2243] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.404832 26408 raft_consensus.cc:2272] T 46e96ef5e406430f9972ca4831f609fe P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.408886 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.409902 26408 raft_consensus.cc:2243] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.410979 26408 raft_consensus.cc:2272] T 4cecdb3fe70f46d8af803d3d7db78976 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.414489  9421 consensus_peers.cc:597] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.415269  9421 consensus_peers.cc:597] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.415877 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.416863 26408 raft_consensus.cc:2243] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.417675 26408 raft_consensus.cc:2272] T 58056f74cd464091a7f28a4a999ce936 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.419634  9495 consensus_peers.cc:597] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.420380  9495 consensus_peers.cc:597] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.421429 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.422052 26408 raft_consensus.cc:2243] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.423207 26408 raft_consensus.cc:2272] T 059f94ed1c1a42a5bc94cff7da97c777 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.425873 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.426345 26408 raft_consensus.cc:2243] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.426810 26408 raft_consensus.cc:2272] T 5a5f846566064f6f932c156f0f9c9d7a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.431082 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.432060 26408 raft_consensus.cc:2243] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.433897 26408 raft_consensus.cc:2272] T 77b8b272744248da9eb46aef40c57d9a P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.438035 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.438951 26408 raft_consensus.cc:2243] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.439720 26408 raft_consensus.cc:2272] T 890c806dd6f94382ba4e14a3d3bc6f85 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.443704 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.444579 26408 raft_consensus.cc:2243] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.445981 26408 raft_consensus.cc:2272] T 6a7c46b071ed4f03bb81dc8a4f55555c P 4e58af1026634b45bafbbfde02a6ecd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.449142 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.449703 26408 raft_consensus.cc:2243] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.450850 26408 raft_consensus.cc:2272] T c9863504e96043ba8fafaa2396daa5f4 P 4e58af1026634b45bafbbfde02a6ecd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.453387 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:38.453924 26408 raft_consensus.cc:2243] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.454391 26408 raft_consensus.cc:2272] T b494d68a603a4ba5a28ee8514cfa3f41 P 4e58af1026634b45bafbbfde02a6ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.468039  9347 consensus_peers.cc:597] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.482489  9495 consensus_peers.cc:597] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.546463  9421 consensus_peers.cc:597] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.551564 10089 connection.cc:466] server connection from 127.0.0.1:53636 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53636 (request call id 431) could send its response
W20260709 12:25:38.564875  9495 consensus_peers.cc:597] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 -> Peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Couldn't send request to peer 4e58af1026634b45bafbbfde02a6ecd0. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.571995 26408 tablet_server.cc:196] TabletServer@127.25.202.1:33407 shutdown complete.
I20260709 12:25:38.696852 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
W20260709 12:25:38.721824  9495 consensus_peers.cc:597] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.763958  9421 consensus_peers.cc:597] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.767944 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:38.769114 26408 tablet_replica.cc:333] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.770233 26408 raft_consensus.cc:2243] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.771168 26408 raft_consensus.cc:2272] T f4ba22da1d594bf19f8b83581343da38 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.774659 26408 tablet_replica.cc:333] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.775590 26408 raft_consensus.cc:2243] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.776196 26408 raft_consensus.cc:2272] T 1a6ceb4648564df0b7e690a8e3343b5b P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.780061 26408 tablet_replica.cc:333] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.780923 26408 raft_consensus.cc:2243] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.782080 26408 raft_consensus.cc:2272] T 9eed29862fa545d7b5bf1232e33f0798 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.785661 26408 tablet_replica.cc:333] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.786662 26408 raft_consensus.cc:2243] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.787389 26408 raft_consensus.cc:2272] T c9863504e96043ba8fafaa2396daa5f4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.790959 26408 tablet_replica.cc:333] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.791989 26408 raft_consensus.cc:2243] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.792698 26408 raft_consensus.cc:2272] T 6a7c46b071ed4f03bb81dc8a4f55555c P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.796375 26408 tablet_replica.cc:333] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.797179 26408 raft_consensus.cc:2243] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.797931 26408 raft_consensus.cc:2272] T f907c1045137407887423995a153fea0 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.801152 26408 tablet_replica.cc:333] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.801968 26408 raft_consensus.cc:2243] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.803035 26408 raft_consensus.cc:2272] T eb1d8de1bb30430eb7ae1bbbf66232df P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.803627  9421 consensus_peers.cc:597] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.805588 26408 tablet_replica.cc:333] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.806308 26408 raft_consensus.cc:2243] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.807173 26408 raft_consensus.cc:2272] T b9508782ff1d4df7867468f207693333 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.809959 26408 tablet_replica.cc:333] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.810772 26408 raft_consensus.cc:2243] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.811896 26408 raft_consensus.cc:2272] T 46e96ef5e406430f9972ca4831f609fe P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.812369  9495 consensus_peers.cc:597] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.815464 26408 tablet_replica.cc:333] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.816627 26408 raft_consensus.cc:2243] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.817696 26408 raft_consensus.cc:2272] T 4cecdb3fe70f46d8af803d3d7db78976 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.821019 26408 tablet_replica.cc:333] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.821668 26408 raft_consensus.cc:2243] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.822284 26408 raft_consensus.cc:2272] T f9fe6f29f3144d21b12db35664f1f69e P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.824791 26408 tablet_replica.cc:333] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.825450 26408 raft_consensus.cc:2243] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.825901 26408 raft_consensus.cc:2272] T 30d357fc75ee4a55a448f070c32e6499 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.828497 26408 tablet_replica.cc:333] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.829270 26408 raft_consensus.cc:2243] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.829921 26408 raft_consensus.cc:2272] T 05a47d242f094b218e731992a142e472 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.832142 26408 tablet_replica.cc:333] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.832782 26408 raft_consensus.cc:2243] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.833467 26408 raft_consensus.cc:2272] T ac21f3e309014f859e6995b20e5414bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.837143 26408 tablet_replica.cc:333] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.838016 26408 raft_consensus.cc:2243] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:25:38.838506  9495 consensus_peers.cc:597] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.839259 26408 raft_consensus.cc:2272] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.842541 26408 tablet_replica.cc:333] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.843212 26408 raft_consensus.cc:2243] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.843734 26408 raft_consensus.cc:2272] T 11e5772eda0648d9a96d2b5755c23569 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.846040 26408 tablet_replica.cc:333] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.846884 26408 raft_consensus.cc:2243] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.847707 26408 raft_consensus.cc:2272] T 109883acf19d4de4b90dfea6f147d368 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.851274 26408 tablet_replica.cc:333] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.852128 26408 raft_consensus.cc:2243] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.852984 26408 raft_consensus.cc:2272] T a97a9ba3c90f464eacdc4e78d6a91f5e P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.856129 26408 tablet_replica.cc:333] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.856928 26408 raft_consensus.cc:2243] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.857594 26408 raft_consensus.cc:2272] T 5a5f846566064f6f932c156f0f9c9d7a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.860845 26408 tablet_replica.cc:333] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.861763 26408 raft_consensus.cc:2243] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.862421 26408 raft_consensus.cc:2272] T 77b8b272744248da9eb46aef40c57d9a P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.865567 26408 tablet_replica.cc:333] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.866544 26408 raft_consensus.cc:2243] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.867174 26408 raft_consensus.cc:2272] T 3cefb8e130fd43a1aba9f4773da6f272 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.867466  9421 consensus_peers.cc:597] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.869441 26408 tablet_replica.cc:333] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.870182 26408 raft_consensus.cc:2243] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.871032 26408 raft_consensus.cc:2272] T 6272a586489844f8a507c2bea1c89107 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.873385 26408 tablet_replica.cc:333] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.874325 26408 raft_consensus.cc:2243] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.874997 26408 raft_consensus.cc:2272] T a53d66736d1046bea06b3935c396942d P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.878526 26408 tablet_replica.cc:333] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.879217 26408 raft_consensus.cc:2243] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.880208 26408 raft_consensus.cc:2272] T 47c1f0a1e81a4ff0a3652fafff1288b4 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.882668 26408 tablet_replica.cc:333] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.883445 26408 raft_consensus.cc:2243] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.884297 26408 raft_consensus.cc:2272] T 68ff98ec6813440cae57fcf054f25732 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.886910 26408 tablet_replica.cc:333] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.887735 26408 raft_consensus.cc:2243] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.888311 26408 raft_consensus.cc:2272] T 84b0698af7d34bc9a3f26a3b75435b16 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.890995 26408 tablet_replica.cc:333] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.891745 26408 raft_consensus.cc:2243] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.892266 26408 raft_consensus.cc:2272] T fc204ad755be48768bf67ef7e84254bb P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.894863 26408 tablet_replica.cc:333] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.895645 26408 raft_consensus.cc:2243] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.896238 26408 raft_consensus.cc:2272] T ccf9f64ab9404078902886a07eef6a66 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.899147 26408 tablet_replica.cc:333] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.900019 26408 raft_consensus.cc:2243] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.901069 26408 raft_consensus.cc:2272] T 1b2fa83e4cf5494ba63becbde8d026b2 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.903824 26408 tablet_replica.cc:333] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.904443 26408 raft_consensus.cc:2243] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.905066 26408 raft_consensus.cc:2272] T 51d19f01907340eb8dc56ed742606710 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.907003  9421 consensus_peers.cc:597] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.907929 26408 tablet_replica.cc:333] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.908758 26408 raft_consensus.cc:2243] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.909356 26408 raft_consensus.cc:2272] T 6dbb3c7d37504398aab89200647dab24 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.912467 26408 tablet_replica.cc:333] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.913261 26408 raft_consensus.cc:2243] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.914409 26408 raft_consensus.cc:2272] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.915969  9495 consensus_peers.cc:597] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.917359 26408 tablet_replica.cc:333] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
W20260709 12:25:38.918048  9495 consensus_peers.cc:597] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.918224 26408 raft_consensus.cc:2243] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.919245 26408 raft_consensus.cc:2272] T 890c806dd6f94382ba4e14a3d3bc6f85 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.922323 26408 tablet_replica.cc:333] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.923055 26408 raft_consensus.cc:2243] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.923676 26408 raft_consensus.cc:2272] T 19cf5127045948269927c8b47835bb6e P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.925721 26408 tablet_replica.cc:333] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.926209 26408 raft_consensus.cc:2243] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.926609 26408 raft_consensus.cc:2272] T c26afa4025e24ea192d424c506f40c56 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.929279 26408 tablet_replica.cc:333] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.929965 26408 raft_consensus.cc:2243] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.931028 26408 raft_consensus.cc:2272] T 6e7e7ca3cc2243559667954882dfa118 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.933306 26408 tablet_replica.cc:333] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.933923 26408 raft_consensus.cc:2243] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.934402 26408 raft_consensus.cc:2272] T b494d68a603a4ba5a28ee8514cfa3f41 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.937259 26408 tablet_replica.cc:333] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.937939 26408 raft_consensus.cc:2243] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.938551 26408 raft_consensus.cc:2272] T 58056f74cd464091a7f28a4a999ce936 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.941596 26408 tablet_replica.cc:333] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.942332 26408 raft_consensus.cc:2243] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.942862 26408 raft_consensus.cc:2272] T 059f94ed1c1a42a5bc94cff7da97c777 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.945407 26408 tablet_replica.cc:333] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.946039 26408 raft_consensus.cc:2243] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.946632 26408 raft_consensus.cc:2272] T 4ae03511a1cc4b369544d0a2b61c2515 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.949388  9495 consensus_peers.cc:597] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:38.949717 26408 tablet_replica.cc:333] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.950383 26408 raft_consensus.cc:2243] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.950878 26408 raft_consensus.cc:2272] T d7076b0f22f54a05838aec7b64bf5575 P e07a645c9ee14f11a5d64fa7b460acf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.953101 26408 tablet_replica.cc:333] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.953624 26408 raft_consensus.cc:2243] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:38.954417 26408 raft_consensus.cc:2272] T e8fabd519bb34fbaae2738d3cfead584 P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.956565 26408 tablet_replica.cc:333] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.957190 26408 raft_consensus.cc:2243] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.957804 26408 raft_consensus.cc:2272] T 8a891dab57c74222928537e93876be86 P e07a645c9ee14f11a5d64fa7b460acf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:38.960809 26408 tablet_replica.cc:333] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9: stopping tablet replica
I20260709 12:25:38.961561 26408 raft_consensus.cc:2243] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:38.962134 26408 raft_consensus.cc:2272] T 160804417e4443b9b25105c759111c8b P e07a645c9ee14f11a5d64fa7b460acf9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:38.966508  9495 consensus_peers.cc:597] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:38.995507  9421 consensus_peers.cc:597] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.019426  9421 consensus_peers.cc:597] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.022526  9495 consensus_peers.cc:597] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.025985  9495 consensus_peers.cc:597] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.056180 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
I20260709 12:25:39.070246 10414 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.070933 10414 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:39.073597 10414 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a5f89c82eda14a9ca425fae398bae986 (127.25.202.4:39855), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:39.074466  9526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9acfb739951443068b4ff6d65ebb3691" candidate_uuid: "074a6ac642e4430ba8045181f0118ee3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a5f89c82eda14a9ca425fae398bae986" is_pre_election: true
W20260709 12:25:39.080657  9421 consensus_peers.cc:597] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.081422  9421 leader_election.cc:336] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:39.081862  9421 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, a5f89c82eda14a9ca425fae398bae986
I20260709 12:25:39.082595 10414 raft_consensus.cc:2749] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:39.138482 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
W20260709 12:25:39.140390  9421 consensus_peers.cc:597] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.141203  9495 consensus_peers.cc:597] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 -> Peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Couldn't send request to peer e07a645c9ee14f11a5d64fa7b460acf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.178372  9494 consensus_peers.cc:597] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.188217 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:39.189405 26408 tablet_replica.cc:333] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.190363 26408 raft_consensus.cc:2243] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.191120 26408 raft_consensus.cc:2272] T aad56d798d414d5eb49051f39b8ae755 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.193763 26408 tablet_replica.cc:333] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.194438 26408 raft_consensus.cc:2243] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.194887 26408 raft_consensus.cc:2272] T f4ba22da1d594bf19f8b83581343da38 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.197152 26408 tablet_replica.cc:333] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.197878 26408 raft_consensus.cc:2243] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.198340 26408 raft_consensus.cc:2272] T 1a6ceb4648564df0b7e690a8e3343b5b P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.200284 26408 tablet_replica.cc:333] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.200922 26408 raft_consensus.cc:2243] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.201371 26408 raft_consensus.cc:2272] T c9863504e96043ba8fafaa2396daa5f4 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.203790 26408 tablet_replica.cc:333] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.204599 26408 raft_consensus.cc:2243] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.204788 10414 raft_consensus.cc:493] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.205166 26408 raft_consensus.cc:2272] T 6a7c46b071ed4f03bb81dc8a4f55555c P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.205425 10414 raft_consensus.cc:515] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } } peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } }
I20260709 12:25:39.207507 10414 leader_election.cc:290] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:39.208254 26408 tablet_replica.cc:333] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.209127 26408 raft_consensus.cc:2243] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.210210 26408 raft_consensus.cc:2272] T f907c1045137407887423995a153fea0 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.212481  9421 leader_election.cc:336] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:39.213240 26408 tablet_replica.cc:333] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
W20260709 12:25:39.213295  9421 leader_election.cc:336] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111)
I20260709 12:25:39.213845  9421 leader_election.cc:304] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074a6ac642e4430ba8045181f0118ee3; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:39.214267 26408 raft_consensus.cc:2243] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.214571 10414 raft_consensus.cc:2749] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:39.215009 26408 raft_consensus.cc:2272] T d7076b0f22f54a05838aec7b64bf5575 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.218344 26408 tablet_replica.cc:333] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.219172 26408 raft_consensus.cc:2243] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.219700 26408 raft_consensus.cc:2272] T e8fabd519bb34fbaae2738d3cfead584 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.222280 26408 tablet_replica.cc:333] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
W20260709 12:25:39.222700  9494 consensus_peers.cc:597] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.223272 26408 raft_consensus.cc:2243] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.224143 26408 raft_consensus.cc:2272] T fd0b217a2c7841638bd1d47ad5ea6783 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.226454 26408 tablet_replica.cc:333] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.227428 26408 raft_consensus.cc:2243] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.228338 26408 raft_consensus.cc:2272] T 36a1d25b3256456ebdf25d5b805c8086 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.231150 26408 tablet_replica.cc:333] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.231945 26408 raft_consensus.cc:2243] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.232561 26408 raft_consensus.cc:2272] T 05a47d242f094b218e731992a142e472 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.235383 26408 tablet_replica.cc:333] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.236207 26408 raft_consensus.cc:2243] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.237000 26408 raft_consensus.cc:2272] T ac21f3e309014f859e6995b20e5414bb P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.239085 26408 tablet_replica.cc:333] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.239727 26408 raft_consensus.cc:2243] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.240193 26408 raft_consensus.cc:2272] T 9acfb739951443068b4ff6d65ebb3691 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.242312 26408 tablet_replica.cc:333] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.242911 26408 raft_consensus.cc:2243] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.243796 26408 raft_consensus.cc:2272] T 642ee9123c0945a7b10a89d4ff3a01c6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.245852 26408 tablet_replica.cc:333] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.246435 26408 raft_consensus.cc:2243] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.247385 26408 raft_consensus.cc:2272] T 11e5772eda0648d9a96d2b5755c23569 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.249939 26408 tablet_replica.cc:333] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.250598 26408 raft_consensus.cc:2243] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.251220 26408 raft_consensus.cc:2272] T bf66e0c0d4ec4b10989c1f0abbbb8eb8 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.252841  9494 consensus_peers.cc:597] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.253799 26408 tablet_replica.cc:333] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.254385 26408 raft_consensus.cc:2243] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.254803 26408 raft_consensus.cc:2272] T b9508782ff1d4df7867468f207693333 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.256771 26408 tablet_replica.cc:333] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.257529 26408 raft_consensus.cc:2243] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.258479 26408 raft_consensus.cc:2272] T 4e722577ca9c4859ac9d6f335a1edfc9 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.261109 26408 tablet_replica.cc:333] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
W20260709 12:25:39.261160  9494 consensus_peers.cc:597] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.261850 26408 raft_consensus.cc:2243] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.262282 26408 raft_consensus.cc:2272] T 30d357fc75ee4a55a448f070c32e6499 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.265408 26408 tablet_replica.cc:333] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.266332 26408 raft_consensus.cc:2243] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.266812 26408 raft_consensus.cc:2272] T 1976a69f7f174bce837050f43e2dd6fe P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.269387 26408 tablet_replica.cc:333] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.269908 26408 raft_consensus.cc:2243] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.270334 26408 raft_consensus.cc:2272] T 68ff98ec6813440cae57fcf054f25732 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.272430 26408 tablet_replica.cc:333] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.272985 26408 raft_consensus.cc:2243] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.273767 26408 raft_consensus.cc:2272] T ccf9f64ab9404078902886a07eef6a66 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.276135 26408 tablet_replica.cc:333] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.276980 26408 raft_consensus.cc:2243] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.277731 26408 raft_consensus.cc:2272] T b303f27117804b4c8dc4ef4e31e0384e P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.280136  9494 consensus_peers.cc:597] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.280220 26408 tablet_replica.cc:333] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.281003 26408 raft_consensus.cc:2243] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.281445 26408 raft_consensus.cc:2272] T 6dbb3c7d37504398aab89200647dab24 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.283411 26408 tablet_replica.cc:333] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.283959 26408 raft_consensus.cc:2243] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.284503 26408 raft_consensus.cc:2272] T f9fe6f29f3144d21b12db35664f1f69e P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.287127 26408 tablet_replica.cc:333] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.288017 26408 raft_consensus.cc:2243] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.288550 26408 raft_consensus.cc:2272] T 2bb9dcd4bb4647b08e76fcc074ad8e40 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.291354 26408 tablet_replica.cc:333] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.292029 26408 raft_consensus.cc:2243] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.292433 26408 raft_consensus.cc:2272] T f950306b3d994137989bf2f2f5512441 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.294245 26408 tablet_replica.cc:333] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.294718 26408 raft_consensus.cc:2243] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.295069 26408 raft_consensus.cc:2272] T 6272a586489844f8a507c2bea1c89107 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.297008 26408 tablet_replica.cc:333] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.297462 26408 raft_consensus.cc:2243] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.298188 26408 raft_consensus.cc:2272] T ffb83e944aed43718284cb48f994efcc P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.300572 26408 tablet_replica.cc:333] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.301424 26408 raft_consensus.cc:2243] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.302284 26408 raft_consensus.cc:2272] T a53d66736d1046bea06b3935c396942d P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.304515 26408 tablet_replica.cc:333] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.305284 26408 raft_consensus.cc:2243] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.305770 26408 raft_consensus.cc:2272] T 47c1f0a1e81a4ff0a3652fafff1288b4 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.308247 26408 tablet_replica.cc:333] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.309042 26408 raft_consensus.cc:2243] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.309612 26408 raft_consensus.cc:2272] T 4ae03511a1cc4b369544d0a2b61c2515 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.311592 26408 tablet_replica.cc:333] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.312091 26408 raft_consensus.cc:2243] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.312453 26408 raft_consensus.cc:2272] T 85baaa21d67f43da9f328b0b51a71215 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.314369 26408 tablet_replica.cc:333] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.314852 26408 raft_consensus.cc:2243] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.315708 26408 raft_consensus.cc:2272] T 19cf5127045948269927c8b47835bb6e P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.317269 10433 raft_consensus.cc:493] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.317785 26408 tablet_replica.cc:333] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.317937 10392 raft_consensus.cc:493] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.318396 26408 raft_consensus.cc:2243] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.317834 10433 raft_consensus.cc:515] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:39.318830 26408 raft_consensus.cc:2272] T 8c302268ead84f94a04993f710b731e6 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.318689 10392 raft_consensus.cc:515] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:39.320557 10433 leader_election.cc:290] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:39.320981 10392 leader_election.cc:290] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
I20260709 12:25:39.321458 26408 tablet_replica.cc:333] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.322280 26408 raft_consensus.cc:2243] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.322916 26408 raft_consensus.cc:2272] T 6e7e7ca3cc2243559667954882dfa118 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.324733  9494 leader_election.cc:336] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.325994 26408 tablet_replica.cc:333] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.326799 26408 raft_consensus.cc:2243] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.328022 26408 raft_consensus.cc:2272] T b494d68a603a4ba5a28ee8514cfa3f41 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.328562  9495 leader_election.cc:336] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111)
W20260709 12:25:39.329201  9495 leader_election.cc:336] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:39.329586  9495 leader_election.cc:304] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:39.330396 10392 raft_consensus.cc:2749] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:39.330490  9495 leader_election.cc:336] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:39.331197 26408 tablet_replica.cc:333] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.331121  9495 leader_election.cc:304] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:39.331862 10392 raft_consensus.cc:2749] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:39.331964 26408 raft_consensus.cc:2243] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.332633 26408 raft_consensus.cc:2272] T 84b0698af7d34bc9a3f26a3b75435b16 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.335779 26408 tablet_replica.cc:333] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.340551 26408 raft_consensus.cc:2243] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.341085 26408 raft_consensus.cc:2272] T fc204ad755be48768bf67ef7e84254bb P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.344003 26408 tablet_replica.cc:333] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.344712 26408 raft_consensus.cc:2243] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.345676 26408 raft_consensus.cc:2272] T 5a5f846566064f6f932c156f0f9c9d7a P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.348549 26408 tablet_replica.cc:333] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.349223 26408 raft_consensus.cc:2243] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.349748 26408 raft_consensus.cc:2272] T c26afa4025e24ea192d424c506f40c56 P 074a6ac642e4430ba8045181f0118ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.352468 26408 tablet_replica.cc:333] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.353200 26408 raft_consensus.cc:2243] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.353830 26408 raft_consensus.cc:2272] T 76060914d3af47e6ad7cb572ad9cc073 P 074a6ac642e4430ba8045181f0118ee3 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.356050  9494 consensus_peers.cc:597] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.356752 26408 tablet_replica.cc:333] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.357581 26408 raft_consensus.cc:2243] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.358605 26408 raft_consensus.cc:2272] T c9515a897d574915ba4d8a478406aa1a P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.361382 26408 tablet_replica.cc:333] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3: stopping tablet replica
I20260709 12:25:39.362025 26408 raft_consensus.cc:2243] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.362629 26408 raft_consensus.cc:2272] T 30315f436f8c4ef4b1ac5381d6a34811 P 074a6ac642e4430ba8045181f0118ee3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:39.379441  9494 consensus_peers.cc:597] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.406623 10433 raft_consensus.cc:493] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.407229 10433 raft_consensus.cc:515] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:39.409574 10433 leader_election.cc:290] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:39.414851  9495 leader_election.cc:336] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:39.417092  9495 leader_election.cc:336] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111)
I20260709 12:25:39.417563  9495 leader_election.cc:304] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:39.418243 10433 raft_consensus.cc:2749] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:39.455968 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:25:39.465193 10433 raft_consensus.cc:493] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.465989 10433 raft_consensus.cc:515] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:39.466670 10392 raft_consensus.cc:493] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.467192 10392 raft_consensus.cc:515] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:39.468447 10433 leader_election.cc:290] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
I20260709 12:25:39.469408 10392 leader_election.cc:290] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:39.475834  9495 leader_election.cc:336] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:39.476473  9495 leader_election.cc:336] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:39.477115  9495 leader_election.cc:336] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111)
I20260709 12:25:39.477432  9495 leader_election.cc:304] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:39.478058 10392 raft_consensus.cc:2749] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:39.478260  9495 leader_election.cc:336] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111)
I20260709 12:25:39.478863  9495 leader_election.cc:304] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:39.479590 10392 raft_consensus.cc:2749] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:39.506239 10392 raft_consensus.cc:493] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.506771 10392 raft_consensus.cc:515] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:39.508780 10392 leader_election.cc:290] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:39.513531  9494 leader_election.cc:336] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Network error: Client connection negotiation failed: client connection to 127.25.202.3:39369: connect: Connection refused (error 111)
W20260709 12:25:39.514364  9495 leader_election.cc:336] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:39.514801  9495 leader_election.cc:304] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:39.515678 10309 raft_consensus.cc:2749] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:39.523128 10309 raft_consensus.cc:493] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.523700 10309 raft_consensus.cc:515] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } } peers { permanent_uuid: "e07a645c9ee14f11a5d64fa7b460acf9" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 42279 } }
I20260709 12:25:39.525926 10309 leader_election.cc:290] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407), e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279)
W20260709 12:25:39.530208  9494 consensus_peers.cc:597] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.3:39369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.531179  9494 consensus_peers.cc:597] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.3:39369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.531711  9494 consensus_peers.cc:597] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.3:39369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.534355  9494 consensus_peers.cc:597] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 -> Peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Couldn't send request to peer 074a6ac642e4430ba8045181f0118ee3. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.3:39369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:39.535102  9495 leader_election.cc:336] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
W20260709 12:25:39.536406  9495 leader_election.cc:336] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e07a645c9ee14f11a5d64fa7b460acf9 (127.25.202.2:42279): Network error: Client connection negotiation failed: client connection to 127.25.202.2:42279: connect: Connection refused (error 111)
I20260709 12:25:39.536871  9495 leader_election.cc:304] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 4e58af1026634b45bafbbfde02a6ecd0, e07a645c9ee14f11a5d64fa7b460acf9
I20260709 12:25:39.537617 10392 raft_consensus.cc:2749] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:39.543680 10433 raft_consensus.cc:493] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e58af1026634b45bafbbfde02a6ecd0)
I20260709 12:25:39.544260 10433 raft_consensus.cc:515] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074a6ac642e4430ba8045181f0118ee3" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 39369 } } peers { permanent_uuid: "a5f89c82eda14a9ca425fae398bae986" member_type: VOTER last_known_addr { host: "127.25.202.4" port: 39855 } } peers { permanent_uuid: "4e58af1026634b45bafbbfde02a6ecd0" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 33407 } }
I20260709 12:25:39.553743 26408 tablet_server.cc:179] TabletServer@127.25.202.4:0 shutting down...
I20260709 12:25:39.554428 10433 leader_election.cc:290] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369), 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407)
W20260709 12:25:39.555238  9494 leader_election.cc:336] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 074a6ac642e4430ba8045181f0118ee3 (127.25.202.3:39369): Network error: Client connection negotiation failed: client connection to 127.25.202.3:39369: connect: Connection refused (error 111)
W20260709 12:25:39.556175  9495 leader_election.cc:336] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4e58af1026634b45bafbbfde02a6ecd0 (127.25.202.1:33407): Network error: Client connection negotiation failed: client connection to 127.25.202.1:33407: connect: Connection refused (error 111)
I20260709 12:25:39.556699  9495 leader_election.cc:304] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f89c82eda14a9ca425fae398bae986; no voters: 074a6ac642e4430ba8045181f0118ee3, 4e58af1026634b45bafbbfde02a6ecd0
I20260709 12:25:39.558115 10433 raft_consensus.cc:2749] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:39.600868 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:39.601990 26408 tablet_replica.cc:333] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.603035 26408 raft_consensus.cc:2243] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.603854 26408 raft_consensus.cc:2272] T aad56d798d414d5eb49051f39b8ae755 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.605830 26408 tablet_replica.cc:333] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.606369 26408 raft_consensus.cc:2243] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.606725 26408 raft_consensus.cc:2272] T 46e96ef5e406430f9972ca4831f609fe P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.608640 26408 tablet_replica.cc:333] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.609143 26408 raft_consensus.cc:2243] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.609515 26408 raft_consensus.cc:2272] T 4cecdb3fe70f46d8af803d3d7db78976 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.611284 26408 tablet_replica.cc:333] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.611847 26408 raft_consensus.cc:2243] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.612244 26408 raft_consensus.cc:2272] T f4ba22da1d594bf19f8b83581343da38 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.614075 26408 tablet_replica.cc:333] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.614558 26408 raft_consensus.cc:2243] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.614900 26408 raft_consensus.cc:2272] T 9eed29862fa545d7b5bf1232e33f0798 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.616953 26408 tablet_replica.cc:333] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.617488 26408 raft_consensus.cc:2243] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.618257 26408 raft_consensus.cc:2272] T 1a6ceb4648564df0b7e690a8e3343b5b P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.620544 26408 tablet_replica.cc:333] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.621140 26408 raft_consensus.cc:2243] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.621578 26408 raft_consensus.cc:2272] T f907c1045137407887423995a153fea0 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.624331 26408 tablet_replica.cc:333] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.625077 26408 raft_consensus.cc:2243] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.625672 26408 raft_consensus.cc:2272] T fd0b217a2c7841638bd1d47ad5ea6783 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.628680 26408 tablet_replica.cc:333] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.629515 26408 raft_consensus.cc:2243] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.629974 26408 raft_consensus.cc:2272] T 36a1d25b3256456ebdf25d5b805c8086 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.632092 26408 tablet_replica.cc:333] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.632683 26408 raft_consensus.cc:2243] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.633118 26408 raft_consensus.cc:2272] T eb1d8de1bb30430eb7ae1bbbf66232df P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.634902 26408 tablet_replica.cc:333] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.635504 26408 raft_consensus.cc:2243] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.636188 26408 raft_consensus.cc:2272] T 58056f74cd464091a7f28a4a999ce936 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.638006 26408 tablet_replica.cc:333] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.638501 26408 raft_consensus.cc:2243] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.638862 26408 raft_consensus.cc:2272] T 059f94ed1c1a42a5bc94cff7da97c777 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.640812 26408 tablet_replica.cc:333] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.641314 26408 raft_consensus.cc:2243] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.641647 26408 raft_consensus.cc:2272] T 642ee9123c0945a7b10a89d4ff3a01c6 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.643539 26408 tablet_replica.cc:333] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.644011 26408 raft_consensus.cc:2243] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.644346 26408 raft_consensus.cc:2272] T 9acfb739951443068b4ff6d65ebb3691 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.646373 26408 tablet_replica.cc:333] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.646860 26408 raft_consensus.cc:2243] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.647222 26408 raft_consensus.cc:2272] T 11e5772eda0648d9a96d2b5755c23569 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.649044 26408 tablet_replica.cc:333] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.649619 26408 raft_consensus.cc:2243] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.650027 26408 raft_consensus.cc:2272] T ac21f3e309014f859e6995b20e5414bb P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.652068 26408 tablet_replica.cc:333] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.652561 26408 raft_consensus.cc:2243] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.652964 26408 raft_consensus.cc:2272] T 76060914d3af47e6ad7cb572ad9cc073 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.654765 26408 tablet_replica.cc:333] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.655277 26408 raft_consensus.cc:2243] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.655721 26408 raft_consensus.cc:2272] T c9515a897d574915ba4d8a478406aa1a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.657534 26408 tablet_replica.cc:333] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.658143 26408 raft_consensus.cc:2243] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.658572 26408 raft_consensus.cc:2272] T 109883acf19d4de4b90dfea6f147d368 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.660320 26408 tablet_replica.cc:333] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.660766 26408 raft_consensus.cc:2243] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.661116 26408 raft_consensus.cc:2272] T f950306b3d994137989bf2f2f5512441 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.664094 26408 tablet_replica.cc:333] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.664533 26408 raft_consensus.cc:2243] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.664883 26408 raft_consensus.cc:2272] T ccf9f64ab9404078902886a07eef6a66 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.666608 26408 tablet_replica.cc:333] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.667065 26408 raft_consensus.cc:2243] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.667496 26408 raft_consensus.cc:2272] T b303f27117804b4c8dc4ef4e31e0384e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.669246 26408 tablet_replica.cc:333] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.669699 26408 raft_consensus.cc:2243] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.670035 26408 raft_consensus.cc:2272] T 1b2fa83e4cf5494ba63becbde8d026b2 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.671851 26408 tablet_replica.cc:333] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.672325 26408 raft_consensus.cc:2243] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.672664 26408 raft_consensus.cc:2272] T 51d19f01907340eb8dc56ed742606710 P a5f89c82eda14a9ca425fae398bae986 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.674495 26408 tablet_replica.cc:333] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.674939 26408 raft_consensus.cc:2243] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.675680 26408 raft_consensus.cc:2272] T 6dbb3c7d37504398aab89200647dab24 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.677551 26408 tablet_replica.cc:333] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.678102 26408 raft_consensus.cc:2243] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.678448 26408 raft_consensus.cc:2272] T ffb83e944aed43718284cb48f994efcc P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.680538 26408 tablet_replica.cc:333] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.681030 26408 raft_consensus.cc:2243] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.681392 26408 raft_consensus.cc:2272] T 47c1f0a1e81a4ff0a3652fafff1288b4 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.683102 26408 tablet_replica.cc:333] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.683601 26408 raft_consensus.cc:2243] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.684273 26408 raft_consensus.cc:2272] T a97a9ba3c90f464eacdc4e78d6a91f5e P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.686059 26408 tablet_replica.cc:333] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.686509 26408 raft_consensus.cc:2243] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.687222 26408 raft_consensus.cc:2272] T 84b0698af7d34bc9a3f26a3b75435b16 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.689066 26408 tablet_replica.cc:333] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.689522 26408 raft_consensus.cc:2243] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.690164 26408 raft_consensus.cc:2272] T fc204ad755be48768bf67ef7e84254bb P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.691917 26408 tablet_replica.cc:333] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.692394 26408 raft_consensus.cc:2243] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.692739 26408 raft_consensus.cc:2272] T b9508782ff1d4df7867468f207693333 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.694577 26408 tablet_replica.cc:333] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.695077 26408 raft_consensus.cc:2243] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.695518 26408 raft_consensus.cc:2272] T 4e722577ca9c4859ac9d6f335a1edfc9 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.697234 26408 tablet_replica.cc:333] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.697685 26408 raft_consensus.cc:2243] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.698382 26408 raft_consensus.cc:2272] T 30d357fc75ee4a55a448f070c32e6499 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.700134 26408 tablet_replica.cc:333] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.700574 26408 raft_consensus.cc:2243] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.701267 26408 raft_consensus.cc:2272] T 1976a69f7f174bce837050f43e2dd6fe P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.703230 26408 tablet_replica.cc:333] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.703832 26408 raft_consensus.cc:2243] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.704564 26408 raft_consensus.cc:2272] T 8c302268ead84f94a04993f710b731e6 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.706401 26408 tablet_replica.cc:333] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.706889 26408 raft_consensus.cc:2243] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.707640 26408 raft_consensus.cc:2272] T 160804417e4443b9b25105c759111c8b P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.709558 26408 tablet_replica.cc:333] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.710057 26408 raft_consensus.cc:2243] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.710471 26408 raft_consensus.cc:2272] T 3cefb8e130fd43a1aba9f4773da6f272 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.712456 26408 tablet_replica.cc:333] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.712965 26408 raft_consensus.cc:2243] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.713755 26408 raft_consensus.cc:2272] T 8a891dab57c74222928537e93876be86 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.715592 26408 tablet_replica.cc:333] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.716102 26408 raft_consensus.cc:2243] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.716977 26408 raft_consensus.cc:2272] T 30315f436f8c4ef4b1ac5381d6a34811 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.719025 26408 tablet_replica.cc:333] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.719605 26408 raft_consensus.cc:2243] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.720039 26408 raft_consensus.cc:2272] T 77b8b272744248da9eb46aef40c57d9a P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.721988 26408 tablet_replica.cc:333] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.722529 26408 raft_consensus.cc:2243] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.723385 26408 raft_consensus.cc:2272] T c26afa4025e24ea192d424c506f40c56 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.725299 26408 tablet_replica.cc:333] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.725708 26408 raft_consensus.cc:2243] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.726060 26408 raft_consensus.cc:2272] T 890c806dd6f94382ba4e14a3d3bc6f85 P a5f89c82eda14a9ca425fae398bae986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.728083 26408 tablet_replica.cc:333] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.728518 26408 raft_consensus.cc:2243] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:25:39.728861  9494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.729275 26408 raft_consensus.cc:2272] T 85baaa21d67f43da9f328b0b51a71215 P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.731380 26408 tablet_replica.cc:333] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986: stopping tablet replica
I20260709 12:25:39.731794 26408 raft_consensus.cc:2243] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:39.732160 26408 raft_consensus.cc:2272] T 19cf5127045948269927c8b47835bb6e P a5f89c82eda14a9ca425fae398bae986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.804003 26408 tablet_server.cc:196] TabletServer@127.25.202.4:0 shutdown complete.
I20260709 12:25:39.870443 26408 master.cc:562] Master@127.25.202.62:39287 shutting down...
I20260709 12:25:39.897626 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:39.898173 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:39.898465 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3de727ba387e472ab4c4baca2c3bd906: stopping tablet replica
I20260709 12:25:39.922093 26408 master.cc:584] Master@127.25.202.62:39287 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (32219 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:25:40.010134 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:40737
I20260709 12:25:40.011533 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:40.017817 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.017815 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.018357 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:40.018674 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:40.020212 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:40.020484 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:40.020665 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599940020649 us; error 0 us; skew 500 ppm
I20260709 12:25:40.021342 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:40.024015 26408 webserver.cc:533] Webserver started at http://127.25.202.62:44145/ using document root <none> and password file <none>
I20260709 12:25:40.024538 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:40.024756 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:40.025044 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:40.026304 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "c473a04b800d4d78a9a2becfd9a41d31"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.031296 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:25:40.035218 10452 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.036100 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20260709 12:25:40.036417 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "c473a04b800d4d78a9a2becfd9a41d31"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.036767 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:40.049263 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:40.050565 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:40.091840 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:40737
I20260709 12:25:40.091956 10503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:40737 every 8 connection(s)
I20260709 12:25:40.096168 10504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.106863 10504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31: Bootstrap starting.
I20260709 12:25:40.111685 10504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:40.115854 10504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31: No bootstrap required, opened a new log
I20260709 12:25:40.118041 10504 raft_consensus.cc:359] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c473a04b800d4d78a9a2becfd9a41d31" member_type: VOTER }
I20260709 12:25:40.118402 10504 raft_consensus.cc:385] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:40.118578 10504 raft_consensus.cc:740] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c473a04b800d4d78a9a2becfd9a41d31, State: Initialized, Role: FOLLOWER
I20260709 12:25:40.119087 10504 consensus_queue.cc:260] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c473a04b800d4d78a9a2becfd9a41d31" member_type: VOTER }
I20260709 12:25:40.119551 10504 raft_consensus.cc:399] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:25:40.119740 10504 raft_consensus.cc:493] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:25:40.119941 10504 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:40.124496 10504 raft_consensus.cc:515] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c473a04b800d4d78a9a2becfd9a41d31" member_type: VOTER }
I20260709 12:25:40.124959 10504 leader_election.cc:304] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c473a04b800d4d78a9a2becfd9a41d31; no voters: 
I20260709 12:25:40.126173 10504 leader_election.cc:290] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:25:40.126557 10507 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:40.127985 10507 raft_consensus.cc:697] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 1 LEADER]: Becoming Leader. State: Replica: c473a04b800d4d78a9a2becfd9a41d31, State: Running, Role: LEADER
I20260709 12:25:40.128710 10507 consensus_queue.cc:237] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c473a04b800d4d78a9a2becfd9a41d31" member_type: VOTER }
I20260709 12:25:40.129288 10504 sys_catalog.cc:565] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:25:40.133842 10509 sys_catalog.cc:455] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c473a04b800d4d78a9a2becfd9a41d31. Latest consensus state: current_term: 1 leader_uuid: "c473a04b800d4d78a9a2becfd9a41d31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c473a04b800d4d78a9a2becfd9a41d31" member_type: VOTER } }
I20260709 12:25:40.134647 10509 sys_catalog.cc:458] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:40.135447 10508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c473a04b800d4d78a9a2becfd9a41d31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c473a04b800d4d78a9a2becfd9a41d31" member_type: VOTER } }
I20260709 12:25:40.136097 10508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:40.137574 10515 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:25:40.143004 10515 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:25:40.144840 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:25:40.152115 10515 catalog_manager.cc:1383] Generated new cluster ID: 81d0106600af472c82f1cb5ad6295c20
I20260709 12:25:40.152470 10515 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:25:40.214315 10515 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:25:40.215932 10515 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:25:40.240691 10515 catalog_manager.cc:6070] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31: Generated new TSK 0
I20260709 12:25:40.241345 10515 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:25:40.276063 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:40.282168 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.283177 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.285027 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:40.285281 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:40.286398 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:40.286607 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:40.286746 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599940286729 us; error 0 us; skew 500 ppm
I20260709 12:25:40.287281 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:40.289834 26408 webserver.cc:533] Webserver started at http://127.25.202.1:36227/ using document root <none> and password file <none>
I20260709 12:25:40.290309 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:40.290485 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:40.290711 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:40.291885 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "717e7628567145a6864ece4fe7048c5a"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.296626 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:25:40.300328 10533 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.301122 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20260709 12:25:40.301450 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "717e7628567145a6864ece4fe7048c5a"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.301748 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:40.320180 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:40.321552 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:40.323289 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:40.325838 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:40.326069 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.326387 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:40.326584 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.370235 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:34205
I20260709 12:25:40.370322 10595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:34205 every 8 connection(s)
I20260709 12:25:40.375829 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:40.384994 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.385543 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.388300 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:40.389150 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:40.390668 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:40.390913 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:40.391090 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599940391073 us; error 0 us; skew 500 ppm
I20260709 12:25:40.391682 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:40.393771 10596 heartbeater.cc:344] Connected to a master server at 127.25.202.62:40737
I20260709 12:25:40.394260 10596 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:40.394424 26408 webserver.cc:533] Webserver started at http://127.25.202.2:43701/ using document root <none> and password file <none>
I20260709 12:25:40.395102 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:40.395121 10596 heartbeater.cc:507] Master 127.25.202.62:40737 requested a full tablet report, sending...
I20260709 12:25:40.395488 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:40.395843 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:40.397311 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "8de5f07284784aeaabd73e28f56e1c79"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.397701 10469 ts_manager.cc:194] Registered new tserver with Master: 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:40.399632 10469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59940
I20260709 12:25:40.402374 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:25:40.406121 10608 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.406862 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:25:40.407153 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "8de5f07284784aeaabd73e28f56e1c79"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.407526 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:40.420835 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:40.422029 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:40.423617 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:40.425792 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:40.426015 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.426249 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:40.426416 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.472723 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:37453
I20260709 12:25:40.472851 10670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:37453 every 8 connection(s)
I20260709 12:25:40.478592 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:40.487735 10674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:40.488314 10675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:40.492341 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:40.492758 10671 heartbeater.cc:344] Connected to a master server at 127.25.202.62:40737
W20260709 12:25:40.493114 10677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:40.493269 10671 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:40.494359 10671 heartbeater.cc:507] Master 127.25.202.62:40737 requested a full tablet report, sending...
I20260709 12:25:40.494359 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:40.494827 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:40.495041 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599940495026 us; error 0 us; skew 500 ppm
I20260709 12:25:40.495760 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:40.496975 10469 ts_manager.cc:194] Registered new tserver with Master: 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:40.499106 10469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59944
I20260709 12:25:40.499876 26408 webserver.cc:533] Webserver started at http://127.25.202.3:44349/ using document root <none> and password file <none>
I20260709 12:25:40.500550 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:40.500787 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:40.501106 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:40.502460 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.507221 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.002s
I20260709 12:25:40.510859 10682 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.511700 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:25:40.512204 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:40.512524 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:40.529675 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:40.531203 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:40.532893 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:40.535156 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:40.535440 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:25:40.535718 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:40.535930 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:40.580850 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:45381
I20260709 12:25:40.580941 10744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:45381 every 8 connection(s)
I20260709 12:25:40.596483 10745 heartbeater.cc:344] Connected to a master server at 127.25.202.62:40737
I20260709 12:25:40.596890 10745 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:40.597608 10745 heartbeater.cc:507] Master 127.25.202.62:40737 requested a full tablet report, sending...
I20260709 12:25:40.600454 10469 ts_manager.cc:194] Registered new tserver with Master: 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:40.600623 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016182059s
I20260709 12:25:40.602389 10469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59946
I20260709 12:25:40.629333 10469 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59956:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:25:40.632453 10469 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:25:40.706940 10636 tablet_service.cc:1511] Processing CreateTablet for tablet dff18e1c3f7a44959883a673cdc34ff8 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:25:40.708521 10636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff18e1c3f7a44959883a673cdc34ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.717401 10710 tablet_service.cc:1511] Processing CreateTablet for tablet dff18e1c3f7a44959883a673cdc34ff8 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:25:40.717653 10635 tablet_service.cc:1511] Processing CreateTablet for tablet d641fa1dbe914a78940cf158d8bc11b3 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:25:40.718963 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff18e1c3f7a44959883a673cdc34ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.720425 10634 tablet_service.cc:1511] Processing CreateTablet for tablet cb8f9a133d8d43babaed9f7d4e4e47c4 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:25:40.721520 10632 tablet_service.cc:1511] Processing CreateTablet for tablet aeeb02ed89f74546808cfc4efe82c88e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:25:40.722895 10633 tablet_service.cc:1511] Processing CreateTablet for tablet c2bbbde330214551a52fa8f29ebe4a7e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:25:40.719138 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d641fa1dbe914a78940cf158d8bc11b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.724484 10709 tablet_service.cc:1511] Processing CreateTablet for tablet d641fa1dbe914a78940cf158d8bc11b3 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:25:40.725840 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d641fa1dbe914a78940cf158d8bc11b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.725692 10631 tablet_service.cc:1511] Processing CreateTablet for tablet a30865ce1f1a443b9b0e21d0b2cfb8e3 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:25:40.727638 10631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30865ce1f1a443b9b0e21d0b2cfb8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.728720 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeb02ed89f74546808cfc4efe82c88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.728570 10708 tablet_service.cc:1511] Processing CreateTablet for tablet cb8f9a133d8d43babaed9f7d4e4e47c4 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:25:40.729925 10708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8f9a133d8d43babaed9f7d4e4e47c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.730504 10634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8f9a133d8d43babaed9f7d4e4e47c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.734436 10630 tablet_service.cc:1511] Processing CreateTablet for tablet 65c4a359537b43cd988c02dbeba52b72 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:25:40.735177 10707 tablet_service.cc:1511] Processing CreateTablet for tablet c2bbbde330214551a52fa8f29ebe4a7e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:25:40.735924 10630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c4a359537b43cd988c02dbeba52b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.736550 10707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2bbbde330214551a52fa8f29ebe4a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.739547 10706 tablet_service.cc:1511] Processing CreateTablet for tablet aeeb02ed89f74546808cfc4efe82c88e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:25:40.740723 10628 tablet_service.cc:1511] Processing CreateTablet for tablet 2b0e3a873b754d0f8041302cb17c816b (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:25:40.742255 10628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0e3a873b754d0f8041302cb17c816b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.742180 10705 tablet_service.cc:1511] Processing CreateTablet for tablet a30865ce1f1a443b9b0e21d0b2cfb8e3 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:25:40.744613 10629 tablet_service.cc:1511] Processing CreateTablet for tablet 22842ec864b84edfb1acd6fe4ed6d63e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:25:40.740972 10706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeb02ed89f74546808cfc4efe82c88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.745155 10627 tablet_service.cc:1511] Processing CreateTablet for tablet 13ce22dbb8e944dabd5c081c64245252 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:25:40.744659 10704 tablet_service.cc:1511] Processing CreateTablet for tablet 65c4a359537b43cd988c02dbeba52b72 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:25:40.747797 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30865ce1f1a443b9b0e21d0b2cfb8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.751726 10704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c4a359537b43cd988c02dbeba52b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.759802 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22842ec864b84edfb1acd6fe4ed6d63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.762041 10703 tablet_service.cc:1511] Processing CreateTablet for tablet 22842ec864b84edfb1acd6fe4ed6d63e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:25:40.763450 10627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ce22dbb8e944dabd5c081c64245252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.765594 10702 tablet_service.cc:1511] Processing CreateTablet for tablet 2b0e3a873b754d0f8041302cb17c816b (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:25:40.767872 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2bbbde330214551a52fa8f29ebe4a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.767975 10703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22842ec864b84edfb1acd6fe4ed6d63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.772794 10760 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:40.775940 10702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0e3a873b754d0f8041302cb17c816b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.787642 10760 tablet_bootstrap.cc:654] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:40.822395 10561 tablet_service.cc:1511] Processing CreateTablet for tablet dff18e1c3f7a44959883a673cdc34ff8 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:25:40.824235 10561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff18e1c3f7a44959883a673cdc34ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.828642 10560 tablet_service.cc:1511] Processing CreateTablet for tablet d641fa1dbe914a78940cf158d8bc11b3 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:25:40.830066 10560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d641fa1dbe914a78940cf158d8bc11b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.832638 10559 tablet_service.cc:1511] Processing CreateTablet for tablet cb8f9a133d8d43babaed9f7d4e4e47c4 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:25:40.834033 10559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8f9a133d8d43babaed9f7d4e4e47c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.837697 10555 tablet_service.cc:1511] Processing CreateTablet for tablet 65c4a359537b43cd988c02dbeba52b72 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:25:40.838013 10556 tablet_service.cc:1511] Processing CreateTablet for tablet a30865ce1f1a443b9b0e21d0b2cfb8e3 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:25:40.839151 10555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c4a359537b43cd988c02dbeba52b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.837733 10554 tablet_service.cc:1511] Processing CreateTablet for tablet 22842ec864b84edfb1acd6fe4ed6d63e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:25:40.841153 10557 tablet_service.cc:1511] Processing CreateTablet for tablet aeeb02ed89f74546808cfc4efe82c88e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:25:40.843015 10557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeb02ed89f74546808cfc4efe82c88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.844085 10558 tablet_service.cc:1511] Processing CreateTablet for tablet c2bbbde330214551a52fa8f29ebe4a7e (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:25:40.850207 10761 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:40.856230 10554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22842ec864b84edfb1acd6fe4ed6d63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.848349 10552 tablet_service.cc:1511] Processing CreateTablet for tablet 13ce22dbb8e944dabd5c081c64245252 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:25:40.869592 10552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ce22dbb8e944dabd5c081c64245252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.850951 10553 tablet_service.cc:1511] Processing CreateTablet for tablet 2b0e3a873b754d0f8041302cb17c816b (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:25:40.872834 10706 tablet_service.cc:1511] Processing CreateTablet for tablet 13ce22dbb8e944dabd5c081c64245252 (DEFAULT_TABLE table=test-workload [id=3d2c7c35eaaa4010b212e13b69295407]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:25:40.873742 10558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2bbbde330214551a52fa8f29ebe4a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.874238 10706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ce22dbb8e944dabd5c081c64245252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.875960 10556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30865ce1f1a443b9b0e21d0b2cfb8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.887843 10553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0e3a873b754d0f8041302cb17c816b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:40.900980 10761 tablet_bootstrap.cc:654] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:40.909067 10760 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:40.909575 10760 ts_tablet_manager.cc:1403] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.137s	user 0.017s	sys 0.040s
I20260709 12:25:40.915093 10760 raft_consensus.cc:359] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:40.915856 10760 raft_consensus.cc:385] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:40.916160 10760 raft_consensus.cc:740] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:40.916406 10763 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:40.917100 10760 consensus_queue.cc:260] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:40.928660 10763 tablet_bootstrap.cc:654] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:40.931768 10761 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:40.932354 10761 ts_tablet_manager.cc:1403] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.083s	user 0.008s	sys 0.005s
I20260709 12:25:40.936471 10761 raft_consensus.cc:359] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:40.937253 10761 raft_consensus.cc:385] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:40.938443 10761 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:40.939473 10761 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:40.945665 10760 ts_tablet_manager.cc:1434] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.036s	user 0.023s	sys 0.009s
I20260709 12:25:40.946830 10745 heartbeater.cc:499] Master 127.25.202.62:40737 was elected leader, sending a full tablet report...
I20260709 12:25:40.957031 10760 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:40.960116 10671 heartbeater.cc:499] Master 127.25.202.62:40737 was elected leader, sending a full tablet report...
I20260709 12:25:40.964464 10760 tablet_bootstrap.cc:654] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:40.974571 10761 ts_tablet_manager.cc:1434] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.042s	user 0.012s	sys 0.008s
I20260709 12:25:40.976224 10761 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
W20260709 12:25:40.977798 10672 tablet.cc:2406] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:25:40.979955 10766 raft_consensus.cc:493] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.980726 10766 raft_consensus.cc:515] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:40.993572 10766 leader_election.cc:290] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:40.995030 10761 tablet_bootstrap.cc:654] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.000222 10760 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.000759 10763 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.001307 10763 ts_tablet_manager.cc:1403] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.085s	user 0.019s	sys 0.023s
I20260709 12:25:41.000782 10760 ts_tablet_manager.cc:1403] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.044s	user 0.005s	sys 0.007s
I20260709 12:25:41.006913 10760 raft_consensus.cc:359] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.007673 10760 raft_consensus.cc:385] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.007997 10760 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.011090 10763 raft_consensus.cc:359] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.011895 10763 raft_consensus.cc:385] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.012204 10763 raft_consensus.cc:740] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.012974 10763 consensus_queue.cc:260] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.014341 10760 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.016806 10760 ts_tablet_manager.cc:1434] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:25:41.017845 10760 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.028079 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.028576 10760 tablet_bootstrap.cc:654] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
W20260709 12:25:41.030031 10610 leader_election.cc:343] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.030195 10761 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.031260 10761 ts_tablet_manager.cc:1403] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.055s	user 0.021s	sys 0.016s
I20260709 12:25:41.031971 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.032792 10596 heartbeater.cc:499] Master 127.25.202.62:40737 was elected leader, sending a full tablet report...
I20260709 12:25:41.033320 10763 ts_tablet_manager.cc:1434] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.006s
I20260709 12:25:41.034343 10763 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.034756 10761 raft_consensus.cc:359] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.035557 10761 raft_consensus.cc:385] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.035874 10761 raft_consensus.cc:740] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
W20260709 12:25:41.036240 10611 leader_election.cc:343] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.036761 10611 leader_election.cc:304] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de5f07284784aeaabd73e28f56e1c79; no voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:41.036623 10761 consensus_queue.cc:260] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.037797 10766 raft_consensus.cc:2749] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:41.040135 10761 ts_tablet_manager.cc:1434] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 12:25:41.041224 10761 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.042330 10763 tablet_bootstrap.cc:654] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.048632 10761 tablet_bootstrap.cc:654] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.051775 10760 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.052428 10760 ts_tablet_manager.cc:1403] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.035s	user 0.012s	sys 0.000s
I20260709 12:25:41.055738 10763 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.056336 10763 ts_tablet_manager.cc:1403] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.005s
I20260709 12:25:41.056473 10760 raft_consensus.cc:359] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.057521 10760 raft_consensus.cc:385] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.057807 10760 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.058974 10760 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.059872 10763 raft_consensus.cc:359] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.060683 10763 raft_consensus.cc:385] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.060966 10763 raft_consensus.cc:740] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.061746 10763 consensus_queue.cc:260] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.064203 10763 ts_tablet_manager.cc:1434] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:41.065246 10763 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.069096 10765 raft_consensus.cc:493] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.069644 10765 raft_consensus.cc:515] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.071302 10763 tablet_bootstrap.cc:654] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.072428 10765 leader_election.cc:290] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.072975 10765 raft_consensus.cc:493] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.073534 10760 ts_tablet_manager.cc:1434] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.002s
I20260709 12:25:41.073432 10765 raft_consensus.cc:515] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.075127 10760 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.084774 10766 raft_consensus.cc:493] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.085489 10766 raft_consensus.cc:515] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.084980 10760 tablet_bootstrap.cc:654] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.098376 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.099354 10571 raft_consensus.cc:2468] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.100715 10766 leader_election.cc:290] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381), 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:41.101735 10611 leader_election.cc:304] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.105252 10766 raft_consensus.cc:2804] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.105729 10766 raft_consensus.cc:493] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.106117 10766 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.116387 10765 leader_election.cc:290] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.124505 10761 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.127877 10761 ts_tablet_manager.cc:1403] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.087s	user 0.023s	sys 0.023s
I20260709 12:25:41.131062 10761 raft_consensus.cc:359] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.136643 10761 raft_consensus.cc:385] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.137677 10761 raft_consensus.cc:740] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.140508 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.141460 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
W20260709 12:25:41.140488 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:41.142721 10761 consensus_queue.cc:260] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.144240 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.145115 10571 raft_consensus.cc:2468] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
W20260709 12:25:41.146813 10610 leader_election.cc:343] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.152457 10761 ts_tablet_manager.cc:1434] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.002s
W20260709 12:25:41.152793 10685 leader_election.cc:343] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.153633 10761 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.152457 10766 raft_consensus.cc:515] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.153520 10685 leader_election.cc:304] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.156566 10765 raft_consensus.cc:2804] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.156956 10765 raft_consensus.cc:493] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.157315 10765 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.168095 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
W20260709 12:25:41.169499 10610 leader_election.cc:343] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.171181 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.171948 10646 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
I20260709 12:25:41.174263 10780 raft_consensus.cc:493] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.174790 10780 raft_consensus.cc:515] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.177806 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.178712 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
W20260709 12:25:41.179612 10611 leader_election.cc:343] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.180014 10685 leader_election.cc:304] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
W20260709 12:25:41.181967 10685 leader_election.cc:343] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.183190 10776 raft_consensus.cc:2804] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.183822 10780 leader_election.cc:290] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.183657 10776 raft_consensus.cc:493] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.184537 10776 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.189561 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.190217 10571 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.190375 10761 tablet_bootstrap.cc:654] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.195468 10766 leader_election.cc:290] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Requested vote from peers 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381), 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:41.199944 10761 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.200590 10761 ts_tablet_manager.cc:1403] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.000s
I20260709 12:25:41.202169 10760 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.202790 10760 ts_tablet_manager.cc:1403] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.128s	user 0.057s	sys 0.065s
I20260709 12:25:41.203090 10763 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.203651 10763 ts_tablet_manager.cc:1403] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.139s	user 0.059s	sys 0.060s
I20260709 12:25:41.203923 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.206316 10760 raft_consensus.cc:359] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.207145 10760 raft_consensus.cc:385] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.206868 10763 raft_consensus.cc:359] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.207710 10763 raft_consensus.cc:385] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.207566 10760 raft_consensus.cc:740] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.208149 10763 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
W20260709 12:25:41.207410 10610 leader_election.cc:343] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.209012 10760 consensus_queue.cc:260] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.209200 10610 leader_election.cc:304] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de5f07284784aeaabd73e28f56e1c79; no voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:41.209164 10761 raft_consensus.cc:359] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.210523 10761 raft_consensus.cc:385] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.210850 10766 raft_consensus.cc:2749] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:41.209244 10763 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.210927 10761 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.216217 10763 ts_tablet_manager.cc:1434] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.004s
I20260709 12:25:41.217487 10783 raft_consensus.cc:493] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.218075 10761 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.218721 10763 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.218256 10783 raft_consensus.cc:515] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.220602 10761 ts_tablet_manager.cc:1434] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 12:25:41.220664 10783 leader_election.cc:290] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.225597 10760 ts_tablet_manager.cc:1434] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.022s	user 0.001s	sys 0.012s
I20260709 12:25:41.225858 10763 tablet_bootstrap.cc:654] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.226907 10760 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.227226 10761 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.202724 10765 raft_consensus.cc:515] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.232949 10571 raft_consensus.cc:2468] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
I20260709 12:25:41.234424 10611 leader_election.cc:304] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:41.235111 10760 tablet_bootstrap.cc:654] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.236392 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.237093 10646 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
I20260709 12:25:41.234671 10761 tablet_bootstrap.cc:654] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.238608 10781 raft_consensus.cc:2804] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.239950 10781 raft_consensus.cc:697] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:41.241163 10781 consensus_queue.cc:237] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.243836 10765 leader_election.cc:290] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.246320 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.247072 10570 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.251569 10763 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.251888 10761 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.252205 10763 ts_tablet_manager.cc:1403] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.034s	user 0.011s	sys 0.008s
I20260709 12:25:41.252450 10761 ts_tablet_manager.cc:1403] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.000s
I20260709 12:25:41.252681 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.254164 10685 leader_election.cc:304] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.255692 10765 raft_consensus.cc:2804] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.255484 10763 raft_consensus.cc:359] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.256214 10765 raft_consensus.cc:493] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.256407 10763 raft_consensus.cc:385] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.256726 10765 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.255497 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.258785 10763 raft_consensus.cc:740] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.261400 10760 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.260761 10761 raft_consensus.cc:359] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.260766 10763 consensus_queue.cc:260] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.261958 10761 raft_consensus.cc:385] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.262266 10760 ts_tablet_manager.cc:1403] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.036s	user 0.017s	sys 0.003s
I20260709 12:25:41.262324 10761 raft_consensus.cc:740] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.263768 10761 consensus_queue.cc:260] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.265101 10765 raft_consensus.cc:515] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.266390 10761 ts_tablet_manager.cc:1434] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:25:41.267505 10761 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.267704 10776 raft_consensus.cc:515] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.270427 10776 leader_election.cc:290] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.267189 10765 leader_election.cc:290] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
W20260709 12:25:41.270429 10685 leader_election.cc:343] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.273818 10763 ts_tablet_manager.cc:1434] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.001s
I20260709 12:25:41.270525 10570 raft_consensus.cc:2468] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.275039 10763 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
W20260709 12:25:41.275101 10685 leader_election.cc:343] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.276569 10685 leader_election.cc:304] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:41.278220 10761 tablet_bootstrap.cc:654] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.278888 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.273100 10760 raft_consensus.cc:359] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.280665 10760 raft_consensus.cc:385] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.281399 10760 raft_consensus.cc:740] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.283741 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.284519 10646 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.284729 10760 consensus_queue.cc:260] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.287207 10763 tablet_bootstrap.cc:654] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.288400 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.290474 10645 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.292387 10760 ts_tablet_manager.cc:1434] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.001s
I20260709 12:25:41.286087 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.293708 10570 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.292685 10467 catalog_manager.cc:5697] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.294487 10646 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.295985 10776 raft_consensus.cc:2804] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.298142 10776 raft_consensus.cc:697] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:41.298403 10645 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.299657 10760 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.299474 10776 consensus_queue.cc:237] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
W20260709 12:25:41.289816 10685 leader_election.cc:343] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:41.303349 10685 leader_election.cc:304] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:41.304991 10685 leader_election.cc:304] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.305334 10765 raft_consensus.cc:2804] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.305931 10765 raft_consensus.cc:697] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:41.306066 10784 raft_consensus.cc:2804] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.306722 10784 raft_consensus.cc:697] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:41.307680 10784 consensus_queue.cc:237] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.306840 10765 consensus_queue.cc:237] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.316603 10760 tablet_bootstrap.cc:654] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.318408 10570 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.319070 10761 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.319669 10761 ts_tablet_manager.cc:1403] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.052s	user 0.017s	sys 0.008s
I20260709 12:25:41.320631 10763 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.321208 10763 ts_tablet_manager.cc:1403] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.046s	user 0.009s	sys 0.005s
I20260709 12:25:41.322896 10761 raft_consensus.cc:359] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.323721 10761 raft_consensus.cc:385] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.324122 10761 raft_consensus.cc:740] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.325050 10761 consensus_queue.cc:260] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.325265 10468 catalog_manager.cc:5697] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.327495 10761 ts_tablet_manager.cc:1434] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20260709 12:25:41.328578 10761 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.329846 10763 raft_consensus.cc:359] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.331938 10763 raft_consensus.cc:385] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.332321 10763 raft_consensus.cc:740] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.333585 10760 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.333339 10763 consensus_queue.cc:260] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.334160 10760 ts_tablet_manager.cc:1403] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.002s
I20260709 12:25:41.336086 10763 ts_tablet_manager.cc:1434] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.004s
I20260709 12:25:41.336274 10761 tablet_bootstrap.cc:654] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.338075 10760 raft_consensus.cc:359] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.338934 10760 raft_consensus.cc:385] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.339346 10760 raft_consensus.cc:740] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.340101 10763 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.340193 10760 consensus_queue.cc:260] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.343020 10760 ts_tablet_manager.cc:1434] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 12:25:41.344151 10760 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.348397 10763 tablet_bootstrap.cc:654] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.351123 10760 tablet_bootstrap.cc:654] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.374969 10761 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.375545 10761 ts_tablet_manager.cc:1403] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.047s	user 0.023s	sys 0.023s
I20260709 12:25:41.381471 10761 raft_consensus.cc:359] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.382261 10761 raft_consensus.cc:385] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.382586 10761 raft_consensus.cc:740] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.383106 10763 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.383600 10781 raft_consensus.cc:493] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.383807 10760 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.383849 10763 ts_tablet_manager.cc:1403] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.028s
I20260709 12:25:41.384550 10760 ts_tablet_manager.cc:1403] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.041s	user 0.020s	sys 0.013s
I20260709 12:25:41.384147 10781 raft_consensus.cc:515] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.386591 10781 leader_election.cc:290] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.387758 10760 raft_consensus.cc:359] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.388584 10760 raft_consensus.cc:385] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.388897 10760 raft_consensus.cc:740] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.388934 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.389729 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.389853 10570 raft_consensus.cc:2393] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de5f07284784aeaabd73e28f56e1c79 in current term 1: Already voted for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in this term.
I20260709 12:25:41.389986 10760 consensus_queue.cc:260] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.390488 10761 consensus_queue.cc:260] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.391767 10611 leader_election.cc:304] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de5f07284784aeaabd73e28f56e1c79; no voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:41.392762 10760 ts_tablet_manager.cc:1434] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:25:41.393139 10781 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.393340 10761 ts_tablet_manager.cc:1434] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:25:41.393915 10760 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.394457 10761 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.401580 10761 tablet_bootstrap.cc:654] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.401572 10760 tablet_bootstrap.cc:654] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.388041 10763 raft_consensus.cc:359] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.402277 10763 raft_consensus.cc:385] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.402702 10763 raft_consensus.cc:740] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.403990 10763 consensus_queue.cc:260] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.401686 10781 raft_consensus.cc:2749] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:41.413513 10761 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.414068 10761 ts_tablet_manager.cc:1403] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.020s	user 0.016s	sys 0.001s
I20260709 12:25:41.416540 10763 ts_tablet_manager.cc:1434] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.001s
I20260709 12:25:41.416865 10761 raft_consensus.cc:359] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.417663 10761 raft_consensus.cc:385] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.418136 10761 raft_consensus.cc:740] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.419134 10760 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.418962 10761 consensus_queue.cc:260] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.419708 10760 ts_tablet_manager.cc:1403] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.004s
I20260709 12:25:41.421044 10761 ts_tablet_manager.cc:1434] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:25:41.422031 10761 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.417635 10466 catalog_manager.cc:5697] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.422640 10760 raft_consensus.cc:359] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.423408 10760 raft_consensus.cc:385] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.423702 10760 raft_consensus.cc:740] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.423841 10466 catalog_manager.cc:5697] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.424687 10760 consensus_queue.cc:260] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.427049 10760 ts_tablet_manager.cc:1434] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:41.428066 10760 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.428566 10763 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.428725 10761 tablet_bootstrap.cc:654] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.434122 10760 tablet_bootstrap.cc:654] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.436501 10763 tablet_bootstrap.cc:654] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.439919 10761 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.440590 10761 ts_tablet_manager.cc:1403] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.019s	user 0.017s	sys 0.000s
I20260709 12:25:41.443289 10761 raft_consensus.cc:359] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.444049 10761 raft_consensus.cc:385] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.444334 10761 raft_consensus.cc:740] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.445176 10761 consensus_queue.cc:260] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.447556 10761 ts_tablet_manager.cc:1434] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260709 12:25:41.448559 10761 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: Bootstrap starting.
I20260709 12:25:41.449458 10760 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.449976 10760 ts_tablet_manager.cc:1403] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.007s
I20260709 12:25:41.452785 10760 raft_consensus.cc:359] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.453399 10761 tablet_bootstrap.cc:654] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.453510 10760 raft_consensus.cc:385] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.453898 10760 raft_consensus.cc:740] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.454612 10760 consensus_queue.cc:260] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.457145 10760 ts_tablet_manager.cc:1434] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:25:41.460669 10760 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34: Bootstrap starting.
I20260709 12:25:41.461849 10761 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: No bootstrap required, opened a new log
I20260709 12:25:41.462481 10761 ts_tablet_manager.cc:1403] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.004s
I20260709 12:25:41.464113 10763 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.464795 10763 ts_tablet_manager.cc:1403] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.042s	user 0.014s	sys 0.005s
I20260709 12:25:41.465456 10761 raft_consensus.cc:359] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.466243 10761 raft_consensus.cc:385] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.466552 10761 raft_consensus.cc:740] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.466676 10760 tablet_bootstrap.cc:654] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.467394 10761 consensus_queue.cc:260] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.468487 10763 raft_consensus.cc:359] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.469375 10763 raft_consensus.cc:385] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.469671 10761 ts_tablet_manager.cc:1434] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:25:41.469825 10763 raft_consensus.cc:740] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.471251 10763 consensus_queue.cc:260] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.475104 10763 ts_tablet_manager.cc:1434] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.004s
I20260709 12:25:41.476262 10763 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.478477 10760 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34: No bootstrap required, opened a new log
I20260709 12:25:41.479055 10760 ts_tablet_manager.cc:1403] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.004s
I20260709 12:25:41.483107 10763 tablet_bootstrap.cc:654] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.486871 10760 raft_consensus.cc:359] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.487720 10760 raft_consensus.cc:385] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.488026 10760 raft_consensus.cc:740] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.488788 10760 consensus_queue.cc:260] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.490978 10760 ts_tablet_manager.cc:1434] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
I20260709 12:25:41.491413 10763 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.491937 10763 ts_tablet_manager.cc:1403] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.005s
I20260709 12:25:41.494009 10763 raft_consensus.cc:359] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.494501 10763 raft_consensus.cc:385] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.494722 10763 raft_consensus.cc:740] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.495497 10763 consensus_queue.cc:260] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.497218 10763 ts_tablet_manager.cc:1434] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:25:41.499172 10763 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.503901 10781 raft_consensus.cc:493] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.504479 10781 raft_consensus.cc:515] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.505208 10763 tablet_bootstrap.cc:654] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.507615 10781 leader_election.cc:290] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.507838 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.507606 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.508882 10570 raft_consensus.cc:2468] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.508891 10720 raft_consensus.cc:2468] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.510282 10772 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.510321 10611 leader_election.cc:304] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.510943 10772 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.511546 10781 raft_consensus.cc:2804] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.511991 10781 raft_consensus.cc:493] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.512275 10781 raft_consensus.cc:3060] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.516680 10772 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.517877 10772 raft_consensus.cc:493] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.518518 10772 raft_consensus.cc:515] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.521513 10772 leader_election.cc:290] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.522279 10781 raft_consensus.cc:515] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.527104 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.527762 10570 raft_consensus.cc:3060] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.527853 10763 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.528643 10763 ts_tablet_manager.cc:1403] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.030s	user 0.010s	sys 0.004s
I20260709 12:25:41.528831 10772 raft_consensus.cc:493] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.529390 10772 raft_consensus.cc:515] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.532502 10763 raft_consensus.cc:359] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.533046 10781 leader_election.cc:290] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.533190 10763 raft_consensus.cc:385] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.533281 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:41.533685 10763 raft_consensus.cc:740] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.534087 10720 raft_consensus.cc:3060] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.534451 10763 consensus_queue.cc:260] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.535939 10570 raft_consensus.cc:2468] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
I20260709 12:25:41.536571 10772 leader_election.cc:290] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.537220 10611 leader_election.cc:304] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.538482 10781 raft_consensus.cc:2804] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.540925 10781 raft_consensus.cc:697] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:41.542510 10781 consensus_queue.cc:237] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.545456 10772 raft_consensus.cc:493] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.545977 10772 raft_consensus.cc:515] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.547503 10766 raft_consensus.cc:493] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.548576 10772 leader_election.cc:290] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.548164 10766 raft_consensus.cc:515] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.549280 10780 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.549922 10780 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.550634 10766 leader_election.cc:290] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.552114 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.552871 10719 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.555238 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.555540 10610 leader_election.cc:304] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.556007 10571 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.555313 10765 raft_consensus.cc:493] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.557399 10765 raft_consensus.cc:515] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.557564 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.557436 10611 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.558344 10570 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.559815 10765 leader_election.cc:290] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.561195 10766 raft_consensus.cc:2804] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.561630 10766 raft_consensus.cc:493] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.561745 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.562033 10766 raft_consensus.cc:3060] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.562440 10570 raft_consensus.cc:2468] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
I20260709 12:25:41.563647 10685 leader_election.cc:304] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.565912 10720 raft_consensus.cc:2468] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
I20260709 12:25:41.552762 10780 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.568974 10780 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.569521 10780 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.570048 10780 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.570215 10715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.571902 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.572705 10718 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 0.
I20260709 12:25:41.574095 10716 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.574832 10716 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 0.
I20260709 12:25:41.575829 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.571072 10715 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 0.
I20260709 12:25:41.556694 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:41.567518 10765 raft_consensus.cc:2804] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.577584 10535 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.578117 10719 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 0.
I20260709 12:25:41.578128 10765 raft_consensus.cc:493] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.578608 10772 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.578979 10765 raft_consensus.cc:3060] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.579082 10772 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.580502 10772 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.571911 10766 raft_consensus.cc:515] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.584569 10766 leader_election.cc:290] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.585942 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:41.586001 10535 leader_election.cc:304] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.587569 10535 leader_election.cc:304] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.586649 10719 raft_consensus.cc:3060] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.590736 10816 raft_consensus.cc:2804] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.591140 10816 raft_consensus.cc:493] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.593964 10816 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.596091 10469 catalog_manager.cc:5697] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.598289 10719 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
I20260709 12:25:41.598417 10772 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.599970 10610 leader_election.cc:304] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.600808 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.601090 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.601785 10570 raft_consensus.cc:3060] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.602154 10772 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 election: Requested vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.602159 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.603080 10646 raft_consensus.cc:2468] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 0.
I20260709 12:25:41.603837 10814 raft_consensus.cc:2804] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.604704 10814 raft_consensus.cc:697] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:41.604954 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:41.606474 10719 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.606213 10814 consensus_queue.cc:237] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.609724 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.609933 10570 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
I20260709 12:25:41.610626 10772 raft_consensus.cc:2764] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:25:41.612653 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.614136 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.610646 10646 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 0.
I20260709 12:25:41.619354 10816 raft_consensus.cc:515] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.619959 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.620951 10644 raft_consensus.cc:2468] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
I20260709 12:25:41.621119 10536 leader_election.cc:304] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:41.622694 10772 raft_consensus.cc:2804] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.622749 10784 raft_consensus.cc:493] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.622959 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:41.623196 10772 raft_consensus.cc:493] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.623349 10784 raft_consensus.cc:515] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.623971 10772 raft_consensus.cc:3060] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.625661 10784 leader_election.cc:290] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.625847 10765 raft_consensus.cc:515] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.627115 10717 raft_consensus.cc:2393] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 717e7628567145a6864ece4fe7048c5a in current term 1: Already voted for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in this term.
I20260709 12:25:41.628235 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d641fa1dbe914a78940cf158d8bc11b3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.628949 10570 raft_consensus.cc:2468] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
I20260709 12:25:41.630265 10685 leader_election.cc:304] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.631138 10784 raft_consensus.cc:2804] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:41.631544 10784 raft_consensus.cc:493] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.631867 10784 raft_consensus.cc:3060] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.635812 10772 raft_consensus.cc:515] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.638350 10816 leader_election.cc:290] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 election: Requested vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.638742 10720 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.641721 10784 raft_consensus.cc:515] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.643867 10784 leader_election.cc:290] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.645012 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d641fa1dbe914a78940cf158d8bc11b3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.645645 10570 raft_consensus.cc:3060] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.645995 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:41.646771 10717 raft_consensus.cc:2393] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 717e7628567145a6864ece4fe7048c5a in current term 1: Already voted for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in this term.
I20260709 12:25:41.648108 10772 leader_election.cc:290] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 election: Requested vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.650470 10720 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 1.
I20260709 12:25:41.652118 10570 raft_consensus.cc:2468] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.652290 10535 leader_election.cc:304] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.653333 10685 leader_election.cc:304] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.653728 10772 raft_consensus.cc:2804] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.654224 10784 raft_consensus.cc:2804] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.654942 10772 raft_consensus.cc:697] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 LEADER]: Becoming Leader. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Running, Role: LEADER
I20260709 12:25:41.654520 10469 catalog_manager.cc:5697] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.656158 10784 raft_consensus.cc:697] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:41.656418 10772 consensus_queue.cc:237] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.657085 10784 consensus_queue.cc:237] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.619174 10780 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.670665 10645 raft_consensus.cc:2393] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 717e7628567145a6864ece4fe7048c5a in current term 1: Already voted for candidate 8de5f07284784aeaabd73e28f56e1c79 in this term.
I20260709 12:25:41.674875 10780 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:41.675570 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:41.676155 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.693950 10645 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.694435 10763 ts_tablet_manager.cc:1434] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.165s	user 0.035s	sys 0.103s
I20260709 12:25:41.695639 10763 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:41.696502 10783 consensus_queue.cc:1048] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:41.697579 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.698303 10642 raft_consensus.cc:3060] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.677654 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "717e7628567145a6864ece4fe7048c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.679774 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d641fa1dbe914a78940cf158d8bc11b3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.699620 10783 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:41.700326 10646 raft_consensus.cc:2468] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 0.
I20260709 12:25:41.681416 10643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d641fa1dbe914a78940cf158d8bc11b3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.701282 10643 raft_consensus.cc:3060] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.702217 10763 tablet_bootstrap.cc:654] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:41.685747 10717 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.688267 10570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.706128 10570 raft_consensus.cc:2393] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de5f07284784aeaabd73e28f56e1c79 in current term 1: Already voted for candidate 717e7628567145a6864ece4fe7048c5a in this term.
I20260709 12:25:41.689720 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.707311 10571 raft_consensus.cc:2393] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 210ea2bced3049a0a26f70ec2b2ecb34 in current term 1: Already voted for candidate 717e7628567145a6864ece4fe7048c5a in this term.
I20260709 12:25:41.690292 10719 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 1.
I20260709 12:25:41.714816 10535 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:41.716086 10816 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.716632 10816 raft_consensus.cc:697] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 LEADER]: Becoming Leader. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Running, Role: LEADER
I20260709 12:25:41.717445 10816 consensus_queue.cc:237] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.718566 10645 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717e7628567145a6864ece4fe7048c5a in term 1.
I20260709 12:25:41.712890 10642 raft_consensus.cc:2468] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.725262 10783 raft_consensus.cc:493] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.725958 10783 raft_consensus.cc:515] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.727033 10536 leader_election.cc:304] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a; no voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:41.728291 10783 leader_election.cc:290] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:41.691984 10765 leader_election.cc:290] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:41.730127 10571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a" is_pre_election: true
I20260709 12:25:41.731096 10571 raft_consensus.cc:2468] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.721971 10643 raft_consensus.cc:2468] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:41.732574 10685 leader_election.cc:304] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:41.732899 10813 raft_consensus.cc:2749] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:25:41.733598 10765 raft_consensus.cc:2804] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:41.734156 10765 raft_consensus.cc:697] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:41.734696 10685 leader_election.cc:304] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:41.736531 10611 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de5f07284784aeaabd73e28f56e1c79; no voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:41.745020 10814 raft_consensus.cc:2749] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:25:41.745816 10765 consensus_queue.cc:237] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.746361 10824 raft_consensus.cc:2804] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:41.747237 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:41.751668 10824 raft_consensus.cc:493] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.752158 10824 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:41.744755 10780 consensus_queue.cc:1048] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:41.765202 10783 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:41.766490 10776 consensus_queue.cc:1048] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:41.767997 10763 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: No bootstrap required, opened a new log
I20260709 12:25:41.768688 10763 ts_tablet_manager.cc:1403] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.073s	user 0.026s	sys 0.022s
I20260709 12:25:41.773515 10763 raft_consensus.cc:359] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.780992 10763 raft_consensus.cc:385] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:41.781288 10763 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:41.781980 10763 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:41.786103 10763 ts_tablet_manager.cc:1434] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.001s
I20260709 12:25:41.789409 10466 catalog_manager.cc:5697] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 717e7628567145a6864ece4fe7048c5a (127.25.202.1). New cstate: current_term: 1 leader_uuid: "717e7628567145a6864ece4fe7048c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.791652 10469 catalog_manager.cc:5697] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.796983 10824 raft_consensus.cc:515] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.798100 10717 raft_consensus.cc:1240] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Rejecting Update request from peer 8de5f07284784aeaabd73e28f56e1c79 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:25:41.802704 10780 consensus_queue.cc:1059] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:41.803581 10780 raft_consensus.cc:3055] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:41.803936 10780 raft_consensus.cc:740] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:41.804756 10780 consensus_queue.cc:260] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.805263 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:41.805923 10780 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:41.816270 10824 leader_election.cc:290] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Requested vote from peers 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453), 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:41.817976 10642 raft_consensus.cc:2410] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 210ea2bced3049a0a26f70ec2b2ecb34 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 12:25:41.817888 10569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:41.818745 10569 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:41.818675 10466 catalog_manager.cc:5697] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 717e7628567145a6864ece4fe7048c5a (127.25.202.1). New cstate: current_term: 1 leader_uuid: "717e7628567145a6864ece4fe7048c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.825800 10569 raft_consensus.cc:2468] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 2.
I20260709 12:25:41.826818 10685 leader_election.cc:304] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:41.827848 10824 raft_consensus.cc:2804] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:41.828349 10824 raft_consensus.cc:697] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:41.829195 10824 consensus_queue.cc:237] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:41.830801 10468 catalog_manager.cc:5697] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.833634 10569 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:41.841480 10824 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:41.846086 10468 catalog_manager.cc:5697] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 1 to 2, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.852759 10824 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:41.948114 10780 consensus_queue.cc:1048] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:41.958827 10814 consensus_queue.cc:1048] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:42.084739 26408 ts_manager.cc:295] Set tserver state for 717e7628567145a6864ece4fe7048c5a to MAINTENANCE_MODE
I20260709 12:25:42.085522 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
W20260709 12:25:42.094777 10611 consensus_peers.cc:597] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:42.111788 10685 consensus_peers.cc:597] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.115511 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:42.116279 26408 tablet_replica.cc:333] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.116940 26408 raft_consensus.cc:2243] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.117405 26408 raft_consensus.cc:2272] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.119949 26408 tablet_replica.cc:333] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.120635 26408 raft_consensus.cc:2243] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.121054 26408 raft_consensus.cc:2272] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.123235 26408 tablet_replica.cc:333] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.123627 10772 consensus_queue.cc:1048] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:42.124069 26408 raft_consensus.cc:2243] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.124655 26408 raft_consensus.cc:2272] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.127103 26408 tablet_replica.cc:333] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.127693 26408 raft_consensus.cc:2243] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.128187 26408 raft_consensus.cc:2272] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.130663 26408 tablet_replica.cc:333] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.131471 26408 raft_consensus.cc:2243] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:42.132084 26408 pending_rounds.cc:70] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Trying to abort 1 pending ops.
I20260709 12:25:42.132313 26408 pending_rounds.cc:77] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305625361039745024 op_type: NO_OP noop_request { }
I20260709 12:25:42.132674 26408 raft_consensus.cc:2889] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:42.132995 26408 raft_consensus.cc:2272] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.134029 10816 consensus_queue.cc:1048] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:42.136121 10814 consensus_queue.cc:1048] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:42.138803 26408 tablet_replica.cc:333] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.141013 26408 raft_consensus.cc:2243] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:42.142174 26408 raft_consensus.cc:2272] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:42.143136 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:42.149946 26408 tablet_replica.cc:333] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.150892 26408 raft_consensus.cc:2243] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.151397 26408 raft_consensus.cc:2272] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.153862 26408 tablet_replica.cc:333] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.154415 26408 raft_consensus.cc:2243] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.154841 26408 raft_consensus.cc:2272] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.158247 26408 tablet_replica.cc:333] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.158998 26408 raft_consensus.cc:2243] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.159487 26408 raft_consensus.cc:2272] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.161401 26408 tablet_replica.cc:333] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: stopping tablet replica
I20260709 12:25:42.161851 26408 raft_consensus.cc:2243] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:42.162248 26408 raft_consensus.cc:2272] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:42.170396 10685 consensus_peers.cc:597] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.193279 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
I20260709 12:25:42.227495 10824 consensus_queue.cc:1048] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:42.230087 10824 consensus_queue.cc:1048] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:25:42.249886 10685 consensus_peers.cc:597] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.273277 10642 raft_consensus.cc:1275] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:25:42.273715 10642 pending_rounds.cc:85] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: Aborting all ops after (but not including) 0
I20260709 12:25:42.273941 10642 pending_rounds.cc:107] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:25:42.274112 10642 raft_consensus.cc:2889] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:42.275406 10783 consensus_queue.cc:1048] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:25:42.287179 10685 consensus_peers.cc:597] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:42.296936 10685 consensus_peers.cc:597] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:42.357517 10685 consensus_peers.cc:597] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:42.553253 10611 consensus_peers.cc:597] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:43.040601 10611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20260709 12:25:43.144405 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:43.637413 10850 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 717e7628567145a6864ece4fe7048c5a)
I20260709 12:25:43.638281 10850 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.639657 10849 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 717e7628567145a6864ece4fe7048c5a)
I20260709 12:25:43.640327 10850 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:43.640211 10849 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.641654 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79" is_pre_election: true
I20260709 12:25:43.642335 10642 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 1.
I20260709 12:25:43.642756 10849 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:43.644276 10685 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:43.644418 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:43.645043 10719 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
W20260709 12:25:43.645156 10685 leader_election.cc:336] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
W20260709 12:25:43.645861 10611 leader_election.cc:336] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
I20260709 12:25:43.645969 10850 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:43.646428 10850 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 717e7628567145a6864ece4fe7048c5a)
I20260709 12:25:43.646425 10610 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:43.647004 10850 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:43.647357 10849 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:43.647782 10849 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 717e7628567145a6864ece4fe7048c5a)
I20260709 12:25:43.648066 10849 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:43.652117 10850 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.652781 10849 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.653761 10850 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:43.655233 10849 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:43.655809 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:43.656569 10642 raft_consensus.cc:2393] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 210ea2bced3049a0a26f70ec2b2ecb34 in current term 2: Already voted for candidate 8de5f07284784aeaabd73e28f56e1c79 in this term.
I20260709 12:25:43.656944 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:43.657750 10719 raft_consensus.cc:2393] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de5f07284784aeaabd73e28f56e1c79 in current term 2: Already voted for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in this term.
W20260709 12:25:43.658407 10685 leader_election.cc:336] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
W20260709 12:25:43.658726 10611 leader_election.cc:336] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
I20260709 12:25:43.659283 10610 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de5f07284784aeaabd73e28f56e1c79; no voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:43.659369 10685 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34; no voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:43.660058 10849 raft_consensus.cc:2749] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:43.660132 10850 raft_consensus.cc:2749] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:43.825438 10849 raft_consensus.cc:493] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.825997 10849 raft_consensus.cc:515] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.827953 10849 leader_election.cc:290] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:43.828977 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:43.829573 10719 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 1.
I20260709 12:25:43.830466 10610 leader_election.cc:304] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:43.831244 10849 raft_consensus.cc:2804] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:25:43.831594 10611 leader_election.cc:336] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
I20260709 12:25:43.831694 10849 raft_consensus.cc:493] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.832154 10849 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:43.837036 10849 raft_consensus.cc:515] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.838596 10849 leader_election.cc:290] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:43.839644 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:43.840242 10719 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:43.842072 10611 leader_election.cc:336] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
I20260709 12:25:43.845700 10719 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:43.846644 10610 leader_election.cc:304] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:43.847247 10849 raft_consensus.cc:2804] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:43.848379 10849 raft_consensus.cc:697] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:43.849279 10849 consensus_queue.cc:237] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:43.857378 10468 catalog_manager.cc:5697] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 1 to 2, leader changed from 717e7628567145a6864ece4fe7048c5a (127.25.202.1) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:44.145478 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:25:44.344589 10611 consensus_peers.cc:597] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:44.428993 10719 raft_consensus.cc:1275] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:25:44.430178 10849 consensus_queue.cc:1048] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:25:44.520071 10685 consensus_peers.cc:597] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.549419 10685 consensus_peers.cc:597] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.569593 10685 consensus_peers.cc:597] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.602308 10685 consensus_peers.cc:597] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.703645 10611 consensus_peers.cc:597] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.728188 10685 consensus_peers.cc:597] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.870726 10611 consensus_peers.cc:597] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:45.052678 10685 consensus_peers.cc:597] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:45.146504 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:45.201205 10849 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.201804 10849 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:45.203598 10849 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:45.205001 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" is_pre_election: true
I20260709 12:25:45.205783 10719 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:45.206923 10610 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:45.207768 10849 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:45.208158 10849 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:45.208173 10611 leader_election.cc:336] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
I20260709 12:25:45.208549 10849 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:45.214012 10849 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:45.215834 10849 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:45.216923 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:45.217543 10719 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:45.219787 10611 leader_election.cc:336] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111)
I20260709 12:25:45.223405 10719 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 3.
I20260709 12:25:45.224407 10610 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:45.225111 10849 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:45.226034 10849 raft_consensus.cc:697] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:45.226864 10849 consensus_queue.cc:237] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:45.234411 10466 catalog_manager.cc:5697] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 1 to 3, leader changed from 717e7628567145a6864ece4fe7048c5a (127.25.202.1) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:45.809319 10719 raft_consensus.cc:1275] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:45.810689 10849 consensus_queue.cc:1048] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:45.828576 10611 consensus_peers.cc:597] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:46.147622 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:25:46.672075 10611 consensus_peers.cc:597] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:46.941087 10685 consensus_peers.cc:597] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:46.951901 10685 consensus_peers.cc:597] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:47.040870 10685 consensus_peers.cc:597] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:47.148432 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:25:47.174973 10685 consensus_peers.cc:597] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:47.198556 10611 consensus_peers.cc:597] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:47.214433 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:47.222113 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:47.222400 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:47.225085 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:47.226356 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:47.227788 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:47.228106 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:47.228394 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599947228372 us; error 0 us; skew 500 ppm
I20260709 12:25:47.229250 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260709 12:25:47.232422 10685 consensus_peers.cc:597] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:47.235884 26408 webserver.cc:533] Webserver started at http://127.25.202.1:36227/ using document root <none> and password file <none>
I20260709 12:25:47.236828 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:47.237164 26408 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:25:47.242079 26408 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:25:47.246493 10882 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:47.247474 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:25:47.247803 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "717e7628567145a6864ece4fe7048c5a"
format_stamp: "Formatted at 2026-07-09 12:25:40 on dist-test-slave-m57n"
I20260709 12:25:47.248097 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:47.265403 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:47.266868 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:47.268953 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:47.274322 10889 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
I20260709 12:25:47.334666 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:25:47.335000 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.062s	user 0.003s	sys 0.000s
I20260709 12:25:47.335292 26408 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 12:25:47.339989 10889 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.352674 10889 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.353440 10889 tablet_bootstrap.cc:492] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.354164 10889 ts_tablet_manager.cc:1403] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20260709 12:25:47.356599 10889 raft_consensus.cc:359] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.357290 10889 raft_consensus.cc:740] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.357987 10889 consensus_queue.cc:260] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.360270 10889 ts_tablet_manager.cc:1434] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.003s
I20260709 12:25:47.360934 10889 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.371543 10889 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.372359 10889 tablet_bootstrap.cc:492] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.372927 10889 ts_tablet_manager.cc:1403] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:25:47.374881 10889 raft_consensus.cc:359] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.375454 10889 raft_consensus.cc:740] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.376078 10889 consensus_queue.cc:260] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.377496 10889 ts_tablet_manager.cc:1434] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.003s
I20260709 12:25:47.378159 10889 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.378631 26408 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:25:47.378975 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.044s	user 0.041s	sys 0.000s
I20260709 12:25:47.389621 10889 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.390617 10889 tablet_bootstrap.cc:492] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.391273 10889 ts_tablet_manager.cc:1403] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.001s
I20260709 12:25:47.393977 10889 raft_consensus.cc:359] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.394728 10889 raft_consensus.cc:740] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.395538 10889 consensus_queue.cc:260] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.397200 10889 ts_tablet_manager.cc:1434] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:25:47.397969 10889 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
W20260709 12:25:47.405941 10611 consensus_peers.cc:597] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:47.411224 10889 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.412079 10889 tablet_bootstrap.cc:492] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.412707 10889 ts_tablet_manager.cc:1403] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20260709 12:25:47.414784 10889 raft_consensus.cc:359] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.415297 10889 raft_consensus.cc:740] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.415930 10889 consensus_queue.cc:260] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.417444 10889 ts_tablet_manager.cc:1434] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20260709 12:25:47.418077 10889 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.430204 10889 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.431087 10889 tablet_bootstrap.cc:492] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.431878 10889 ts_tablet_manager.cc:1403] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.007s
I20260709 12:25:47.434813 10889 raft_consensus.cc:359] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:47.435660 10889 raft_consensus.cc:740] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.436558 10889 consensus_queue.cc:260] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:47.438817 10889 ts_tablet_manager.cc:1434] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:25:47.439868 10889 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.442191 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:34205
I20260709 12:25:47.442305 10951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:34205 every 8 connection(s)
I20260709 12:25:47.447620 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:47.460089 10889 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.461498 10889 tablet_bootstrap.cc:492] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.462425 10889 ts_tablet_manager.cc:1403] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.023s	user 0.017s	sys 0.004s
I20260709 12:25:47.465915 10889 raft_consensus.cc:359] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.466898 10889 raft_consensus.cc:740] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.467752 10889 consensus_queue.cc:260] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.469584 10889 ts_tablet_manager.cc:1434] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:47.470331 10889 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.481694 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:47.482586 10719 raft_consensus.cc:606] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:47.483763 10952 heartbeater.cc:344] Connected to a master server at 127.25.202.62:40737
I20260709 12:25:47.484259 10952 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:47.485302 10952 heartbeater.cc:507] Master 127.25.202.62:40737 requested a full tablet report, sending...
I20260709 12:25:47.487042 10889 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:25:47.488157 10889 tablet_bootstrap.cc:492] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.488935 10889 ts_tablet_manager.cc:1403] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.000s
I20260709 12:25:47.491832 10466 ts_manager.cc:194] Re-registered known tserver with Master: 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:47.491254 10889 raft_consensus.cc:359] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.493427 10889 raft_consensus.cc:740] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.494313 10889 consensus_queue.cc:260] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.496706 10889 ts_tablet_manager.cc:1434] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:25:47.497817 10889 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.506533 10466 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47832
I20260709 12:25:47.511247 10952 heartbeater.cc:499] Master 127.25.202.62:40737 was elected leader, sending a full tablet report...
I20260709 12:25:47.520331 10642 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:47.521046 10642 raft_consensus.cc:606] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:47.525861 10717 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:47.526768 10717 raft_consensus.cc:606] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:47.528533 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:47.529538 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:47.530673 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:47.537424 10889 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.538226 10889 tablet_bootstrap.cc:492] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.539038 10889 ts_tablet_manager.cc:1403] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.041s	user 0.021s	sys 0.013s
I20260709 12:25:47.540858 10889 raft_consensus.cc:359] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.541407 10889 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.542053 10889 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.543655 10889 ts_tablet_manager.cc:1434] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:25:47.544417 10889 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.557858 10889 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.558660 10889 tablet_bootstrap.cc:492] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.559363 10889 ts_tablet_manager.cc:1403] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.004s
I20260709 12:25:47.561019 10889 raft_consensus.cc:359] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.561476 10889 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.561913 10889 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.563359 10889 ts_tablet_manager.cc:1434] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.000s
I20260709 12:25:47.564185 10889 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap starting.
I20260709 12:25:47.575281 10889 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:47.575996 10889 tablet_bootstrap.cc:492] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Bootstrap complete.
I20260709 12:25:47.576543 10889 ts_tablet_manager.cc:1403] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20260709 12:25:47.578281 10889 raft_consensus.cc:359] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.578781 10889 raft_consensus.cc:740] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e7628567145a6864ece4fe7048c5a, State: Initialized, Role: FOLLOWER
I20260709 12:25:47.579326 10889 consensus_queue.cc:260] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.581256 10889 ts_tablet_manager.cc:1434] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:25:47.620790 10968 raft_consensus.cc:993] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:47.621248 10864 raft_consensus.cc:1081] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:47.622618 10717 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:47.623211 10717 raft_consensus.cc:493] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:47.623554 10717 raft_consensus.cc:3060] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.628556 10717 raft_consensus.cc:515] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.630205 10717 leader_election.cc:290] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:47.631031 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:47.631182 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2bbbde330214551a52fa8f29ebe4a7e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:47.631861 10926 raft_consensus.cc:3060] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.631876 10642 raft_consensus.cc:3055] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:47.632419 10642 raft_consensus.cc:740] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:47.633283 10642 consensus_queue.cc:260] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.634601 10642 raft_consensus.cc:3060] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.639436 10926 raft_consensus.cc:2468] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 2.
I20260709 12:25:47.640798 10685 leader_election.cc:304] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:47.641541 10858 raft_consensus.cc:2804] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:47.641947 10642 raft_consensus.cc:2468] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 2.
I20260709 12:25:47.642781 10858 raft_consensus.cc:697] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:47.643661 10858 consensus_queue.cc:237] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.650331 10466 catalog_manager.cc:5697] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 1 to 2, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.664151 10969 raft_consensus.cc:993] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:47.664595 10858 raft_consensus.cc:1081] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:47.665918 10642 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:47.666425 10642 raft_consensus.cc:493] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:47.666715 10642 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.671981 10642 raft_consensus.cc:515] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.673530 10642 leader_election.cc:290] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:47.675256 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:47.675930 10717 raft_consensus.cc:3055] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:47.676280 10717 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:47.677103 10717 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.678344 10717 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.684253 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:47.684592 10717 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:47.685207 10926 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.686091 10610 leader_election.cc:304] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:47.686965 10864 raft_consensus.cc:2804] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:47.687484 10864 raft_consensus.cc:697] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:47.688326 10864 consensus_queue.cc:237] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:47.692463 10926 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:47.696681 10466 catalog_manager.cc:5697] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 1 to 2, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:47.739989 10926 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:25:47.743760 10925 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:47.753611 10925 pending_rounds.cc:85] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Aborting all ops after (but not including) 0
I20260709 12:25:47.753901 10925 pending_rounds.cc:107] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:25:47.754140 10925 raft_consensus.cc:2889] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:47.782408 10858 raft_consensus.cc:993] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:47.782900 10969 raft_consensus.cc:1081] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:47.784442 10642 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:47.784914 10642 raft_consensus.cc:493] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:47.785208 10642 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:47.790300 10642 raft_consensus.cc:515] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:47.792088 10642 leader_election.cc:290] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 election: Requested vote from peers 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381), 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205)
I20260709 12:25:47.792806 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:47.793109 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dff18e1c3f7a44959883a673cdc34ff8" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:47.793466 10717 raft_consensus.cc:3055] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:47.793689 10925 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:47.793819 10717 raft_consensus.cc:740] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:47.794723 10717 consensus_queue.cc:260] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:47.796020 10717 raft_consensus.cc:3060] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:47.800995 10925 raft_consensus.cc:2468] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 3.
I20260709 12:25:47.802434 10611 leader_election.cc:304] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:47.803201 10864 raft_consensus.cc:2804] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:47.803386 10717 raft_consensus.cc:2468] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 3.
I20260709 12:25:47.803730 10864 raft_consensus.cc:697] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:47.804669 10864 consensus_queue.cc:237] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } }
I20260709 12:25:47.813241 10468 catalog_manager.cc:5697] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 2 to 3, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 3 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.121601 10926 raft_consensus.cc:1275] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:48.123293 10858 consensus_queue.cc:1048] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:48.133855 10642 raft_consensus.cc:1275] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:48.136207 10858 consensus_queue.cc:1048] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:48.149521 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:48.226763 10926 raft_consensus.cc:1275] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:48.229319 10864 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:48.248370 10717 raft_consensus.cc:1275] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:48.250787 10988 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:48.257732 10717 raft_consensus.cc:1275] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:48.260110 10988 consensus_queue.cc:1048] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:48.272868 10926 raft_consensus.cc:1275] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:48.275045 10864 consensus_queue.cc:1048] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:48.542555 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:48.553373 10642 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:48.554049 10642 raft_consensus.cc:606] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:48.557067 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:48.557520 10719 raft_consensus.cc:606] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:48.560177 10642 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:48.560585 10642 raft_consensus.cc:606] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:48.561599 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:48.562049 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:48.562804 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:48.693588 10988 raft_consensus.cc:993] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:48.694130 10864 raft_consensus.cc:1081] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:48.695840 10719 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:48.696372 10719 raft_consensus.cc:493] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:48.696707 10719 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:48.704530 10719 raft_consensus.cc:515] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.706454 10719 leader_election.cc:290] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:48.707494 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:48.707310 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:48.708129 10642 raft_consensus.cc:3055] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:48.708190 10926 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:48.708570 10642 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:48.709553 10642 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.711050 10642 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:48.718345 10926 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:48.719915 10685 leader_election.cc:304] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:48.720788 10969 raft_consensus.cc:2804] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:48.720949 10642 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:48.721266 10969 raft_consensus.cc:697] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:48.722977 10969 consensus_queue.cc:237] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.730434 10858 raft_consensus.cc:993] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:48.730902 10969 raft_consensus.cc:1081] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:48.730705 10468 catalog_manager.cc:5697] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 2 to 3, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.732479 10642 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:48.732945 10642 raft_consensus.cc:493] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:48.733214 10642 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:48.739946 10642 raft_consensus.cc:515] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.741750 10642 leader_election.cc:290] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:48.742862 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:48.743600 10925 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:48.743721 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:48.744481 10719 raft_consensus.cc:3055] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:48.744783 10719 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:48.745811 10719 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.747525 10719 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:48.754395 10925 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:48.754408 10719 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:48.755895 10611 leader_election.cc:304] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:48.757053 10864 raft_consensus.cc:2804] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:48.757476 10864 raft_consensus.cc:697] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:48.758507 10864 consensus_queue.cc:237] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.765134 10468 catalog_manager.cc:5697] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 1 to 2, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.889988 10979 raft_consensus.cc:993] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:48.890512 10864 raft_consensus.cc:1081] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:48.892045 10719 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:48.892568 10719 raft_consensus.cc:493] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:48.892877 10719 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:48.898833 10719 raft_consensus.cc:515] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.900935 10719 leader_election.cc:290] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:48.901783 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:48.902314 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:48.902655 10925 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:48.903002 10642 raft_consensus.cc:3055] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:48.903390 10642 raft_consensus.cc:740] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:48.904367 10642 consensus_queue.cc:260] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.905884 10642 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:48.911062 10925 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:48.912333 10685 leader_election.cc:304] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:48.913172 10969 raft_consensus.cc:2804] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:48.913314 10642 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:48.913740 10969 raft_consensus.cc:697] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:48.914963 10969 consensus_queue.cc:237] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:48.923238 10468 catalog_manager.cc:5697] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 2 to 3, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:49.150578 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:49.159580 10925 raft_consensus.cc:1275] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:49.160777 10969 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:49.170138 10642 raft_consensus.cc:1275] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:49.171602 10969 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:49.182755 10719 raft_consensus.cc:1275] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:49.184232 10864 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:49.195379 10925 raft_consensus.cc:1275] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:49.197342 10864 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:49.411499 10925 raft_consensus.cc:1275] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:49.412726 10969 consensus_queue.cc:1048] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:49.422246 10642 raft_consensus.cc:1275] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:49.423938 10969 consensus_queue.cc:1048] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:49.570067 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:49.581713 10642 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:49.582211 10642 raft_consensus.cc:606] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:49.584895 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:49.585366 10719 raft_consensus.cc:606] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:49.588021 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:49.588464 10719 raft_consensus.cc:606] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:49.589476 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:49.589964 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:49.590512 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:49.605644 10988 raft_consensus.cc:993] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:49.606257 10864 raft_consensus.cc:1081] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:49.607944 10719 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:49.608417 10719 raft_consensus.cc:493] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:49.608695 10719 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:49.616410 10719 raft_consensus.cc:515] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:49.618052 10719 leader_election.cc:290] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 4 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:49.618896 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:49.619130 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ce22dbb8e944dabd5c081c64245252" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:49.619630 10925 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:49.619853 10642 raft_consensus.cc:3055] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:49.620261 10642 raft_consensus.cc:740] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:49.621055 10642 consensus_queue.cc:260] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:49.622032 10642 raft_consensus.cc:3060] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:49.625258 10925 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 4.
I20260709 12:25:49.626514 10685 leader_election.cc:304] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:49.627291 10969 raft_consensus.cc:2804] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:49.627519 10642 raft_consensus.cc:2468] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 4.
I20260709 12:25:49.627804 10969 raft_consensus.cc:697] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:49.628917 10969 consensus_queue.cc:237] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:49.636246 10466 catalog_manager.cc:5697] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 3 to 4, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 4 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:49.918131 10982 raft_consensus.cc:993] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:49.918633 10969 raft_consensus.cc:1081] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:49.919979 10642 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:49.920466 10642 raft_consensus.cc:493] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:49.920817 10642 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:49.926119 10642 raft_consensus.cc:515] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:49.927688 10642 leader_election.cc:290] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 4 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:49.928678 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:49.928678 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22842ec864b84edfb1acd6fe4ed6d63e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:49.929550 10719 raft_consensus.cc:3055] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:49.929579 10925 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:49.930070 10719 raft_consensus.cc:740] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:49.930943 10719 consensus_queue.cc:260] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:49.932264 10719 raft_consensus.cc:3060] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:49.937557 10925 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 4.
I20260709 12:25:49.938985 10611 leader_election.cc:304] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:49.939585 10719 raft_consensus.cc:2468] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 4.
I20260709 12:25:49.939744 10864 raft_consensus.cc:2804] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:49.940320 10864 raft_consensus.cc:697] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 4 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:49.941603 10864 consensus_queue.cc:237] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:49.948168 10466 catalog_manager.cc:5697] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 3 to 4, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:50.145166 10643 raft_consensus.cc:1275] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:50.146487 10982 consensus_queue.cc:1048] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:25:50.151684 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:50.157915 10925 raft_consensus.cc:1275] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:50.159976 10982 consensus_queue.cc:1048] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:50.172534 10858 raft_consensus.cc:993] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:50.173036 10858 raft_consensus.cc:1081] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:50.174571 10643 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:50.175122 10643 raft_consensus.cc:493] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:50.175480 10643 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:50.182379 10643 raft_consensus.cc:515] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.184280 10643 leader_election.cc:290] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:50.185046 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:50.185304 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:50.185731 10925 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:50.186015 10719 raft_consensus.cc:3055] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:50.186321 10719 raft_consensus.cc:740] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:50.187244 10719 consensus_queue.cc:260] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.188746 10719 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:50.191650 10925 raft_consensus.cc:2468] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:50.192693 10611 leader_election.cc:304] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:50.193569 10864 raft_consensus.cc:2804] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:50.194000 10864 raft_consensus.cc:697] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:50.194098 10719 raft_consensus.cc:2468] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:50.195020 10864 consensus_queue.cc:237] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.203112 10468 catalog_manager.cc:5697] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 1 to 2, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:50.441501 10925 raft_consensus.cc:1275] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:50.442744 10864 consensus_queue.cc:1048] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:50.452529 10719 raft_consensus.cc:1275] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:50.454118 10988 consensus_queue.cc:1048] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:50.597806 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:50.606612 10643 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:50.607201 10643 raft_consensus.cc:606] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:50.609925 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:50.610466 10719 raft_consensus.cc:606] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:50.613379 10643 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:50.613991 10643 raft_consensus.cc:606] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:50.615360 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:50.616007 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:50.616608 10864 raft_consensus.cc:993] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:50.616707 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:50.617322 10988 raft_consensus.cc:1081] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:50.618821 10719 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:50.619248 10719 raft_consensus.cc:493] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:50.619576 10719 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:50.624320 10719 raft_consensus.cc:515] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.626111 10719 leader_election.cc:290] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:50.626919 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:50.627504 10925 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:50.628197 10643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:50.628706 10643 raft_consensus.cc:3055] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:50.628969 10643 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:50.629676 10643 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.630854 10643 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:50.631705 10719 raft_consensus.cc:1275] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:50.634177 10925 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:50.635259 10988 consensus_queue.cc:1048] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:50.637106 10685 leader_election.cc:304] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:50.637825 10982 raft_consensus.cc:2804] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:50.638306 10982 raft_consensus.cc:697] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:50.639153 10643 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:50.639132 10982 consensus_queue.cc:237] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.645979 10864 raft_consensus.cc:993] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:50.647444 10988 raft_consensus.cc:1081] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:50.649338 10466 catalog_manager.cc:5697] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 2 to 3, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.650877 10717 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:50.651482 10717 raft_consensus.cc:493] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:50.651854 10717 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:50.653868 10925 raft_consensus.cc:1275] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:50.655282 11017 consensus_queue.cc:1048] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:50.660435 10717 raft_consensus.cc:515] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.662998 10717 leader_election.cc:290] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:50.664114 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:50.667378 10643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb02ed89f74546808cfc4efe82c88e" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:50.668083 10643 raft_consensus.cc:3055] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:50.668514 10643 raft_consensus.cc:740] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:50.669569 10643 consensus_queue.cc:260] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.671103 10643 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:50.677989 10643 raft_consensus.cc:2468] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 3.
I20260709 12:25:50.679091 10685 leader_election.cc:304] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 717e7628567145a6864ece4fe7048c5a
I20260709 12:25:50.679862 10982 raft_consensus.cc:2804] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:50.680269 10982 raft_consensus.cc:697] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:50.681027 10982 consensus_queue.cc:237] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.686924 10468 catalog_manager.cc:5697] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 2 to 3, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.986720 10969 raft_consensus.cc:993] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:50.987178 10982 raft_consensus.cc:1081] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:50.988708 10642 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:50.989266 10642 raft_consensus.cc:493] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:50.989559 10642 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:50.994323 10642 raft_consensus.cc:515] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:50.995750 10642 leader_election.cc:290] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 4 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:50.996855 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:50.996732 10925 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c4a359537b43cd988c02dbeba52b72" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:50.997558 10717 raft_consensus.cc:3055] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:50.997597 10925 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:50.998080 10717 raft_consensus.cc:740] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:50.998948 10717 consensus_queue.cc:260] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.000309 10717 raft_consensus.cc:3060] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:51.005971 10925 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 4.
I20260709 12:25:51.007395 10611 leader_election.cc:304] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:51.007859 10717 raft_consensus.cc:2468] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 4.
I20260709 12:25:51.008116 10988 raft_consensus.cc:2804] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:51.008667 10988 raft_consensus.cc:697] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:51.009541 10988 consensus_queue.cc:237] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.018755 10468 catalog_manager.cc:5697] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 3 to 4, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:51.096379 10925 raft_consensus.cc:3060] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:51.101895 10925 raft_consensus.cc:1275] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:51.103243 10982 consensus_queue.cc:1048] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:51.111604 10642 raft_consensus.cc:1275] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:51.113130 10969 consensus_queue.cc:1048] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:25:51.152684 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:51.213109 10642 raft_consensus.cc:1275] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:51.214618 10969 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:51.223426 10925 raft_consensus.cc:1275] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:51.225234 10858 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:51.462304 10925 raft_consensus.cc:1275] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:51.463497 10988 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:51.473758 10717 raft_consensus.cc:1275] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:51.475916 10864 consensus_queue.cc:1048] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:51.623648 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:51.631445 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:51.632063 10719 raft_consensus.cc:606] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:51.634511 10719 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
mode: GRACEFUL
new_leader_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:51436
I20260709 12:25:51.635129 10719 raft_consensus.cc:606] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Received request to transfer leadership to TS 8de5f07284784aeaabd73e28f56e1c79
I20260709 12:25:51.637516 10642 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
mode: GRACEFUL
new_leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:25:51.638006 10642 raft_consensus.cc:606] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Received request to transfer leadership to TS 210ea2bced3049a0a26f70ec2b2ecb34
I20260709 12:25:51.639235 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:51.639750 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:51.640326 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:51.762740 10982 raft_consensus.cc:993] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:51.763245 10858 raft_consensus.cc:1081] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:51.764695 10642 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:51.765172 10642 raft_consensus.cc:493] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:51.765455 10642 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:51.770098 10642 raft_consensus.cc:515] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.771616 10642 leader_election.cc:290] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 4 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:51.772625 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:51.772686 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b0e3a873b754d0f8041302cb17c816b" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:51.773414 10926 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:51.773424 10719 raft_consensus.cc:3055] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:51.773967 10719 raft_consensus.cc:740] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:51.774736 10719 consensus_queue.cc:260] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.776002 10719 raft_consensus.cc:3060] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:51.780833 10719 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 4.
I20260709 12:25:51.781024 10926 raft_consensus.cc:2468] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 4.
I20260709 12:25:51.781994 10610 leader_election.cc:304] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:51.782948 10988 raft_consensus.cc:2804] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:51.783301 10988 raft_consensus.cc:697] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 4 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:51.784221 10988 consensus_queue.cc:237] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.789925 10466 catalog_manager.cc:5697] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 3 to 4, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 4 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:51.939265 10864 raft_consensus.cc:993] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: : Instructing follower 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:51.939728 10988 raft_consensus.cc:1081] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Signalling peer 210ea2bced3049a0a26f70ec2b2ecb34 to start an election
I20260709 12:25:51.941253 10719 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4"
dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
 from {username='slave'} at 127.0.0.1:37864
I20260709 12:25:51.941695 10719 raft_consensus.cc:493] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:51.941992 10719 raft_consensus.cc:3060] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:51.947489 10719 raft_consensus.cc:515] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.949124 10719 leader_election.cc:290] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453)
I20260709 12:25:51.950093 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:51.950095 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb8f9a133d8d43babaed9f7d4e4e47c4" candidate_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
I20260709 12:25:51.950999 10926 raft_consensus.cc:3060] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:51.951114 10642 raft_consensus.cc:3055] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:51.951337 10858 raft_consensus.cc:993] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: : Instructing follower 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:51.951614 10642 raft_consensus.cc:740] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:51.951894 10858 raft_consensus.cc:1081] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Signalling peer 8de5f07284784aeaabd73e28f56e1c79 to start an election
I20260709 12:25:51.952540 10642 consensus_queue.cc:260] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.953877 10642 raft_consensus.cc:3060] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:51.953928 10643 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3"
dest_uuid: "8de5f07284784aeaabd73e28f56e1c79"
 from {username='slave'} at 127.0.0.1:60490
I20260709 12:25:51.954586 10643 raft_consensus.cc:493] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:51.955010 10643 raft_consensus.cc:3060] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:51.957995 10926 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 2.
I20260709 12:25:51.959014 10685 leader_election.cc:304] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210ea2bced3049a0a26f70ec2b2ecb34, 717e7628567145a6864ece4fe7048c5a; no voters: 
I20260709 12:25:51.959771 10858 raft_consensus.cc:2804] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:51.960217 10858 raft_consensus.cc:697] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Becoming Leader. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:51.960877 10642 raft_consensus.cc:2468] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210ea2bced3049a0a26f70ec2b2ecb34 in term 2.
I20260709 12:25:51.961071 10858 consensus_queue.cc:237] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.962625 10643 raft_consensus.cc:515] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.964982 10643 leader_election.cc:290] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Requested vote from peers 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205), 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3:45381)
I20260709 12:25:51.965349 10926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "717e7628567145a6864ece4fe7048c5a"
I20260709 12:25:51.966049 10926 raft_consensus.cc:3060] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:51.967299 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30865ce1f1a443b9b0e21d0b2cfb8e3" candidate_uuid: "8de5f07284784aeaabd73e28f56e1c79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "210ea2bced3049a0a26f70ec2b2ecb34"
I20260709 12:25:51.967924 10719 raft_consensus.cc:3055] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:51.968168 10719 raft_consensus.cc:740] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 210ea2bced3049a0a26f70ec2b2ecb34, State: Running, Role: LEADER
I20260709 12:25:51.969177 10719 consensus_queue.cc:260] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.970193 10719 raft_consensus.cc:3060] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:51.969642 10466 catalog_manager.cc:5697] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 reported cstate change: term changed from 1 to 2, leader changed from 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2) to 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:51.973551 10926 raft_consensus.cc:2468] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:51.974669 10611 leader_election.cc:304] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717e7628567145a6864ece4fe7048c5a, 8de5f07284784aeaabd73e28f56e1c79; no voters: 
I20260709 12:25:51.976061 11017 raft_consensus.cc:2804] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:51.976588 11017 raft_consensus.cc:697] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Becoming Leader. State: Replica: 8de5f07284784aeaabd73e28f56e1c79, State: Running, Role: LEADER
I20260709 12:25:51.977417 10719 raft_consensus.cc:2468] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de5f07284784aeaabd73e28f56e1c79 in term 2.
I20260709 12:25:51.977893 11017 consensus_queue.cc:237] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } }
I20260709 12:25:51.987607 10466 catalog_manager.cc:5697] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 reported cstate change: term changed from 1 to 2, leader changed from 210ea2bced3049a0a26f70ec2b2ecb34 (127.25.202.3) to 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "8de5f07284784aeaabd73e28f56e1c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:52.153680 10517 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:25:52.208860 10926 raft_consensus.cc:1275] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:52.210191 11017 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:52.221827 10719 raft_consensus.cc:1275] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:52.223479 11017 consensus_queue.cc:1048] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:52.374374 10926 raft_consensus.cc:1275] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:52.375653 11017 consensus_queue.cc:1048] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:52.385293 10719 raft_consensus.cc:1275] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Refusing update from remote peer 8de5f07284784aeaabd73e28f56e1c79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:52.386826 10988 consensus_queue.cc:1048] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210ea2bced3049a0a26f70ec2b2ecb34" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:52.485251 10926 raft_consensus.cc:1275] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:52.486521 10982 consensus_queue.cc:1048] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717e7628567145a6864ece4fe7048c5a" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:52.498075 10643 raft_consensus.cc:1275] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Refusing update from remote peer 210ea2bced3049a0a26f70ec2b2ecb34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:52.499958 10858 consensus_queue.cc:1048] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de5f07284784aeaabd73e28f56e1c79" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:52.647392 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:52.678192 26408 tablet_server.cc:179] TabletServer@127.25.202.1:34205 shutting down...
W20260709 12:25:52.689337 10611 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 88 similar messages]
W20260709 12:25:52.693212 10611 consensus_peers.cc:597] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.705397 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:52.706230 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.706964 26408 raft_consensus.cc:2243] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.707530 26408 raft_consensus.cc:2272] T 2b0e3a873b754d0f8041302cb17c816b P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.710788 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.711459 26408 raft_consensus.cc:2243] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.711922 26408 raft_consensus.cc:2272] T 22842ec864b84edfb1acd6fe4ed6d63e P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.714311 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.715047 26408 raft_consensus.cc:2243] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.715548 26408 raft_consensus.cc:2272] T c2bbbde330214551a52fa8f29ebe4a7e P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.717666 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.718298 26408 raft_consensus.cc:2243] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.718750 26408 raft_consensus.cc:2272] T dff18e1c3f7a44959883a673cdc34ff8 P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.721107 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.721698 26408 raft_consensus.cc:2243] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.722157 26408 raft_consensus.cc:2272] T d641fa1dbe914a78940cf158d8bc11b3 P 717e7628567145a6864ece4fe7048c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.724560 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.725196 26408 raft_consensus.cc:2243] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.725641 26408 raft_consensus.cc:2272] T aeeb02ed89f74546808cfc4efe82c88e P 717e7628567145a6864ece4fe7048c5a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.728031 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.728560 26408 raft_consensus.cc:2243] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.729012 26408 raft_consensus.cc:2272] T 65c4a359537b43cd988c02dbeba52b72 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.731353 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.731982 26408 raft_consensus.cc:2243] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.732379 26408 raft_consensus.cc:2272] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.735136 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.735720 26408 raft_consensus.cc:2243] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.736106 26408 raft_consensus.cc:2272] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 717e7628567145a6864ece4fe7048c5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.738839 26408 tablet_replica.cc:333] stopping tablet replica
I20260709 12:25:52.739562 26408 raft_consensus.cc:2243] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.739933 26408 raft_consensus.cc:2272] T 13ce22dbb8e944dabd5c081c64245252 P 717e7628567145a6864ece4fe7048c5a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.774616 26408 tablet_server.cc:196] TabletServer@127.25.202.1:34205 shutdown complete.
W20260709 12:25:52.796347 10611 consensus_peers.cc:597] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.807595 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
W20260709 12:25:52.825802 10685 consensus_peers.cc:597] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.840044 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:52.840883 26408 tablet_replica.cc:333] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.841571 26408 raft_consensus.cc:2243] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.842025 26408 raft_consensus.cc:2272] T 13ce22dbb8e944dabd5c081c64245252 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.844437 26408 tablet_replica.cc:333] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.845031 26408 raft_consensus.cc:2243] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.845824 26408 raft_consensus.cc:2272] T 2b0e3a873b754d0f8041302cb17c816b P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.848733 26408 tablet_replica.cc:333] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.849355 26408 raft_consensus.cc:2243] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.850142 26408 raft_consensus.cc:2272] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.852803 26408 tablet_replica.cc:333] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.853456 26408 raft_consensus.cc:2243] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.853935 26408 raft_consensus.cc:2272] T d641fa1dbe914a78940cf158d8bc11b3 P 8de5f07284784aeaabd73e28f56e1c79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.855991 26408 tablet_replica.cc:333] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.856534 26408 raft_consensus.cc:2243] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.856995 26408 raft_consensus.cc:2272] T aeeb02ed89f74546808cfc4efe82c88e P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.859136 26408 tablet_replica.cc:333] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.859757 26408 raft_consensus.cc:2243] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.860260 26408 raft_consensus.cc:2272] T c2bbbde330214551a52fa8f29ebe4a7e P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.862214 26408 tablet_replica.cc:333] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.862740 26408 raft_consensus.cc:2243] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.863584 26408 raft_consensus.cc:2272] T 22842ec864b84edfb1acd6fe4ed6d63e P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.865585 26408 tablet_replica.cc:333] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.866112 26408 raft_consensus.cc:2243] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.867000 26408 raft_consensus.cc:2272] T dff18e1c3f7a44959883a673cdc34ff8 P 8de5f07284784aeaabd73e28f56e1c79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.869446 26408 tablet_replica.cc:333] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.870012 26408 raft_consensus.cc:2243] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.870419 26408 raft_consensus.cc:2272] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 8de5f07284784aeaabd73e28f56e1c79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.873674 26408 tablet_replica.cc:333] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79: stopping tablet replica
I20260709 12:25:52.874117 26408 raft_consensus.cc:2243] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.874923 26408 raft_consensus.cc:2272] T 65c4a359537b43cd988c02dbeba52b72 P 8de5f07284784aeaabd73e28f56e1c79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.900122 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
I20260709 12:25:52.925005 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
W20260709 12:25:52.959190 10685 consensus_peers.cc:597] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 717e7628567145a6864ece4fe7048c5a (127.25.202.1:34205): Couldn't send request to peer 717e7628567145a6864ece4fe7048c5a. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:34205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:52.960069 10685 consensus_peers.cc:597] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 -> Peer 8de5f07284784aeaabd73e28f56e1c79 (127.25.202.2:37453): Couldn't send request to peer 8de5f07284784aeaabd73e28f56e1c79. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.961438 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:52.962190 26408 tablet_replica.cc:333] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.963090 26408 raft_consensus.cc:2243] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.963972 26408 raft_consensus.cc:2272] T 13ce22dbb8e944dabd5c081c64245252 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.967064 26408 tablet_replica.cc:333] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.967733 26408 raft_consensus.cc:2243] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.968228 26408 raft_consensus.cc:2272] T 22842ec864b84edfb1acd6fe4ed6d63e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.970520 26408 tablet_replica.cc:333] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.971154 26408 raft_consensus.cc:2243] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.971681 26408 raft_consensus.cc:2272] T dff18e1c3f7a44959883a673cdc34ff8 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.973716 26408 tablet_replica.cc:333] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.974288 26408 raft_consensus.cc:2243] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.975104 26408 raft_consensus.cc:2272] T c2bbbde330214551a52fa8f29ebe4a7e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.977295 26408 tablet_replica.cc:333] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.977962 26408 raft_consensus.cc:2243] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.978725 26408 raft_consensus.cc:2272] T aeeb02ed89f74546808cfc4efe82c88e P 210ea2bced3049a0a26f70ec2b2ecb34 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.981079 26408 tablet_replica.cc:333] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.981612 26408 raft_consensus.cc:2243] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.982438 26408 raft_consensus.cc:2272] T d641fa1dbe914a78940cf158d8bc11b3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.984601 26408 tablet_replica.cc:333] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.985167 26408 raft_consensus.cc:2243] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.985582 26408 raft_consensus.cc:2272] T a30865ce1f1a443b9b0e21d0b2cfb8e3 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.988252 26408 tablet_replica.cc:333] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.988797 26408 raft_consensus.cc:2243] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:52.989598 26408 raft_consensus.cc:2272] T cb8f9a133d8d43babaed9f7d4e4e47c4 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.992105 26408 tablet_replica.cc:333] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.992604 26408 raft_consensus.cc:2243] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.993021 26408 raft_consensus.cc:2272] T 65c4a359537b43cd988c02dbeba52b72 P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.995285 26408 tablet_replica.cc:333] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34: stopping tablet replica
I20260709 12:25:52.995814 26408 raft_consensus.cc:2243] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.996461 26408 raft_consensus.cc:2272] T 2b0e3a873b754d0f8041302cb17c816b P 210ea2bced3049a0a26f70ec2b2ecb34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:53.031272 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:25:53.058357 26408 master.cc:562] Master@127.25.202.62:40737 shutting down...
I20260709 12:25:53.084070 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:53.084663 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:53.085006 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P c473a04b800d4d78a9a2becfd9a41d31: stopping tablet replica
I20260709 12:25:53.106133 26408 master.cc:584] Master@127.25.202.62:40737 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13143 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:25:53.153262 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:37423
I20260709 12:25:53.154462 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:53.160204 11049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.160320 11050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.160884 11052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:53.163115 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:53.164083 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:53.164254 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:53.164364 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599953164354 us; error 0 us; skew 500 ppm
I20260709 12:25:53.164840 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:53.167268 26408 webserver.cc:533] Webserver started at http://127.25.202.62:35193/ using document root <none> and password file <none>
I20260709 12:25:53.167802 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:53.167966 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:53.168181 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:53.169246 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "a1989576226e4c719723b4a00433924d"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.173751 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:25:53.177160 11057 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.177974 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:25:53.178267 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "a1989576226e4c719723b4a00433924d"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.178555 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:53.197271 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:53.198519 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:53.239355 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:37423
I20260709 12:25:53.239707 11108 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:37423 every 8 connection(s)
I20260709 12:25:53.243813 11109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.256601 11109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d: Bootstrap starting.
I20260709 12:25:53.262079 11109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.266870 11109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d: No bootstrap required, opened a new log
I20260709 12:25:53.269400 11109 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1989576226e4c719723b4a00433924d" member_type: VOTER }
I20260709 12:25:53.269886 11109 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.270186 11109 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1989576226e4c719723b4a00433924d, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.271004 11109 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1989576226e4c719723b4a00433924d" member_type: VOTER }
I20260709 12:25:53.271549 11109 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:25:53.271814 11109 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:25:53.272123 11109 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:53.277412 11109 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1989576226e4c719723b4a00433924d" member_type: VOTER }
I20260709 12:25:53.277997 11109 leader_election.cc:304] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1989576226e4c719723b4a00433924d; no voters: 
I20260709 12:25:53.279237 11109 leader_election.cc:290] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:25:53.279628 11112 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:53.281067 11112 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 1 LEADER]: Becoming Leader. State: Replica: a1989576226e4c719723b4a00433924d, State: Running, Role: LEADER
I20260709 12:25:53.281774 11112 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1989576226e4c719723b4a00433924d" member_type: VOTER }
I20260709 12:25:53.282852 11109 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:25:53.288654 11113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1989576226e4c719723b4a00433924d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1989576226e4c719723b4a00433924d" member_type: VOTER } }
I20260709 12:25:53.288553 11114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1989576226e4c719723b4a00433924d. Latest consensus state: current_term: 1 leader_uuid: "a1989576226e4c719723b4a00433924d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1989576226e4c719723b4a00433924d" member_type: VOTER } }
I20260709 12:25:53.289417 11113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:53.289530 11114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:53.292469 11121 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:25:53.300443 11121 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:25:53.301833 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:25:53.312279 11121 catalog_manager.cc:1383] Generated new cluster ID: 0ed8970e3f0e45f2b666396da1e2bf8c
I20260709 12:25:53.312592 11121 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:25:53.354590 11121 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:25:53.355924 11121 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:25:53.367952 11121 catalog_manager.cc:6070] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d: Generated new TSK 0
I20260709 12:25:53.368566 11121 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:25:53.433831 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:53.440623 11130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.441280 11131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.444096 11133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:53.447435 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:53.450856 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:53.451063 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:53.451189 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599953451178 us; error 0 us; skew 500 ppm
I20260709 12:25:53.451973 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:53.454389 26408 webserver.cc:533] Webserver started at http://127.25.202.1:34439/ using document root <none> and password file <none>
I20260709 12:25:53.454864 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:53.455026 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:53.455236 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:53.456453 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "ece17c3d816940f192e055b2c6f87a84"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.461009 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.005s
I20260709 12:25:53.464874 11138 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.465705 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20260709 12:25:53.466158 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "ece17c3d816940f192e055b2c6f87a84"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.466465 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:53.484582 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:53.485903 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:53.487426 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:53.489732 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:53.489940 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.490233 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:53.490411 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.531783 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:39677
I20260709 12:25:53.531896 11200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:39677 every 8 connection(s)
I20260709 12:25:53.536990 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:53.545140 11205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.545326 11206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:53.551177 26408 server_base.cc:1061] running on GCE node
W20260709 12:25:53.551415 11208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:53.552980 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:53.553260 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:53.553488 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599953553468 us; error 0 us; skew 500 ppm
I20260709 12:25:53.553557 11201 heartbeater.cc:344] Connected to a master server at 127.25.202.62:37423
I20260709 12:25:53.554014 11201 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:53.554157 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:53.554898 11201 heartbeater.cc:507] Master 127.25.202.62:37423 requested a full tablet report, sending...
I20260709 12:25:53.557251 11074 ts_manager.cc:194] Registered new tserver with Master: ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677)
I20260709 12:25:53.557343 26408 webserver.cc:533] Webserver started at http://127.25.202.2:44065/ using document root <none> and password file <none>
I20260709 12:25:53.558025 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:53.558219 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:53.558521 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:53.559230 11074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33536
I20260709 12:25:53.559870 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "0ae87234d66346cf813d2ceb0951efa7"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.564527 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:25:53.567929 11213 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.568747 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:25:53.569096 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "0ae87234d66346cf813d2ceb0951efa7"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.569401 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:53.587532 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:53.588789 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:53.590338 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:53.592630 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:53.592857 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.593132 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:53.593318 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.633678 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:46187
I20260709 12:25:53.633800 11275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:46187 every 8 connection(s)
I20260709 12:25:53.638520 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:53.645344 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.648052 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:53.651283 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:53.651798 11276 heartbeater.cc:344] Connected to a master server at 127.25.202.62:37423
I20260709 12:25:53.652271 11276 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:53.652338 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:53.653234 11276 heartbeater.cc:507] Master 127.25.202.62:37423 requested a full tablet report, sending...
I20260709 12:25:53.653373 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:53.653707 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:53.653951 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599953653931 us; error 0 us; skew 500 ppm
I20260709 12:25:53.654551 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:53.655536 11074 ts_manager.cc:194] Registered new tserver with Master: 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187)
I20260709 12:25:53.657193 26408 webserver.cc:533] Webserver started at http://127.25.202.3:37925/ using document root <none> and password file <none>
I20260709 12:25:53.657243 11074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33550
I20260709 12:25:53.657902 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:53.658141 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:53.658489 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:53.659821 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "6306229d5c88442ba392bac64b07ca8d"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.664073 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:25:53.667292 11287 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.668263 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:25:53.668576 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "6306229d5c88442ba392bac64b07ca8d"
format_stamp: "Formatted at 2026-07-09 12:25:53 on dist-test-slave-m57n"
I20260709 12:25:53.668859 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:53.713795 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:53.715071 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:53.716408 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:53.718851 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:53.719064 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:53.719352 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:53.719532 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260709 12:25:53.760955 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:44977
I20260709 12:25:53.761061 11349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:44977 every 8 connection(s)
I20260709 12:25:53.775463 11350 heartbeater.cc:344] Connected to a master server at 127.25.202.62:37423
I20260709 12:25:53.775938 11350 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:53.776795 11350 heartbeater.cc:507] Master 127.25.202.62:37423 requested a full tablet report, sending...
I20260709 12:25:53.779016 11074 ts_manager.cc:194] Registered new tserver with Master: 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:53.779656 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015523239s
I20260709 12:25:53.780680 11074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33560
I20260709 12:25:53.808264 11074 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33576:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:25:53.810720 11074 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:25:53.860564 11240 tablet_service.cc:1511] Processing CreateTablet for tablet f27b00a4689544e694a5725d3661d3b3 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=110d677cf59b46958b8f31da4c19684a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:25:53.860564 11241 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfc510575a24b458b19bded69eb8576 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=110d677cf59b46958b8f31da4c19684a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:25:53.862317 11241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfc510575a24b458b19bded69eb8576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.865792 11240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27b00a4689544e694a5725d3661d3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.869416 11166 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfc510575a24b458b19bded69eb8576 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=110d677cf59b46958b8f31da4c19684a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:25:53.870613 11165 tablet_service.cc:1511] Processing CreateTablet for tablet f27b00a4689544e694a5725d3661d3b3 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=110d677cf59b46958b8f31da4c19684a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:25:53.872067 11315 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfc510575a24b458b19bded69eb8576 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=110d677cf59b46958b8f31da4c19684a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:25:53.870887 11166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfc510575a24b458b19bded69eb8576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.873658 11315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfc510575a24b458b19bded69eb8576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.874508 11165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27b00a4689544e694a5725d3661d3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.875092 11314 tablet_service.cc:1511] Processing CreateTablet for tablet f27b00a4689544e694a5725d3661d3b3 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=110d677cf59b46958b8f31da4c19684a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:25:53.880321 11314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27b00a4689544e694a5725d3661d3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:53.917105 11365 tablet_bootstrap.cc:492] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7: Bootstrap starting.
I20260709 12:25:53.918268 11366 tablet_bootstrap.cc:492] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84: Bootstrap starting.
I20260709 12:25:53.925143 11367 tablet_bootstrap.cc:492] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: Bootstrap starting.
I20260709 12:25:53.926479 11366 tablet_bootstrap.cc:654] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.926884 11365 tablet_bootstrap.cc:654] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.931703 11366 tablet_bootstrap.cc:492] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84: No bootstrap required, opened a new log
I20260709 12:25:53.932278 11366 ts_tablet_manager.cc:1403] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84: Time spent bootstrapping tablet: real 0.014s	user 0.013s	sys 0.000s
I20260709 12:25:53.933228 11367 tablet_bootstrap.cc:654] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.933918 11365 tablet_bootstrap.cc:492] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7: No bootstrap required, opened a new log
I20260709 12:25:53.934526 11365 ts_tablet_manager.cc:1403] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.004s
I20260709 12:25:53.934917 11366 raft_consensus.cc:359] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:53.935627 11366 raft_consensus.cc:385] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.935894 11366 raft_consensus.cc:740] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.936707 11366 consensus_queue.cc:260] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:53.938495 11365 raft_consensus.cc:359] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:53.939607 11365 raft_consensus.cc:385] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.939993 11365 raft_consensus.cc:740] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.941218 11365 consensus_queue.cc:260] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:53.942588 11201 heartbeater.cc:499] Master 127.25.202.62:37423 was elected leader, sending a full tablet report...
I20260709 12:25:53.943864 11367 tablet_bootstrap.cc:492] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: No bootstrap required, opened a new log
I20260709 12:25:53.943951 11276 heartbeater.cc:499] Master 127.25.202.62:37423 was elected leader, sending a full tablet report...
I20260709 12:25:53.944458 11367 ts_tablet_manager.cc:1403] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: Time spent bootstrapping tablet: real 0.020s	user 0.018s	sys 0.000s
I20260709 12:25:53.945760 11366 ts_tablet_manager.cc:1434] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.004s
I20260709 12:25:53.946853 11366 tablet_bootstrap.cc:492] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84: Bootstrap starting.
I20260709 12:25:53.947916 11367 raft_consensus.cc:359] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:53.948719 11367 raft_consensus.cc:385] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.949028 11367 raft_consensus.cc:740] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.944402 11365 ts_tablet_manager.cc:1434] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.006s
I20260709 12:25:53.949841 11367 consensus_queue.cc:260] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:53.951292 11365 tablet_bootstrap.cc:492] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7: Bootstrap starting.
I20260709 12:25:53.956136 11366 tablet_bootstrap.cc:654] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.958585 11350 heartbeater.cc:499] Master 127.25.202.62:37423 was elected leader, sending a full tablet report...
I20260709 12:25:53.959781 11367 ts_tablet_manager.cc:1434] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.008s
I20260709 12:25:53.960635 11365 tablet_bootstrap.cc:654] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.960922 11367 tablet_bootstrap.cc:492] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d: Bootstrap starting.
I20260709 12:25:53.970078 11367 tablet_bootstrap.cc:654] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:53.970270 11366 tablet_bootstrap.cc:492] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84: No bootstrap required, opened a new log
I20260709 12:25:53.970949 11365 tablet_bootstrap.cc:492] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7: No bootstrap required, opened a new log
I20260709 12:25:53.971014 11366 ts_tablet_manager.cc:1403] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.008s
I20260709 12:25:53.971733 11365 ts_tablet_manager.cc:1403] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7: Time spent bootstrapping tablet: real 0.021s	user 0.001s	sys 0.013s
I20260709 12:25:53.975274 11366 raft_consensus.cc:359] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:53.978121 11366 raft_consensus.cc:385] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.978554 11366 raft_consensus.cc:740] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.976902 11365 raft_consensus.cc:359] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:53.979794 11366 consensus_queue.cc:260] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:53.980916 11365 raft_consensus.cc:385] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.981312 11365 raft_consensus.cc:740] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.982127 11365 consensus_queue.cc:260] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:53.983108 11366 ts_tablet_manager.cc:1434] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:25:53.984336 11365 ts_tablet_manager.cc:1434] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.002s
I20260709 12:25:53.990017 11367 tablet_bootstrap.cc:492] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d: No bootstrap required, opened a new log
I20260709 12:25:53.990691 11367 ts_tablet_manager.cc:1403] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d: Time spent bootstrapping tablet: real 0.030s	user 0.020s	sys 0.005s
I20260709 12:25:53.993849 11367 raft_consensus.cc:359] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:53.994436 11367 raft_consensus.cc:385] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:53.994700 11367 raft_consensus.cc:740] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Initialized, Role: FOLLOWER
I20260709 12:25:53.995369 11367 consensus_queue.cc:260] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:53.997284 11367 ts_tablet_manager.cc:1434] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:25:54.023787 11371 raft_consensus.cc:493] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.024353 11371 raft_consensus.cc:515] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:54.026439 11371 leader_election.cc:290] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.036235 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfc510575a24b458b19bded69eb8576" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae87234d66346cf813d2ceb0951efa7" is_pre_election: true
I20260709 12:25:54.036938 11251 raft_consensus.cc:2468] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 0.
I20260709 12:25:54.037586 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfc510575a24b458b19bded69eb8576" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6306229d5c88442ba392bac64b07ca8d" is_pre_election: true
I20260709 12:25:54.038199 11142 leader_election.cc:304] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.038379 11325 raft_consensus.cc:2468] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 0.
I20260709 12:25:54.039111 11371 raft_consensus.cc:2804] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:54.039656 11371 raft_consensus.cc:493] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.039994 11371 raft_consensus.cc:3060] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.045729 11371 raft_consensus.cc:515] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:54.047566 11371 leader_election.cc:290] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 1 election: Requested vote from peers 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.048420 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfc510575a24b458b19bded69eb8576" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
I20260709 12:25:54.048493 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfc510575a24b458b19bded69eb8576" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
I20260709 12:25:54.049077 11251 raft_consensus.cc:3060] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.049152 11325 raft_consensus.cc:3060] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.054816 11325 raft_consensus.cc:2468] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 1.
I20260709 12:25:54.055122 11251 raft_consensus.cc:2468] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 1.
I20260709 12:25:54.056051 11139 leader_election.cc:304] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6306229d5c88442ba392bac64b07ca8d, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.056797 11371 raft_consensus.cc:2804] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:54.057983 11371 raft_consensus.cc:697] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 1 LEADER]: Becoming Leader. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Running, Role: LEADER
I20260709 12:25:54.058871 11371 consensus_queue.cc:237] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } }
I20260709 12:25:54.062812 11373 raft_consensus.cc:493] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.063544 11373 raft_consensus.cc:515] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.066358 11373 leader_election.cc:290] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187)
I20260709 12:25:54.069579 11073 catalog_manager.cc:5697] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to ece17c3d816940f192e055b2c6f87a84 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "ece17c3d816940f192e055b2c6f87a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:54.076421 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3" candidate_uuid: "6306229d5c88442ba392bac64b07ca8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece17c3d816940f192e055b2c6f87a84" is_pre_election: true
I20260709 12:25:54.077204 11176 raft_consensus.cc:2468] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6306229d5c88442ba392bac64b07ca8d in term 0.
I20260709 12:25:54.078502 11291 leader_election.cc:304] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6306229d5c88442ba392bac64b07ca8d, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.079360 11373 raft_consensus.cc:2804] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:54.079748 11373 raft_consensus.cc:493] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.080109 11373 raft_consensus.cc:3060] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.080313 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3" candidate_uuid: "6306229d5c88442ba392bac64b07ca8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae87234d66346cf813d2ceb0951efa7" is_pre_election: true
I20260709 12:25:54.080909 11251 raft_consensus.cc:2468] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6306229d5c88442ba392bac64b07ca8d in term 0.
I20260709 12:25:54.086117 11373 raft_consensus.cc:515] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.087805 11373 leader_election.cc:290] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [CANDIDATE]: Term 1 election: Requested vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187)
I20260709 12:25:54.088869 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3" candidate_uuid: "6306229d5c88442ba392bac64b07ca8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
I20260709 12:25:54.089203 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3" candidate_uuid: "6306229d5c88442ba392bac64b07ca8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
I20260709 12:25:54.089532 11176 raft_consensus.cc:3060] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.089895 11251 raft_consensus.cc:3060] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.095257 11251 raft_consensus.cc:2468] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6306229d5c88442ba392bac64b07ca8d in term 1.
I20260709 12:25:54.096053 11176 raft_consensus.cc:2468] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6306229d5c88442ba392bac64b07ca8d in term 1.
I20260709 12:25:54.096400 11291 leader_election.cc:304] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, 6306229d5c88442ba392bac64b07ca8d; no voters: 
I20260709 12:25:54.097360 11373 raft_consensus.cc:2804] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:54.098673 11373 raft_consensus.cc:697] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 LEADER]: Becoming Leader. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Running, Role: LEADER
I20260709 12:25:54.099640 11373 consensus_queue.cc:237] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.107843 11073 catalog_manager.cc:5697] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6306229d5c88442ba392bac64b07ca8d (127.25.202.3). New cstate: current_term: 1 leader_uuid: "6306229d5c88442ba392bac64b07ca8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:54.190817 11073 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33582:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:25:54.192858 11073 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:25:54.216248 11166 tablet_service.cc:1511] Processing CreateTablet for tablet 410412fadbea4ebf838b275fa228bfa2 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=234b37a331c144bb8d42d87507692f02]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:25:54.217554 11166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410412fadbea4ebf838b275fa228bfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:54.217370 11241 tablet_service.cc:1511] Processing CreateTablet for tablet 410412fadbea4ebf838b275fa228bfa2 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=234b37a331c144bb8d42d87507692f02]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:25:54.218650 11241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410412fadbea4ebf838b275fa228bfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:54.218780 11314 tablet_service.cc:1511] Processing CreateTablet for tablet 410412fadbea4ebf838b275fa228bfa2 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=234b37a331c144bb8d42d87507692f02]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:25:54.220077 11314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410412fadbea4ebf838b275fa228bfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:54.221602 11165 tablet_service.cc:1511] Processing CreateTablet for tablet a120a6f8529248a8b310cbf9d141ad05 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=234b37a331c144bb8d42d87507692f02]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:25:54.222915 11165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a120a6f8529248a8b310cbf9d141ad05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:54.223919 11240 tablet_service.cc:1511] Processing CreateTablet for tablet a120a6f8529248a8b310cbf9d141ad05 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=234b37a331c144bb8d42d87507692f02]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:25:54.225164 11240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a120a6f8529248a8b310cbf9d141ad05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:54.229275 11315 tablet_service.cc:1511] Processing CreateTablet for tablet a120a6f8529248a8b310cbf9d141ad05 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=234b37a331c144bb8d42d87507692f02]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:25:54.230839 11315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a120a6f8529248a8b310cbf9d141ad05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:54.241416 11365 tablet_bootstrap.cc:492] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: Bootstrap starting.
I20260709 12:25:54.248898 11365 tablet_bootstrap.cc:654] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:54.251969 11367 tablet_bootstrap.cc:492] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d: Bootstrap starting.
I20260709 12:25:54.258158 11366 tablet_bootstrap.cc:492] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84: Bootstrap starting.
I20260709 12:25:54.259513 11367 tablet_bootstrap.cc:654] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:54.259881 11365 tablet_bootstrap.cc:492] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: No bootstrap required, opened a new log
I20260709 12:25:54.260465 11365 ts_tablet_manager.cc:1403] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.001s
I20260709 12:25:54.263777 11365 raft_consensus.cc:359] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.264442 11366 tablet_bootstrap.cc:654] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:54.264647 11365 raft_consensus.cc:385] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:54.265085 11365 raft_consensus.cc:740] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Initialized, Role: FOLLOWER
I20260709 12:25:54.265956 11365 consensus_queue.cc:260] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.267529 11367 tablet_bootstrap.cc:492] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d: No bootstrap required, opened a new log
I20260709 12:25:54.268194 11367 ts_tablet_manager.cc:1403] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.009s
I20260709 12:25:54.268350 11365 ts_tablet_manager.cc:1434] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:25:54.269649 11365 tablet_bootstrap.cc:492] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: Bootstrap starting.
I20260709 12:25:54.271746 11367 raft_consensus.cc:359] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.272557 11367 raft_consensus.cc:385] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:54.272912 11367 raft_consensus.cc:740] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Initialized, Role: FOLLOWER
I20260709 12:25:54.273991 11367 consensus_queue.cc:260] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.276603 11365 tablet_bootstrap.cc:654] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:54.276826 11366 tablet_bootstrap.cc:492] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84: No bootstrap required, opened a new log
I20260709 12:25:54.277426 11366 ts_tablet_manager.cc:1403] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.003s
I20260709 12:25:54.280630 11366 raft_consensus.cc:359] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.281466 11366 raft_consensus.cc:385] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:54.281844 11366 raft_consensus.cc:740] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Initialized, Role: FOLLOWER
I20260709 12:25:54.282706 11366 consensus_queue.cc:260] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.284763 11367 ts_tablet_manager.cc:1434] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.003s
I20260709 12:25:54.285449 11365 tablet_bootstrap.cc:492] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: No bootstrap required, opened a new log
I20260709 12:25:54.286008 11367 tablet_bootstrap.cc:492] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: Bootstrap starting.
I20260709 12:25:54.287513 11366 ts_tablet_manager.cc:1434] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:25:54.288688 11366 tablet_bootstrap.cc:492] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: Bootstrap starting.
I20260709 12:25:54.286034 11365 ts_tablet_manager.cc:1403] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.005s
W20260709 12:25:54.293790 11122 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:54.294866 11365 raft_consensus.cc:359] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.295653 11367 tablet_bootstrap.cc:654] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:54.295768 11365 raft_consensus.cc:385] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:54.296279 11365 raft_consensus.cc:740] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Initialized, Role: FOLLOWER
I20260709 12:25:54.297149 11365 consensus_queue.cc:260] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.298480 11366 tablet_bootstrap.cc:654] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:54.299567 11365 ts_tablet_manager.cc:1434] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:25:54.303212 11367 tablet_bootstrap.cc:492] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: No bootstrap required, opened a new log
I20260709 12:25:54.303836 11367 ts_tablet_manager.cc:1403] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
I20260709 12:25:54.305989 11366 tablet_bootstrap.cc:492] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: No bootstrap required, opened a new log
I20260709 12:25:54.306529 11366 ts_tablet_manager.cc:1403] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.003s
I20260709 12:25:54.306991 11367 raft_consensus.cc:359] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.307842 11367 raft_consensus.cc:385] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:54.308135 11367 raft_consensus.cc:740] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Initialized, Role: FOLLOWER
I20260709 12:25:54.308960 11367 consensus_queue.cc:260] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.309736 11366 raft_consensus.cc:359] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.310534 11366 raft_consensus.cc:385] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:54.310838 11366 raft_consensus.cc:740] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Initialized, Role: FOLLOWER
I20260709 12:25:54.311431 11367 ts_tablet_manager.cc:1434] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:25:54.311790 11366 consensus_queue.cc:260] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.315842 11366 ts_tablet_manager.cc:1434] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:25:54.340416 11372 raft_consensus.cc:493] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.340977 11372 raft_consensus.cc:515] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.343107 11372 leader_election.cc:290] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.356303 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece17c3d816940f192e055b2c6f87a84" is_pre_election: true
I20260709 12:25:54.356585 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6306229d5c88442ba392bac64b07ca8d" is_pre_election: true
I20260709 12:25:54.357014 11176 raft_consensus.cc:2468] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 0.
I20260709 12:25:54.357191 11325 raft_consensus.cc:2468] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 0.
I20260709 12:25:54.358165 11217 leader_election.cc:304] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.358946 11372 raft_consensus.cc:2804] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:54.359223 11372 raft_consensus.cc:493] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.359566 11372 raft_consensus.cc:3060] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.364956 11372 raft_consensus.cc:515] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.366648 11372 leader_election.cc:290] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 election: Requested vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.367390 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
I20260709 12:25:54.367609 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
I20260709 12:25:54.367944 11176 raft_consensus.cc:3060] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.368196 11325 raft_consensus.cc:3060] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.373726 11176 raft_consensus.cc:2468] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 1.
I20260709 12:25:54.373726 11325 raft_consensus.cc:2468] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 1.
I20260709 12:25:54.374814 11217 leader_election.cc:304] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.375553 11372 raft_consensus.cc:2804] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:54.376499 11372 raft_consensus.cc:697] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Becoming Leader. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Running, Role: LEADER
I20260709 12:25:54.377321 11372 consensus_queue.cc:237] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.384248 11074 catalog_manager.cc:5697] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "0ae87234d66346cf813d2ceb0951efa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:54.394938 11372 raft_consensus.cc:493] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.395466 11372 raft_consensus.cc:515] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.397217 11372 leader_election.cc:290] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.398002 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece17c3d816940f192e055b2c6f87a84" is_pre_election: true
I20260709 12:25:54.398253 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6306229d5c88442ba392bac64b07ca8d" is_pre_election: true
I20260709 12:25:54.398599 11176 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 0.
I20260709 12:25:54.399111 11325 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 0.
I20260709 12:25:54.399755 11217 leader_election.cc:304] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.400616 11372 raft_consensus.cc:2804] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:54.400987 11372 raft_consensus.cc:493] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.401249 11372 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.406492 11372 raft_consensus.cc:515] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.408349 11372 leader_election.cc:290] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 election: Requested vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.409258 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
I20260709 12:25:54.409420 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
I20260709 12:25:54.409894 11176 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.410050 11325 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:54.415302 11176 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 1.
I20260709 12:25:54.415302 11325 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 1.
I20260709 12:25:54.416653 11217 leader_election.cc:304] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.417374 11372 raft_consensus.cc:2804] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:54.417845 11372 raft_consensus.cc:697] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Becoming Leader. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Running, Role: LEADER
I20260709 12:25:54.418509 11372 consensus_queue.cc:237] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.425698 11074 catalog_manager.cc:5697] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "0ae87234d66346cf813d2ceb0951efa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:54.493475 11325 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3"
dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
mode: GRACEFUL
new_leader_uuid: "0ae87234d66346cf813d2ceb0951efa7"
 from {username='slave'} at 127.0.0.1:50714
I20260709 12:25:54.494185 11325 raft_consensus.cc:606] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 LEADER]: Received request to transfer leadership to TS 0ae87234d66346cf813d2ceb0951efa7
I20260709 12:25:54.501865 11378 consensus_queue.cc:1048] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:54.504415 11378 consensus_queue.cc:1048] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:54.517242 11250 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2"
dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
mode: GRACEFUL
new_leader_uuid: "ece17c3d816940f192e055b2c6f87a84"
 from {username='slave'} at 127.0.0.1:60240
I20260709 12:25:54.518193 11250 raft_consensus.cc:606] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Received request to transfer leadership to TS ece17c3d816940f192e055b2c6f87a84
I20260709 12:25:54.526714 11250 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05"
dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
mode: GRACEFUL
new_leader_uuid: "ece17c3d816940f192e055b2c6f87a84"
 from {username='slave'} at 127.0.0.1:60240
I20260709 12:25:54.527375 11250 raft_consensus.cc:606] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Received request to transfer leadership to TS ece17c3d816940f192e055b2c6f87a84
I20260709 12:25:54.598371 11373 consensus_queue.cc:1048] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:54.606626 11407 raft_consensus.cc:993] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: : Instructing follower 0ae87234d66346cf813d2ceb0951efa7 to start an election
I20260709 12:25:54.607187 11373 raft_consensus.cc:1081] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 LEADER]: Signalling peer 0ae87234d66346cf813d2ceb0951efa7 to start an election
I20260709 12:25:54.610342 11251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3"
dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
 from {username='slave'} at 127.0.0.1:60224
I20260709 12:25:54.610978 11251 raft_consensus.cc:493] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:54.612583 11251 raft_consensus.cc:3060] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.623667 11251 raft_consensus.cc:515] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.625958 11408 consensus_queue.cc:1048] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:54.626912 11176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
I20260709 12:25:54.627574 11176 raft_consensus.cc:3060] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.627820 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f27b00a4689544e694a5725d3661d3b3" candidate_uuid: "0ae87234d66346cf813d2ceb0951efa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
I20260709 12:25:54.628446 11325 raft_consensus.cc:3055] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:54.628774 11325 raft_consensus.cc:740] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Running, Role: LEADER
I20260709 12:25:54.629606 11325 consensus_queue.cc:260] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.630642 11325 raft_consensus.cc:3060] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.635957 11176 raft_consensus.cc:2468] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 2.
I20260709 12:25:54.637734 11175 raft_consensus.cc:1240] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Rejecting Update request from peer 6306229d5c88442ba392bac64b07ca8d for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:25:54.637799 11217 leader_election.cc:304] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.638576 11325 raft_consensus.cc:2468] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae87234d66346cf813d2ceb0951efa7 in term 2.
I20260709 12:25:54.638845 11372 raft_consensus.cc:2804] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:54.639452 11372 raft_consensus.cc:697] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 LEADER]: Becoming Leader. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Running, Role: LEADER
I20260709 12:25:54.639497 11251 leader_election.cc:290] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [CANDIDATE]: Term 2 election: Requested vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.640389 11372 consensus_queue.cc:237] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.646781 11073 catalog_manager.cc:5697] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 reported cstate change: term changed from 1 to 2, leader changed from 6306229d5c88442ba392bac64b07ca8d (127.25.202.3) to 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "0ae87234d66346cf813d2ceb0951efa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:54.845952 11372 consensus_queue.cc:1048] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:54.852607 11396 raft_consensus.cc:993] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: : Instructing follower ece17c3d816940f192e055b2c6f87a84 to start an election
I20260709 12:25:54.853291 11372 raft_consensus.cc:1081] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Signalling peer ece17c3d816940f192e055b2c6f87a84 to start an election
I20260709 12:25:54.855391 11176 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2"
dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
 from {username='slave'} at 127.0.0.1:42660
I20260709 12:25:54.856462 11176 raft_consensus.cc:493] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:54.856928 11176 raft_consensus.cc:3060] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.861356 11372 consensus_queue.cc:1048] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:54.865821 11176 raft_consensus.cc:515] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.868647 11176 leader_election.cc:290] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 2 election: Requested vote from peers 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:54.868904 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
I20260709 12:25:54.869541 11251 raft_consensus.cc:3055] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:54.869750 11251 raft_consensus.cc:740] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Running, Role: LEADER
I20260709 12:25:54.870394 11251 consensus_queue.cc:260] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.872108 11324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410412fadbea4ebf838b275fa228bfa2" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
I20260709 12:25:54.872762 11324 raft_consensus.cc:3060] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.873337 11251 raft_consensus.cc:3060] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.880950 11324 raft_consensus.cc:2468] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 2.
I20260709 12:25:54.881433 11251 raft_consensus.cc:2468] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 2.
I20260709 12:25:54.882310 11139 leader_election.cc:304] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6306229d5c88442ba392bac64b07ca8d, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:54.883222 11371 raft_consensus.cc:2804] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:54.883719 11371 raft_consensus.cc:697] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Becoming Leader. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Running, Role: LEADER
I20260709 12:25:54.884536 11371 consensus_queue.cc:237] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:54.890770 11071 catalog_manager.cc:5697] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 reported cstate change: term changed from 1 to 2, leader changed from 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2) to ece17c3d816940f192e055b2c6f87a84 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "ece17c3d816940f192e055b2c6f87a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:54.978837 11372 consensus_queue.cc:1048] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:54.986765 11396 raft_consensus.cc:993] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: : Instructing follower ece17c3d816940f192e055b2c6f87a84 to start an election
I20260709 12:25:54.987282 11396 raft_consensus.cc:1081] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Signalling peer ece17c3d816940f192e055b2c6f87a84 to start an election
I20260709 12:25:54.989411 11175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05"
dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
 from {username='slave'} at 127.0.0.1:42660
I20260709 12:25:54.990422 11175 raft_consensus.cc:493] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:54.990795 11175 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:54.991843 11396 consensus_queue.cc:1048] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:55.000573 11175 raft_consensus.cc:515] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:55.001817 11176 raft_consensus.cc:1240] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Rejecting Update request from peer 0ae87234d66346cf813d2ceb0951efa7 for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:55.003880 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
I20260709 12:25:55.004097 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "ece17c3d816940f192e055b2c6f87a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
I20260709 12:25:55.004568 11251 raft_consensus.cc:3055] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:55.004767 11325 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:55.004956 11251 raft_consensus.cc:740] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ae87234d66346cf813d2ceb0951efa7, State: Running, Role: LEADER
I20260709 12:25:55.005760 11251 consensus_queue.cc:260] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:55.006593 11251 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:55.011153 11325 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 2.
W20260709 12:25:55.011687 11410 consensus_queue.cc:1175] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ece17c3d816940f192e055b2c6f87a84" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0ae87234d66346cf813d2ceb0951efa7 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:55.012387 11251 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece17c3d816940f192e055b2c6f87a84 in term 2.
I20260709 12:25:55.012521 11139 leader_election.cc:304] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6306229d5c88442ba392bac64b07ca8d, ece17c3d816940f192e055b2c6f87a84; no voters: 
I20260709 12:25:55.013403 11175 leader_election.cc:290] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [CANDIDATE]: Term 2 election: Requested vote from peers 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187), 6306229d5c88442ba392bac64b07ca8d (127.25.202.3:44977)
I20260709 12:25:55.014029 11371 raft_consensus.cc:2804] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:55.014817 11371 raft_consensus.cc:697] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Becoming Leader. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Running, Role: LEADER
I20260709 12:25:55.015793 11371 consensus_queue.cc:237] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:55.023242 11071 catalog_manager.cc:5697] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 reported cstate change: term changed from 1 to 2, leader changed from 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2) to ece17c3d816940f192e055b2c6f87a84 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "ece17c3d816940f192e055b2c6f87a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:55.038017 11175 raft_consensus.cc:1275] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Refusing update from remote peer 0ae87234d66346cf813d2ceb0951efa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:55.039451 11372 consensus_queue.cc:1048] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:55.049207 11325 raft_consensus.cc:1275] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Refusing update from remote peer 0ae87234d66346cf813d2ceb0951efa7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:55.050782 11372 consensus_queue.cc:1048] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:55.273257 26408 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:25:55.275110 26408 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:25:55.275653 26408 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:25:55.289882 11175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05"
dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
mode: GRACEFUL
new_leader_uuid: "6306229d5c88442ba392bac64b07ca8d"
 from {username='slave'} at 127.0.0.1:44736
I20260709 12:25:55.290513 11175 raft_consensus.cc:606] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Received request to transfer leadership to TS 6306229d5c88442ba392bac64b07ca8d
I20260709 12:25:55.291983 26408 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:25:55.372200 11251 raft_consensus.cc:1275] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Refusing update from remote peer ece17c3d816940f192e055b2c6f87a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:55.373476 11371 consensus_queue.cc:1048] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:55.381439 11325 raft_consensus.cc:1275] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Refusing update from remote peer ece17c3d816940f192e055b2c6f87a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:55.384847 11378 consensus_queue.cc:1048] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:55.409013 11325 raft_consensus.cc:1275] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Refusing update from remote peer ece17c3d816940f192e055b2c6f87a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:55.410260 11417 consensus_queue.cc:1048] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:55.415632 11378 raft_consensus.cc:993] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: : Instructing follower 6306229d5c88442ba392bac64b07ca8d to start an election
I20260709 12:25:55.416278 11371 raft_consensus.cc:1081] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Signalling peer 6306229d5c88442ba392bac64b07ca8d to start an election
I20260709 12:25:55.418164 11324 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05"
dest_uuid: "6306229d5c88442ba392bac64b07ca8d"
 from {username='slave'} at 127.0.0.1:50708
I20260709 12:25:55.419760 11324 raft_consensus.cc:493] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:55.420192 11324 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:55.421259 11251 raft_consensus.cc:1275] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Refusing update from remote peer ece17c3d816940f192e055b2c6f87a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:55.422710 11417 consensus_queue.cc:1048] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:55.428664 11324 raft_consensus.cc:515] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:55.434232 11324 leader_election.cc:290] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [CANDIDATE]: Term 3 election: Requested vote from peers ece17c3d816940f192e055b2c6f87a84 (127.25.202.1:39677), 0ae87234d66346cf813d2ceb0951efa7 (127.25.202.2:46187)
I20260709 12:25:55.435014 11175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "6306229d5c88442ba392bac64b07ca8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ece17c3d816940f192e055b2c6f87a84"
I20260709 12:25:55.435863 11175 raft_consensus.cc:3055] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:55.436280 11175 raft_consensus.cc:740] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ece17c3d816940f192e055b2c6f87a84, State: Running, Role: LEADER
I20260709 12:25:55.436587 11251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a120a6f8529248a8b310cbf9d141ad05" candidate_uuid: "6306229d5c88442ba392bac64b07ca8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0ae87234d66346cf813d2ceb0951efa7"
I20260709 12:25:55.437281 11251 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:55.438941 11175 consensus_queue.cc:260] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:55.440562 11175 raft_consensus.cc:3060] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:55.446162 11251 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6306229d5c88442ba392bac64b07ca8d in term 3.
I20260709 12:25:55.447606 11291 leader_election.cc:304] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae87234d66346cf813d2ceb0951efa7, 6306229d5c88442ba392bac64b07ca8d; no voters: 
I20260709 12:25:55.448534 11418 raft_consensus.cc:2804] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:55.448764 11175 raft_consensus.cc:2468] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6306229d5c88442ba392bac64b07ca8d in term 3.
I20260709 12:25:55.449131 11418 raft_consensus.cc:697] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 3 LEADER]: Becoming Leader. State: Replica: 6306229d5c88442ba392bac64b07ca8d, State: Running, Role: LEADER
I20260709 12:25:55.450161 11418 consensus_queue.cc:237] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } }
I20260709 12:25:55.456068 11073 catalog_manager.cc:5697] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d reported cstate change: term changed from 2 to 3, leader changed from ece17c3d816940f192e055b2c6f87a84 (127.25.202.1) to 6306229d5c88442ba392bac64b07ca8d (127.25.202.3). New cstate: current_term: 3 leader_uuid: "6306229d5c88442ba392bac64b07ca8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece17c3d816940f192e055b2c6f87a84" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae87234d66346cf813d2ceb0951efa7" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6306229d5c88442ba392bac64b07ca8d" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 44977 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:55.513895 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
I20260709 12:25:55.535753 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:55.536473 26408 tablet_replica.cc:333] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84: stopping tablet replica
I20260709 12:25:55.537034 26408 raft_consensus.cc:2243] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.537503 26408 raft_consensus.cc:2272] T f27b00a4689544e694a5725d3661d3b3 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.539863 26408 tablet_replica.cc:333] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84: stopping tablet replica
I20260709 12:25:55.540460 26408 raft_consensus.cc:2243] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:55.541280 26408 raft_consensus.cc:2272] T 410412fadbea4ebf838b275fa228bfa2 P ece17c3d816940f192e055b2c6f87a84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.543637 26408 tablet_replica.cc:333] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84: stopping tablet replica
I20260709 12:25:55.544148 26408 raft_consensus.cc:2243] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.544566 26408 raft_consensus.cc:2272] T a120a6f8529248a8b310cbf9d141ad05 P ece17c3d816940f192e055b2c6f87a84 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.546818 26408 tablet_replica.cc:333] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84: stopping tablet replica
I20260709 12:25:55.547392 26408 raft_consensus.cc:2243] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:55.548220 26408 raft_consensus.cc:2272] T 8dfc510575a24b458b19bded69eb8576 P ece17c3d816940f192e055b2c6f87a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.562837 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
I20260709 12:25:55.576201 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
I20260709 12:25:55.594480 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:55.595124 26408 tablet_replica.cc:333] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7: stopping tablet replica
I20260709 12:25:55.595765 26408 raft_consensus.cc:2243] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:25:55.596624 26408 raft_consensus.cc:2272] T f27b00a4689544e694a5725d3661d3b3 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.598853 26408 tablet_replica.cc:333] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7: stopping tablet replica
I20260709 12:25:55.599474 26408 raft_consensus.cc:2243] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.599911 26408 raft_consensus.cc:2272] T 410412fadbea4ebf838b275fa228bfa2 P 0ae87234d66346cf813d2ceb0951efa7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.602017 26408 tablet_replica.cc:333] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7: stopping tablet replica
I20260709 12:25:55.602530 26408 raft_consensus.cc:2243] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.602965 26408 raft_consensus.cc:2272] T a120a6f8529248a8b310cbf9d141ad05 P 0ae87234d66346cf813d2ceb0951efa7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.604990 26408 tablet_replica.cc:333] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7: stopping tablet replica
I20260709 12:25:55.605479 26408 raft_consensus.cc:2243] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.605849 26408 raft_consensus.cc:2272] T 8dfc510575a24b458b19bded69eb8576 P 0ae87234d66346cf813d2ceb0951efa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.627820 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
I20260709 12:25:55.640494 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
I20260709 12:25:55.660307 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:55.660996 26408 tablet_replica.cc:333] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: stopping tablet replica
I20260709 12:25:55.661617 26408 raft_consensus.cc:2243] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:25:55.662298 26408 pending_rounds.cc:70] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: Trying to abort 1 pending ops.
I20260709 12:25:55.662519 26408 pending_rounds.cc:77] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305625417532768256 op_type: NO_OP noop_request { }
I20260709 12:25:55.662830 26408 raft_consensus.cc:2889] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:55.663115 26408 raft_consensus.cc:2272] T a120a6f8529248a8b310cbf9d141ad05 P 6306229d5c88442ba392bac64b07ca8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.665448 26408 tablet_replica.cc:333] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d: stopping tablet replica
I20260709 12:25:55.665998 26408 raft_consensus.cc:2243] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.666446 26408 raft_consensus.cc:2272] T 410412fadbea4ebf838b275fa228bfa2 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.669190 26408 tablet_replica.cc:333] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d: stopping tablet replica
I20260709 12:25:55.669698 26408 raft_consensus.cc:2243] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.670109 26408 raft_consensus.cc:2272] T 8dfc510575a24b458b19bded69eb8576 P 6306229d5c88442ba392bac64b07ca8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.672256 26408 tablet_replica.cc:333] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d: stopping tablet replica
I20260709 12:25:55.672835 26408 raft_consensus.cc:2243] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:55.673254 26408 raft_consensus.cc:2272] T f27b00a4689544e694a5725d3661d3b3 P 6306229d5c88442ba392bac64b07ca8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.696226 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:25:55.709738 26408 master.cc:562] Master@127.25.202.62:37423 shutting down...
I20260709 12:25:55.726621 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:25:55.727210 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:55.727617 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P a1989576226e4c719723b4a00433924d: stopping tablet replica
I20260709 12:25:55.746100 26408 master.cc:584] Master@127.25.202.62:37423 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (2623 ms)
[----------] 6 tests from LeaderRebalancerTest (567771 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:25:55.777107 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:33011
I20260709 12:25:55.778452 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:55.784209 11419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:55.784437 11420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:55.785609 11422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:55.787206 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:55.788036 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:55.788208 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:55.788411 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599955788369 us; error 0 us; skew 500 ppm
I20260709 12:25:55.788992 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:55.791499 26408 webserver.cc:533] Webserver started at http://127.25.202.62:37145/ using document root <none> and password file <none>
I20260709 12:25:55.791997 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:55.792186 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:55.792452 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:55.793507 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "b44b70e854b446b4956c66823c526813"
format_stamp: "Formatted at 2026-07-09 12:25:55 on dist-test-slave-m57n"
I20260709 12:25:55.798087 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 12:25:55.801594 11427 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:55.802425 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:25:55.802716 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "b44b70e854b446b4956c66823c526813"
format_stamp: "Formatted at 2026-07-09 12:25:55 on dist-test-slave-m57n"
I20260709 12:25:55.803015 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:55.813716 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:55.814718 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:55.853276 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:33011
I20260709 12:25:55.853400 11478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:33011 every 8 connection(s)
I20260709 12:25:55.857329 11479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:55.867928 11479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813: Bootstrap starting.
I20260709 12:25:55.872404 11479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:55.876569 11479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813: No bootstrap required, opened a new log
I20260709 12:25:55.878686 11479 raft_consensus.cc:359] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44b70e854b446b4956c66823c526813" member_type: VOTER }
I20260709 12:25:55.879088 11479 raft_consensus.cc:385] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:55.879323 11479 raft_consensus.cc:740] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44b70e854b446b4956c66823c526813, State: Initialized, Role: FOLLOWER
I20260709 12:25:55.879935 11479 consensus_queue.cc:260] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44b70e854b446b4956c66823c526813" member_type: VOTER }
I20260709 12:25:55.880424 11479 raft_consensus.cc:399] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:25:55.880664 11479 raft_consensus.cc:493] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:25:55.880920 11479 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:55.886023 11479 raft_consensus.cc:515] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44b70e854b446b4956c66823c526813" member_type: VOTER }
I20260709 12:25:55.886576 11479 leader_election.cc:304] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b44b70e854b446b4956c66823c526813; no voters: 
I20260709 12:25:55.887907 11479 leader_election.cc:290] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:25:55.888329 11482 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:55.889706 11482 raft_consensus.cc:697] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 1 LEADER]: Becoming Leader. State: Replica: b44b70e854b446b4956c66823c526813, State: Running, Role: LEADER
I20260709 12:25:55.890369 11482 consensus_queue.cc:237] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44b70e854b446b4956c66823c526813" member_type: VOTER }
I20260709 12:25:55.890815 11479 sys_catalog.cc:565] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:25:55.893237 11483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b44b70e854b446b4956c66823c526813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44b70e854b446b4956c66823c526813" member_type: VOTER } }
I20260709 12:25:55.893251 11484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b44b70e854b446b4956c66823c526813. Latest consensus state: current_term: 1 leader_uuid: "b44b70e854b446b4956c66823c526813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44b70e854b446b4956c66823c526813" member_type: VOTER } }
I20260709 12:25:55.894001 11483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:55.894073 11484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [sys.catalog]: This master's current role is: LEADER
I20260709 12:25:55.897209 11487 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:25:55.901958 11487 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:25:55.907423 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:25:55.911448 11487 catalog_manager.cc:1383] Generated new cluster ID: 03a706e352d8425f84c3cd7effc1697d
I20260709 12:25:55.911720 11487 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:25:55.937623 11487 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:25:55.938887 11487 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:25:55.959203 11487 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813: Generated new TSK 0
I20260709 12:25:55.959964 11487 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:25:55.974884 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:55.981637 11500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:55.982923 11501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:55.990896 26408 server_base.cc:1061] running on GCE node
W20260709 12:25:55.991062 11503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:55.992298 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:55.992574 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:55.992766 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599955992747 us; error 0 us; skew 500 ppm
I20260709 12:25:55.993327 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:55.995849 26408 webserver.cc:533] Webserver started at http://127.25.202.1:32817/ using document root <none> and password file <none>
I20260709 12:25:55.996332 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:55.996520 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:55.996747 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:55.997874 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "f00bca05e5804522901cee258b1290a1"
format_stamp: "Formatted at 2026-07-09 12:25:55 on dist-test-slave-m57n"
I20260709 12:25:56.002915 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:25:56.006959 11508 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.008042 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:25:56.008414 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "f00bca05e5804522901cee258b1290a1"
format_stamp: "Formatted at 2026-07-09 12:25:55 on dist-test-slave-m57n"
I20260709 12:25:56.008739 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:56.025372 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:56.026535 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:56.028074 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:56.030378 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:56.030613 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.030920 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:56.031090 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.070739 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:42087
I20260709 12:25:56.070827 11570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:42087 every 8 connection(s)
I20260709 12:25:56.076327 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:56.084651 11575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:56.086874 11576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:56.095588 11571 heartbeater.cc:344] Connected to a master server at 127.25.202.62:33011
W20260709 12:25:56.096041 11578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:56.096192 11571 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:56.097184 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:56.097633 11571 heartbeater.cc:507] Master 127.25.202.62:33011 requested a full tablet report, sending...
I20260709 12:25:56.099283 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:56.099654 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:56.099869 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599956099829 us; error 0 us; skew 500 ppm
I20260709 12:25:56.100404 11444 ts_manager.cc:194] Registered new tserver with Master: f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087)
I20260709 12:25:56.100551 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:56.102420 11444 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33266
I20260709 12:25:56.104971 26408 webserver.cc:533] Webserver started at http://127.25.202.2:40681/ using document root <none> and password file <none>
I20260709 12:25:56.105743 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:56.105937 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:56.106164 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:56.107661 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
format_stamp: "Formatted at 2026-07-09 12:25:56 on dist-test-slave-m57n"
I20260709 12:25:56.112825 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:25:56.116505 11583 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.117506 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:25:56.117764 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
format_stamp: "Formatted at 2026-07-09 12:25:56 on dist-test-slave-m57n"
I20260709 12:25:56.118086 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:56.138476 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:56.139631 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:56.141085 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:56.143386 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:56.143611 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.143887 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:56.144047 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.182098 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:34629
I20260709 12:25:56.182205 11645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:34629 every 8 connection(s)
I20260709 12:25:56.187198 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:25:56.194690 11649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:56.196271 11650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:25:56.198813 11652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:25:56.199550 26408 server_base.cc:1061] running on GCE node
I20260709 12:25:56.200680 11646 heartbeater.cc:344] Connected to a master server at 127.25.202.62:33011
I20260709 12:25:56.200737 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:25:56.201175 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:25:56.201233 11646 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:56.201527 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599956201506 us; error 0 us; skew 500 ppm
I20260709 12:25:56.202196 11646 heartbeater.cc:507] Master 127.25.202.62:33011 requested a full tablet report, sending...
I20260709 12:25:56.202385 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:25:56.204579 11444 ts_manager.cc:194] Registered new tserver with Master: 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:25:56.205942 11444 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33272
I20260709 12:25:56.207093 26408 webserver.cc:533] Webserver started at http://127.25.202.3:44669/ using document root <none> and password file <none>
I20260709 12:25:56.207683 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:25:56.207942 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:25:56.208211 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:25:56.209323 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "3c9e1ca3ff074690b75a039a85d60299"
format_stamp: "Formatted at 2026-07-09 12:25:56 on dist-test-slave-m57n"
I20260709 12:25:56.214046 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:25:56.217324 11657 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.218179 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:25:56.218518 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "3c9e1ca3ff074690b75a039a85d60299"
format_stamp: "Formatted at 2026-07-09 12:25:56 on dist-test-slave-m57n"
I20260709 12:25:56.218825 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:25:56.239162 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:25:56.240443 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:25:56.242048 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:25:56.244619 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:25:56.244886 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.245177 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:25:56.245385 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:25:56.286391 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:37239
I20260709 12:25:56.286492 11719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:37239 every 8 connection(s)
I20260709 12:25:56.301100 11720 heartbeater.cc:344] Connected to a master server at 127.25.202.62:33011
I20260709 12:25:56.301687 11720 heartbeater.cc:461] Registering TS with master...
I20260709 12:25:56.302387 11720 heartbeater.cc:507] Master 127.25.202.62:33011 requested a full tablet report, sending...
I20260709 12:25:56.304452 11444 ts_manager.cc:194] Registered new tserver with Master: 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:56.304992 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014945401s
I20260709 12:25:56.306206 11444 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33278
I20260709 12:25:56.332041 11444 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33292:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:25:56.334635 11444 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:25:56.416630 11611 tablet_service.cc:1511] Processing CreateTablet for tablet b5a95d90e44047ad8c258903bdb1f9eb (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:25:56.417397 11608 tablet_service.cc:1511] Processing CreateTablet for tablet c725053672514cf69f7eec8351d1759e (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:25:56.418061 11611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a95d90e44047ad8c258903bdb1f9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.418800 11608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c725053672514cf69f7eec8351d1759e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.420859 11607 tablet_service.cc:1511] Processing CreateTablet for tablet d24b1b461ca143e4a8f686d24d9b0751 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:25:56.422312 11607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d24b1b461ca143e4a8f686d24d9b0751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.422575 11685 tablet_service.cc:1511] Processing CreateTablet for tablet b5a95d90e44047ad8c258903bdb1f9eb (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:25:56.423156 11684 tablet_service.cc:1511] Processing CreateTablet for tablet 22f1c13eb88b4a56ad4eca20821a17e9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:25:56.424696 11684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f1c13eb88b4a56ad4eca20821a17e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.424402 11609 tablet_service.cc:1511] Processing CreateTablet for tablet 6d7b3decd14c4619bb62776d919d8ae9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:25:56.425776 11609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7b3decd14c4619bb62776d919d8ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.427249 11685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a95d90e44047ad8c258903bdb1f9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.432514 11610 tablet_service.cc:1511] Processing CreateTablet for tablet 22f1c13eb88b4a56ad4eca20821a17e9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:25:56.434036 11610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f1c13eb88b4a56ad4eca20821a17e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.437054 11683 tablet_service.cc:1511] Processing CreateTablet for tablet 6d7b3decd14c4619bb62776d919d8ae9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:25:56.438270 11606 tablet_service.cc:1511] Processing CreateTablet for tablet 80924118d6e74310b99c0478496ae0b9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:25:56.439787 11606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80924118d6e74310b99c0478496ae0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.440680 11682 tablet_service.cc:1511] Processing CreateTablet for tablet c725053672514cf69f7eec8351d1759e (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:25:56.438684 11683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7b3decd14c4619bb62776d919d8ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.443819 11681 tablet_service.cc:1511] Processing CreateTablet for tablet d24b1b461ca143e4a8f686d24d9b0751 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:25:56.445346 11681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d24b1b461ca143e4a8f686d24d9b0751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.447716 11682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c725053672514cf69f7eec8351d1759e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.457141 11605 tablet_service.cc:1511] Processing CreateTablet for tablet 5a17e2a0a7964ec4a956743fe816de45 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:25:56.458652 11605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a17e2a0a7964ec4a956743fe816de45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.465286 11536 tablet_service.cc:1511] Processing CreateTablet for tablet b5a95d90e44047ad8c258903bdb1f9eb (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:25:56.466886 11536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a95d90e44047ad8c258903bdb1f9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.466531 11604 tablet_service.cc:1511] Processing CreateTablet for tablet 735953c802fa471ba0d68511ee3495e5 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:25:56.468011 11604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735953c802fa471ba0d68511ee3495e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.473209 11603 tablet_service.cc:1511] Processing CreateTablet for tablet c3d00baea3d94637ab044e58ce707954 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:25:56.475186 11603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d00baea3d94637ab044e58ce707954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.489871 11535 tablet_service.cc:1511] Processing CreateTablet for tablet 22f1c13eb88b4a56ad4eca20821a17e9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:25:56.493633 11535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f1c13eb88b4a56ad4eca20821a17e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.501925 11534 tablet_service.cc:1511] Processing CreateTablet for tablet 6d7b3decd14c4619bb62776d919d8ae9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:25:56.504123 11534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7b3decd14c4619bb62776d919d8ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.514946 11533 tablet_service.cc:1511] Processing CreateTablet for tablet c725053672514cf69f7eec8351d1759e (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:25:56.516465 11533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c725053672514cf69f7eec8351d1759e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.520536 11532 tablet_service.cc:1511] Processing CreateTablet for tablet d24b1b461ca143e4a8f686d24d9b0751 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:25:56.521313 11680 tablet_service.cc:1511] Processing CreateTablet for tablet 80924118d6e74310b99c0478496ae0b9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:25:56.522006 11532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d24b1b461ca143e4a8f686d24d9b0751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.522701 11680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80924118d6e74310b99c0478496ae0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.524397 11531 tablet_service.cc:1511] Processing CreateTablet for tablet 80924118d6e74310b99c0478496ae0b9 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:25:56.527495 11530 tablet_service.cc:1511] Processing CreateTablet for tablet 5a17e2a0a7964ec4a956743fe816de45 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:25:56.528836 11530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a17e2a0a7964ec4a956743fe816de45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.531677 11531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80924118d6e74310b99c0478496ae0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.532047 11679 tablet_service.cc:1511] Processing CreateTablet for tablet 5a17e2a0a7964ec4a956743fe816de45 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:25:56.533385 11679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a17e2a0a7964ec4a956743fe816de45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.537938 11736 tablet_bootstrap.cc:492] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.545213 11529 tablet_service.cc:1511] Processing CreateTablet for tablet 735953c802fa471ba0d68511ee3495e5 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:25:56.545953 11678 tablet_service.cc:1511] Processing CreateTablet for tablet 735953c802fa471ba0d68511ee3495e5 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:25:56.546711 11529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735953c802fa471ba0d68511ee3495e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.547351 11678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735953c802fa471ba0d68511ee3495e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.553325 11528 tablet_service.cc:1511] Processing CreateTablet for tablet c3d00baea3d94637ab044e58ce707954 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:25:56.554746 11528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d00baea3d94637ab044e58ce707954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.556579 11677 tablet_service.cc:1511] Processing CreateTablet for tablet c3d00baea3d94637ab044e58ce707954 (DEFAULT_TABLE table=test-workload [id=cfe6143711f746c58474ab9e705279f7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:25:56.558291 11677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d00baea3d94637ab044e58ce707954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:56.565963 11736 tablet_bootstrap.cc:654] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.580679 11735 tablet_bootstrap.cc:492] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:56.631201 11735 tablet_bootstrap.cc:654] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.641276 11737 tablet_bootstrap.cc:492] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.653973 11736 tablet_bootstrap.cc:492] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.654568 11736 ts_tablet_manager.cc:1403] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.117s	user 0.016s	sys 0.000s
I20260709 12:25:56.655753 11735 tablet_bootstrap.cc:492] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:56.656323 11735 ts_tablet_manager.cc:1403] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.076s	user 0.005s	sys 0.015s
I20260709 12:25:56.657109 11737 tablet_bootstrap.cc:654] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.657589 11736 raft_consensus.cc:359] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.658372 11736 raft_consensus.cc:385] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.658681 11736 raft_consensus.cc:740] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.659526 11736 consensus_queue.cc:260] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.659772 11735 raft_consensus.cc:359] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.660595 11735 raft_consensus.cc:385] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.660953 11735 raft_consensus.cc:740] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.662173 11737 tablet_bootstrap.cc:492] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:56.662133 11646 heartbeater.cc:499] Master 127.25.202.62:33011 was elected leader, sending a full tablet report...
I20260709 12:25:56.661994 11735 consensus_queue.cc:260] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.662652 11736 ts_tablet_manager.cc:1434] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:25:56.663640 11737 ts_tablet_manager.cc:1403] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.001s
I20260709 12:25:56.663968 11736 tablet_bootstrap.cc:492] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.666725 11737 raft_consensus.cc:359] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.667521 11737 raft_consensus.cc:385] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.667869 11737 raft_consensus.cc:740] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.668659 11737 consensus_queue.cc:260] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.671715 11736 tablet_bootstrap.cc:654] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.672056 11720 heartbeater.cc:499] Master 127.25.202.62:33011 was elected leader, sending a full tablet report...
I20260709 12:25:56.671741 11735 ts_tablet_manager.cc:1434] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.012s
I20260709 12:25:56.684118 11735 tablet_bootstrap.cc:492] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:56.694600 11735 tablet_bootstrap.cc:654] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.704523 11737 ts_tablet_manager.cc:1434] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.040s	user 0.013s	sys 0.023s
I20260709 12:25:56.705708 11571 heartbeater.cc:499] Master 127.25.202.62:33011 was elected leader, sending a full tablet report...
I20260709 12:25:56.712001 11735 tablet_bootstrap.cc:492] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:56.712456 11736 tablet_bootstrap.cc:492] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.712679 11735 ts_tablet_manager.cc:1403] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.006s
I20260709 12:25:56.713058 11736 ts_tablet_manager.cc:1403] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.049s	user 0.023s	sys 0.017s
I20260709 12:25:56.714493 11737 tablet_bootstrap.cc:492] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.715706 11736 raft_consensus.cc:359] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.716359 11736 raft_consensus.cc:385] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.716109 11735 raft_consensus.cc:359] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.716765 11736 raft_consensus.cc:740] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.717001 11735 raft_consensus.cc:385] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.717401 11735 raft_consensus.cc:740] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.717633 11736 consensus_queue.cc:260] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.718329 11735 consensus_queue.cc:260] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.719864 11736 ts_tablet_manager.cc:1434] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.006s
I20260709 12:25:56.721074 11736 tablet_bootstrap.cc:492] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.724661 11735 ts_tablet_manager.cc:1434] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:25:56.725903 11735 tablet_bootstrap.cc:492] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:56.725937 11736 tablet_bootstrap.cc:654] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.728993 11737 tablet_bootstrap.cc:654] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.743619 11735 tablet_bootstrap.cc:654] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.750213 11736 tablet_bootstrap.cc:492] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.750705 11736 ts_tablet_manager.cc:1403] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.030s	user 0.020s	sys 0.008s
I20260709 12:25:56.752992 11736 raft_consensus.cc:359] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.753578 11736 raft_consensus.cc:385] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.753906 11736 raft_consensus.cc:740] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.754868 11736 consensus_queue.cc:260] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.756526 11737 tablet_bootstrap.cc:492] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:56.757133 11737 ts_tablet_manager.cc:1403] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.019s
I20260709 12:25:56.759948 11737 raft_consensus.cc:359] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.760581 11736 ts_tablet_manager.cc:1434] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:25:56.760707 11737 raft_consensus.cc:385] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.761332 11737 raft_consensus.cc:740] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.762262 11736 tablet_bootstrap.cc:492] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.762121 11737 consensus_queue.cc:260] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.764379 11737 ts_tablet_manager.cc:1434] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:25:56.765408 11737 tablet_bootstrap.cc:492] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.772079 11736 tablet_bootstrap.cc:654] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.772348 11737 tablet_bootstrap.cc:654] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.774173 11735 tablet_bootstrap.cc:492] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:56.774745 11735 ts_tablet_manager.cc:1403] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.049s	user 0.013s	sys 0.016s
I20260709 12:25:56.778889 11735 raft_consensus.cc:359] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.779912 11735 raft_consensus.cc:385] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.780293 11735 raft_consensus.cc:740] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.781399 11735 consensus_queue.cc:260] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.784409 11735 ts_tablet_manager.cc:1434] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:25:56.784603 11737 tablet_bootstrap.cc:492] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:56.785365 11737 ts_tablet_manager.cc:1403] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.004s
I20260709 12:25:56.785844 11735 tablet_bootstrap.cc:492] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:56.788442 11737 raft_consensus.cc:359] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.789203 11737 raft_consensus.cc:385] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.789562 11737 raft_consensus.cc:740] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.791374 11743 raft_consensus.cc:493] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.791880 11743 raft_consensus.cc:515] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.790452 11737 consensus_queue.cc:260] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.795063 11736 tablet_bootstrap.cc:492] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.796861 11736 ts_tablet_manager.cc:1403] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.035s	user 0.021s	sys 0.011s
I20260709 12:25:56.797740 11743 leader_election.cc:290] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:56.795065 11735 tablet_bootstrap.cc:654] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.800190 11736 raft_consensus.cc:359] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.801039 11736 raft_consensus.cc:385] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.801324 11736 raft_consensus.cc:740] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.802140 11736 consensus_queue.cc:260] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.805505 11736 ts_tablet_manager.cc:1434] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:25:56.806891 11736 tablet_bootstrap.cc:492] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.809489 11737 ts_tablet_manager.cc:1434] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.007s
I20260709 12:25:56.810925 11737 tablet_bootstrap.cc:492] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.815558 11736 tablet_bootstrap.cc:654] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.826190 11737 tablet_bootstrap.cc:654] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.829005 11736 tablet_bootstrap.cc:492] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.829797 11736 ts_tablet_manager.cc:1403] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.012s
I20260709 12:25:56.833966 11736 raft_consensus.cc:359] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.835106 11736 raft_consensus.cc:385] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.835568 11736 raft_consensus.cc:740] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.836856 11736 consensus_queue.cc:260] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.839152 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d7b3decd14c4619bb62776d919d8ae9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:56.839962 11695 raft_consensus.cc:2468] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:56.840945 11742 raft_consensus.cc:493] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.841398 11510 leader_election.cc:304] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:56.841907 11736 ts_tablet_manager.cc:1434] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.004s
I20260709 12:25:56.841471 11742 raft_consensus.cc:515] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.842644 11751 raft_consensus.cc:2804] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:56.843065 11751 raft_consensus.cc:493] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.843381 11736 tablet_bootstrap.cc:492] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.843418 11751 raft_consensus.cc:3060] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:56.850466 11736 tablet_bootstrap.cc:654] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.851506 11751 raft_consensus.cc:515] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.853989 11742 leader_election.cc:290] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087)
I20260709 12:25:56.855373 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d7b3decd14c4619bb62776d919d8ae9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:56.856201 11695 raft_consensus.cc:3060] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:56.860667 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d7b3decd14c4619bb62776d919d8ae9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:56.861622 11621 raft_consensus.cc:2468] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:56.862588 11737 tablet_bootstrap.cc:492] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:56.863137 11737 ts_tablet_manager.cc:1403] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.053s	user 0.011s	sys 0.000s
I20260709 12:25:56.863528 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d7b3decd14c4619bb62776d919d8ae9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:56.864153 11621 raft_consensus.cc:3060] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:56.866410 11737 raft_consensus.cc:359] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.867460 11751 leader_election.cc:290] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:56.868378 11695 raft_consensus.cc:2468] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:56.870040 11510 leader_election.cc:304] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:56.871183 11743 raft_consensus.cc:493] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.871855 11743 raft_consensus.cc:515] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.875860 11743 leader_election.cc:290] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:56.877017 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:56.880309 11621 raft_consensus.cc:2468] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:56.867167 11737 raft_consensus.cc:385] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.880604 11694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
W20260709 12:25:56.881807 11509 leader_election.cc:343] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:25:56.882702 11510 leader_election.cc:343] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:56.883298 11510 leader_election.cc:304] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f00bca05e5804522901cee258b1290a1; no voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:56.886845 11735 tablet_bootstrap.cc:492] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:56.887480 11743 raft_consensus.cc:2804] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:56.888793 11735 ts_tablet_manager.cc:1403] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.103s	user 0.020s	sys 0.028s
I20260709 12:25:56.890050 11737 raft_consensus.cc:740] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.891531 11751 raft_consensus.cc:2749] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:56.891079 11737 consensus_queue.cc:260] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.892486 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
W20260709 12:25:56.893832 11658 leader_election.cc:343] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:56.894908 11737 ts_tablet_manager.cc:1434] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.031s	user 0.003s	sys 0.004s
I20260709 12:25:56.896090 11737 tablet_bootstrap.cc:492] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.895749 11735 raft_consensus.cc:359] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.899773 11735 raft_consensus.cc:385] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.900164 11735 raft_consensus.cc:740] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.900619 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:56.901350 11743 raft_consensus.cc:697] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
W20260709 12:25:56.902328 11492 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:25:56.902279 11743 consensus_queue.cc:237] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
W20260709 12:25:56.902364 11659 leader_election.cc:343] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:56.903226 11659 leader_election.cc:304] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299; no voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1
I20260709 12:25:56.904079 11742 raft_consensus.cc:2749] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:56.905689 11737 tablet_bootstrap.cc:654] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.905983 11736 tablet_bootstrap.cc:492] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.906554 11736 ts_tablet_manager.cc:1403] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.063s	user 0.032s	sys 0.024s
I20260709 12:25:56.901036 11735 consensus_queue.cc:260] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.909925 11736 raft_consensus.cc:359] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.910672 11736 raft_consensus.cc:385] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.910985 11736 raft_consensus.cc:740] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.912111 11735 ts_tablet_manager.cc:1434] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 12:25:56.911888 11736 consensus_queue.cc:260] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.912214 11444 catalog_manager.cc:5697] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:56.913358 11735 tablet_bootstrap.cc:492] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:56.915004 11736 ts_tablet_manager.cc:1434] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:25:56.916175 11736 tablet_bootstrap.cc:492] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.921270 11735 tablet_bootstrap.cc:654] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.921274 11737 tablet_bootstrap.cc:492] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:56.922632 11737 ts_tablet_manager.cc:1403] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.027s	user 0.013s	sys 0.001s
I20260709 12:25:56.922874 11736 tablet_bootstrap.cc:654] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.928864 11737 raft_consensus.cc:359] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.930004 11737 raft_consensus.cc:385] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.930536 11737 raft_consensus.cc:740] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.941386 11737 consensus_queue.cc:260] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.945688 11735 tablet_bootstrap.cc:492] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:56.946441 11735 ts_tablet_manager.cc:1403] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.019s
I20260709 12:25:56.947793 11737 ts_tablet_manager.cc:1434] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.004s
I20260709 12:25:56.948961 11737 tablet_bootstrap.cc:492] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.949644 11735 raft_consensus.cc:359] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.950452 11735 raft_consensus.cc:385] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.950752 11735 raft_consensus.cc:740] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.945840 11736 tablet_bootstrap.cc:492] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.951748 11736 ts_tablet_manager.cc:1403] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.036s	user 0.017s	sys 0.012s
I20260709 12:25:56.951717 11735 consensus_queue.cc:260] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.955202 11736 raft_consensus.cc:359] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.956007 11736 raft_consensus.cc:385] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.956305 11736 raft_consensus.cc:740] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.956568 11737 tablet_bootstrap.cc:654] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.957106 11736 consensus_queue.cc:260] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.959635 11736 ts_tablet_manager.cc:1434] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:25:56.960669 11736 tablet_bootstrap.cc:492] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:56.966419 11735 ts_tablet_manager.cc:1434] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.004s
I20260709 12:25:56.967417 11736 tablet_bootstrap.cc:654] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.967777 11735 tablet_bootstrap.cc:492] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:56.970423 11743 raft_consensus.cc:493] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.971135 11743 raft_consensus.cc:515] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.975121 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
W20260709 12:25:56.977150 11509 leader_election.cc:343] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:56.977155 11737 tablet_bootstrap.cc:492] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:56.978564 11737 ts_tablet_manager.cc:1403] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.030s	user 0.007s	sys 0.012s
I20260709 12:25:56.980434 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:56.981132 11695 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:56.982630 11736 tablet_bootstrap.cc:492] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:56.982159 11737 raft_consensus.cc:359] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.975299 11735 tablet_bootstrap.cc:654] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.983021 11737 raft_consensus.cc:385] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.982465 11510 leader_election.cc:304] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:56.983426 11737 raft_consensus.cc:740] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.983932 11760 raft_consensus.cc:2804] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:56.984344 11760 raft_consensus.cc:493] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.984229 11737 consensus_queue.cc:260] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.984746 11760 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:56.986819 11736 ts_tablet_manager.cc:1403] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.012s
I20260709 12:25:56.986933 11737 ts_tablet_manager.cc:1434] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:25:56.988104 11737 tablet_bootstrap.cc:492] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:56.988922 11735 tablet_bootstrap.cc:492] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:56.989450 11735 ts_tablet_manager.cc:1403] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.000s
I20260709 12:25:56.989859 11736 raft_consensus.cc:359] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.990597 11736 raft_consensus.cc:385] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.990882 11736 raft_consensus.cc:740] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.991643 11736 consensus_queue.cc:260] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.992753 11735 raft_consensus.cc:359] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.993610 11735 raft_consensus.cc:385] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:56.993922 11735 raft_consensus.cc:740] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:56.994880 11737 tablet_bootstrap.cc:654] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:56.994918 11735 consensus_queue.cc:260] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:56.998258 11735 ts_tablet_manager.cc:1434] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:25:56.999372 11735 tablet_bootstrap.cc:492] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:57.002295 11736 ts_tablet_manager.cc:1434] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.015s	user 0.000s	sys 0.006s
I20260709 12:25:57.002857 11743 leader_election.cc:290] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.008044 11736 tablet_bootstrap.cc:492] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:25:57.006081 11735 tablet_bootstrap.cc:654] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:57.014068 11737 tablet_bootstrap.cc:492] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:57.013639 11760 raft_consensus.cc:515] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.014639 11737 ts_tablet_manager.cc:1403] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.027s	user 0.022s	sys 0.001s
I20260709 12:25:57.014978 11736 tablet_bootstrap.cc:654] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:57.016757 11760 leader_election.cc:290] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.017244 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.017391 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.018131 11695 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.018143 11621 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.022528 11735 tablet_bootstrap.cc:492] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:57.023119 11735 ts_tablet_manager.cc:1403] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.006s
I20260709 12:25:57.026355 11735 raft_consensus.cc:359] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.027083 11735 raft_consensus.cc:385] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.027428 11735 raft_consensus.cc:740] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.027485 11621 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:57.028416 11735 consensus_queue.cc:260] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.028858 11509 leader_election.cc:304] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.029122 11695 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:57.029901 11760 raft_consensus.cc:2804] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.031204 11760 raft_consensus.cc:697] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:57.032289 11736 tablet_bootstrap.cc:492] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:25:57.032101 11760 consensus_queue.cc:237] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.032948 11736 ts_tablet_manager.cc:1403] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.008s
I20260709 12:25:57.032305 11737 raft_consensus.cc:359] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.033514 11737 raft_consensus.cc:385] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.035676 11737 raft_consensus.cc:740] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.036056 11736 raft_consensus.cc:359] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.036480 11737 consensus_queue.cc:260] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.037166 11736 raft_consensus.cc:385] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.037606 11736 raft_consensus.cc:740] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.038364 11736 consensus_queue.cc:260] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.039347 11737 ts_tablet_manager.cc:1434] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260709 12:25:57.040689 11736 ts_tablet_manager.cc:1434] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:25:57.040891 11737 tablet_bootstrap.cc:492] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:57.042584 11735 ts_tablet_manager.cc:1434] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 12:25:57.043730 11735 tablet_bootstrap.cc:492] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:57.045529 11741 raft_consensus.cc:493] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.046090 11741 raft_consensus.cc:515] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.048939 11737 tablet_bootstrap.cc:654] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:57.050493 11735 tablet_bootstrap.cc:654] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:57.051965 11741 leader_election.cc:290] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.054610 11444 catalog_manager.cc:5697] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.061920 11742 raft_consensus.cc:493] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.062561 11742 raft_consensus.cc:515] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.064997 11742 leader_election.cc:290] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:25:57.068328 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.069377 11546 raft_consensus.cc:2468] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:25:57.070452 11737 tablet_bootstrap.cc:492] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:57.071115 11737 ts_tablet_manager.cc:1403] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.030s	user 0.016s	sys 0.008s
I20260709 12:25:57.071647 11743 raft_consensus.cc:493] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.072386 11743 raft_consensus.cc:515] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.074656 11737 raft_consensus.cc:359] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.075564 11737 raft_consensus.cc:385] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.075930 11737 raft_consensus.cc:740] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.076396 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22f1c13eb88b4a56ad4eca20821a17e9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:57.077378 11620 raft_consensus.cc:2468] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:57.079478 11509 leader_election.cc:304] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.080673 11751 raft_consensus.cc:493] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.076799 11737 consensus_queue.cc:260] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.081326 11751 raft_consensus.cc:515] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.089671 11760 raft_consensus.cc:2804] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.090389 11751 leader_election.cc:290] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.092156 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c725053672514cf69f7eec8351d1759e" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.093166 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c725053672514cf69f7eec8351d1759e" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:57.093268 11695 raft_consensus.cc:2468] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:57.094293 11620 raft_consensus.cc:2468] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:57.095515 11510 leader_election.cc:304] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.096755 11743 leader_election.cc:290] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.097606 11743 raft_consensus.cc:2804] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.098078 11743 raft_consensus.cc:493] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.098516 11743 raft_consensus.cc:3060] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.098887 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22f1c13eb88b4a56ad4eca20821a17e9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.071343 11659 leader_election.cc:304] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.100790 11742 raft_consensus.cc:2804] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.101305 11742 raft_consensus.cc:493] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:57.101487 11510 leader_election.cc:343] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:57.101897 11742 raft_consensus.cc:3060] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.103056 11741 raft_consensus.cc:493] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.103821 11741 raft_consensus.cc:515] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.107072 11741 leader_election.cc:290] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.111363 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:57.112413 11621 raft_consensus.cc:2468] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:25:57.090308 11760 raft_consensus.cc:493] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.114133 11760 raft_consensus.cc:3060] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.143882 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.145038 11546 raft_consensus.cc:2468] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:25:57.147799 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.148797 11546 raft_consensus.cc:2468] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:25:57.149628 11743 raft_consensus.cc:515] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.150892 11585 leader_election.cc:304] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.154726 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c725053672514cf69f7eec8351d1759e" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.155683 11621 raft_consensus.cc:3060] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.156987 11741 raft_consensus.cc:2804] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.157473 11741 raft_consensus.cc:493] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.157862 11741 raft_consensus.cc:3060] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.158064 11737 ts_tablet_manager.cc:1434] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.086s	user 0.024s	sys 0.054s
I20260709 12:25:57.158385 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c725053672514cf69f7eec8351d1759e" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.159013 11695 raft_consensus.cc:3060] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.159564 11737 tablet_bootstrap.cc:492] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:25:57.166462 11737 tablet_bootstrap.cc:654] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:57.167490 11695 raft_consensus.cc:2468] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:57.167783 11585 leader_election.cc:304] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.168824 11510 leader_election.cc:304] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.146162 11742 raft_consensus.cc:515] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.169871 11778 raft_consensus.cc:2804] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.170598 11778 raft_consensus.cc:697] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:57.171527 11778 consensus_queue.cc:237] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.172926 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.173520 11743 leader_election.cc:290] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.177321 11735 tablet_bootstrap.cc:492] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:57.177455 11780 raft_consensus.cc:2804] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.178050 11780 raft_consensus.cc:493] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.178158 11735 ts_tablet_manager.cc:1403] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.135s	user 0.053s	sys 0.057s
I20260709 12:25:57.178439 11780 raft_consensus.cc:3060] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.181207 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.176013 11546 raft_consensus.cc:3060] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.182691 11742 leader_election.cc:290] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:25:57.179075 11621 raft_consensus.cc:2468] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:57.182186 11735 raft_consensus.cc:359] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.183735 11735 raft_consensus.cc:385] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.166819 11741 raft_consensus.cc:515] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.184116 11735 raft_consensus.cc:740] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.184382 11760 raft_consensus.cc:515] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.184942 11735 consensus_queue.cc:260] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.185701 11741 leader_election.cc:290] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.186307 11741 raft_consensus.cc:493] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.186702 11741 raft_consensus.cc:515] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.187695 11735 ts_tablet_manager.cc:1434] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.003s
I20260709 12:25:57.218654 11735 tablet_bootstrap.cc:492] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:25:57.193504 11694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.187193 11760 leader_election.cc:290] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.215677 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.194900 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.219548 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22f1c13eb88b4a56ad4eca20821a17e9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.196920 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.222864 11694 raft_consensus.cc:2468] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:25:57.229322 11737 tablet_bootstrap.cc:492] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:25:57.229744 11780 raft_consensus.cc:515] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.234714 11737 ts_tablet_manager.cc:1403] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.076s	user 0.013s	sys 0.019s
I20260709 12:25:57.235249 11781 raft_consensus.cc:493] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.229802 11443 catalog_manager.cc:5697] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.235780 11781 raft_consensus.cc:515] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.231547 11546 raft_consensus.cc:2468] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 1.
I20260709 12:25:57.231858 11693 raft_consensus.cc:2393] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e119e0d85d645adbc00fd8fa7590ef2 in current term 1: Already voted for candidate 3c9e1ca3ff074690b75a039a85d60299 in this term.
I20260709 12:25:57.190572 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22f1c13eb88b4a56ad4eca20821a17e9" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.237783 11619 raft_consensus.cc:3060] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.238097 11781 leader_election.cc:290] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.231206 11695 raft_consensus.cc:3060] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.239908 11543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.240717 11543 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:25:57.239923 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.242393 11692 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:25:57.232205 11545 raft_consensus.cc:3060] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.235033 11544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.233376 11741 leader_election.cc:290] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.242563 11694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.244419 11585 leader_election.cc:304] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
W20260709 12:25:57.246475 11585 leader_election.cc:343] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:57.247514 11620 raft_consensus.cc:2393] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c9e1ca3ff074690b75a039a85d60299 in current term 1: Already voted for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in this term.
I20260709 12:25:57.249722 11659 leader_election.cc:304] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.250506 11780 leader_election.cc:290] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.250795 11742 raft_consensus.cc:2804] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.251065 11741 raft_consensus.cc:2804] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.255856 11741 raft_consensus.cc:493] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.256382 11741 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.264528 11544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.264343 11737 raft_consensus.cc:359] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.265627 11544 raft_consensus.cc:2393] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e119e0d85d645adbc00fd8fa7590ef2 in current term 1: Already voted for candidate 3c9e1ca3ff074690b75a039a85d60299 in this term.
I20260709 12:25:57.265914 11735 tablet_bootstrap.cc:654] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:25:57.265938 11737 raft_consensus.cc:385] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.266541 11737 raft_consensus.cc:740] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.267004 11742 raft_consensus.cc:697] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:25:57.267525 11737 consensus_queue.cc:260] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.268415 11619 raft_consensus.cc:2468] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
W20260709 12:25:57.268610 11585 leader_election.cc:343] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:25:57.269532 11509 leader_election.cc:304] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.269905 11737 ts_tablet_manager.cc:1434] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
I20260709 12:25:57.270182 11545 raft_consensus.cc:2468] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:25:57.270533 11760 raft_consensus.cc:2804] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.271044 11760 raft_consensus.cc:697] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:57.268098 11694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.271965 11760 consensus_queue.cc:237] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.272866 11780 raft_consensus.cc:493] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.273497 11780 raft_consensus.cc:515] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.275943 11780 leader_election.cc:290] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.268149 11742 consensus_queue.cc:237] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.254757 11695 raft_consensus.cc:2468] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:57.269093 11585 leader_election.cc:304] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1
I20260709 12:25:57.280889 11778 raft_consensus.cc:493] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.281598 11778 raft_consensus.cc:515] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.284017 11778 leader_election.cc:290] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.284253 11735 tablet_bootstrap.cc:492] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:25:57.284989 11735 ts_tablet_manager.cc:1403] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.069s	user 0.010s	sys 0.007s
I20260709 12:25:57.285580 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.285656 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:57.286170 11780 raft_consensus.cc:2749] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:25:57.286443 11695 raft_consensus.cc:2468] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:57.286559 11619 raft_consensus.cc:2468] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:25:57.288105 11510 leader_election.cc:304] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.289120 11778 raft_consensus.cc:2804] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.289482 11778 raft_consensus.cc:493] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.289804 11778 raft_consensus.cc:3060] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.291834 11735 raft_consensus.cc:359] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.292621 11735 raft_consensus.cc:385] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:25:57.293130 11735 raft_consensus.cc:740] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:25:57.294057 11735 consensus_queue.cc:260] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.301404 11735 ts_tablet_manager.cc:1434] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 12:25:57.306469 11741 raft_consensus.cc:515] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.286793 11585 leader_election.cc:304] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:25:57.308661 11780 raft_consensus.cc:2804] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.309216 11780 raft_consensus.cc:697] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:57.310552 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.310113 11780 consensus_queue.cc:237] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.311415 11778 raft_consensus.cc:515] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.312726 11545 raft_consensus.cc:2393] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e119e0d85d645adbc00fd8fa7590ef2 in current term 1: Already voted for candidate f00bca05e5804522901cee258b1290a1 in this term.
I20260709 12:25:57.313364 11443 catalog_manager.cc:5697] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:57.310091 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:25:57.315187 11695 raft_consensus.cc:2468] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:25:57.315056 11585 leader_election.cc:304] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1
I20260709 12:25:57.322867 11780 raft_consensus.cc:2749] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:25:57.323735 11585 leader_election.cc:304] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: f00bca05e5804522901cee258b1290a1
I20260709 12:25:57.324707 11780 raft_consensus.cc:2804] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.325178 11780 raft_consensus.cc:493] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.325613 11780 raft_consensus.cc:3060] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.327339 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.328081 11741 leader_election.cc:290] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.329156 11778 leader_election.cc:290] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.330529 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.330796 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.331172 11695 raft_consensus.cc:3060] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.331518 11545 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.335162 11441 catalog_manager.cc:5697] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.339558 11545 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:25:57.345448 11780 raft_consensus.cc:515] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.341212 11694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.346474 11694 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.346777 11619 raft_consensus.cc:2393] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f00bca05e5804522901cee258b1290a1 in current term 1: Already voted for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in this term.
I20260709 12:25:57.348901 11585 leader_election.cc:304] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.349862 11741 raft_consensus.cc:2804] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.350363 11741 raft_consensus.cc:697] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:57.351035 11780 leader_election.cc:290] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.351223 11741 consensus_queue.cc:237] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.353505 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.353865 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.354563 11545 raft_consensus.cc:2393] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e119e0d85d645adbc00fd8fa7590ef2 in current term 1: Already voted for candidate f00bca05e5804522901cee258b1290a1 in this term.
I20260709 12:25:57.355856 11585 leader_election.cc:304] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1
I20260709 12:25:57.356742 11781 raft_consensus.cc:2749] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:25:57.358556 11694 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:25:57.365120 11441 catalog_manager.cc:5697] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.342701 11695 raft_consensus.cc:2468] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:25:57.342206 11444 catalog_manager.cc:5697] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.367952 11510 leader_election.cc:304] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:57.368937 11778 raft_consensus.cc:2804] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.369444 11778 raft_consensus.cc:697] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:57.370275 11778 consensus_queue.cc:237] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.387421 11441 catalog_manager.cc:5697] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.409786 11778 consensus_queue.cc:1048] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.422056 11743 consensus_queue.cc:1048] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.426133 11742 raft_consensus.cc:493] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.426667 11742 raft_consensus.cc:515] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.430958 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:25:57.431679 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:25:57.431802 11545 raft_consensus.cc:2468] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:25:57.432663 11619 raft_consensus.cc:2468] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:25:57.433754 11659 leader_election.cc:304] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.433878 11742 leader_election.cc:290] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:25:57.435598 11742 raft_consensus.cc:2804] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:25:57.436050 11742 raft_consensus.cc:493] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.436389 11742 raft_consensus.cc:3060] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.442590 11742 raft_consensus.cc:515] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.445174 11742 leader_election.cc:290] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:25:57.445870 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.446620 11545 raft_consensus.cc:3060] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.446734 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.447515 11619 raft_consensus.cc:3060] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.452742 11619 raft_consensus.cc:2468] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 1.
I20260709 12:25:57.452890 11545 raft_consensus.cc:2468] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 1.
I20260709 12:25:57.453783 11658 leader_election.cc:304] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:25:57.454658 11742 raft_consensus.cc:2804] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:25:57.455183 11742 raft_consensus.cc:697] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:25:57.456005 11742 consensus_queue.cc:237] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.464390 11443 catalog_manager.cc:5697] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:57.499975 11743 consensus_queue.cc:1048] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.511035 11760 consensus_queue.cc:1048] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.534812 11619 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:41716
I20260709 12:25:57.535424 11619 raft_consensus.cc:606] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Received request to transfer leadership to TS f00bca05e5804522901cee258b1290a1
I20260709 12:25:57.557525 11694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:53832
I20260709 12:25:57.558131 11694 raft_consensus.cc:606] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:57.572144 11545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:34906
I20260709 12:25:57.572782 11545 raft_consensus.cc:606] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:57.578436 11694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:53832
I20260709 12:25:57.578979 11694 raft_consensus.cc:606] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:57.584364 11545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:34906
I20260709 12:25:57.584878 11545 raft_consensus.cc:606] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:25:57.590116 11619 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:41716
I20260709 12:25:57.590673 11619 raft_consensus.cc:606] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Received request to transfer leadership to TS 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:25:57.717753 11778 consensus_queue.cc:1048] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.733388 11778 consensus_queue.cc:1048] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.736205 11742 consensus_queue.cc:1048] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.749821 11778 consensus_queue.cc:1048] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.752925 11784 consensus_queue.cc:1048] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.765425 11784 raft_consensus.cc:993] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.766459 11742 raft_consensus.cc:1081] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.768157 11751 consensus_queue.cc:1048] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.769027 11621 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41700
I20260709 12:25:57.769701 11621 raft_consensus.cc:493] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:57.770103 11621 raft_consensus.cc:3060] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.777257 11621 raft_consensus.cc:515] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.779587 11621 leader_election.cc:290] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.780874 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.781131 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d24b1b461ca143e4a8f686d24d9b0751" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.781596 11695 raft_consensus.cc:3055] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:57.781733 11545 raft_consensus.cc:3060] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.782008 11695 raft_consensus.cc:740] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:25:57.783035 11695 consensus_queue.cc:260] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.784206 11695 raft_consensus.cc:3060] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.787923 11545 raft_consensus.cc:2468] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:25:57.789373 11585 leader_election.cc:304] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:57.790324 11741 raft_consensus.cc:2804] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:57.790861 11741 raft_consensus.cc:697] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:57.792022 11741 consensus_queue.cc:237] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.790735 11695 raft_consensus.cc:2468] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:25:57.800196 11441 catalog_manager.cc:5697] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.813817 11778 consensus_queue.cc:1048] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.820942 11751 raft_consensus.cc:993] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.821473 11751 raft_consensus.cc:1081] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.826547 11621 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41698
I20260709 12:25:57.826845 11778 consensus_queue.cc:1048] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.827189 11741 consensus_queue.cc:1048] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.829934 11621 raft_consensus.cc:493] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:57.830310 11695 raft_consensus.cc:3060] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:25:57.830437 11621 raft_consensus.cc:3060] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.842410 11741 consensus_queue.cc:1048] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.844714 11621 raft_consensus.cc:515] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.849435 11544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.850303 11544 raft_consensus.cc:3055] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:57.850821 11544 raft_consensus.cc:740] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:57.852170 11544 consensus_queue.cc:260] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.853746 11544 raft_consensus.cc:3060] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.860447 11621 leader_election.cc:290] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.863543 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735953c802fa471ba0d68511ee3495e5" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.866775 11781 raft_consensus.cc:993] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:25:57.869757 11544 raft_consensus.cc:2468] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:25:57.875972 11585 leader_election.cc:304] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:25:57.876133 11741 raft_consensus.cc:1081] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Signalling peer 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:25:57.877290 11741 raft_consensus.cc:2804] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:57.877810 11741 raft_consensus.cc:697] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:57.878873 11693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:53822
I20260709 12:25:57.880774 11693 raft_consensus.cc:493] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:57.878775 11741 consensus_queue.cc:237] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.882402 11693 raft_consensus.cc:3060] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.890782 11693 raft_consensus.cc:515] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.893103 11444 catalog_manager.cc:5697] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from f00bca05e5804522901cee258b1290a1 (127.25.202.1) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:57.898232 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.898908 11621 raft_consensus.cc:3055] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:57.899230 11621 raft_consensus.cc:740] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:57.895475 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d00baea3d94637ab044e58ce707954" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.899782 11545 raft_consensus.cc:3060] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.900012 11621 consensus_queue.cc:260] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.901733 11621 raft_consensus.cc:3060] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.909128 11621 raft_consensus.cc:2468] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 2.
I20260709 12:25:57.909308 11693 leader_election.cc:290] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:25:57.914709 11658 leader_election.cc:304] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:25:57.915693 11804 raft_consensus.cc:2804] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:57.916225 11804 raft_consensus.cc:697] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:25:57.917089 11545 raft_consensus.cc:2468] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 2.
I20260709 12:25:57.917125 11804 consensus_queue.cc:237] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.919077 11741 consensus_queue.cc:1048] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.925859 11441 catalog_manager.cc:5697] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 1 to 2, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:57.926788 11804 consensus_queue.cc:1048] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.936460 11778 raft_consensus.cc:993] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.936973 11778 raft_consensus.cc:1081] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.939898 11812 raft_consensus.cc:993] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:25:57.940466 11741 raft_consensus.cc:1081] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Signalling peer f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:25:57.940665 11619 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41698
I20260709 12:25:57.941398 11619 raft_consensus.cc:493] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:57.941794 11619 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.940935 11784 raft_consensus.cc:993] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.943071 11784 raft_consensus.cc:1081] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:25:57.944938 11620 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41700
I20260709 12:25:57.945788 11545 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:34892
I20260709 12:25:57.946417 11545 raft_consensus.cc:493] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:57.946846 11545 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.948282 11742 consensus_queue.cc:1048] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.953496 11620 raft_consensus.cc:493] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:57.954046 11620 raft_consensus.cc:3060] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.955173 11545 raft_consensus.cc:515] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.957392 11741 consensus_queue.cc:1048] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:57.959664 11545 leader_election.cc:290] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.960117 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.963582 11693 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.965173 11619 raft_consensus.cc:515] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.970062 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.970912 11546 raft_consensus.cc:3055] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:57.971344 11546 raft_consensus.cc:740] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:57.972000 11619 leader_election.cc:290] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.972150 11620 raft_consensus.cc:515] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.972484 11546 consensus_queue.cc:260] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.974402 11546 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.975436 11620 leader_election.cc:290] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:25:57.976876 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:25:57.986685 11693 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:25:57.987838 11694 raft_consensus.cc:1240] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Rejecting Update request from peer 4e119e0d85d645adbc00fd8fa7590ef2 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:25:57.988284 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.988674 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:25:57.988988 11695 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.989418 11692 raft_consensus.cc:3055] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:57.989797 11692 raft_consensus.cc:740] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:25:57.990685 11692 consensus_queue.cc:260] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:57.991101 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a17e2a0a7964ec4a956743fe816de45" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:25:57.991955 11692 raft_consensus.cc:3060] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:57.994076 11741 consensus_queue.cc:1059] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:25:57.998656 11695 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:25:58.000120 11585 leader_election.cc:304] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:25:58.001084 11815 raft_consensus.cc:2804] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:58.002512 11621 raft_consensus.cc:3055] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:58.002892 11815 raft_consensus.cc:697] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:58.002945 11621 raft_consensus.cc:740] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:58.003762 11510 leader_election.cc:304] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:25:58.004730 11743 raft_consensus.cc:2804] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:58.004156 11621 consensus_queue.cc:260] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:58.005270 11743 raft_consensus.cc:697] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:25:58.005877 11546 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:25:58.006183 11743 consensus_queue.cc:237] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:58.004061 11815 consensus_queue.cc:237] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:58.005900 11621 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:58.000072 11692 raft_consensus.cc:2468] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:25:58.009137 11585 leader_election.cc:304] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: f00bca05e5804522901cee258b1290a1
I20260709 12:25:58.010447 11741 raft_consensus.cc:2804] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:58.010965 11741 raft_consensus.cc:697] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:25:58.011787 11741 consensus_queue.cc:237] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:25:58.018237 11444 catalog_manager.cc:5697] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from f00bca05e5804522901cee258b1290a1 (127.25.202.1) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:58.020607 11621 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:25:58.021514 11441 catalog_manager.cc:5697] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 1 to 2, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:58.024235 11812 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:25:58.033310 11444 catalog_manager.cc:5697] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:58.281086 11695 raft_consensus.cc:3060] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:58.282128 11692 raft_consensus.cc:1275] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.285182 11741 consensus_queue.cc:1048] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:58.299301 11695 raft_consensus.cc:1275] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.305024 11546 raft_consensus.cc:1275] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.306082 11812 consensus_queue.cc:1048] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:58.319881 11781 consensus_queue.cc:1048] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:58.326121 11545 raft_consensus.cc:1275] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.328714 11812 consensus_queue.cc:1048] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:58.344376 11545 raft_consensus.cc:1275] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.346141 11804 consensus_queue.cc:1048] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:58.356520 11621 raft_consensus.cc:1275] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.358500 11804 consensus_queue.cc:1048] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:58.493124 11692 raft_consensus.cc:1275] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.494508 11818 consensus_queue.cc:1048] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:58.504122 11545 raft_consensus.cc:3060] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:58.515507 11545 raft_consensus.cc:1275] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.516760 11741 consensus_queue.cc:1048] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:58.533318 11545 raft_consensus.cc:1275] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.534497 11741 consensus_queue.cc:1048] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:58.546324 11692 raft_consensus.cc:1275] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.547942 11741 consensus_queue.cc:1048] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:58.553431 11695 raft_consensus.cc:1275] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.555630 11621 raft_consensus.cc:1275] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:58.555698 11743 consensus_queue.cc:1048] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:25:58.558101 11778 consensus_queue.cc:1048] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:00.626880 11441 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:26:00.629474 11441 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:26:00.689219 11530 tablet_service.cc:1511] Processing CreateTablet for tablet 984debe0dc4c4b4f91387db0e911a5ae (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:00.690055 11603 tablet_service.cc:1511] Processing CreateTablet for tablet 984debe0dc4c4b4f91387db0e911a5ae (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:00.691263 11530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 984debe0dc4c4b4f91387db0e911a5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.691768 11603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 984debe0dc4c4b4f91387db0e911a5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.692360 11679 tablet_service.cc:1511] Processing CreateTablet for tablet 984debe0dc4c4b4f91387db0e911a5ae (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:00.693822 11679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 984debe0dc4c4b4f91387db0e911a5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.695580 11606 tablet_service.cc:1511] Processing CreateTablet for tablet 3e495b6c9db94ccda94d34a8c66192e7 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:00.697211 11606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e495b6c9db94ccda94d34a8c66192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.698001 11684 tablet_service.cc:1511] Processing CreateTablet for tablet 3e495b6c9db94ccda94d34a8c66192e7 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:00.699467 11684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e495b6c9db94ccda94d34a8c66192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.699540 11535 tablet_service.cc:1511] Processing CreateTablet for tablet 621bc6cfa6634625b67f91f29db76b81 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:00.702920 11535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 621bc6cfa6634625b67f91f29db76b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.704035 11529 tablet_service.cc:1511] Processing CreateTablet for tablet 3e495b6c9db94ccda94d34a8c66192e7 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:00.704494 11536 tablet_service.cc:1511] Processing CreateTablet for tablet 9817dc25684c4781a16c88f50f925ce1 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:00.705502 11529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e495b6c9db94ccda94d34a8c66192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.707062 11677 tablet_service.cc:1511] Processing CreateTablet for tablet 621bc6cfa6634625b67f91f29db76b81 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:00.708578 11677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 621bc6cfa6634625b67f91f29db76b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.708271 11678 tablet_service.cc:1511] Processing CreateTablet for tablet 9817dc25684c4781a16c88f50f925ce1 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:00.709738 11678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9817dc25684c4781a16c88f50f925ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.710805 11680 tablet_service.cc:1511] Processing CreateTablet for tablet 419a49ea5b5146bea4b3eea2a507fa10 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:00.712307 11680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419a49ea5b5146bea4b3eea2a507fa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.714829 11683 tablet_service.cc:1511] Processing CreateTablet for tablet f701a63f43f34f65b2f848f84bb73752 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:00.716370 11683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f701a63f43f34f65b2f848f84bb73752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.716670 11532 tablet_service.cc:1511] Processing CreateTablet for tablet 419a49ea5b5146bea4b3eea2a507fa10 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:00.719183 11533 tablet_service.cc:1511] Processing CreateTablet for tablet f701a63f43f34f65b2f848f84bb73752 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:00.720631 11531 tablet_service.cc:1511] Processing CreateTablet for tablet c60cfd059ea5413182db993dbd80c73b (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:00.722429 11534 tablet_service.cc:1511] Processing CreateTablet for tablet 0c969b48c58646c2ac21bb9b275d7b5b (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:00.719187 11681 tablet_service.cc:1511] Processing CreateTablet for tablet c60cfd059ea5413182db993dbd80c73b (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:00.727640 11611 tablet_service.cc:1511] Processing CreateTablet for tablet 621bc6cfa6634625b67f91f29db76b81 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:00.729254 11611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 621bc6cfa6634625b67f91f29db76b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.729652 11607 tablet_service.cc:1511] Processing CreateTablet for tablet 9817dc25684c4781a16c88f50f925ce1 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:00.730108 11605 tablet_service.cc:1511] Processing CreateTablet for tablet c60cfd059ea5413182db993dbd80c73b (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:00.731091 11607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9817dc25684c4781a16c88f50f925ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.732700 11609 tablet_service.cc:1511] Processing CreateTablet for tablet 419a49ea5b5146bea4b3eea2a507fa10 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:00.733224 11604 tablet_service.cc:1511] Processing CreateTablet for tablet f701a63f43f34f65b2f848f84bb73752 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:00.732789 11608 tablet_service.cc:1511] Processing CreateTablet for tablet 0c969b48c58646c2ac21bb9b275d7b5b (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:00.736347 11681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c60cfd059ea5413182db993dbd80c73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.738749 11532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419a49ea5b5146bea4b3eea2a507fa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.739709 11531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c60cfd059ea5413182db993dbd80c73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.741322 11533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f701a63f43f34f65b2f848f84bb73752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.742293 11534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c969b48c58646c2ac21bb9b275d7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.737257 11605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c60cfd059ea5413182db993dbd80c73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.755674 11604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f701a63f43f34f65b2f848f84bb73752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.759739 11609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419a49ea5b5146bea4b3eea2a507fa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.767715 11608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c969b48c58646c2ac21bb9b275d7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.779982 11682 tablet_service.cc:1511] Processing CreateTablet for tablet 0c969b48c58646c2ac21bb9b275d7b5b (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:00.781970 11682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c969b48c58646c2ac21bb9b275d7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.787937 11536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9817dc25684c4781a16c88f50f925ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.802862 11833 tablet_bootstrap.cc:492] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:00.824476 11610 tablet_service.cc:1511] Processing CreateTablet for tablet 2b09c76b8a494d56abce9d46ea4d5a17 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:00.825937 11610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b09c76b8a494d56abce9d46ea4d5a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.839826 11677 tablet_service.cc:1511] Processing CreateTablet for tablet 2b09c76b8a494d56abce9d46ea4d5a17 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:00.850556 11677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b09c76b8a494d56abce9d46ea4d5a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.843483 11834 tablet_bootstrap.cc:492] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:00.858444 11834 tablet_bootstrap.cc:654] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.868600 11833 tablet_bootstrap.cc:654] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.870095 11532 tablet_service.cc:1511] Processing CreateTablet for tablet 2b09c76b8a494d56abce9d46ea4d5a17 (DEFAULT_TABLE table=soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:00.908219 11833 tablet_bootstrap.cc:492] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:00.908834 11833 ts_tablet_manager.cc:1403] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.106s	user 0.009s	sys 0.007s
I20260709 12:26:00.909868 11532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b09c76b8a494d56abce9d46ea4d5a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:00.911574 11835 tablet_bootstrap.cc:492] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:00.912504 11833 raft_consensus.cc:359] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.920203 11833 raft_consensus.cc:385] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:00.920624 11833 raft_consensus.cc:740] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:00.921649 11833 consensus_queue.cc:260] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.928084 11835 tablet_bootstrap.cc:654] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.937943 11833 ts_tablet_manager.cc:1434] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.004s
I20260709 12:26:00.940896 11835 tablet_bootstrap.cc:492] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:00.941529 11835 ts_tablet_manager.cc:1403] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.056s	user 0.009s	sys 0.010s
I20260709 12:26:00.943378 11833 tablet_bootstrap.cc:492] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:00.945124 11835 raft_consensus.cc:359] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.945994 11835 raft_consensus.cc:385] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:00.946338 11835 raft_consensus.cc:740] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
W20260709 12:26:00.949572 11492 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:00.947192 11835 consensus_queue.cc:260] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.952737 11835 ts_tablet_manager.cc:1434] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:26:00.953926 11835 tablet_bootstrap.cc:492] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:00.955834 11833 tablet_bootstrap.cc:654] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.956768 11834 tablet_bootstrap.cc:492] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:00.957345 11834 ts_tablet_manager.cc:1403] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.114s	user 0.019s	sys 0.023s
I20260709 12:26:00.960752 11834 raft_consensus.cc:359] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.961546 11834 raft_consensus.cc:385] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:00.961840 11834 raft_consensus.cc:740] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:00.962577 11834 consensus_queue.cc:260] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.962966 11835 tablet_bootstrap.cc:654] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.965512 11834 ts_tablet_manager.cc:1434] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:26:00.966784 11834 tablet_bootstrap.cc:492] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:00.973268 11833 tablet_bootstrap.cc:492] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:00.973821 11835 tablet_bootstrap.cc:492] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:00.973955 11833 ts_tablet_manager.cc:1403] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.006s
I20260709 12:26:00.974581 11835 ts_tablet_manager.cc:1403] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.000s
I20260709 12:26:00.976975 11834 tablet_bootstrap.cc:654] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.977747 11833 raft_consensus.cc:359] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.978631 11833 raft_consensus.cc:385] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:00.978286 11835 raft_consensus.cc:359] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.979041 11833 raft_consensus.cc:740] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:00.979497 11835 raft_consensus.cc:385] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:00.980131 11835 raft_consensus.cc:740] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:00.980111 11833 consensus_queue.cc:260] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.983048 11833 ts_tablet_manager.cc:1434] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:26:00.984216 11833 tablet_bootstrap.cc:492] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:00.984952 11835 consensus_queue.cc:260] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:00.987793 11835 ts_tablet_manager.cc:1434] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.003s
I20260709 12:26:00.989352 11835 tablet_bootstrap.cc:492] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:00.994838 11834 tablet_bootstrap.cc:492] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:00.995589 11834 ts_tablet_manager.cc:1403] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.029s	user 0.019s	sys 0.006s
I20260709 12:26:00.996838 11833 tablet_bootstrap.cc:654] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.997370 11778 raft_consensus.cc:493] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.997668 11835 tablet_bootstrap.cc:654] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:00.997938 11778 raft_consensus.cc:515] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.002465 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
W20260709 12:26:01.004465 11509 leader_election.cc:343] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.005189 11834 raft_consensus.cc:359] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.006047 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.006448 11834 raft_consensus.cc:385] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.006968 11834 raft_consensus.cc:740] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
W20260709 12:26:01.008056 11510 leader_election.cc:343] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.008592 11510 leader_election.cc:304] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f00bca05e5804522901cee258b1290a1; no voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:01.010959 11778 leader_election.cc:290] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.014475 11778 raft_consensus.cc:2749] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.015838 11804 raft_consensus.cc:493] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.016654 11804 raft_consensus.cc:515] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.021207 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
W20260709 12:26:01.023737 11659 leader_election.cc:343] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.008082 11834 consensus_queue.cc:260] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.025591 11804 leader_election.cc:290] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:01.027626 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.029717 11834 ts_tablet_manager.cc:1434] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.000s
W20260709 12:26:01.030133 11658 leader_election.cc:343] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.030825 11834 tablet_bootstrap.cc:492] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.030853 11658 leader_election.cc:304] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299; no voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.032814 11804 raft_consensus.cc:2749] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.034140 11804 raft_consensus.cc:493] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.034643 11804 raft_consensus.cc:515] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.035494 11833 tablet_bootstrap.cc:492] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.036128 11833 ts_tablet_manager.cc:1403] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.052s	user 0.024s	sys 0.008s
I20260709 12:26:01.037911 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.039633 11833 raft_consensus.cc:359] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
W20260709 12:26:01.039644 11659 leader_election.cc:343] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.040499 11833 raft_consensus.cc:385] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.040891 11833 raft_consensus.cc:740] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.041850 11833 consensus_queue.cc:260] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.046237 11804 leader_election.cc:290] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:01.047510 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
W20260709 12:26:01.049523 11658 leader_election.cc:343] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.050222 11658 leader_election.cc:304] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299; no voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.046976 11834 tablet_bootstrap.cc:654] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.051957 11804 raft_consensus.cc:2749] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.059584 11833 ts_tablet_manager.cc:1434] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.023s	user 0.012s	sys 0.004s
I20260709 12:26:01.061791 11834 tablet_bootstrap.cc:492] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:01.061904 11833 tablet_bootstrap.cc:492] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:01.062471 11834 ts_tablet_manager.cc:1403] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.032s	user 0.008s	sys 0.004s
I20260709 12:26:01.065828 11834 raft_consensus.cc:359] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.066570 11834 raft_consensus.cc:385] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.066864 11834 raft_consensus.cc:740] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.073467 11833 tablet_bootstrap.cc:654] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.074759 11834 consensus_queue.cc:260] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
W20260709 12:26:01.083067 11510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.3:37239, user_credentials={real_user=slave}} blocked reactor thread for 75156.2us
I20260709 12:26:01.086930 11835 tablet_bootstrap.cc:492] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.087815 11835 ts_tablet_manager.cc:1403] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.099s	user 0.045s	sys 0.024s
I20260709 12:26:01.092818 11835 raft_consensus.cc:359] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.093534 11835 raft_consensus.cc:385] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.093900 11835 raft_consensus.cc:740] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.095214 11834 ts_tablet_manager.cc:1434] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.000s
I20260709 12:26:01.094937 11835 consensus_queue.cc:260] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.095692 11833 tablet_bootstrap.cc:492] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.096496 11834 tablet_bootstrap.cc:492] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.096508 11833 ts_tablet_manager.cc:1403] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.035s	user 0.021s	sys 0.011s
I20260709 12:26:01.100670 11833 raft_consensus.cc:359] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.101665 11833 raft_consensus.cc:385] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.102092 11833 raft_consensus.cc:740] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.103083 11833 consensus_queue.cc:260] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.103698 11834 tablet_bootstrap.cc:654] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.105624 11833 ts_tablet_manager.cc:1434] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 12:26:01.106806 11835 ts_tablet_manager.cc:1434] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 12:26:01.106962 11833 tablet_bootstrap.cc:492] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:01.115789 11833 tablet_bootstrap.cc:654] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.117936 11835 tablet_bootstrap.cc:492] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:01.124099 11834 tablet_bootstrap.cc:492] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:01.124104 11833 tablet_bootstrap.cc:492] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.124960 11834 ts_tablet_manager.cc:1403] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.007s
I20260709 12:26:01.125046 11833 ts_tablet_manager.cc:1403] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.000s
I20260709 12:26:01.126937 11835 tablet_bootstrap.cc:654] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.128898 11833 raft_consensus.cc:359] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.129662 11833 raft_consensus.cc:385] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.129964 11833 raft_consensus.cc:740] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.130709 11833 consensus_queue.cc:260] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.130841 11834 raft_consensus.cc:359] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.132252 11834 raft_consensus.cc:385] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.132561 11834 raft_consensus.cc:740] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.133033 11833 ts_tablet_manager.cc:1434] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:26:01.133399 11834 consensus_queue.cc:260] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.134099 11833 tablet_bootstrap.cc:492] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:01.136721 11835 tablet_bootstrap.cc:492] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.137357 11835 ts_tablet_manager.cc:1403] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.007s
I20260709 12:26:01.138773 11834 ts_tablet_manager.cc:1434] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 12:26:01.140079 11834 tablet_bootstrap.cc:492] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.141913 11833 tablet_bootstrap.cc:654] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.141206 11835 raft_consensus.cc:359] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.142431 11835 raft_consensus.cc:385] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.142827 11835 raft_consensus.cc:740] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.143787 11835 consensus_queue.cc:260] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.149040 11834 tablet_bootstrap.cc:654] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.158469 11835 ts_tablet_manager.cc:1434] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:26:01.161391 11834 tablet_bootstrap.cc:492] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:01.162055 11834 ts_tablet_manager.cc:1403] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.014s
I20260709 12:26:01.165743 11834 raft_consensus.cc:359] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.166527 11834 raft_consensus.cc:385] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.167018 11834 raft_consensus.cc:740] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.167976 11834 consensus_queue.cc:260] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.170629 11834 ts_tablet_manager.cc:1434] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260709 12:26:01.171648 11835 tablet_bootstrap.cc:492] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:01.171851 11834 tablet_bootstrap.cc:492] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.180022 11846 raft_consensus.cc:493] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.180644 11846 raft_consensus.cc:515] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.183139 11835 tablet_bootstrap.cc:654] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.183941 11833 tablet_bootstrap.cc:492] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.184109 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.184546 11833 ts_tablet_manager.cc:1403] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.051s	user 0.013s	sys 0.034s
I20260709 12:26:01.184676 11834 tablet_bootstrap.cc:654] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.185030 11545 raft_consensus.cc:2468] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:26:01.186704 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.187127 11659 leader_election.cc:304] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.184129 11846 leader_election.cc:290] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
W20260709 12:26:01.188416 11658 leader_election.cc:343] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.188843 11846 raft_consensus.cc:2804] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.188704 11833 raft_consensus.cc:359] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.190970 11833 raft_consensus.cc:385] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.190879 11812 raft_consensus.cc:493] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.191569 11833 raft_consensus.cc:740] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.191920 11812 raft_consensus.cc:515] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.192709 11835 tablet_bootstrap.cc:492] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.192426 11833 consensus_queue.cc:260] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.193252 11835 ts_tablet_manager.cc:1403] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.004s
I20260709 12:26:01.194891 11833 ts_tablet_manager.cc:1434] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:26:01.196723 11835 raft_consensus.cc:359] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.197474 11835 raft_consensus.cc:385] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.197742 11835 raft_consensus.cc:740] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.198082 11833 tablet_bootstrap.cc:492] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:01.198537 11835 consensus_queue.cc:260] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.199905 11846 raft_consensus.cc:493] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.200275 11846 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.201097 11835 ts_tablet_manager.cc:1434] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:26:01.202157 11835 tablet_bootstrap.cc:492] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:01.206837 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.207953 11545 raft_consensus.cc:2468] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.209362 11835 tablet_bootstrap.cc:654] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.210311 11585 leader_election.cc:304] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.211758 11845 raft_consensus.cc:2804] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.212373 11845 raft_consensus.cc:493] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.212839 11845 raft_consensus.cc:3060] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.212991 11833 tablet_bootstrap.cc:654] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.216429 11846 raft_consensus.cc:515] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.218264 11812 leader_election.cc:290] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.220405 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:01.221637 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.219120 11846 leader_election.cc:290] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
W20260709 12:26:01.222146 11658 leader_election.cc:343] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.222371 11692 raft_consensus.cc:2468] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.229336 11846 raft_consensus.cc:493] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.230000 11846 raft_consensus.cc:515] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.232857 11846 leader_election.cc:290] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:01.233875 11833 tablet_bootstrap.cc:492] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.234658 11833 ts_tablet_manager.cc:1403] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.004s
I20260709 12:26:01.238981 11833 raft_consensus.cc:359] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.240077 11833 raft_consensus.cc:385] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.240337 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.240464 11833 raft_consensus.cc:740] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.242065 11833 consensus_queue.cc:260] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
W20260709 12:26:01.242632 11659 leader_election.cc:343] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.246006 11833 ts_tablet_manager.cc:1434] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:26:01.247557 11833 tablet_bootstrap.cc:492] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:01.247888 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
W20260709 12:26:01.250037 11658 leader_election.cc:343] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.250654 11658 leader_election.cc:304] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299; no voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.251720 11846 raft_consensus.cc:2749] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.252565 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:01.253783 11545 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.258698 11778 raft_consensus.cc:493] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.259621 11778 raft_consensus.cc:515] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.261190 11833 tablet_bootstrap.cc:654] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.265092 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e495b6c9db94ccda94d34a8c66192e7" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.265625 11778 leader_election.cc:290] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.265892 11621 raft_consensus.cc:2468] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.266578 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e495b6c9db94ccda94d34a8c66192e7" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.267409 11692 raft_consensus.cc:2468] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.269122 11510 leader_election.cc:304] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.271627 11844 raft_consensus.cc:2804] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.272269 11844 raft_consensus.cc:493] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.272818 11844 raft_consensus.cc:3060] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.274618 11833 tablet_bootstrap.cc:492] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.275373 11833 ts_tablet_manager.cc:1403] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.028s	user 0.016s	sys 0.007s
I20260709 12:26:01.279261 11833 raft_consensus.cc:359] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.280227 11833 raft_consensus.cc:385] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.280571 11833 raft_consensus.cc:740] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.281733 11833 consensus_queue.cc:260] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.284946 11833 ts_tablet_manager.cc:1434] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:26:01.285702 11778 raft_consensus.cc:493] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.286204 11833 tablet_bootstrap.cc:492] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: Bootstrap starting.
I20260709 12:26:01.286733 11778 raft_consensus.cc:515] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.291062 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.292057 11621 raft_consensus.cc:2468] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.294092 11509 leader_election.cc:304] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.295461 11833 tablet_bootstrap.cc:654] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.324337 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.325143 11692 raft_consensus.cc:2468] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.331146 11545 raft_consensus.cc:2468] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 1.
I20260709 12:26:01.337893 11659 leader_election.cc:304] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:01.321441 11778 leader_election.cc:290] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.341464 11846 raft_consensus.cc:2804] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.342101 11846 raft_consensus.cc:697] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:01.341048 11833 tablet_bootstrap.cc:492] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: No bootstrap required, opened a new log
I20260709 12:26:01.343214 11833 ts_tablet_manager.cc:1403] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: Time spent bootstrapping tablet: real 0.057s	user 0.014s	sys 0.012s
I20260709 12:26:01.344730 11846 consensus_queue.cc:237] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.341785 11778 raft_consensus.cc:2804] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.345798 11778 raft_consensus.cc:493] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.346138 11778 raft_consensus.cc:3060] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.346382 11833 raft_consensus.cc:359] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.347218 11833 raft_consensus.cc:385] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.347536 11833 raft_consensus.cc:740] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.348366 11833 consensus_queue.cc:260] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.345000 11845 raft_consensus.cc:515] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.354233 11833 ts_tablet_manager.cc:1434] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.004s
I20260709 12:26:01.355576 11845 leader_election.cc:290] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.357420 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:01.358074 11545 raft_consensus.cc:3060] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.361516 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.362139 11692 raft_consensus.cc:3060] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.363658 11845 raft_consensus.cc:493] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.364223 11845 raft_consensus.cc:515] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.366577 11845 leader_election.cc:290] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.368561 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.368912 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.368569 11778 raft_consensus.cc:515] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.369454 11546 raft_consensus.cc:2468] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.369652 11695 raft_consensus.cc:2468] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.371043 11778 leader_election.cc:290] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.370986 11585 leader_election.cc:304] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:26:01.373694 11845 raft_consensus.cc:2804] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.374253 11845 raft_consensus.cc:493] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.375450 11845 raft_consensus.cc:3060] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.376845 11834 tablet_bootstrap.cc:492] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:01.377460 11834 ts_tablet_manager.cc:1403] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.206s	user 0.046s	sys 0.077s
I20260709 12:26:01.373558 11692 raft_consensus.cc:2468] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.374179 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.379866 11695 raft_consensus.cc:3060] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.382100 11844 raft_consensus.cc:515] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.380362 11585 leader_election.cc:304] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:26:01.387856 11812 raft_consensus.cc:2804] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.383527 11834 raft_consensus.cc:359] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.386854 11545 raft_consensus.cc:2468] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.384531 11845 raft_consensus.cc:515] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.389118 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:01.392680 11621 raft_consensus.cc:3060] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.393922 11845 leader_election.cc:290] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.393486 11440 catalog_manager.cc:5697] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:01.395818 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.388399 11812 raft_consensus.cc:697] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:01.396436 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:01.397126 11546 raft_consensus.cc:3060] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.397527 11695 raft_consensus.cc:2468] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:26:01.397316 11812 consensus_queue.cc:237] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.391809 11835 tablet_bootstrap.cc:492] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.398684 11835 ts_tablet_manager.cc:1403] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.197s	user 0.037s	sys 0.090s
I20260709 12:26:01.391171 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e495b6c9db94ccda94d34a8c66192e7" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:01.388935 11834 raft_consensus.cc:385] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.399641 11510 leader_election.cc:304] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.399803 11834 raft_consensus.cc:740] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.399827 11620 raft_consensus.cc:3060] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.400691 11861 raft_consensus.cc:2804] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.401412 11861 raft_consensus.cc:697] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:01.400918 11834 consensus_queue.cc:260] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.402364 11861 consensus_queue.cc:237] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.396508 11693 raft_consensus.cc:3060] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.403676 11834 ts_tablet_manager.cc:1434] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.001s
I20260709 12:26:01.404975 11834 tablet_bootstrap.cc:492] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.405560 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e495b6c9db94ccda94d34a8c66192e7" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.406199 11692 raft_consensus.cc:3060] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.415222 11835 raft_consensus.cc:359] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.416832 11835 raft_consensus.cc:385] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.417693 11835 raft_consensus.cc:740] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.391749 11844 leader_election.cc:290] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.421883 11441 catalog_manager.cc:5697] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.424528 11621 raft_consensus.cc:2468] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:26:01.431988 11620 raft_consensus.cc:2468] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:26:01.430769 11442 catalog_manager.cc:5697] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.433364 11509 leader_election.cc:304] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.434321 11844 raft_consensus.cc:2804] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.434856 11844 raft_consensus.cc:697] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:01.436416 11835 consensus_queue.cc:260] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.436942 11845 raft_consensus.cc:493] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.437912 11844 consensus_queue.cc:237] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.438750 11845 raft_consensus.cc:515] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.445178 11544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.446134 11692 raft_consensus.cc:2468] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:26:01.449239 11546 raft_consensus.cc:2468] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.449927 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.439131 11835 ts_tablet_manager.cc:1434] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.040s	user 0.003s	sys 0.002s
I20260709 12:26:01.450644 11695 raft_consensus.cc:2468] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.442983 11845 leader_election.cc:290] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
W20260709 12:26:01.449553 11585 leader_election.cc:343] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.453693 11835 tablet_bootstrap.cc:492] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:01.457082 11585 leader_election.cc:304] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.458729 11834 tablet_bootstrap.cc:654] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.459249 11845 raft_consensus.cc:2804] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.460325 11845 raft_consensus.cc:697] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:01.461481 11845 consensus_queue.cc:237] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.458492 11585 leader_election.cc:304] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.464691 11812 raft_consensus.cc:2804] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.465109 11812 raft_consensus.cc:493] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.465505 11812 raft_consensus.cc:3060] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.470338 11693 raft_consensus.cc:2468] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.480286 11835 tablet_bootstrap.cc:654] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.482458 11812 raft_consensus.cc:515] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.485922 11812 leader_election.cc:290] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.486649 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:01.489038 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.489863 11693 raft_consensus.cc:3060] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.492508 11834 tablet_bootstrap.cc:492] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
W20260709 12:26:01.493057 11585 leader_election.cc:343] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:01.493896 11834 ts_tablet_manager.cc:1403] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.089s	user 0.021s	sys 0.001s
I20260709 12:26:01.498488 11834 raft_consensus.cc:359] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.499423 11834 raft_consensus.cc:385] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.499851 11834 raft_consensus.cc:740] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.500702 11834 consensus_queue.cc:260] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.503655 11834 ts_tablet_manager.cc:1434] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:26:01.504824 11834 tablet_bootstrap.cc:492] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.507357 11693 raft_consensus.cc:2468] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.508726 11585 leader_election.cc:304] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.510387 11845 raft_consensus.cc:2804] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.510932 11845 raft_consensus.cc:697] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:01.512583 11834 tablet_bootstrap.cc:654] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.512261 11845 consensus_queue.cc:237] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.513025 11441 catalog_manager.cc:5697] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.514659 11442 catalog_manager.cc:5697] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.521060 11835 tablet_bootstrap.cc:492] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.522176 11835 ts_tablet_manager.cc:1403] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.069s	user 0.011s	sys 0.006s
I20260709 12:26:01.523758 11834 tablet_bootstrap.cc:492] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:01.524372 11834 ts_tablet_manager.cc:1403] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.003s
I20260709 12:26:01.527616 11834 raft_consensus.cc:359] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.528402 11834 raft_consensus.cc:385] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.528700 11834 raft_consensus.cc:740] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.529659 11834 consensus_queue.cc:260] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.532042 11834 ts_tablet_manager.cc:1434] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:01.533355 11834 tablet_bootstrap.cc:492] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: Bootstrap starting.
I20260709 12:26:01.526379 11835 raft_consensus.cc:359] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.538048 11835 raft_consensus.cc:385] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.538396 11835 raft_consensus.cc:740] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.538811 11834 tablet_bootstrap.cc:654] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.539476 11835 consensus_queue.cc:260] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.546619 11835 ts_tablet_manager.cc:1434] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:26:01.547086 11441 catalog_manager.cc:5697] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.548362 11835 tablet_bootstrap.cc:492] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:01.550832 11834 tablet_bootstrap.cc:492] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: No bootstrap required, opened a new log
I20260709 12:26:01.551429 11834 ts_tablet_manager.cc:1403] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.003s
I20260709 12:26:01.554522 11834 raft_consensus.cc:359] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.555290 11834 raft_consensus.cc:385] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.555655 11834 raft_consensus.cc:740] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.556473 11834 consensus_queue.cc:260] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.558120 11835 tablet_bootstrap.cc:654] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.558835 11834 ts_tablet_manager.cc:1434] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 12:26:01.564829 11835 tablet_bootstrap.cc:492] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.565295 11835 ts_tablet_manager.cc:1403] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.000s
I20260709 12:26:01.567682 11835 raft_consensus.cc:359] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.568259 11845 raft_consensus.cc:493] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.568363 11835 raft_consensus.cc:385] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.569137 11835 raft_consensus.cc:740] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.568925 11845 raft_consensus.cc:515] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.569813 11835 consensus_queue.cc:260] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.572033 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.571403 11845 leader_election.cc:290] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.572955 11546 raft_consensus.cc:2393] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e119e0d85d645adbc00fd8fa7590ef2 in current term 1: Already voted for candidate 3c9e1ca3ff074690b75a039a85d60299 in this term.
I20260709 12:26:01.572897 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.575753 11835 ts_tablet_manager.cc:1434] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:26:01.575999 11585 leader_election.cc:304] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.576812 11835 tablet_bootstrap.cc:492] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: Bootstrap starting.
I20260709 12:26:01.577873 11822 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.580554 11846 raft_consensus.cc:493] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.581106 11846 raft_consensus.cc:515] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.583388 11846 leader_election.cc:290] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:01.584893 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.585539 11620 raft_consensus.cc:2468] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:26:01.586596 11822 raft_consensus.cc:2749] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.586750 11658 leader_election.cc:304] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:26:01.587791 11846 raft_consensus.cc:2804] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.588241 11846 raft_consensus.cc:493] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.588596 11846 raft_consensus.cc:3060] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.588930 11844 raft_consensus.cc:493] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.589167 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.589530 11844 raft_consensus.cc:515] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.590626 11546 raft_consensus.cc:2468] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 0.
I20260709 12:26:01.592554 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.592828 11844 leader_election.cc:290] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.593254 11620 raft_consensus.cc:2468] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.593454 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.594841 11509 leader_election.cc:304] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.595708 11844 raft_consensus.cc:2804] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.596304 11844 raft_consensus.cc:493] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.596336 11846 raft_consensus.cc:515] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.596787 11844 raft_consensus.cc:3060] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.597222 11835 tablet_bootstrap.cc:654] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:01.597549 11692 raft_consensus.cc:2393] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f00bca05e5804522901cee258b1290a1 in current term 1: Already voted for candidate 3c9e1ca3ff074690b75a039a85d60299 in this term.
I20260709 12:26:01.598603 11846 leader_election.cc:290] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:01.600049 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:01.600859 11620 raft_consensus.cc:3060] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.602744 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:01.606029 11835 tablet_bootstrap.cc:492] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: No bootstrap required, opened a new log
I20260709 12:26:01.605698 11844 raft_consensus.cc:515] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.606653 11835 ts_tablet_manager.cc:1403] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent bootstrapping tablet: real 0.030s	user 0.013s	sys 0.003s
I20260709 12:26:01.606961 11546 raft_consensus.cc:2393] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c9e1ca3ff074690b75a039a85d60299 in current term 1: Already voted for candidate f00bca05e5804522901cee258b1290a1 in this term.
I20260709 12:26:01.609704 11835 raft_consensus.cc:359] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.610240 11620 raft_consensus.cc:2468] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 1.
I20260709 12:26:01.610531 11835 raft_consensus.cc:385] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:01.611102 11835 raft_consensus.cc:740] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Initialized, Role: FOLLOWER
I20260709 12:26:01.611691 11844 leader_election.cc:290] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.611680 11658 leader_election.cc:304] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: f00bca05e5804522901cee258b1290a1
I20260709 12:26:01.611558 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:01.611932 11835 consensus_queue.cc:260] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.611611 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.613304 11621 raft_consensus.cc:2393] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f00bca05e5804522901cee258b1290a1 in current term 1: Already voted for candidate 3c9e1ca3ff074690b75a039a85d60299 in this term.
I20260709 12:26:01.613780 11692 raft_consensus.cc:2393] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f00bca05e5804522901cee258b1290a1 in current term 1: Already voted for candidate 3c9e1ca3ff074690b75a039a85d60299 in this term.
I20260709 12:26:01.614663 11835 ts_tablet_manager.cc:1434] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 12:26:01.615041 11510 leader_election.cc:304] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f00bca05e5804522901cee258b1290a1; no voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:01.615285 11846 raft_consensus.cc:2804] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.615938 11846 raft_consensus.cc:697] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:01.616068 11861 raft_consensus.cc:2749] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.615015 11844 raft_consensus.cc:493] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.616997 11844 raft_consensus.cc:515] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.617519 11846 consensus_queue.cc:237] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.620111 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.621887 11844 leader_election.cc:290] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.622010 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.622970 11692 raft_consensus.cc:2393] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f00bca05e5804522901cee258b1290a1 in current term 1: Already voted for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in this term.
I20260709 12:26:01.624095 11510 leader_election.cc:304] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f00bca05e5804522901cee258b1290a1; no voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:01.625340 11844 raft_consensus.cc:3060] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.631953 11844 raft_consensus.cc:2749] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:01.634593 11440 catalog_manager.cc:5697] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:01.701268 11822 raft_consensus.cc:493] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.701905 11822 raft_consensus.cc:515] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.703742 11822 leader_election.cc:290] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.704604 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1" is_pre_election: true
I20260709 12:26:01.705085 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.705345 11545 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.705742 11692 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 0.
I20260709 12:26:01.706682 11585 leader_election.cc:304] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.707504 11822 raft_consensus.cc:2804] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.707791 11822 raft_consensus.cc:493] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.708083 11822 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.713064 11822 raft_consensus.cc:515] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.714661 11822 leader_election.cc:290] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.715624 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:01.716002 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.716210 11545 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.716634 11692 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.721467 11545 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.721932 11692 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 1.
I20260709 12:26:01.722558 11585 leader_election.cc:304] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.723415 11822 raft_consensus.cc:2804] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.723887 11822 raft_consensus.cc:697] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:01.724686 11822 consensus_queue.cc:237] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.732829 11440 catalog_manager.cc:5697] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 1 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.809579 11844 raft_consensus.cc:493] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.810313 11844 raft_consensus.cc:515] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.812284 11844 leader_election.cc:290] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.813102 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" is_pre_election: true
I20260709 12:26:01.813377 11692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299" is_pre_election: true
I20260709 12:26:01.813875 11620 raft_consensus.cc:2468] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.814018 11692 raft_consensus.cc:2468] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 0.
I20260709 12:26:01.815234 11509 leader_election.cc:304] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.815951 11844 raft_consensus.cc:2804] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:01.816318 11844 raft_consensus.cc:493] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.816601 11844 raft_consensus.cc:3060] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.820108 11861 consensus_queue.cc:1048] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:01.823567 11844 raft_consensus.cc:515] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.825958 11844 leader_election.cc:290] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:01.826510 11844 consensus_queue.cc:1048] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:01.826853 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:01.827775 11621 raft_consensus.cc:3060] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.826972 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:01.831352 11693 raft_consensus.cc:3060] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:01.831512 11822 consensus_queue.cc:1048] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.835940 11621 raft_consensus.cc:2468] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:26:01.837805 11509 leader_election.cc:304] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:01.840078 11861 raft_consensus.cc:2804] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:01.840607 11861 raft_consensus.cc:697] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:01.841439 11861 consensus_queue.cc:237] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:01.847033 11693 raft_consensus.cc:2468] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 1.
I20260709 12:26:01.854102 11861 consensus_queue.cc:1048] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.857736 11440 catalog_manager.cc:5697] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:01.877827 11778 consensus_queue.cc:1048] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:01.883976 11846 consensus_queue.cc:1048] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.891388 11822 consensus_queue.cc:1048] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.919802 11845 consensus_queue.cc:1048] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.941039 11812 consensus_queue.cc:1048] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.972893 11846 consensus_queue.cc:1048] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:01.973312 11621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:41716
I20260709 12:26:01.974642 11621 raft_consensus.cc:606] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Received request to transfer leadership to TS f00bca05e5804522901cee258b1290a1
I20260709 12:26:02.011119 11621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:41716
I20260709 12:26:02.011745 11621 raft_consensus.cc:606] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Received request to transfer leadership to TS f00bca05e5804522901cee258b1290a1
I20260709 12:26:02.021965 11621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:41716
I20260709 12:26:02.022730 11621 raft_consensus.cc:606] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Received request to transfer leadership to TS f00bca05e5804522901cee258b1290a1
I20260709 12:26:02.030511 11694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:53832
I20260709 12:26:02.032321 11694 raft_consensus.cc:606] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:02.034305 11812 raft_consensus.cc:993] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:26:02.039641 11845 raft_consensus.cc:1081] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Signalling peer f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:26:02.044823 11546 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:34892
I20260709 12:26:02.045253 11845 consensus_queue.cc:1048] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.045513 11546 raft_consensus.cc:493] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.046216 11546 raft_consensus.cc:3060] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.049957 11545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:34906
I20260709 12:26:02.050565 11545 raft_consensus.cc:606] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:02.057384 11695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:53832
I20260709 12:26:02.058207 11695 raft_consensus.cc:606] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:02.057505 11546 raft_consensus.cc:515] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.063376 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:02.063932 11621 raft_consensus.cc:3055] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.064230 11621 raft_consensus.cc:740] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.065012 11621 consensus_queue.cc:260] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.066089 11621 raft_consensus.cc:3060] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.067019 11545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
mode: GRACEFUL
new_leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:34906
I20260709 12:26:02.067696 11545 raft_consensus.cc:606] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Received request to transfer leadership to TS 4e119e0d85d645adbc00fd8fa7590ef2
I20260709 12:26:02.068790 11845 consensus_queue.cc:1048] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.072913 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "621bc6cfa6634625b67f91f29db76b81" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.073580 11695 raft_consensus.cc:3060] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.076133 11620 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:41716
I20260709 12:26:02.077098 11620 raft_consensus.cc:606] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Received request to transfer leadership to TS 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:26:02.085084 11546 leader_election.cc:290] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.090098 11695 raft_consensus.cc:2468] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:26:02.091452 11510 leader_election.cc:304] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.092348 11861 raft_consensus.cc:2804] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.092818 11861 raft_consensus.cc:697] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:02.093624 11861 consensus_queue.cc:237] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.097295 11621 raft_consensus.cc:2468] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:26:02.100057 11845 raft_consensus.cc:993] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:26:02.100556 11822 raft_consensus.cc:1081] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Signalling peer f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:26:02.103554 11546 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:34892
I20260709 12:26:02.105129 11546 raft_consensus.cc:493] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.105659 11546 raft_consensus.cc:3060] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.113919 11546 raft_consensus.cc:515] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.117529 11546 leader_election.cc:290] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.117707 11621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:02.118459 11621 raft_consensus.cc:3055] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.118724 11621 raft_consensus.cc:740] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.119745 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984debe0dc4c4b4f91387db0e911a5ae" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.119508 11621 consensus_queue.cc:260] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.120366 11695 raft_consensus.cc:3060] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.121009 11621 raft_consensus.cc:3060] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.121173 11441 catalog_manager.cc:5697] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 1 to 2, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.127506 11621 raft_consensus.cc:2468] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:26:02.127506 11695 raft_consensus.cc:2468] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:26:02.128928 11509 leader_election.cc:304] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.129748 11861 raft_consensus.cc:2804] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.130237 11861 raft_consensus.cc:697] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:02.130872 11861 consensus_queue.cc:237] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.137655 11441 catalog_manager.cc:5697] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 1 to 2, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.162083 11822 consensus_queue.cc:1048] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.174271 11845 consensus_queue.cc:1048] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.182488 11845 raft_consensus.cc:993] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:26:02.182986 11822 raft_consensus.cc:1081] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Signalling peer f00bca05e5804522901cee258b1290a1 to start an election
I20260709 12:26:02.184979 11546 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
 from {username='slave'} at 127.0.0.1:34892
I20260709 12:26:02.185549 11546 raft_consensus.cc:493] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.185962 11546 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.185866 11898 consensus_queue.cc:1048] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.193267 11546 raft_consensus.cc:515] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.197160 11897 consensus_queue.cc:1048] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.197177 11620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:02.197594 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "f00bca05e5804522901cee258b1290a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.198096 11620 raft_consensus.cc:3055] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.198180 11695 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.198496 11620 raft_consensus.cc:740] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.199452 11620 consensus_queue.cc:260] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.200587 11620 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.213420 11620 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:26:02.214712 11509 leader_election.cc:304] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.206310 11695 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00bca05e5804522901cee258b1290a1 in term 2.
I20260709 12:26:02.217417 11778 raft_consensus.cc:2804] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.217907 11778 raft_consensus.cc:697] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Becoming Leader. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:02.218715 11778 consensus_queue.cc:237] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.219666 11546 leader_election.cc:290] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [CANDIDATE]: Term 2 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.227912 11441 catalog_manager.cc:5697] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 reported cstate change: term changed from 1 to 2, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to f00bca05e5804522901cee258b1290a1 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "f00bca05e5804522901cee258b1290a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.229672 11778 consensus_queue.cc:1048] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:02.231952 11861 raft_consensus.cc:993] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.232468 11861 raft_consensus.cc:1081] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.234807 11897 raft_consensus.cc:993] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.235299 11862 raft_consensus.cc:1081] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.237643 11619 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41700
I20260709 12:26:02.238330 11619 raft_consensus.cc:493] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.238710 11778 consensus_queue.cc:1048] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:02.238734 11620 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41698
I20260709 12:26:02.239851 11620 raft_consensus.cc:493] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.240502 11620 raft_consensus.cc:3060] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.238751 11619 raft_consensus.cc:3060] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.256861 11861 raft_consensus.cc:993] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.257495 11861 raft_consensus.cc:1081] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.262420 11619 raft_consensus.cc:515] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.263933 11618 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41698
I20260709 12:26:02.264561 11618 raft_consensus.cc:493] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.264918 11618 raft_consensus.cc:3060] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.266958 11694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.267570 11694 raft_consensus.cc:3055] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.267855 11694 raft_consensus.cc:740] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:02.268807 11694 consensus_queue.cc:260] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.269855 11694 raft_consensus.cc:3060] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.263954 11620 raft_consensus.cc:515] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.275490 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c60cfd059ea5413182db993dbd80c73b" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:02.276153 11546 raft_consensus.cc:3060] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.276716 11694 raft_consensus.cc:2468] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.277712 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:02.278296 11545 raft_consensus.cc:3055] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.278609 11545 raft_consensus.cc:740] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:02.279434 11545 consensus_queue.cc:260] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.280138 11620 leader_election.cc:290] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.280555 11545 raft_consensus.cc:3060] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.282912 11618 raft_consensus.cc:515] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.283442 11546 raft_consensus.cc:2468] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.284988 11585 leader_election.cc:304] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.285857 11822 raft_consensus.cc:2804] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.286345 11822 raft_consensus.cc:697] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.287170 11822 consensus_queue.cc:237] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.288070 11545 raft_consensus.cc:2468] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.288861 11619 leader_election.cc:290] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.289877 11695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f701a63f43f34f65b2f848f84bb73752" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.290444 11695 raft_consensus.cc:3060] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.292230 11585 leader_election.cc:304] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.293047 11822 raft_consensus.cc:2804] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.293557 11822 raft_consensus.cc:697] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.294382 11822 consensus_queue.cc:237] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.295569 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.296201 11693 raft_consensus.cc:3060] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.296948 11441 catalog_manager.cc:5697] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.297894 11695 raft_consensus.cc:2468] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.301147 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c969b48c58646c2ac21bb9b275d7b5b" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:02.301744 11545 raft_consensus.cc:3055] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.302035 11545 raft_consensus.cc:740] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:02.302954 11693 raft_consensus.cc:2468] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.303927 11618 leader_election.cc:290] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.304291 11585 leader_election.cc:304] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:26:02.302781 11545 consensus_queue.cc:260] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.305284 11822 raft_consensus.cc:2804] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.305814 11545 raft_consensus.cc:3060] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.305950 11822 raft_consensus.cc:697] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.306941 11822 consensus_queue.cc:237] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.312085 11545 raft_consensus.cc:2468] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.315559 11441 catalog_manager.cc:5697] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from f00bca05e5804522901cee258b1290a1 (127.25.202.1) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.316682 11441 catalog_manager.cc:5697] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from f00bca05e5804522901cee258b1290a1 (127.25.202.1) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.485530 11822 raft_consensus.cc:993] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:02.486043 11845 raft_consensus.cc:1081] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Signalling peer 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:02.487453 11693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:53822
I20260709 12:26:02.487983 11693 raft_consensus.cc:493] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.488304 11693 raft_consensus.cc:3060] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.491698 11897 raft_consensus.cc:993] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: : Instructing follower 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.492300 11862 raft_consensus.cc:1081] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Signalling peer 4e119e0d85d645adbc00fd8fa7590ef2 to start an election
I20260709 12:26:02.493788 11619 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
 from {username='slave'} at 127.0.0.1:41700
I20260709 12:26:02.494359 11619 raft_consensus.cc:493] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:02.494805 11619 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.494709 11693 raft_consensus.cc:515] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.497203 11693 leader_election.cc:290] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:02.497887 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:02.498494 11545 raft_consensus.cc:3060] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.498435 11618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b09c76b8a494d56abce9d46ea4d5a17" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:02.499214 11618 raft_consensus.cc:3055] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.499600 11618 raft_consensus.cc:740] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.500422 11618 consensus_queue.cc:260] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.501677 11618 raft_consensus.cc:3060] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.502431 11619 raft_consensus.cc:515] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.504297 11545 raft_consensus.cc:2468] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 2.
I20260709 12:26:02.504887 11619 leader_election.cc:290] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3:37239)
I20260709 12:26:02.507160 11659 leader_election.cc:304] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.507287 11693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
I20260709 12:26:02.508069 11862 raft_consensus.cc:2804] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.508067 11693 raft_consensus.cc:3055] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:02.507249 11545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:02.508705 11693 raft_consensus.cc:740] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:02.509058 11545 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.508922 11618 raft_consensus.cc:2468] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 2.
I20260709 12:26:02.508796 11862 raft_consensus.cc:697] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:02.509599 11693 consensus_queue.cc:260] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.511162 11693 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:02.510891 11862 consensus_queue.cc:237] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.514730 11545 raft_consensus.cc:2468] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.515836 11585 leader_election.cc:304] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e119e0d85d645adbc00fd8fa7590ef2, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:02.516572 11845 raft_consensus.cc:2804] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:02.517135 11845 raft_consensus.cc:697] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:02.517979 11845 consensus_queue.cc:237] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:02.518754 11693 raft_consensus.cc:2468] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e119e0d85d645adbc00fd8fa7590ef2 in term 2.
I20260709 12:26:02.519948 11441 catalog_manager.cc:5697] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 1 to 2, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:02.525883 11618 raft_consensus.cc:1275] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.527602 11861 consensus_queue.cc:1048] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.527935 11442 catalog_manager.cc:5697] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 reported cstate change: term changed from 1 to 2, leader changed from 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3) to 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2). New cstate: current_term: 2 leader_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:02.540979 11693 raft_consensus.cc:1275] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.542737 11844 consensus_queue.cc:1048] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.576159 11618 raft_consensus.cc:1275] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.577556 11844 consensus_queue.cc:1048] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:02.586659 11693 raft_consensus.cc:1275] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.588328 11861 consensus_queue.cc:1048] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.700683 11693 raft_consensus.cc:1275] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.702180 11861 consensus_queue.cc:1048] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.710273 11618 raft_consensus.cc:1275] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Refusing update from remote peer f00bca05e5804522901cee258b1290a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.711872 11861 consensus_queue.cc:1048] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:02.759953 11695 raft_consensus.cc:1275] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.761474 11822 consensus_queue.cc:1048] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.769675 11545 raft_consensus.cc:1275] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.770991 11910 consensus_queue.cc:1048] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.802520 11695 raft_consensus.cc:1275] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.803025 11693 raft_consensus.cc:1275] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.803778 11845 consensus_queue.cc:1048] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.804441 11910 consensus_queue.cc:1048] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.812984 11545 raft_consensus.cc:1275] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.815697 11545 raft_consensus.cc:1275] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.816023 11822 consensus_queue.cc:1048] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:02.821136 11822 consensus_queue.cc:1048] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.913616 11545 raft_consensus.cc:1275] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.914809 11862 consensus_queue.cc:1048] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.922900 11619 raft_consensus.cc:1275] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.924592 11911 consensus_queue.cc:1048] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.935981 11545 raft_consensus.cc:1275] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.937198 11845 consensus_queue.cc:1048] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:02.946238 11695 raft_consensus.cc:1275] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Refusing update from remote peer 4e119e0d85d645adbc00fd8fa7590ef2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:02.948031 11822 consensus_queue.cc:1048] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:05.082257 11444 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33308:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:26:05.107931 11444 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33308:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:26:05.178627 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:26:05.376278 11929 log.cc:927] Time spent T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260709 12:26:05.409199 11942 log.cc:927] Time spent T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:26:05.402287 11932 log.cc:927] Time spent T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
W20260709 12:26:05.444348 11931 log.cc:927] Time spent T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: Append to log took a long time: real 0.112s	user 0.002s	sys 0.000s
W20260709 12:26:05.458216 11949 log.cc:927] Time spent T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:26:05.462042 11958 tablet.cc:1724] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:05.523355 11950 log.cc:927] Time spent T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
W20260709 12:26:05.526742 11954 log.cc:927] Time spent T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:26:05.536989 11961 tablet.cc:1724] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.509467 11968 tablet.cc:1724] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.564263 11965 tablet.cc:1724] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:05.568758 11960 log.cc:927] Time spent T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:26:05.570870 11618 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:38850
I20260709 12:26:05.572450 11618 raft_consensus.cc:606] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Received request to transfer leadership to TS 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:26:05.509248 11966 tablet.cc:1724] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:05.602422 11510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.3:37239, user_credentials={real_user=slave}} blocked reactor thread for 89135.6us
W20260709 12:26:05.611613 11721 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:621bc6cfa6634625b67f91f29db76b81)
I20260709 12:26:05.616595 11970 tablet.cc:1724] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.617002 11973 tablet.cc:1724] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:05.624801 11721 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:621bc6cfa6634625b67f91f29db76b81)
W20260709 12:26:05.630811 11721 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:621bc6cfa6634625b67f91f29db76b81)
W20260709 12:26:05.650162 11509 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.202.2:34629, user_credentials={real_user=slave}} blocked reactor thread for 76011.5us
I20260709 12:26:05.667445 11961 tablet.cc:1724] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.669565 11977 tablet.cc:1724] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.676682 11965 tablet.cc:1724] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.684767 11980 tablet.cc:1724] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.678798 11982 tablet.cc:1724] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.710686 11977 tablet.cc:1724] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.684778 11970 tablet.cc:1724] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.740069 11913 raft_consensus.cc:993] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:05.740597 11913 raft_consensus.cc:1081] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Signalling peer 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:05.745012 11693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:53822
I20260709 12:26:05.745822 11693 raft_consensus.cc:493] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:05.746311 11693 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:05.775033 11693 raft_consensus.cc:515] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:05.780233 11977 tablet.cc:1724] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.797856 11544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:05.798816 11544 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:05.804992 11977 tablet.cc:1724] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.807302 11968 tablet.cc:1724] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.821306 11693 leader_election.cc:290] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:05.827211 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "419a49ea5b5146bea4b3eea2a507fa10" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:05.830731 11619 raft_consensus.cc:3055] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:05.831091 11619 raft_consensus.cc:740] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:05.820946 11966 tablet.cc:1724] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.832957 11619 consensus_queue.cc:260] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:05.835827 11619 raft_consensus.cc:3060] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:05.833743 11984 tablet.cc:1724] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.837779 11544 raft_consensus.cc:2468] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:05.840863 11542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
mode: GRACEFUL
new_leader_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:57980
I20260709 12:26:05.841805 11983 tablet.cc:1724] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.847752 11542 raft_consensus.cc:606] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Received request to transfer leadership to TS 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:26:05.845438 11958 tablet.cc:1724] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.852138 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 12:26:05.852813 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:05.853255 11659 leader_election.cc:304] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:05.854212 11967 raft_consensus.cc:2804] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:05.854682 11967 raft_consensus.cc:697] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:05.855505 11967 consensus_queue.cc:237] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:05.859304 11973 tablet.cc:1724] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.860826 11961 tablet.cc:1724] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.868903 11617 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9"
dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
mode: GRACEFUL
new_leader_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:38850
I20260709 12:26:05.870886 11617 raft_consensus.cc:606] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Received request to transfer leadership to TS 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:26:05.865721 11968 tablet.cc:1724] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.870347 11982 tablet.cc:1724] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.877084 11542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb"
dest_uuid: "f00bca05e5804522901cee258b1290a1"
mode: GRACEFUL
new_leader_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:57980
I20260709 12:26:05.877812 11542 raft_consensus.cc:606] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Received request to transfer leadership to TS 3c9e1ca3ff074690b75a039a85d60299
I20260709 12:26:05.875314 11970 tablet.cc:1724] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.874969 11619 raft_consensus.cc:2468] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:05.881908 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:26:05.882539 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:05.908979 11441 catalog_manager.cc:5697] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 2 to 3, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.901808 11965 tablet.cc:1724] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:05.939122 11444 catalog_manager.cc:6242] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813: soft_deleted_table [id=2ac8e19a36794adaad71302484c47bb9] alter complete (version 1)
I20260709 12:26:05.941352 11918 raft_consensus.cc:993] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: : Instructing follower 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:05.942065 11974 raft_consensus.cc:1081] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Signalling peer 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:05.944219 11692 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:53808
I20260709 12:26:05.944752 11692 raft_consensus.cc:493] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:05.945183 11692 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:05.951741 11692 raft_consensus.cc:515] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:05.954182 11692 leader_election.cc:290] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Requested vote from peers 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629), f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087)
I20260709 12:26:05.954473 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:05.954891 11542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a95d90e44047ad8c258903bdb1f9eb" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:05.955112 11619 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:05.955534 11542 raft_consensus.cc:3055] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:05.955875 11542 raft_consensus.cc:740] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:05.956660 11542 consensus_queue.cc:260] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:05.957952 11542 raft_consensus.cc:3060] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:05.962179 11619 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:05.964493 11658 leader_election.cc:304] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:26:05.964898 11542 raft_consensus.cc:2468] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:05.965492 11987 raft_consensus.cc:2804] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:05.966105 11987 raft_consensus.cc:697] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:05.966946 11987 consensus_queue.cc:237] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:05.974741 11441 catalog_manager.cc:5697] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 2 to 3, leader changed from f00bca05e5804522901cee258b1290a1 (127.25.202.1) to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:06.132287 11979 raft_consensus.cc:993] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: : Instructing follower 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:06.132742 11974 raft_consensus.cc:1081] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Signalling peer 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:06.134310 11692 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:53808
I20260709 12:26:06.134792 11692 raft_consensus.cc:493] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:06.135120 11692 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:06.140779 11692 raft_consensus.cc:515] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:06.143153 11692 leader_election.cc:290] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:06.143654 11542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:06.144345 11542 raft_consensus.cc:3055] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:06.144687 11542 raft_consensus.cc:740] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f00bca05e5804522901cee258b1290a1, State: Running, Role: LEADER
I20260709 12:26:06.144734 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9817dc25684c4781a16c88f50f925ce1" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:06.145351 11619 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:06.145479 11542 consensus_queue.cc:260] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:06.146648 11542 raft_consensus.cc:3060] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:06.150838 11619 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:06.152033 11658 leader_election.cc:304] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, 4e119e0d85d645adbc00fd8fa7590ef2; no voters: 
I20260709 12:26:06.152173 11542 raft_consensus.cc:2468] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:06.152822 11987 raft_consensus.cc:2804] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:06.153318 11987 raft_consensus.cc:697] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:06.154096 11987 consensus_queue.cc:237] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:06.161070 11441 catalog_manager.cc:5697] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 2 to 3, leader changed from f00bca05e5804522901cee258b1290a1 (127.25.202.1) to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:06.242456 11975 raft_consensus.cc:993] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: : Instructing follower 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:06.243237 11951 raft_consensus.cc:1081] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Signalling peer 3c9e1ca3ff074690b75a039a85d60299 to start an election
I20260709 12:26:06.245124 11692 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9"
dest_uuid: "3c9e1ca3ff074690b75a039a85d60299"
 from {username='slave'} at 127.0.0.1:53822
I20260709 12:26:06.245898 11692 raft_consensus.cc:493] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:06.246220 11692 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:06.251884 11692 raft_consensus.cc:515] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:06.253530 11692 leader_election.cc:290] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Requested vote from peers f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087), 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629)
I20260709 12:26:06.254374 11542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f00bca05e5804522901cee258b1290a1"
I20260709 12:26:06.254616 11619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80924118d6e74310b99c0478496ae0b9" candidate_uuid: "3c9e1ca3ff074690b75a039a85d60299" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4e119e0d85d645adbc00fd8fa7590ef2"
I20260709 12:26:06.255160 11542 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:06.255443 11619 raft_consensus.cc:3055] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:06.255808 11619 raft_consensus.cc:740] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e119e0d85d645adbc00fd8fa7590ef2, State: Running, Role: LEADER
I20260709 12:26:06.256633 11619 consensus_queue.cc:260] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:06.258004 11619 raft_consensus.cc:3060] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:06.263523 11542 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:06.264989 11659 leader_election.cc:304] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9e1ca3ff074690b75a039a85d60299, f00bca05e5804522901cee258b1290a1; no voters: 
I20260709 12:26:06.265250 11619 raft_consensus.cc:2468] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9e1ca3ff074690b75a039a85d60299 in term 3.
I20260709 12:26:06.265861 11987 raft_consensus.cc:2804] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:06.266427 11987 raft_consensus.cc:697] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Becoming Leader. State: Replica: 3c9e1ca3ff074690b75a039a85d60299, State: Running, Role: LEADER
I20260709 12:26:06.267292 11987 consensus_queue.cc:237] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } }
I20260709 12:26:06.275770 11441 catalog_manager.cc:5697] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 reported cstate change: term changed from 2 to 3, leader changed from 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2) to 3c9e1ca3ff074690b75a039a85d60299 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "3c9e1ca3ff074690b75a039a85d60299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9e1ca3ff074690b75a039a85d60299" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:06.359546 11619 raft_consensus.cc:1275] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:26:06.361129 11987 consensus_queue.cc:1048] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:26:06.370685 11542 raft_consensus.cc:1275] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:26:06.372164 11987 consensus_queue.cc:1048] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:26:06.501362 11619 raft_consensus.cc:1275] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:06.502616 11987 consensus_queue.cc:1048] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:06.512104 11544 raft_consensus.cc:1275] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:06.514163 11911 consensus_queue.cc:1048] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:06.646646 11544 raft_consensus.cc:1275] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:06.648332 11911 consensus_queue.cc:1048] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:06.658022 11617 raft_consensus.cc:1275] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:06.659900 11998 consensus_queue.cc:1048] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:06.722621 11617 raft_consensus.cc:1275] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:26:06.724056 11967 consensus_queue.cc:1048] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e119e0d85d645adbc00fd8fa7590ef2" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:26:06.728258 11544 raft_consensus.cc:1275] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Refusing update from remote peer 3c9e1ca3ff074690b75a039a85d60299: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:26:06.730484 11998 consensus_queue.cc:1048] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00bca05e5804522901cee258b1290a1" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:26:06.883984 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:06.889184 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:06.889618 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:06.894591 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:06.895016 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:07.896121 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:07.901666 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:07.902120 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:07.907902 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:07.908519 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:08.910513 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:08.919588 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:08.920077 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:08.928509 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:08.928973 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:09.930274 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:09.935454 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:09.935921 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:09.940574 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:09.940975 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:10.942096 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:10.947083 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:10.947566 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:10.952281 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:10.952708 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:11.953799 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:11.959556 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:11.960134 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:11.965982 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:11.966459 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:12.967712 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:12.974082 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:12.974547 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:12.979887 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:12.980329 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:13.981369 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:13.986816 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:13.987274 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:13.992067 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:13.992477 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:14.993672 26408 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:26:14.999225 26408 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:26:14.999857 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:15.007004 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:15.007480 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:16.075146 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
W20260709 12:26:16.086503 11585 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20260709 12:26:16.092569 11585 consensus_peers.cc:597] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.126109 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:26:16.126245 11659 consensus_peers.cc:597] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.127233 26408 tablet_replica.cc:333] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.128361 26408 raft_consensus.cc:2243] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.129070 26408 raft_consensus.cc:2272] T 2b09c76b8a494d56abce9d46ea4d5a17 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.131932 26408 tablet_replica.cc:333] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.132614 26408 raft_consensus.cc:2243] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.133136 26408 raft_consensus.cc:2272] T 9817dc25684c4781a16c88f50f925ce1 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.135718 26408 tablet_replica.cc:333] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.136401 26408 raft_consensus.cc:2243] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.136965 26408 raft_consensus.cc:2272] T 0c969b48c58646c2ac21bb9b275d7b5b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.139549 26408 tablet_replica.cc:333] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.140182 26408 raft_consensus.cc:2243] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.141129 26408 raft_consensus.cc:2272] T 3e495b6c9db94ccda94d34a8c66192e7 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.145116 26408 tablet_replica.cc:333] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.145803 26408 raft_consensus.cc:2243] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.146365 26408 raft_consensus.cc:2272] T 419a49ea5b5146bea4b3eea2a507fa10 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.149302 26408 tablet_replica.cc:333] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.149976 26408 raft_consensus.cc:2243] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.150554 26408 raft_consensus.cc:2272] T 80924118d6e74310b99c0478496ae0b9 P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.153051 26408 tablet_replica.cc:333] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.153697 26408 raft_consensus.cc:2243] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.154517 26408 raft_consensus.cc:2272] T f701a63f43f34f65b2f848f84bb73752 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.156925 26408 tablet_replica.cc:333] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.157804 26408 raft_consensus.cc:2243] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.159056 26408 raft_consensus.cc:2272] T 6d7b3decd14c4619bb62776d919d8ae9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:16.162261 11585 consensus_peers.cc:597] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.162727 26408 tablet_replica.cc:333] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.163767 26408 raft_consensus.cc:2243] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.164752 26408 raft_consensus.cc:2272] T 984debe0dc4c4b4f91387db0e911a5ae P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.167892 26408 tablet_replica.cc:333] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.168470 26408 raft_consensus.cc:2243] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.169152 26408 raft_consensus.cc:2272] T 5a17e2a0a7964ec4a956743fe816de45 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.171849 26408 tablet_replica.cc:333] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.172570 26408 raft_consensus.cc:2243] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.173496 26408 raft_consensus.cc:2272] T 22f1c13eb88b4a56ad4eca20821a17e9 P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.175976 26408 tablet_replica.cc:333] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.176563 26408 raft_consensus.cc:2243] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.177470 26408 raft_consensus.cc:2272] T 621bc6cfa6634625b67f91f29db76b81 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.180048 26408 tablet_replica.cc:333] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.180648 26408 raft_consensus.cc:2243] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.181211 26408 raft_consensus.cc:2272] T b5a95d90e44047ad8c258903bdb1f9eb P f00bca05e5804522901cee258b1290a1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.183585 26408 tablet_replica.cc:333] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.184175 26408 raft_consensus.cc:2243] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.184757 26408 raft_consensus.cc:2272] T c60cfd059ea5413182db993dbd80c73b P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.187296 26408 tablet_replica.cc:333] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.187892 26408 raft_consensus.cc:2243] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.188437 26408 raft_consensus.cc:2272] T 735953c802fa471ba0d68511ee3495e5 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.190742 26408 tablet_replica.cc:333] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.191282 26408 raft_consensus.cc:2243] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.192482 26408 raft_consensus.cc:2272] T c725053672514cf69f7eec8351d1759e P f00bca05e5804522901cee258b1290a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.194929 26408 tablet_replica.cc:333] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.195497 26408 raft_consensus.cc:2243] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.196107 26408 raft_consensus.cc:2272] T d24b1b461ca143e4a8f686d24d9b0751 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.198611 26408 tablet_replica.cc:333] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1: stopping tablet replica
I20260709 12:26:16.199120 26408 raft_consensus.cc:2243] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.199654 26408 raft_consensus.cc:2272] T c3d00baea3d94637ab044e58ce707954 P f00bca05e5804522901cee258b1290a1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:16.225023 11585 consensus_peers.cc:597] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.249827 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
I20260709 12:26:16.296553 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
W20260709 12:26:16.319013 11659 consensus_peers.cc:597] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.319734 11659 consensus_peers.cc:597] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.320291 11659 consensus_peers.cc:597] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.335649 11658 consensus_peers.cc:597] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Couldn't send request to peer 4e119e0d85d645adbc00fd8fa7590ef2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.342686 11658 consensus_peers.cc:597] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Couldn't send request to peer 4e119e0d85d645adbc00fd8fa7590ef2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.348886 11585 consensus_peers.cc:597] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.349576 11585 consensus_peers.cc:597] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.351121 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:16.351953 26408 tablet_replica.cc:333] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.352995 26408 raft_consensus.cc:2243] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.354063 26408 raft_consensus.cc:2272] T 0c969b48c58646c2ac21bb9b275d7b5b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.356652 26408 tablet_replica.cc:333] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.357388 26408 raft_consensus.cc:2243] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.357987 26408 raft_consensus.cc:2272] T 419a49ea5b5146bea4b3eea2a507fa10 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.360545 26408 tablet_replica.cc:333] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.361274 26408 raft_consensus.cc:2243] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.361790 26408 raft_consensus.cc:2272] T 9817dc25684c4781a16c88f50f925ce1 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.365200 26408 tablet_replica.cc:333] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.365931 26408 raft_consensus.cc:2243] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.366622 26408 raft_consensus.cc:2272] T 2b09c76b8a494d56abce9d46ea4d5a17 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.369208 26408 tablet_replica.cc:333] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.369814 26408 raft_consensus.cc:2243] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.370302 26408 raft_consensus.cc:2272] T 80924118d6e74310b99c0478496ae0b9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.372875 26408 tablet_replica.cc:333] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.373450 26408 raft_consensus.cc:2243] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.374413 26408 raft_consensus.cc:2272] T f701a63f43f34f65b2f848f84bb73752 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.377101 26408 tablet_replica.cc:333] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.377748 26408 raft_consensus.cc:2243] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.378249 26408 raft_consensus.cc:2272] T c3d00baea3d94637ab044e58ce707954 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.380872 26408 tablet_replica.cc:333] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.381662 26408 raft_consensus.cc:2243] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.382210 26408 raft_consensus.cc:2272] T 621bc6cfa6634625b67f91f29db76b81 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.384817 26408 tablet_replica.cc:333] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.385452 26408 raft_consensus.cc:2243] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.386026 26408 raft_consensus.cc:2272] T b5a95d90e44047ad8c258903bdb1f9eb P 4e119e0d85d645adbc00fd8fa7590ef2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.389073 26408 tablet_replica.cc:333] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.389673 26408 raft_consensus.cc:2243] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.390157 26408 raft_consensus.cc:2272] T 22f1c13eb88b4a56ad4eca20821a17e9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.393085 26408 tablet_replica.cc:333] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.393750 26408 raft_consensus.cc:2243] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.394686 26408 raft_consensus.cc:2272] T 5a17e2a0a7964ec4a956743fe816de45 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:16.396888 11658 consensus_peers.cc:597] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Couldn't send request to peer 4e119e0d85d645adbc00fd8fa7590ef2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.397907 26408 tablet_replica.cc:333] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.398569 26408 raft_consensus.cc:2243] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.399663 26408 raft_consensus.cc:2272] T d24b1b461ca143e4a8f686d24d9b0751 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:16.401046 11659 consensus_peers.cc:597] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.402271 26408 tablet_replica.cc:333] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.402951 26408 raft_consensus.cc:2243] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.403474 26408 raft_consensus.cc:2272] T c725053672514cf69f7eec8351d1759e P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.405973 26408 tablet_replica.cc:333] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.406607 26408 raft_consensus.cc:2243] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.407531 26408 raft_consensus.cc:2272] T c60cfd059ea5413182db993dbd80c73b P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.409793 26408 tablet_replica.cc:333] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.410324 26408 raft_consensus.cc:2243] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.411204 26408 raft_consensus.cc:2272] T 735953c802fa471ba0d68511ee3495e5 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.413585 26408 tablet_replica.cc:333] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.414148 26408 raft_consensus.cc:2243] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.414752 26408 raft_consensus.cc:2272] T 3e495b6c9db94ccda94d34a8c66192e7 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.417021 26408 tablet_replica.cc:333] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.417553 26408 raft_consensus.cc:2243] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.418121 26408 raft_consensus.cc:2272] T 984debe0dc4c4b4f91387db0e911a5ae P 4e119e0d85d645adbc00fd8fa7590ef2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.420672 26408 tablet_replica.cc:333] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2: stopping tablet replica
I20260709 12:26:16.421211 26408 raft_consensus.cc:2243] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.421741 26408 raft_consensus.cc:2272] T 6d7b3decd14c4619bb62776d919d8ae9 P 4e119e0d85d645adbc00fd8fa7590ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:16.447522 11658 consensus_peers.cc:597] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 -> Peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Couldn't send request to peer 4e119e0d85d645adbc00fd8fa7590ef2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.457978 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
W20260709 12:26:16.458606 11658 consensus_peers.cc:597] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Couldn't send request to peer 4e119e0d85d645adbc00fd8fa7590ef2. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:34629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:16.465178 11659 consensus_peers.cc:597] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer f00bca05e5804522901cee258b1290a1 (127.25.202.1:42087): Couldn't send request to peer f00bca05e5804522901cee258b1290a1. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:42087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.501758 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
W20260709 12:26:16.531845 11658 consensus_peers.cc:597] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 -> Peer 4e119e0d85d645adbc00fd8fa7590ef2 (127.25.202.2:34629): Couldn't send request to peer 4e119e0d85d645adbc00fd8fa7590ef2. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:34629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:16.537459 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:16.538242 26408 tablet_replica.cc:333] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.539143 26408 raft_consensus.cc:2243] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.540300 26408 raft_consensus.cc:2272] T 2b09c76b8a494d56abce9d46ea4d5a17 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.542784 26408 tablet_replica.cc:333] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.543550 26408 raft_consensus.cc:2243] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.544126 26408 raft_consensus.cc:2272] T 0c969b48c58646c2ac21bb9b275d7b5b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.546551 26408 tablet_replica.cc:333] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.547171 26408 raft_consensus.cc:2243] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.547818 26408 raft_consensus.cc:2272] T 3e495b6c9db94ccda94d34a8c66192e7 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.550197 26408 tablet_replica.cc:333] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.550781 26408 raft_consensus.cc:2243] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.551774 26408 raft_consensus.cc:2272] T 419a49ea5b5146bea4b3eea2a507fa10 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.554216 26408 tablet_replica.cc:333] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.554906 26408 raft_consensus.cc:2243] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.555851 26408 raft_consensus.cc:2272] T b5a95d90e44047ad8c258903bdb1f9eb P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.558199 26408 tablet_replica.cc:333] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.559029 26408 raft_consensus.cc:2243] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.559607 26408 raft_consensus.cc:2272] T 621bc6cfa6634625b67f91f29db76b81 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.562021 26408 tablet_replica.cc:333] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.562654 26408 raft_consensus.cc:2243] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.563153 26408 raft_consensus.cc:2272] T 22f1c13eb88b4a56ad4eca20821a17e9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.565390 26408 tablet_replica.cc:333] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.566005 26408 raft_consensus.cc:2243] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.566524 26408 raft_consensus.cc:2272] T 5a17e2a0a7964ec4a956743fe816de45 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.568765 26408 tablet_replica.cc:333] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.569414 26408 raft_consensus.cc:2243] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.570276 26408 raft_consensus.cc:2272] T 9817dc25684c4781a16c88f50f925ce1 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.572537 26408 tablet_replica.cc:333] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.573159 26408 raft_consensus.cc:2243] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.574064 26408 raft_consensus.cc:2272] T c3d00baea3d94637ab044e58ce707954 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.576407 26408 tablet_replica.cc:333] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.577059 26408 raft_consensus.cc:2243] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.577555 26408 raft_consensus.cc:2272] T c725053672514cf69f7eec8351d1759e P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.579984 26408 tablet_replica.cc:333] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.580588 26408 raft_consensus.cc:2243] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.581147 26408 raft_consensus.cc:2272] T d24b1b461ca143e4a8f686d24d9b0751 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.583361 26408 tablet_replica.cc:333] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.583933 26408 raft_consensus.cc:2243] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.584465 26408 raft_consensus.cc:2272] T 984debe0dc4c4b4f91387db0e911a5ae P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.586747 26408 tablet_replica.cc:333] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.587365 26408 raft_consensus.cc:2243] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.587896 26408 raft_consensus.cc:2272] T 6d7b3decd14c4619bb62776d919d8ae9 P 3c9e1ca3ff074690b75a039a85d60299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.590420 26408 tablet_replica.cc:333] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.590997 26408 raft_consensus.cc:2243] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.591539 26408 raft_consensus.cc:2272] T c60cfd059ea5413182db993dbd80c73b P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.593998 26408 tablet_replica.cc:333] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.594569 26408 raft_consensus.cc:2243] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.595060 26408 raft_consensus.cc:2272] T 735953c802fa471ba0d68511ee3495e5 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.597417 26408 tablet_replica.cc:333] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.597955 26408 raft_consensus.cc:2243] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:16.598464 26408 raft_consensus.cc:2272] T f701a63f43f34f65b2f848f84bb73752 P 3c9e1ca3ff074690b75a039a85d60299 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.600729 26408 tablet_replica.cc:333] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299: stopping tablet replica
I20260709 12:26:16.601322 26408 raft_consensus.cc:2243] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.602243 26408 raft_consensus.cc:2272] T 80924118d6e74310b99c0478496ae0b9 P 3c9e1ca3ff074690b75a039a85d60299 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.649349 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:26:16.687806 26408 master.cc:562] Master@127.25.202.62:33011 shutting down...
I20260709 12:26:16.713755 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:16.714432 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:16.714784 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P b44b70e854b446b4956c66823c526813: stopping tablet replica
I20260709 12:26:16.736138 26408 master.cc:584] Master@127.25.202.62:33011 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (21015 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:26:16.792464 26408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.202.62:41237
I20260709 12:26:16.793649 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:16.799825 12030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:16.799808 12031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:16.800760 12033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:16.802995 26408 server_base.cc:1061] running on GCE node
I20260709 12:26:16.804260 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:16.804435 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:16.804636 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599976804603 us; error 0 us; skew 500 ppm
I20260709 12:26:16.805176 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:16.807631 26408 webserver.cc:533] Webserver started at http://127.25.202.62:44171/ using document root <none> and password file <none>
I20260709 12:26:16.808158 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:16.808347 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:16.808625 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:16.809731 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/master-0-root/instance:
uuid: "4672144bf9f64bccaf81431afd23fabc"
format_stamp: "Formatted at 2026-07-09 12:26:16 on dist-test-slave-m57n"
I20260709 12:26:16.814545 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.006s
I20260709 12:26:16.817956 12038 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:16.818641 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:26:16.818936 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/master-0-root
uuid: "4672144bf9f64bccaf81431afd23fabc"
format_stamp: "Formatted at 2026-07-09 12:26:16 on dist-test-slave-m57n"
I20260709 12:26:16.819201 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:16.834429 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:16.835878 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:16.873627 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.62:41237
I20260709 12:26:16.873736 12089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.62:41237 every 8 connection(s)
I20260709 12:26:16.877869 12090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:16.890482 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc: Bootstrap starting.
I20260709 12:26:16.895731 12090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:16.900296 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc: No bootstrap required, opened a new log
I20260709 12:26:16.903224 12090 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4672144bf9f64bccaf81431afd23fabc" member_type: VOTER }
I20260709 12:26:16.903790 12090 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:16.904011 12090 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4672144bf9f64bccaf81431afd23fabc, State: Initialized, Role: FOLLOWER
I20260709 12:26:16.904650 12090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4672144bf9f64bccaf81431afd23fabc" member_type: VOTER }
I20260709 12:26:16.905141 12090 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:26:16.905372 12090 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:26:16.905730 12090 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:16.910712 12090 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4672144bf9f64bccaf81431afd23fabc" member_type: VOTER }
I20260709 12:26:16.911207 12090 leader_election.cc:304] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4672144bf9f64bccaf81431afd23fabc; no voters: 
I20260709 12:26:16.912259 12090 leader_election.cc:290] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:26:16.912719 12093 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:16.914103 12093 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 1 LEADER]: Becoming Leader. State: Replica: 4672144bf9f64bccaf81431afd23fabc, State: Running, Role: LEADER
I20260709 12:26:16.914812 12093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4672144bf9f64bccaf81431afd23fabc" member_type: VOTER }
I20260709 12:26:16.915277 12090 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:26:16.921048 12094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4672144bf9f64bccaf81431afd23fabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4672144bf9f64bccaf81431afd23fabc" member_type: VOTER } }
I20260709 12:26:16.921620 12094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [sys.catalog]: This master's current role is: LEADER
I20260709 12:26:16.920286 12095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4672144bf9f64bccaf81431afd23fabc. Latest consensus state: current_term: 1 leader_uuid: "4672144bf9f64bccaf81431afd23fabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4672144bf9f64bccaf81431afd23fabc" member_type: VOTER } }
I20260709 12:26:16.922648 12095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [sys.catalog]: This master's current role is: LEADER
I20260709 12:26:16.923666 12101 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:26:16.931663 12101 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:26:16.932638 26408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:26:16.940716 12101 catalog_manager.cc:1383] Generated new cluster ID: 7c575228a9a64d93a0679011cc13a3cd
I20260709 12:26:16.941008 12101 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:26:16.970044 12101 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:26:16.971622 12101 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:26:16.985700 12101 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc: Generated new TSK 0
I20260709 12:26:16.986407 12101 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:26:16.999111 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:17.005541 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:17.006261 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:17.009963 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:17.010255 26408 server_base.cc:1061] running on GCE node
I20260709 12:26:17.011210 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:17.011490 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:17.011678 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599977011659 us; error 0 us; skew 500 ppm
I20260709 12:26:17.012229 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:17.014680 26408 webserver.cc:533] Webserver started at http://127.25.202.1:39513/ using document root <none> and password file <none>
I20260709 12:26:17.015206 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:17.015448 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:17.015728 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:17.016887 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-0-root/instance:
uuid: "62022a2fa120494cbe56659b74b966f8"
format_stamp: "Formatted at 2026-07-09 12:26:17 on dist-test-slave-m57n"
I20260709 12:26:17.021723 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:26:17.024991 12119 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.025712 26408 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:26:17.026038 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-0-root
uuid: "62022a2fa120494cbe56659b74b966f8"
format_stamp: "Formatted at 2026-07-09 12:26:17 on dist-test-slave-m57n"
I20260709 12:26:17.026364 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:17.040612 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:17.041982 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:17.043679 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:17.046249 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:17.046504 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.046825 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:17.047055 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.093770 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.1:44853
I20260709 12:26:17.093912 12181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.1:44853 every 8 connection(s)
I20260709 12:26:17.099685 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:17.108579 12186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:17.108911 12187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:17.112370 12189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:17.112623 26408 server_base.cc:1061] running on GCE node
I20260709 12:26:17.114205 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:17.114500 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:17.114743 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599977114724 us; error 0 us; skew 500 ppm
I20260709 12:26:17.115600 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:17.118428 12182 heartbeater.cc:344] Connected to a master server at 127.25.202.62:41237
I20260709 12:26:17.118686 26408 webserver.cc:533] Webserver started at http://127.25.202.2:34151/ using document root <none> and password file <none>
I20260709 12:26:17.118896 12182 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:17.119503 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:17.119772 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:17.120085 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:17.120012 12182 heartbeater.cc:507] Master 127.25.202.62:41237 requested a full tablet report, sending...
I20260709 12:26:17.121811 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-1-root/instance:
uuid: "588391676e574efa9b7da64a5ca954aa"
format_stamp: "Formatted at 2026-07-09 12:26:17 on dist-test-slave-m57n"
I20260709 12:26:17.122622 12055 ts_manager.cc:194] Registered new tserver with Master: 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853)
I20260709 12:26:17.124635 12055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37380
I20260709 12:26:17.127668 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 12:26:17.131951 12194 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.132941 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:26:17.133292 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-1-root
uuid: "588391676e574efa9b7da64a5ca954aa"
format_stamp: "Formatted at 2026-07-09 12:26:17 on dist-test-slave-m57n"
I20260709 12:26:17.133630 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:17.164211 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:17.165591 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:17.167397 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:17.169950 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:17.170209 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.170517 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:17.170742 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.218104 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.2:34351
I20260709 12:26:17.218237 12256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.2:34351 every 8 connection(s)
I20260709 12:26:17.223937 26408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:17.232507 12260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:17.234793 12261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:17.236194 12263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:17.237493 26408 server_base.cc:1061] running on GCE node
I20260709 12:26:17.238521 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:17.238775 26408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:17.238982 26408 hybrid_clock.cc:648] HybridClock initialized: now 1783599977238960 us; error 0 us; skew 500 ppm
I20260709 12:26:17.239636 26408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:17.242437 26408 webserver.cc:533] Webserver started at http://127.25.202.3:43543/ using document root <none> and password file <none>
I20260709 12:26:17.243005 12257 heartbeater.cc:344] Connected to a master server at 127.25.202.62:41237
I20260709 12:26:17.243155 26408 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:17.243507 26408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:17.243530 12257 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:17.243958 26408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:17.244554 12257 heartbeater.cc:507] Master 127.25.202.62:41237 requested a full tablet report, sending...
I20260709 12:26:17.245620 26408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-2-root/instance:
uuid: "3a09a0f9c16e402b8237f06400fc1b36"
format_stamp: "Formatted at 2026-07-09 12:26:17 on dist-test-slave-m57n"
I20260709 12:26:17.246879 12055 ts_manager.cc:194] Registered new tserver with Master: 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:17.248400 12055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37388
I20260709 12:26:17.250856 26408 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:26:17.254791 12268 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.255846 26408 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:26:17.256197 26408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-2-root
uuid: "3a09a0f9c16e402b8237f06400fc1b36"
format_stamp: "Formatted at 2026-07-09 12:26:17 on dist-test-slave-m57n"
I20260709 12:26:17.256511 26408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfHEpW1/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599387827064-26408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:17.285952 26408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:17.287200 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:17.288724 26408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:17.290984 26408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:17.291215 26408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.291491 26408 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:17.291687 26408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:17.343159 26408 rpc_server.cc:307] RPC server started. Bound to: 127.25.202.3:46063
I20260709 12:26:17.343271 12330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.202.3:46063 every 8 connection(s)
I20260709 12:26:17.358553 12331 heartbeater.cc:344] Connected to a master server at 127.25.202.62:41237
I20260709 12:26:17.359117 12331 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:17.360029 12331 heartbeater.cc:507] Master 127.25.202.62:41237 requested a full tablet report, sending...
I20260709 12:26:17.362160 12055 ts_manager.cc:194] Registered new tserver with Master: 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:17.362406 26408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015797885s
I20260709 12:26:17.364300 12055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37394
I20260709 12:26:17.389858 12055 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37410:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:26:17.392692 12055 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:26:17.482203 12146 tablet_service.cc:1511] Processing CreateTablet for tablet cf6db4244c274ed2b604f852a7267a95 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:17.483742 12147 tablet_service.cc:1511] Processing CreateTablet for tablet 61488d77945649d59f085e9f2035be1a (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:17.484280 12146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6db4244c274ed2b604f852a7267a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.485291 12147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61488d77945649d59f085e9f2035be1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.494038 12145 tablet_service.cc:1511] Processing CreateTablet for tablet e9689cd771ee44b2808c2d7c52d83766 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:17.496268 12145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9689cd771ee44b2808c2d7c52d83766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.505744 12144 tablet_service.cc:1511] Processing CreateTablet for tablet d499c7e4809645de92dfc91281414bdd (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:17.507295 12143 tablet_service.cc:1511] Processing CreateTablet for tablet 3ecef6444dfd41e1bbb5b1534a4626be (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:17.507352 12144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d499c7e4809645de92dfc91281414bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.511729 12142 tablet_service.cc:1511] Processing CreateTablet for tablet 2a9e43f88aef4273a7a00ced5ba8e744 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:17.512797 12143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecef6444dfd41e1bbb5b1534a4626be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.513463 12142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9e43f88aef4273a7a00ced5ba8e744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.519730 12141 tablet_service.cc:1511] Processing CreateTablet for tablet 781cab6f3dbf45a58fa529278460248f (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:17.521193 12141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781cab6f3dbf45a58fa529278460248f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.537894 12140 tablet_service.cc:1511] Processing CreateTablet for tablet 8d871c96be864188b0b1da87fd438d00 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:17.542225 12139 tablet_service.cc:1511] Processing CreateTablet for tablet 503e4fb15a654c938a923e5c6808c209 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:17.544129 12140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d871c96be864188b0b1da87fd438d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.548002 12139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503e4fb15a654c938a923e5c6808c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.554369 12346 tablet_bootstrap.cc:492] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:17.561951 12222 tablet_service.cc:1511] Processing CreateTablet for tablet 61488d77945649d59f085e9f2035be1a (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:17.565183 12219 tablet_service.cc:1511] Processing CreateTablet for tablet d499c7e4809645de92dfc91281414bdd (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:17.565495 12221 tablet_service.cc:1511] Processing CreateTablet for tablet cf6db4244c274ed2b604f852a7267a95 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:17.568251 12216 tablet_service.cc:1511] Processing CreateTablet for tablet 781cab6f3dbf45a58fa529278460248f (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:17.568602 12215 tablet_service.cc:1511] Processing CreateTablet for tablet 8d871c96be864188b0b1da87fd438d00 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:17.569285 12214 tablet_service.cc:1511] Processing CreateTablet for tablet 503e4fb15a654c938a923e5c6808c209 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:17.570744 12220 tablet_service.cc:1511] Processing CreateTablet for tablet e9689cd771ee44b2808c2d7c52d83766 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:17.573493 12222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61488d77945649d59f085e9f2035be1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.573343 12217 tablet_service.cc:1511] Processing CreateTablet for tablet 2a9e43f88aef4273a7a00ced5ba8e744 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:17.574484 12215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d871c96be864188b0b1da87fd438d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.575251 12217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9e43f88aef4273a7a00ced5ba8e744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.573395 12218 tablet_service.cc:1511] Processing CreateTablet for tablet 3ecef6444dfd41e1bbb5b1534a4626be (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:17.576821 12216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781cab6f3dbf45a58fa529278460248f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.577455 12221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6db4244c274ed2b604f852a7267a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.579648 12220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9689cd771ee44b2808c2d7c52d83766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.580302 12219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d499c7e4809645de92dfc91281414bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.582803 12218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecef6444dfd41e1bbb5b1534a4626be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.590468 12214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503e4fb15a654c938a923e5c6808c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.603080 12296 tablet_service.cc:1511] Processing CreateTablet for tablet 61488d77945649d59f085e9f2035be1a (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:17.604640 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61488d77945649d59f085e9f2035be1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.606874 12290 tablet_service.cc:1511] Processing CreateTablet for tablet 781cab6f3dbf45a58fa529278460248f (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:17.608412 12290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781cab6f3dbf45a58fa529278460248f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.611950 12291 tablet_service.cc:1511] Processing CreateTablet for tablet 2a9e43f88aef4273a7a00ced5ba8e744 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:17.613507 12291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9e43f88aef4273a7a00ced5ba8e744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.613147 12292 tablet_service.cc:1511] Processing CreateTablet for tablet 3ecef6444dfd41e1bbb5b1534a4626be (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:17.614962 12293 tablet_service.cc:1511] Processing CreateTablet for tablet d499c7e4809645de92dfc91281414bdd (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:17.616135 12292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecef6444dfd41e1bbb5b1534a4626be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.611935 12346 tablet_bootstrap.cc:654] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.622169 12293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d499c7e4809645de92dfc91281414bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.621948 12295 tablet_service.cc:1511] Processing CreateTablet for tablet cf6db4244c274ed2b604f852a7267a95 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:17.623044 12288 tablet_service.cc:1511] Processing CreateTablet for tablet 503e4fb15a654c938a923e5c6808c209 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:17.623631 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6db4244c274ed2b604f852a7267a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.624384 12288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503e4fb15a654c938a923e5c6808c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.625620 12289 tablet_service.cc:1511] Processing CreateTablet for tablet 8d871c96be864188b0b1da87fd438d00 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:17.626977 12289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d871c96be864188b0b1da87fd438d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.647475 12294 tablet_service.cc:1511] Processing CreateTablet for tablet e9689cd771ee44b2808c2d7c52d83766 (DEFAULT_TABLE table=test-workload [id=467e4be2f33643d4be51b42364e651c0]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:17.649927 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9689cd771ee44b2808c2d7c52d83766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:17.722218 12346 tablet_bootstrap.cc:492] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:17.722836 12346 ts_tablet_manager.cc:1403] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.169s	user 0.031s	sys 0.061s
I20260709 12:26:17.725363 12348 tablet_bootstrap.cc:492] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:17.727134 12346 raft_consensus.cc:359] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.728380 12346 raft_consensus.cc:385] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.728758 12346 raft_consensus.cc:740] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.733476 12348 tablet_bootstrap.cc:654] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.737378 12346 consensus_queue.cc:260] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.752274 12346 ts_tablet_manager.cc:1434] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.013s
I20260709 12:26:17.753243 12182 heartbeater.cc:499] Master 127.25.202.62:41237 was elected leader, sending a full tablet report...
I20260709 12:26:17.753435 12346 tablet_bootstrap.cc:492] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:17.760408 12346 tablet_bootstrap.cc:654] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.766536 12349 tablet_bootstrap.cc:492] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:17.781157 12348 tablet_bootstrap.cc:492] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:17.781816 12348 ts_tablet_manager.cc:1403] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.057s	user 0.006s	sys 0.008s
I20260709 12:26:17.781865 12349 tablet_bootstrap.cc:654] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.783680 12346 tablet_bootstrap.cc:492] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:17.784240 12346 ts_tablet_manager.cc:1403] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.031s	user 0.019s	sys 0.005s
I20260709 12:26:17.787431 12348 raft_consensus.cc:359] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.788102 12346 raft_consensus.cc:359] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.789130 12348 raft_consensus.cc:385] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.789342 12346 raft_consensus.cc:385] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.789543 12348 raft_consensus.cc:740] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.789595 12346 raft_consensus.cc:740] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.790493 12348 consensus_queue.cc:260] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.790618 12346 consensus_queue.cc:260] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.792876 12346 ts_tablet_manager.cc:1434] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:26:17.793949 12346 tablet_bootstrap.cc:492] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:17.798063 12349 tablet_bootstrap.cc:492] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:17.798804 12349 ts_tablet_manager.cc:1403] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.000s
I20260709 12:26:17.801688 12346 tablet_bootstrap.cc:654] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.803182 12349 raft_consensus.cc:359] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.807139 12349 raft_consensus.cc:385] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.807572 12349 raft_consensus.cc:740] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.808936 12257 heartbeater.cc:499] Master 127.25.202.62:41237 was elected leader, sending a full tablet report...
I20260709 12:26:17.808583 12349 consensus_queue.cc:260] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.812252 12348 ts_tablet_manager.cc:1434] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.030s	user 0.016s	sys 0.007s
I20260709 12:26:17.813886 12348 tablet_bootstrap.cc:492] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:17.818336 12349 ts_tablet_manager.cc:1434] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.019s	user 0.014s	sys 0.004s
I20260709 12:26:17.819186 12331 heartbeater.cc:499] Master 127.25.202.62:41237 was elected leader, sending a full tablet report...
I20260709 12:26:17.821892 12346 tablet_bootstrap.cc:492] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:17.822513 12346 ts_tablet_manager.cc:1403] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.014s
I20260709 12:26:17.822705 12348 tablet_bootstrap.cc:654] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.826052 12346 raft_consensus.cc:359] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.827169 12346 raft_consensus.cc:385] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.827740 12346 raft_consensus.cc:740] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.829051 12346 consensus_queue.cc:260] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.830097 12349 tablet_bootstrap.cc:492] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:17.836086 12346 ts_tablet_manager.cc:1434] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.003s
I20260709 12:26:17.837718 12346 tablet_bootstrap.cc:492] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:17.838104 12349 tablet_bootstrap.cc:654] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.845091 12348 tablet_bootstrap.cc:492] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:17.845651 12348 ts_tablet_manager.cc:1403] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.032s	user 0.018s	sys 0.010s
I20260709 12:26:17.846593 12346 tablet_bootstrap.cc:654] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:17.849210 12183 tablet.cc:2406] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:26:17.849100 12348 raft_consensus.cc:359] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.849823 12349 tablet_bootstrap.cc:492] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:17.850054 12348 raft_consensus.cc:385] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.850407 12348 raft_consensus.cc:740] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.850417 12349 ts_tablet_manager.cc:1403] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.004s
I20260709 12:26:17.851410 12348 consensus_queue.cc:260] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.853497 12350 raft_consensus.cc:493] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.853812 12349 raft_consensus.cc:359] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.854592 12349 raft_consensus.cc:385] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.854202 12350 raft_consensus.cc:515] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.854938 12349 raft_consensus.cc:740] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.855955 12349 consensus_queue.cc:260] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.859094 12349 ts_tablet_manager.cc:1434] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:26:17.860170 12349 tablet_bootstrap.cc:492] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:17.861161 12350 leader_election.cc:290] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:17.867547 12349 tablet_bootstrap.cc:654] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.870586 12348 ts_tablet_manager.cc:1434] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.003s
I20260709 12:26:17.872118 12348 tablet_bootstrap.cc:492] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:17.901036 12349 tablet_bootstrap.cc:492] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:17.901731 12349 ts_tablet_manager.cc:1403] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.042s	user 0.005s	sys 0.006s
I20260709 12:26:17.901564 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61488d77945649d59f085e9f2035be1a" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:17.902518 12348 tablet_bootstrap.cc:654] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.902443 12232 raft_consensus.cc:2468] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:17.903961 12346 tablet_bootstrap.cc:492] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:17.904181 12122 leader_election.cc:304] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:17.904539 12346 ts_tablet_manager.cc:1403] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.067s	user 0.024s	sys 0.037s
I20260709 12:26:17.905158 12350 raft_consensus.cc:2804] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:17.905007 12349 raft_consensus.cc:359] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.905625 12350 raft_consensus.cc:493] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.905824 12349 raft_consensus.cc:385] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.906070 12350 raft_consensus.cc:3060] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:17.906226 12349 raft_consensus.cc:740] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.907620 12346 raft_consensus.cc:359] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.908399 12346 raft_consensus.cc:385] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.908843 12346 raft_consensus.cc:740] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.909637 12346 consensus_queue.cc:260] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.914188 12349 consensus_queue.cc:260] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.918329 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61488d77945649d59f085e9f2035be1a" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:17.919267 12306 raft_consensus.cc:2468] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:17.918686 12350 raft_consensus.cc:515] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.923588 12349 ts_tablet_manager.cc:1434] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.004s
I20260709 12:26:17.924917 12349 tablet_bootstrap.cc:492] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:17.926390 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61488d77945649d59f085e9f2035be1a" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
W20260709 12:26:17.926928 12103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:17.927013 12232 raft_consensus.cc:3060] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:17.938668 12232 raft_consensus.cc:2468] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:17.941567 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61488d77945649d59f085e9f2035be1a" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:17.945202 12306 raft_consensus.cc:3060] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:17.945483 12346 ts_tablet_manager.cc:1434] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.041s	user 0.015s	sys 0.024s
I20260709 12:26:17.946559 12346 tablet_bootstrap.cc:492] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:17.949095 12348 tablet_bootstrap.cc:492] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:17.949736 12348 ts_tablet_manager.cc:1403] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.078s	user 0.010s	sys 0.005s
I20260709 12:26:17.952919 12348 raft_consensus.cc:359] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.955183 12348 raft_consensus.cc:385] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.955610 12348 raft_consensus.cc:740] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.956462 12348 consensus_queue.cc:260] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.942852 12349 tablet_bootstrap.cc:654] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.953605 12306 raft_consensus.cc:2468] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:17.954602 12122 leader_election.cc:304] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:17.942801 12350 leader_election.cc:290] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:17.959101 12350 raft_consensus.cc:2804] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:17.959640 12350 raft_consensus.cc:697] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:17.960878 12346 tablet_bootstrap.cc:654] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.960494 12350 consensus_queue.cc:237] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.965961 12348 ts_tablet_manager.cc:1434] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:26:17.970216 12349 tablet_bootstrap.cc:492] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:17.970816 12349 ts_tablet_manager.cc:1403] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.046s	user 0.006s	sys 0.007s
I20260709 12:26:17.975203 12349 raft_consensus.cc:359] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.975893 12349 raft_consensus.cc:385] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.976217 12349 raft_consensus.cc:740] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.977031 12349 consensus_queue.cc:260] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.978243 12346 tablet_bootstrap.cc:492] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:17.977209 12054 catalog_manager.cc:5697] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:17.978950 12346 ts_tablet_manager.cc:1403] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.033s	user 0.014s	sys 0.001s
I20260709 12:26:17.979480 12349 ts_tablet_manager.cc:1434] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:26:17.980607 12349 tablet_bootstrap.cc:492] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:17.982352 12346 raft_consensus.cc:359] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.983155 12346 raft_consensus.cc:385] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:17.983560 12346 raft_consensus.cc:740] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:17.984576 12346 consensus_queue.cc:260] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:17.987280 12349 tablet_bootstrap.cc:654] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.987634 12346 ts_tablet_manager.cc:1434] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:26:17.988862 12346 tablet_bootstrap.cc:492] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:17.996948 12346 tablet_bootstrap.cc:654] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:17.997916 12349 tablet_bootstrap.cc:492] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:17.998586 12349 ts_tablet_manager.cc:1403] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.007s
I20260709 12:26:18.001323 12348 tablet_bootstrap.cc:492] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:18.002094 12354 raft_consensus.cc:493] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.002625 12354 raft_consensus.cc:515] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.009500 12348 tablet_bootstrap.cc:654] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.010334 12346 tablet_bootstrap.cc:492] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:18.013839 12346 ts_tablet_manager.cc:1403] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.011s
I20260709 12:26:18.017325 12346 raft_consensus.cc:359] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.018159 12346 raft_consensus.cc:385] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.018481 12346 raft_consensus.cc:740] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.019230 12346 consensus_queue.cc:260] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.021612 12346 ts_tablet_manager.cc:1434] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:26:18.021889 12354 leader_election.cc:290] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.022583 12346 tablet_bootstrap.cc:492] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:18.029497 12346 tablet_bootstrap.cc:654] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.036705 12346 tablet_bootstrap.cc:492] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:18.037225 12346 ts_tablet_manager.cc:1403] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.003s
I20260709 12:26:18.001547 12349 raft_consensus.cc:359] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.040365 12346 raft_consensus.cc:359] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.041216 12346 raft_consensus.cc:385] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.041524 12346 raft_consensus.cc:740] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.042388 12346 consensus_queue.cc:260] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.040356 12349 raft_consensus.cc:385] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.044019 12349 raft_consensus.cc:740] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.045542 12346 ts_tablet_manager.cc:1434] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:26:18.045220 12349 consensus_queue.cc:260] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.046823 12346 tablet_bootstrap.cc:492] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:18.054669 12346 tablet_bootstrap.cc:654] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.060387 12346 tablet_bootstrap.cc:492] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:18.060953 12346 ts_tablet_manager.cc:1403] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.007s
I20260709 12:26:18.063941 12346 raft_consensus.cc:359] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.064797 12346 raft_consensus.cc:385] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.065091 12346 raft_consensus.cc:740] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.065841 12346 consensus_queue.cc:260] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.068401 12346 ts_tablet_manager.cc:1434] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:18.069516 12346 tablet_bootstrap.cc:492] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:18.071857 12350 raft_consensus.cc:493] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.072455 12350 raft_consensus.cc:515] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.073596 12349 ts_tablet_manager.cc:1434] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.075s	user 0.009s	sys 0.000s
I20260709 12:26:18.074847 12349 tablet_bootstrap.cc:492] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:18.077937 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9689cd771ee44b2808c2d7c52d83766" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:18.078890 12232 raft_consensus.cc:2468] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:18.080746 12122 leader_election.cc:304] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.074741 12350 leader_election.cc:290] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.081686 12363 raft_consensus.cc:2804] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.083391 12363 raft_consensus.cc:493] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.083734 12363 raft_consensus.cc:3060] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.085911 12346 tablet_bootstrap.cc:654] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.081089 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9689cd771ee44b2808c2d7c52d83766" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
W20260709 12:26:18.088465 12123 leader_election.cc:343] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.099129 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecef6444dfd41e1bbb5b1534a4626be" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:18.100675 12306 raft_consensus.cc:2468] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:18.101994 12348 tablet_bootstrap.cc:492] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:18.102106 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecef6444dfd41e1bbb5b1534a4626be" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.102607 12348 ts_tablet_manager.cc:1403] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.136s	user 0.011s	sys 0.042s
I20260709 12:26:18.102900 12157 raft_consensus.cc:2468] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:18.104207 12198 leader_election.cc:304] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:18.105363 12354 raft_consensus.cc:2804] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.105744 12354 raft_consensus.cc:493] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.106209 12354 raft_consensus.cc:3060] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.107088 12349 tablet_bootstrap.cc:654] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.107136 12363 raft_consensus.cc:515] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.108338 12346 tablet_bootstrap.cc:492] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:18.108851 12346 ts_tablet_manager.cc:1403] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.040s	user 0.010s	sys 0.000s
I20260709 12:26:18.110719 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9689cd771ee44b2808c2d7c52d83766" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:18.111596 12232 raft_consensus.cc:3060] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.105789 12348 raft_consensus.cc:359] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.115732 12348 raft_consensus.cc:385] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.117189 12346 raft_consensus.cc:359] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.119052 12348 raft_consensus.cc:740] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.119212 12346 raft_consensus.cc:385] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.119721 12346 raft_consensus.cc:740] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.120759 12350 raft_consensus.cc:493] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.120430 12348 consensus_queue.cc:260] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.121290 12350 raft_consensus.cc:515] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.121902 12363 leader_election.cc:290] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.118019 12354 raft_consensus.cc:515] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.122200 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9689cd771ee44b2808c2d7c52d83766" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.123575 12350 leader_election.cc:290] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
W20260709 12:26:18.124155 12123 leader_election.cc:343] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.120755 12346 consensus_queue.cc:260] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.125975 12231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d499c7e4809645de92dfc91281414bdd" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:18.126446 12232 raft_consensus.cc:2468] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:18.127185 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d499c7e4809645de92dfc91281414bdd" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
W20260709 12:26:18.128140 12122 leader_election.cc:343] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.129036 12122 leader_election.cc:304] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:18.130146 12346 ts_tablet_manager.cc:1434] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:26:18.130476 12363 raft_consensus.cc:2804] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.130985 12363 raft_consensus.cc:697] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:18.131871 12363 consensus_queue.cc:237] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.132556 12356 raft_consensus.cc:493] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.130483 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecef6444dfd41e1bbb5b1534a4626be" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.133122 12356 raft_consensus.cc:515] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.133865 12157 raft_consensus.cc:3060] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.133109 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecef6444dfd41e1bbb5b1534a4626be" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.139717 12356 leader_election.cc:290] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:18.140116 12305 raft_consensus.cc:3060] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.143011 12157 raft_consensus.cc:2468] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:18.128095 12306 raft_consensus.cc:2468] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:18.144496 12195 leader_election.cc:304] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.132567 12349 tablet_bootstrap.cc:492] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:18.145896 12349 ts_tablet_manager.cc:1403] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.071s	user 0.015s	sys 0.010s
I20260709 12:26:18.146301 12123 leader_election.cc:304] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.147418 12348 ts_tablet_manager.cc:1434] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.003s
I20260709 12:26:18.147696 12350 raft_consensus.cc:2804] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.148656 12350 raft_consensus.cc:493] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.149006 12350 raft_consensus.cc:3060] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.130496 12354 leader_election.cc:290] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.150709 12354 raft_consensus.cc:2804] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.151135 12354 raft_consensus.cc:697] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:18.152108 12305 raft_consensus.cc:2468] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:18.151996 12354 consensus_queue.cc:237] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.160257 12348 tablet_bootstrap.cc:492] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:18.160611 12349 raft_consensus.cc:359] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.161396 12349 raft_consensus.cc:385] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.161672 12349 raft_consensus.cc:740] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.166954 12348 tablet_bootstrap.cc:654] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.168232 12350 raft_consensus.cc:515] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.171221 12350 leader_election.cc:290] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.172142 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d499c7e4809645de92dfc91281414bdd" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:18.173496 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d499c7e4809645de92dfc91281414bdd" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.174181 12305 raft_consensus.cc:3060] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:18.176863 12122 leader_election.cc:343] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.180743 12305 raft_consensus.cc:2468] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:18.181821 12123 leader_election.cc:304] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.182567 12363 raft_consensus.cc:2804] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.183068 12363 raft_consensus.cc:697] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:18.183836 12363 consensus_queue.cc:237] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.189512 12349 consensus_queue.cc:260] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.196692 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "503e4fb15a654c938a923e5c6808c209" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.196112 12052 catalog_manager.cc:5697] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.197413 12157 raft_consensus.cc:2468] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:18.198869 12269 leader_election.cc:304] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.199807 12356 raft_consensus.cc:2804] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.200284 12356 raft_consensus.cc:493] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.200711 12356 raft_consensus.cc:3060] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.209517 12349 ts_tablet_manager.cc:1434] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.003s
I20260709 12:26:18.211009 12349 tablet_bootstrap.cc:492] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:18.227638 12363 raft_consensus.cc:493] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.236027 12363 raft_consensus.cc:515] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.240192 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:18.240442 12363 leader_election.cc:290] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
W20260709 12:26:18.242280 12122 leader_election.cc:343] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.242429 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:18.244577 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "503e4fb15a654c938a923e5c6808c209" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
W20260709 12:26:18.245114 12123 leader_election.cc:343] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.245383 12232 raft_consensus.cc:2468] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:18.245657 12123 leader_election.cc:304] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.246912 12363 raft_consensus.cc:2749] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:18.250478 12356 raft_consensus.cc:515] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.252840 12356 leader_election.cc:290] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:18.253219 12349 tablet_bootstrap.cc:654] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.253654 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "503e4fb15a654c938a923e5c6808c209" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.253997 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "503e4fb15a654c938a923e5c6808c209" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:18.254351 12157 raft_consensus.cc:3060] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.254774 12232 raft_consensus.cc:3060] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.257671 12348 tablet_bootstrap.cc:492] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:18.258352 12348 ts_tablet_manager.cc:1403] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.098s	user 0.036s	sys 0.029s
I20260709 12:26:18.259258 12054 catalog_manager.cc:5697] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.261286 12356 raft_consensus.cc:493] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.261837 12356 raft_consensus.cc:515] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.263164 12157 raft_consensus.cc:2468] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:18.264232 12356 leader_election.cc:290] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:18.265175 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.265509 12348 raft_consensus.cc:359] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.265787 12231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:18.265938 12156 raft_consensus.cc:2468] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:18.267721 12269 leader_election.cc:304] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
W20260709 12:26:18.268432 12271 leader_election.cc:343] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.268699 12269 leader_election.cc:304] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.270311 12356 raft_consensus.cc:2804] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.270781 12356 raft_consensus.cc:493] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.271068 12390 raft_consensus.cc:2804] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.271377 12356 raft_consensus.cc:3060] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.273334 12232 raft_consensus.cc:2468] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:18.275556 12052 catalog_manager.cc:5697] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 1 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.266258 12348 raft_consensus.cc:385] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.278765 12348 raft_consensus.cc:740] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.279656 12348 consensus_queue.cc:260] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.280100 12356 raft_consensus.cc:515] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.280972 12390 raft_consensus.cc:697] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:18.282111 12348 ts_tablet_manager.cc:1434] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:26:18.282261 12390 consensus_queue.cc:237] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.283515 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.284242 12157 raft_consensus.cc:3060] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.283260 12348 tablet_bootstrap.cc:492] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:18.297537 12348 tablet_bootstrap.cc:654] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.300112 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
W20260709 12:26:18.301482 12271 leader_election.cc:343] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:18.302209 12356 leader_election.cc:290] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:18.303817 12054 catalog_manager.cc:5697] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:18.306241 12349 tablet_bootstrap.cc:492] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:18.306811 12349 ts_tablet_manager.cc:1403] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.096s	user 0.019s	sys 0.031s
I20260709 12:26:18.306854 12157 raft_consensus.cc:2468] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:18.308112 12269 leader_election.cc:304] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.309307 12356 raft_consensus.cc:2804] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.309762 12356 raft_consensus.cc:697] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:18.310248 12348 tablet_bootstrap.cc:492] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:18.309968 12349 raft_consensus.cc:359] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.310743 12349 raft_consensus.cc:385] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.310753 12348 ts_tablet_manager.cc:1403] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.028s	user 0.007s	sys 0.007s
I20260709 12:26:18.310596 12356 consensus_queue.cc:237] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.311223 12349 raft_consensus.cc:740] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.312122 12349 consensus_queue.cc:260] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.314034 12348 raft_consensus.cc:359] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.314548 12349 ts_tablet_manager.cc:1434] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:26:18.314810 12348 raft_consensus.cc:385] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.315191 12348 raft_consensus.cc:740] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.316044 12349 tablet_bootstrap.cc:492] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:18.316056 12348 consensus_queue.cc:260] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.318446 12348 ts_tablet_manager.cc:1434] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:26:18.319268 12348 tablet_bootstrap.cc:492] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:18.324298 12348 tablet_bootstrap.cc:654] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.331542 12349 tablet_bootstrap.cc:654] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.331633 12348 tablet_bootstrap.cc:492] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:18.332571 12348 ts_tablet_manager.cc:1403] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.013s	user 0.012s	sys 0.000s
I20260709 12:26:18.331620 12052 catalog_manager.cc:5697] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:18.335129 12348 raft_consensus.cc:359] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.335887 12348 raft_consensus.cc:385] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.336175 12348 raft_consensus.cc:740] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.336995 12348 consensus_queue.cc:260] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.339635 12348 ts_tablet_manager.cc:1434] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:26:18.340763 12348 tablet_bootstrap.cc:492] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:18.341140 12349 tablet_bootstrap.cc:492] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:18.341603 12349 ts_tablet_manager.cc:1403] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.026s	user 0.004s	sys 0.009s
I20260709 12:26:18.344192 12349 raft_consensus.cc:359] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.344777 12349 raft_consensus.cc:385] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.345012 12349 raft_consensus.cc:740] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.345538 12349 consensus_queue.cc:260] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.346695 12390 raft_consensus.cc:493] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.347280 12348 tablet_bootstrap.cc:654] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.347155 12390 raft_consensus.cc:515] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.348003 12349 ts_tablet_manager.cc:1434] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:26:18.349045 12349 tablet_bootstrap.cc:492] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:18.349483 12390 leader_election.cc:290] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:18.350428 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.351094 12157 raft_consensus.cc:2468] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:18.351428 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:18.352339 12232 raft_consensus.cc:2468] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:18.352892 12269 leader_election.cc:304] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.353940 12390 raft_consensus.cc:2804] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.354357 12390 raft_consensus.cc:493] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.354717 12390 raft_consensus.cc:3060] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.358559 12349 tablet_bootstrap.cc:654] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.358675 12348 tablet_bootstrap.cc:492] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:18.359278 12348 ts_tablet_manager.cc:1403] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.003s
I20260709 12:26:18.361683 12348 raft_consensus.cc:359] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.362452 12348 raft_consensus.cc:385] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.362703 12348 raft_consensus.cc:740] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.362844 12390 raft_consensus.cc:515] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.363648 12349 tablet_bootstrap.cc:492] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:18.363452 12348 consensus_queue.cc:260] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.364343 12349 ts_tablet_manager.cc:1403] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.000s
I20260709 12:26:18.365926 12348 ts_tablet_manager.cc:1434] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.002s
I20260709 12:26:18.366165 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.366765 12348 tablet_bootstrap.cc:492] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:18.366758 12157 raft_consensus.cc:3060] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.367283 12349 raft_consensus.cc:359] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.368037 12349 raft_consensus.cc:385] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.368126 12390 leader_election.cc:290] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:18.368512 12349 raft_consensus.cc:740] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.368291 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:18.369203 12232 raft_consensus.cc:3060] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.369398 12349 consensus_queue.cc:260] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.371860 12349 ts_tablet_manager.cc:1434] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:26:18.431102 12157 raft_consensus.cc:2468] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:18.434346 12269 leader_election.cc:304] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.436259 12348 tablet_bootstrap.cc:654] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:18.442624 12363 raft_consensus.cc:493] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.444849 12390 raft_consensus.cc:2804] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.445822 12390 raft_consensus.cc:697] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:18.443622 12363 raft_consensus.cc:515] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.451757 12390 consensus_queue.cc:237] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.452343 12363 leader_election.cc:290] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.454197 12231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:18.456547 12231 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:18.447011 12232 raft_consensus.cc:2468] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:18.457636 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:18.458738 12305 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:18.461269 12122 leader_election.cc:304] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.462203 12363 raft_consensus.cc:2804] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.462584 12363 raft_consensus.cc:493] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.462895 12363 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.464208 12354 raft_consensus.cc:493] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.464984 12354 raft_consensus.cc:515] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.466881 12348 tablet_bootstrap.cc:492] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:18.468014 12348 ts_tablet_manager.cc:1403] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.018s
I20260709 12:26:18.471516 12354 leader_election.cc:290] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.472359 12348 raft_consensus.cc:359] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.473191 12348 raft_consensus.cc:385] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:18.473449 12348 raft_consensus.cc:740] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:18.473552 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:18.474174 12348 consensus_queue.cc:260] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.474388 12305 raft_consensus.cc:2468] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:18.476099 12198 leader_election.cc:304] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:18.477583 12385 raft_consensus.cc:2804] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:18.477794 12348 ts_tablet_manager.cc:1434] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:26:18.478240 12385 raft_consensus.cc:493] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.478777 12385 raft_consensus.cc:3060] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.476718 12363 raft_consensus.cc:515] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.480962 12363 leader_election.cc:290] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.482661 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:18.482846 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.483462 12232 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.487798 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.488816 12157 raft_consensus.cc:2468] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:18.483515 12305 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.494920 12385 raft_consensus.cc:515] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.497181 12232 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:18.499756 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.500473 12122 leader_election.cc:304] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.500504 12157 raft_consensus.cc:3060] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.500926 12054 catalog_manager.cc:5697] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:18.502092 12305 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:18.502429 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.503281 12306 raft_consensus.cc:3060] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.503568 12363 raft_consensus.cc:2804] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.504283 12363 raft_consensus.cc:697] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:18.505519 12363 consensus_queue.cc:237] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.517190 12385 leader_election.cc:290] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.516631 12306 raft_consensus.cc:2468] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:18.520161 12198 leader_election.cc:304] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:18.520744 12305 raft_consensus.cc:3060] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.521152 12385 raft_consensus.cc:2804] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:18.520334 12157 raft_consensus.cc:2468] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:18.523124 12385 raft_consensus.cc:697] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:18.524654 12385 consensus_queue.cc:237] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.529107 12350 consensus_queue.cc:1048] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:18.536160 12054 catalog_manager.cc:5697] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.540164 12350 consensus_queue.cc:1048] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.540071 12052 catalog_manager.cc:5697] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 1 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.565116 12407 consensus_queue.cc:1048] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:18.568645 12407 consensus_queue.cc:1048] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.578097 12385 raft_consensus.cc:493] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.578598 12385 raft_consensus.cc:515] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.581660 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d499c7e4809645de92dfc91281414bdd" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.591137 12385 leader_election.cc:290] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.593039 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d499c7e4809645de92dfc91281414bdd" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:18.594226 12305 raft_consensus.cc:2393] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 588391676e574efa9b7da64a5ca954aa in current term 1: Already voted for candidate 62022a2fa120494cbe56659b74b966f8 in this term.
I20260709 12:26:18.595389 12198 leader_election.cc:304] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa; no voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:18.596300 12385 raft_consensus.cc:3060] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.604067 12385 raft_consensus.cc:2749] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:18.608371 12407 consensus_queue.cc:1048] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.622548 12363 consensus_queue.cc:1048] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.654206 12232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
mode: GRACEFUL
new_leader_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:57238
I20260709 12:26:18.654922 12232 raft_consensus.cc:606] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Received request to transfer leadership to TS 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:18.677817 12157 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
mode: GRACEFUL
new_leader_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:37902
I20260709 12:26:18.678390 12157 raft_consensus.cc:606] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Received request to transfer leadership to TS 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.693459 12305 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
mode: GRACEFUL
new_leader_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:43432
I20260709 12:26:18.694017 12305 raft_consensus.cc:606] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Received request to transfer leadership to TS 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.699935 12305 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
mode: GRACEFUL
new_leader_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:43432
I20260709 12:26:18.700636 12305 raft_consensus.cc:606] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Received request to transfer leadership to TS 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:18.726636 12390 consensus_queue.cc:1048] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.731472 12385 consensus_queue.cc:1048] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.746351 12390 consensus_queue.cc:1048] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.753271 12385 consensus_queue.cc:1048] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:18.805070 12419 raft_consensus.cc:493] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.805925 12419 raft_consensus.cc:515] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.807921 12419 leader_election.cc:290] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.808966 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:18.809032 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:18.810143 12157 raft_consensus.cc:2393] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 588391676e574efa9b7da64a5ca954aa in current term 1: Already voted for candidate 3a09a0f9c16e402b8237f06400fc1b36 in this term.
I20260709 12:26:18.811153 12195 leader_election.cc:304] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa; no voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:18.811861 12419 raft_consensus.cc:3060] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:18.818136 12419 raft_consensus.cc:2749] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:18.879603 12390 consensus_queue.cc:1048] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.880365 12418 consensus_queue.cc:1048] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.888829 12391 raft_consensus.cc:993] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: : Instructing follower 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:18.889379 12418 raft_consensus.cc:1081] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Signalling peer 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:18.892599 12230 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:26:18.893496 12230 raft_consensus.cc:493] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:18.893908 12230 raft_consensus.cc:3060] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:18.902594 12230 raft_consensus.cc:515] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.906018 12157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.906606 12157 raft_consensus.cc:3060] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:18.906869 12363 consensus_queue.cc:1048] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:18.907728 12230 leader_election.cc:290] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.910259 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "781cab6f3dbf45a58fa529278460248f" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.910832 12306 raft_consensus.cc:3055] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:18.911156 12306 raft_consensus.cc:740] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:18.912307 12306 consensus_queue.cc:260] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.913728 12306 raft_consensus.cc:3060] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:18.914872 12157 raft_consensus.cc:2468] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:18.916116 12195 leader_election.cc:304] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:18.916999 12419 raft_consensus.cc:2804] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:18.917501 12419 raft_consensus.cc:697] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:18.918555 12419 consensus_queue.cc:237] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.922569 12407 consensus_queue.cc:1048] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.927662 12054 catalog_manager.cc:5697] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 1 to 2, leader changed from 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3) to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 2 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.930109 12306 raft_consensus.cc:2468] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:18.938853 12363 raft_consensus.cc:993] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: : Instructing follower 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:18.939458 12407 raft_consensus.cc:1081] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Signalling peer 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:18.941460 12230 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:57224
I20260709 12:26:18.942142 12230 raft_consensus.cc:493] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:18.942581 12230 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:18.943815 12390 consensus_queue.cc:1048] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:18.951234 12230 raft_consensus.cc:515] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.958184 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:18.958660 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:18.958953 12156 raft_consensus.cc:3055] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:18.959474 12305 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:18.959619 12156 raft_consensus.cc:740] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:18.961086 12230 leader_election.cc:290] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:18.967391 12419 consensus_queue.cc:1048] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:18.969605 12305 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:18.960783 12156 consensus_queue.cc:260] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.971236 12198 leader_election.cc:304] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:18.971750 12390 consensus_queue.cc:1048] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:18.972147 12156 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:18.972491 12419 raft_consensus.cc:2804] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:18.973120 12419 raft_consensus.cc:697] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:18.974259 12419 consensus_queue.cc:237] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:18.992952 12390 raft_consensus.cc:993] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: : Instructing follower 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:18.995609 12356 raft_consensus.cc:1081] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Signalling peer 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:18.995466 12055 catalog_manager.cc:5697] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 1 to 2, leader changed from 62022a2fa120494cbe56659b74b966f8 (127.25.202.1) to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 2 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:19.000425 12156 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:19.006791 12385 raft_consensus.cc:993] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: : Instructing follower 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:19.008965 12385 raft_consensus.cc:1081] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Signalling peer 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:19.007994 12354 consensus_queue.cc:1048] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:26:19.015012 12155 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:37884
I20260709 12:26:19.019963 12155 raft_consensus.cc:493] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:19.020402 12155 raft_consensus.cc:3060] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:19.000495 12232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:26:19.022190 12232 raft_consensus.cc:493] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:19.022578 12232 raft_consensus.cc:3060] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:19.035204 12232 raft_consensus.cc:515] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:19.037286 12232 leader_election.cc:290] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:19.037864 12155 raft_consensus.cc:515] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:19.038130 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:19.038542 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d871c96be864188b0b1da87fd438d00" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:19.038903 12156 raft_consensus.cc:3060] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:19.039202 12306 raft_consensus.cc:3055] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:19.039614 12306 raft_consensus.cc:740] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:19.040802 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:19.040582 12306 consensus_queue.cc:260] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:19.041531 12232 raft_consensus.cc:3055] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:19.041956 12232 raft_consensus.cc:740] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:19.042503 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6db4244c274ed2b604f852a7267a95" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:19.043138 12305 raft_consensus.cc:3060] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:19.043159 12232 consensus_queue.cc:260] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:19.044754 12232 raft_consensus.cc:3060] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:19.046545 12155 leader_election.cc:290] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:19.047389 12306 raft_consensus.cc:3060] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:19.047595 12156 raft_consensus.cc:2468] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:19.048978 12195 leader_election.cc:304] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:19.050771 12354 raft_consensus.cc:2804] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:19.051239 12354 raft_consensus.cc:697] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:19.052246 12354 consensus_queue.cc:237] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:19.052690 12305 raft_consensus.cc:2468] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:19.053025 12232 raft_consensus.cc:2468] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:19.054340 12123 leader_election.cc:304] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:19.055287 12350 raft_consensus.cc:2804] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:19.055871 12350 raft_consensus.cc:697] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:19.056828 12350 consensus_queue.cc:237] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:19.060846 12306 raft_consensus.cc:2468] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:19.061374 12054 catalog_manager.cc:5697] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 1 to 2, leader changed from 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3) to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 2 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:19.064989 12052 catalog_manager.cc:5697] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 1 to 2, leader changed from 588391676e574efa9b7da64a5ca954aa (127.25.202.2) to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:19.425695 12306 raft_consensus.cc:1275] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.427053 12354 consensus_queue.cc:1048] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:19.436142 12156 raft_consensus.cc:1275] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.437914 12354 consensus_queue.cc:1048] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:19.439534 12306 raft_consensus.cc:1275] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.441103 12350 consensus_queue.cc:1048] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:19.449373 12156 raft_consensus.cc:1275] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.450644 12232 raft_consensus.cc:1275] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.451138 12354 consensus_queue.cc:1048] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:19.452543 12407 consensus_queue.cc:1048] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:19.470854 12306 raft_consensus.cc:1275] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.472239 12385 consensus_queue.cc:1048] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:19.517764 12306 raft_consensus.cc:1275] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.519155 12385 consensus_queue.cc:1048] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:19.527165 12155 raft_consensus.cc:1275] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:19.528967 12354 consensus_queue.cc:1048] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:21.740008 12054 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37416:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:26:21.742587 12054 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:26:21.793469 12140 tablet_service.cc:1511] Processing CreateTablet for tablet 6953952a4d8e4a1b91693794a5c5c486 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:21.795599 12140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6953952a4d8e4a1b91693794a5c5c486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.795066 12219 tablet_service.cc:1511] Processing CreateTablet for tablet 6953952a4d8e4a1b91693794a5c5c486 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:21.796325 12294 tablet_service.cc:1511] Processing CreateTablet for tablet 6953952a4d8e4a1b91693794a5c5c486 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:26:21.797101 12219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6953952a4d8e4a1b91693794a5c5c486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.797828 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6953952a4d8e4a1b91693794a5c5c486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.802009 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 56ab8e7a609e421c83735ff7134913a0 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:21.803565 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ab8e7a609e421c83735ff7134913a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.809415 12216 tablet_service.cc:1511] Processing CreateTablet for tablet 56ab8e7a609e421c83735ff7134913a0 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:21.812460 12142 tablet_service.cc:1511] Processing CreateTablet for tablet a9b5d597d28b46bebede3f34afc92b92 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:21.814025 12142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b5d597d28b46bebede3f34afc92b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.815663 12217 tablet_service.cc:1511] Processing CreateTablet for tablet 25f7a207a27d4252b403a7aa91597d3c (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:21.812530 12143 tablet_service.cc:1511] Processing CreateTablet for tablet 56ab8e7a609e421c83735ff7134913a0 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:26:21.818257 12221 tablet_service.cc:1511] Processing CreateTablet for tablet a9b5d597d28b46bebede3f34afc92b92 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:21.818966 12143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ab8e7a609e421c83735ff7134913a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.819288 12291 tablet_service.cc:1511] Processing CreateTablet for tablet a9b5d597d28b46bebede3f34afc92b92 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:26:21.820107 12292 tablet_service.cc:1511] Processing CreateTablet for tablet 25f7a207a27d4252b403a7aa91597d3c (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:21.820786 12291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b5d597d28b46bebede3f34afc92b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.822042 12141 tablet_service.cc:1511] Processing CreateTablet for tablet 25f7a207a27d4252b403a7aa91597d3c (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:26:21.822691 12292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f7a207a27d4252b403a7aa91597d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.823271 12222 tablet_service.cc:1511] Processing CreateTablet for tablet 9f7ca2294c8845eabca8500e7ba7a22a (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:21.824656 12144 tablet_service.cc:1511] Processing CreateTablet for tablet 9f7ca2294c8845eabca8500e7ba7a22a (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:21.824718 12289 tablet_service.cc:1511] Processing CreateTablet for tablet 984de35a63d14587ae1333d131c23433 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:21.825501 12296 tablet_service.cc:1511] Processing CreateTablet for tablet 9f7ca2294c8845eabca8500e7ba7a22a (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:26:21.826118 12289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 984de35a63d14587ae1333d131c23433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.826923 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7ca2294c8845eabca8500e7ba7a22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.817734 12216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ab8e7a609e421c83735ff7134913a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.827450 12141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f7a207a27d4252b403a7aa91597d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.827457 12139 tablet_service.cc:1511] Processing CreateTablet for tablet 984de35a63d14587ae1333d131c23433 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:21.829094 12214 tablet_service.cc:1511] Processing CreateTablet for tablet 984de35a63d14587ae1333d131c23433 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:26:21.829290 12146 tablet_service.cc:1511] Processing CreateTablet for tablet e2f95c3487484c0fb9e5b270a4623904 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:21.830183 12139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 984de35a63d14587ae1333d131c23433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.830498 12214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 984de35a63d14587ae1333d131c23433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.831250 12146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2f95c3487484c0fb9e5b270a4623904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.832890 12217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f7a207a27d4252b403a7aa91597d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.835691 12222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7ca2294c8845eabca8500e7ba7a22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.836297 12144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7ca2294c8845eabca8500e7ba7a22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.837832 12221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b5d597d28b46bebede3f34afc92b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.842372 12215 tablet_service.cc:1511] Processing CreateTablet for tablet e2f95c3487484c0fb9e5b270a4623904 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:21.844117 12215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2f95c3487484c0fb9e5b270a4623904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.886338 12290 tablet_service.cc:1511] Processing CreateTablet for tablet e2f95c3487484c0fb9e5b270a4623904 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:26:21.888042 12290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2f95c3487484c0fb9e5b270a4623904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.929342 12142 tablet_service.cc:1511] Processing CreateTablet for tablet f59313bf8b2a4592b508d3fe3f261445 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:21.937428 12142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59313bf8b2a4592b508d3fe3f261445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.937618 12445 tablet_bootstrap.cc:492] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:21.934867 12219 tablet_service.cc:1511] Processing CreateTablet for tablet f59313bf8b2a4592b508d3fe3f261445 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:21.944799 12219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59313bf8b2a4592b508d3fe3f261445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.960093 12140 tablet_service.cc:1511] Processing CreateTablet for tablet 2802579dd4404e269ebfc771503f56e9 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:21.961313 12217 tablet_service.cc:1511] Processing CreateTablet for tablet 2802579dd4404e269ebfc771503f56e9 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:21.962076 12140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2802579dd4404e269ebfc771503f56e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.962586 12217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2802579dd4404e269ebfc771503f56e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.966135 12296 tablet_service.cc:1511] Processing CreateTablet for tablet 2802579dd4404e269ebfc771503f56e9 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:26:21.967576 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2802579dd4404e269ebfc771503f56e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:21.986295 12446 tablet_bootstrap.cc:492] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
W20260709 12:26:21.987262 12103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:21.990176 12447 tablet_bootstrap.cc:492] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:21.997148 12447 tablet_bootstrap.cc:654] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.005558 12446 tablet_bootstrap.cc:654] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.012993 12445 tablet_bootstrap.cc:654] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:21.987413 12295 tablet_service.cc:1511] Processing CreateTablet for tablet f59313bf8b2a4592b508d3fe3f261445 (DEFAULT_TABLE table=soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:26:22.016525 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59313bf8b2a4592b508d3fe3f261445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:22.023432 12446 tablet_bootstrap.cc:492] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.024052 12446 ts_tablet_manager.cc:1403] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.038s	user 0.014s	sys 0.000s
I20260709 12:26:22.026980 12447 tablet_bootstrap.cc:492] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.027774 12447 ts_tablet_manager.cc:1403] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.038s	user 0.006s	sys 0.016s
I20260709 12:26:22.027694 12446 raft_consensus.cc:359] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.028947 12446 raft_consensus.cc:385] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.029260 12446 raft_consensus.cc:740] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.030107 12446 consensus_queue.cc:260] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.033012 12446 ts_tablet_manager.cc:1434] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260709 12:26:22.033679 12445 tablet_bootstrap.cc:492] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.032998 12447 raft_consensus.cc:359] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.034195 12447 raft_consensus.cc:385] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.034509 12446 tablet_bootstrap.cc:492] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.034670 12447 raft_consensus.cc:740] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.035928 12447 consensus_queue.cc:260] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.038195 12445 ts_tablet_manager.cc:1403] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.101s	user 0.022s	sys 0.000s
I20260709 12:26:22.039091 12447 ts_tablet_manager.cc:1434] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.004s
I20260709 12:26:22.041074 12447 tablet_bootstrap.cc:492] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.041460 12445 raft_consensus.cc:359] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.042268 12445 raft_consensus.cc:385] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.042366 12446 tablet_bootstrap.cc:654] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.042599 12445 raft_consensus.cc:740] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.043493 12445 consensus_queue.cc:260] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.045849 12445 ts_tablet_manager.cc:1434] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:26:22.046904 12445 tablet_bootstrap.cc:492] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.051729 12447 tablet_bootstrap.cc:654] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.055855 12445 tablet_bootstrap.cc:654] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.063687 12446 tablet_bootstrap.cc:492] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.064282 12446 ts_tablet_manager.cc:1403] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.030s	user 0.013s	sys 0.010s
I20260709 12:26:22.066208 12445 tablet_bootstrap.cc:492] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.066807 12445 ts_tablet_manager.cc:1403] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.004s
I20260709 12:26:22.066236 12447 tablet_bootstrap.cc:492] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.067736 12446 raft_consensus.cc:359] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.068548 12447 ts_tablet_manager.cc:1403] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.011s
I20260709 12:26:22.068697 12446 raft_consensus.cc:385] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.069180 12446 raft_consensus.cc:740] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.070111 12445 raft_consensus.cc:359] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.070214 12446 consensus_queue.cc:260] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.071164 12445 raft_consensus.cc:385] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.071621 12445 raft_consensus.cc:740] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.072611 12445 consensus_queue.cc:260] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.072639 12447 raft_consensus.cc:359] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.073237 12446 ts_tablet_manager.cc:1434] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:26:22.073726 12447 raft_consensus.cc:385] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.074209 12447 raft_consensus.cc:740] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.074822 12446 tablet_bootstrap.cc:492] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.075146 12447 consensus_queue.cc:260] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.076659 12445 ts_tablet_manager.cc:1434] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 12:26:22.077697 12445 tablet_bootstrap.cc:492] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.078763 12447 ts_tablet_manager.cc:1434] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:26:22.080148 12447 tablet_bootstrap.cc:492] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.083451 12446 tablet_bootstrap.cc:654] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.084476 12445 tablet_bootstrap.cc:654] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.087996 12447 tablet_bootstrap.cc:654] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.096717 12445 tablet_bootstrap.cc:492] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.097270 12445 ts_tablet_manager.cc:1403] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.020s	user 0.005s	sys 0.011s
I20260709 12:26:22.100834 12447 tablet_bootstrap.cc:492] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.100713 12445 raft_consensus.cc:359] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.101073 12446 tablet_bootstrap.cc:492] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.101531 12445 raft_consensus.cc:385] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.101765 12447 ts_tablet_manager.cc:1403] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.022s	user 0.010s	sys 0.010s
I20260709 12:26:22.101941 12445 raft_consensus.cc:740] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.103262 12445 consensus_queue.cc:260] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.101799 12446 ts_tablet_manager.cc:1403] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.027s	user 0.016s	sys 0.004s
I20260709 12:26:22.105808 12445 ts_tablet_manager.cc:1434] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.005s
I20260709 12:26:22.105697 12447 raft_consensus.cc:359] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.106742 12447 raft_consensus.cc:385] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.107062 12445 tablet_bootstrap.cc:492] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.108142 12446 raft_consensus.cc:359] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.109153 12446 raft_consensus.cc:385] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.109558 12446 raft_consensus.cc:740] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.110503 12446 consensus_queue.cc:260] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.107120 12447 raft_consensus.cc:740] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.113849 12446 ts_tablet_manager.cc:1434] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:26:22.113637 12447 consensus_queue.cc:260] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.114917 12445 tablet_bootstrap.cc:654] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.115269 12446 tablet_bootstrap.cc:492] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.116617 12447 ts_tablet_manager.cc:1434] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:26:22.120095 12447 tablet_bootstrap.cc:492] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.125437 12446 tablet_bootstrap.cc:654] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.127642 12447 tablet_bootstrap.cc:654] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.166428 12446 tablet_bootstrap.cc:492] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.167201 12446 ts_tablet_manager.cc:1403] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.052s	user 0.007s	sys 0.013s
I20260709 12:26:22.171286 12445 tablet_bootstrap.cc:492] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.172132 12445 ts_tablet_manager.cc:1403] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.065s	user 0.028s	sys 0.021s
I20260709 12:26:22.171281 12446 raft_consensus.cc:359] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.173605 12446 raft_consensus.cc:385] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.173920 12446 raft_consensus.cc:740] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.174746 12446 consensus_queue.cc:260] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.177028 12446 ts_tablet_manager.cc:1434] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:26:22.177340 12447 tablet_bootstrap.cc:492] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.177228 12445 raft_consensus.cc:359] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.177963 12447 ts_tablet_manager.cc:1403] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.060s	user 0.008s	sys 0.030s
I20260709 12:26:22.178058 12446 tablet_bootstrap.cc:492] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.178175 12445 raft_consensus.cc:385] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.178697 12445 raft_consensus.cc:740] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.179558 12445 consensus_queue.cc:260] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.182219 12445 ts_tablet_manager.cc:1434] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:26:22.181746 12447 raft_consensus.cc:359] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.182649 12447 raft_consensus.cc:385] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.182941 12447 raft_consensus.cc:740] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.183286 12445 tablet_bootstrap.cc:492] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.183707 12447 consensus_queue.cc:260] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.186381 12447 ts_tablet_manager.cc:1434] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 12:26:22.187568 12447 tablet_bootstrap.cc:492] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.189980 12446 tablet_bootstrap.cc:654] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.190887 12445 tablet_bootstrap.cc:654] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.194555 12447 tablet_bootstrap.cc:654] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.197889 12354 raft_consensus.cc:493] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.198665 12354 raft_consensus.cc:515] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.201807 12354 leader_election.cc:290] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.202818 12391 raft_consensus.cc:493] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.203250 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.204371 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:22.204808 12155 raft_consensus.cc:2468] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:22.205277 12305 raft_consensus.cc:2468] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:22.206434 12195 leader_election.cc:304] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.203521 12391 raft_consensus.cc:515] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.209649 12391 leader_election.cc:290] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.213131 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.213932 12232 raft_consensus.cc:2468] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:22.215571 12271 leader_election.cc:304] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:22.216817 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.217984 12391 raft_consensus.cc:2804] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:26:22.218459 12269 leader_election.cc:343] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.218490 12391 raft_consensus.cc:493] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.219439 12391 raft_consensus.cc:3060] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.219529 12363 raft_consensus.cc:493] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.220041 12363 raft_consensus.cc:515] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.218660 12354 raft_consensus.cc:2804] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.221235 12354 raft_consensus.cc:493] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.221709 12354 raft_consensus.cc:3060] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.222852 12363 leader_election.cc:290] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.224273 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f7a207a27d4252b403a7aa91597d3c" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.224985 12232 raft_consensus.cc:2468] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:22.225732 12446 tablet_bootstrap.cc:492] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.226375 12446 ts_tablet_manager.cc:1403] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.049s	user 0.010s	sys 0.009s
I20260709 12:26:22.226317 12122 leader_election.cc:304] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.228396 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f7a207a27d4252b403a7aa91597d3c" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
W20260709 12:26:22.230012 12123 leader_election.cc:343] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.229854 12446 raft_consensus.cc:359] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.230597 12446 raft_consensus.cc:385] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.230959 12446 raft_consensus.cc:740] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.231604 12363 raft_consensus.cc:2804] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.232218 12363 raft_consensus.cc:493] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.232795 12363 raft_consensus.cc:3060] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.232923 12354 raft_consensus.cc:515] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.237366 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.238014 12155 raft_consensus.cc:3060] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.238446 12354 leader_election.cc:290] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.237807 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:22.239755 12305 raft_consensus.cc:3060] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.245906 12155 raft_consensus.cc:2468] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:22.247293 12195 leader_election.cc:304] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.248172 12461 raft_consensus.cc:493] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.248629 12461 raft_consensus.cc:515] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.248695 12391 raft_consensus.cc:515] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.250738 12461 leader_election.cc:290] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.235524 12446 consensus_queue.cc:260] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.251713 12363 raft_consensus.cc:515] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.252199 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.254412 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2f95c3487484c0fb9e5b270a4623904" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:22.255323 12306 raft_consensus.cc:2468] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:22.256305 12305 raft_consensus.cc:2468] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:22.256738 12198 leader_election.cc:304] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:22.254515 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f7a207a27d4252b403a7aa91597d3c" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:22.257575 12363 leader_election.cc:290] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.257838 12447 tablet_bootstrap.cc:492] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.258445 12447 ts_tablet_manager.cc:1403] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.071s	user 0.011s	sys 0.023s
I20260709 12:26:22.259024 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:22.259836 12230 raft_consensus.cc:3060] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.260149 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f7a207a27d4252b403a7aa91597d3c" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:22.257771 12232 raft_consensus.cc:3060] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:22.261619 12123 leader_election.cc:343] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.261685 12447 raft_consensus.cc:359] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.260128 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2f95c3487484c0fb9e5b270a4623904" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.262519 12447 raft_consensus.cc:385] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.262876 12447 raft_consensus.cc:740] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.251266 12461 raft_consensus.cc:2804] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:26:22.263602 12195 leader_election.cc:343] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.265849 12445 tablet_bootstrap.cc:492] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.266415 12445 ts_tablet_manager.cc:1403] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.083s	user 0.026s	sys 0.018s
W20260709 12:26:22.266345 12269 leader_election.cc:343] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.269869 12232 raft_consensus.cc:2468] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:22.270912 12445 raft_consensus.cc:359] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.271734 12445 raft_consensus.cc:385] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.271067 12354 raft_consensus.cc:2804] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.272075 12445 raft_consensus.cc:740] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.272272 12354 raft_consensus.cc:493] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.272662 12122 leader_election.cc:304] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:22.273103 12354 raft_consensus.cc:3060] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.273674 12363 raft_consensus.cc:2804] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.263645 12447 consensus_queue.cc:260] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.274209 12363 raft_consensus.cc:697] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:22.272661 12461 raft_consensus.cc:697] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:22.273005 12445 consensus_queue.cc:260] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.275067 12363 consensus_queue.cc:237] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.275605 12461 consensus_queue.cc:237] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.277223 12445 ts_tablet_manager.cc:1434] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.003s
I20260709 12:26:22.278321 12445 tablet_bootstrap.cc:492] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.279511 12230 raft_consensus.cc:2468] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:22.281342 12271 leader_election.cc:304] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.282225 12465 raft_consensus.cc:2804] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.285979 12446 ts_tablet_manager.cc:1434] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260709 12:26:22.292148 12445 tablet_bootstrap.cc:654] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.296123 12446 tablet_bootstrap.cc:492] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.306309 12446 tablet_bootstrap.cc:654] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.288179 12391 leader_election.cc:290] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.309288 12354 raft_consensus.cc:515] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.316588 12466 raft_consensus.cc:493] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.317340 12466 raft_consensus.cc:515] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.320288 12447 ts_tablet_manager.cc:1434] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.062s	user 0.014s	sys 0.022s
I20260709 12:26:22.322081 12391 raft_consensus.cc:493] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.322638 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.322841 12391 raft_consensus.cc:515] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.323650 12155 raft_consensus.cc:2468] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:22.325920 12195 leader_election.cc:304] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.326622 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.327345 12155 raft_consensus.cc:2468] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:22.328806 12269 leader_election.cc:304] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.330295 12391 leader_election.cc:290] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.330717 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:22.321974 12447 tablet_bootstrap.cc:492] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.331431 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.331209 12391 raft_consensus.cc:2804] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.332152 12391 raft_consensus.cc:493] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.332563 12391 raft_consensus.cc:3060] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:22.332818 12198 leader_election.cc:343] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:22.332980 12271 leader_election.cc:343] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.333788 12466 leader_election.cc:290] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.334520 12466 raft_consensus.cc:2804] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.334956 12466 raft_consensus.cc:493] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.335427 12466 raft_consensus.cc:3060] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.342144 12391 raft_consensus.cc:515] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.310745 12465 raft_consensus.cc:697] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:22.347205 12465 consensus_queue.cc:237] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.352893 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.353842 12155 raft_consensus.cc:3060] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.354074 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2f95c3487484c0fb9e5b270a4623904" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:22.354775 12305 raft_consensus.cc:3060] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.356384 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
W20260709 12:26:22.359647 12271 leader_election.cc:343] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.359848 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2f95c3487484c0fb9e5b270a4623904" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.352910 12354 leader_election.cc:290] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.362254 12052 catalog_manager.cc:5697] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:22.366379 12470 raft_consensus.cc:493] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.366926 12470 raft_consensus.cc:515] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.369234 12470 leader_election.cc:290] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.370630 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.371469 12156 raft_consensus.cc:2468] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 0.
I20260709 12:26:22.372306 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
W20260709 12:26:22.370582 12195 leader_election.cc:343] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.373811 12269 leader_election.cc:304] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.373853 12305 raft_consensus.cc:2468] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:22.374785 12470 raft_consensus.cc:2804] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.375345 12470 raft_consensus.cc:493] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.375484 12446 tablet_bootstrap.cc:492] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.375859 12470 raft_consensus.cc:3060] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.376145 12446 ts_tablet_manager.cc:1403] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.080s	user 0.033s	sys 0.006s
W20260709 12:26:22.375017 12271 leader_election.cc:343] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.380137 12447 tablet_bootstrap.cc:654] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.380048 12446 raft_consensus.cc:359] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.383524 12446 raft_consensus.cc:385] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.383853 12446 raft_consensus.cc:740] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.345922 12391 leader_election.cc:290] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.384423 12198 leader_election.cc:304] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.384850 12446 consensus_queue.cc:260] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.382944 12155 raft_consensus.cc:2468] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:22.385778 12354 raft_consensus.cc:2804] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.385752 12466 raft_consensus.cc:515] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.386502 12354 raft_consensus.cc:697] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:22.386893 12269 leader_election.cc:304] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:22.387662 12446 ts_tablet_manager.cc:1434] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:26:22.387405 12354 consensus_queue.cc:237] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.388983 12446 tablet_bootstrap.cc:492] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.391110 12391 raft_consensus.cc:2804] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.394454 12391 raft_consensus.cc:697] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:22.395393 12470 raft_consensus.cc:515] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.396821 12391 consensus_queue.cc:237] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.399174 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
W20260709 12:26:22.400759 12271 leader_election.cc:343] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.397639 12446 tablet_bootstrap.cc:654] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.407847 12466 leader_election.cc:290] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.408056 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
W20260709 12:26:22.409610 12198 leader_election.cc:343] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.410600 12054 catalog_manager.cc:5697] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:22.397791 12470 leader_election.cc:290] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.372418 12445 tablet_bootstrap.cc:492] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.414340 12445 ts_tablet_manager.cc:1403] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.136s	user 0.013s	sys 0.040s
I20260709 12:26:22.403307 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.420010 12155 raft_consensus.cc:3060] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.423640 12445 raft_consensus.cc:359] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.425738 12445 raft_consensus.cc:385] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.426246 12445 raft_consensus.cc:740] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.427572 12445 consensus_queue.cc:260] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
W20260709 12:26:22.430897 12269 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.202.1:44853, user_credentials={real_user=slave}} blocked reactor thread for 44099.6us
I20260709 12:26:22.432327 12445 ts_tablet_manager.cc:1434] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.001s
I20260709 12:26:22.437449 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.438120 12156 raft_consensus.cc:3060] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.442706 12155 raft_consensus.cc:2468] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:22.444937 12195 leader_election.cc:304] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:22.446859 12466 raft_consensus.cc:2804] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.447664 12466 raft_consensus.cc:697] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:22.448918 12466 consensus_queue.cc:237] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.462311 12445 tablet_bootstrap.cc:492] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.462677 12156 raft_consensus.cc:2468] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 1.
I20260709 12:26:22.465044 12269 leader_election.cc:304] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:22.466452 12476 raft_consensus.cc:2804] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.467116 12476 raft_consensus.cc:697] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:22.468222 12476 consensus_queue.cc:237] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.469715 12447 tablet_bootstrap.cc:492] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.470319 12447 ts_tablet_manager.cc:1403] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.149s	user 0.012s	sys 0.003s
I20260709 12:26:22.475049 12447 raft_consensus.cc:359] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.476217 12447 raft_consensus.cc:385] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.476668 12447 raft_consensus.cc:740] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.483827 12447 consensus_queue.cc:260] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.491276 12445 tablet_bootstrap.cc:654] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.493469 12447 ts_tablet_manager.cc:1434] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.023s	user 0.011s	sys 0.002s
I20260709 12:26:22.495126 12447 tablet_bootstrap.cc:492] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.500967 12055 catalog_manager.cc:5697] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 1 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:22.502775 12447 tablet_bootstrap.cc:654] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.503690 12446 tablet_bootstrap.cc:492] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.504284 12446 ts_tablet_manager.cc:1403] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.116s	user 0.041s	sys 0.042s
I20260709 12:26:22.507587 12446 raft_consensus.cc:359] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.508505 12446 raft_consensus.cc:385] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.508857 12446 raft_consensus.cc:740] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.509637 12446 consensus_queue.cc:260] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.512383 12446 ts_tablet_manager.cc:1434] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:26:22.514115 12446 tablet_bootstrap.cc:492] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.514588 12447 tablet_bootstrap.cc:492] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.515175 12447 ts_tablet_manager.cc:1403] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.014s
I20260709 12:26:22.518359 12447 raft_consensus.cc:359] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.518923 12476 raft_consensus.cc:493] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.519208 12447 raft_consensus.cc:385] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.519716 12447 raft_consensus.cc:740] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.519505 12476 raft_consensus.cc:515] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.520490 12447 consensus_queue.cc:260] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.521378 12446 tablet_bootstrap.cc:654] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.523208 12447 ts_tablet_manager.cc:1434] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20260709 12:26:22.524147 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f7a207a27d4252b403a7aa91597d3c" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.524356 12447 tablet_bootstrap.cc:492] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.525548 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f7a207a27d4252b403a7aa91597d3c" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.526353 12230 raft_consensus.cc:2393] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a09a0f9c16e402b8237f06400fc1b36 in current term 1: Already voted for candidate 62022a2fa120494cbe56659b74b966f8 in this term.
I20260709 12:26:22.529241 12269 leader_election.cc:304] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36; no voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.528052 12053 catalog_manager.cc:5697] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:22.529891 12363 raft_consensus.cc:493] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.530164 12391 raft_consensus.cc:3060] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.530428 12363 raft_consensus.cc:515] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.518229 12445 tablet_bootstrap.cc:492] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.533097 12445 ts_tablet_manager.cc:1403] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.071s	user 0.017s	sys 0.002s
I20260709 12:26:22.534515 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:22.521824 12476 leader_election.cc:290] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
W20260709 12:26:22.536258 12123 leader_election.cc:343] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.536484 12445 raft_consensus.cc:359] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.537326 12445 raft_consensus.cc:385] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.537631 12445 raft_consensus.cc:740] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.538980 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.539194 12391 raft_consensus.cc:2749] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:22.539743 12230 raft_consensus.cc:2468] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 0.
I20260709 12:26:22.541069 12122 leader_election.cc:304] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:22.532655 12363 leader_election.cc:290] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.538349 12445 consensus_queue.cc:260] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.541967 12467 raft_consensus.cc:2804] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.542415 12467 raft_consensus.cc:493] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.542587 12447 tablet_bootstrap.cc:654] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.542795 12467 raft_consensus.cc:3060] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.546828 12446 tablet_bootstrap.cc:492] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.547377 12446 ts_tablet_manager.cc:1403] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.006s
I20260709 12:26:22.550257 12446 raft_consensus.cc:359] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.551036 12446 raft_consensus.cc:385] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.551425 12446 raft_consensus.cc:740] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.552232 12446 consensus_queue.cc:260] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.554165 12447 tablet_bootstrap.cc:492] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.553484 12052 catalog_manager.cc:5697] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 1 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:22.554479 12446 ts_tablet_manager.cc:1434] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:26:22.554672 12447 ts_tablet_manager.cc:1403] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.004s
I20260709 12:26:22.531303 12053 catalog_manager.cc:5697] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 1 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:22.555251 12052 catalog_manager.cc:5697] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 1 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:22.555727 12446 tablet_bootstrap.cc:492] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: Bootstrap starting.
I20260709 12:26:22.557546 12447 raft_consensus.cc:359] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.558378 12447 raft_consensus.cc:385] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.558650 12447 raft_consensus.cc:740] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.563199 12467 raft_consensus.cc:515] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.564211 12446 tablet_bootstrap.cc:654] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.553435 12445 ts_tablet_manager.cc:1434] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.001s
I20260709 12:26:22.566051 12445 tablet_bootstrap.cc:492] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.559369 12447 consensus_queue.cc:260] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.568651 12447 ts_tablet_manager.cc:1434] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 12:26:22.569583 12447 tablet_bootstrap.cc:492] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: Bootstrap starting.
I20260709 12:26:22.574865 12467 leader_election.cc:290] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.575727 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:22.575974 12447 tablet_bootstrap.cc:654] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.576397 12230 raft_consensus.cc:3060] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.576998 12445 tablet_bootstrap.cc:654] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.583935 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
W20260709 12:26:22.585767 12123 leader_election.cc:343] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:22.587388 12447 tablet_bootstrap.cc:492] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: No bootstrap required, opened a new log
I20260709 12:26:22.587950 12447 ts_tablet_manager.cc:1403] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.004s
I20260709 12:26:22.590960 12447 raft_consensus.cc:359] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.594925 12447 raft_consensus.cc:385] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.595252 12447 raft_consensus.cc:740] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.596149 12447 consensus_queue.cc:260] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.599028 12447 ts_tablet_manager.cc:1434] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260709 12:26:22.594053 12391 raft_consensus.cc:493] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.600099 12391 raft_consensus.cc:515] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.602285 12391 leader_election.cc:290] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.603505 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.604048 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.604732 12156 raft_consensus.cc:2393] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a09a0f9c16e402b8237f06400fc1b36 in current term 1: Already voted for candidate 588391676e574efa9b7da64a5ca954aa in this term.
I20260709 12:26:22.605897 12269 leader_election.cc:304] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36; no voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.606545 12446 tablet_bootstrap.cc:492] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: No bootstrap required, opened a new log
I20260709 12:26:22.607085 12446 ts_tablet_manager.cc:1403] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent bootstrapping tablet: real 0.052s	user 0.013s	sys 0.003s
I20260709 12:26:22.607592 12391 raft_consensus.cc:3060] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.589934 12230 raft_consensus.cc:2468] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 1.
I20260709 12:26:22.610051 12122 leader_election.cc:304] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:22.610859 12467 raft_consensus.cc:2804] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.610891 12446 raft_consensus.cc:359] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.611471 12467 raft_consensus.cc:697] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:22.611753 12446 raft_consensus.cc:385] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.612128 12446 raft_consensus.cc:740] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.612361 12467 consensus_queue.cc:237] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.612767 12446 consensus_queue.cc:260] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.614835 12446 ts_tablet_manager.cc:1434] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20260709 12:26:22.616338 12391 raft_consensus.cc:2749] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:22.619779 12445 tablet_bootstrap.cc:492] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.620489 12445 ts_tablet_manager.cc:1403] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.055s	user 0.008s	sys 0.018s
I20260709 12:26:22.625723 12445 raft_consensus.cc:359] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.626437 12445 raft_consensus.cc:385] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.626787 12445 raft_consensus.cc:740] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.627611 12445 consensus_queue.cc:260] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.631835 12445 ts_tablet_manager.cc:1434] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.001s
I20260709 12:26:22.632809 12445 tablet_bootstrap.cc:492] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: Bootstrap starting.
I20260709 12:26:22.638774 12445 tablet_bootstrap.cc:654] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:22.639382 12054 catalog_manager.cc:5697] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 1 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:22.643890 12445 tablet_bootstrap.cc:492] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: No bootstrap required, opened a new log
I20260709 12:26:22.644418 12445 ts_tablet_manager.cc:1403] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:26:22.646777 12445 raft_consensus.cc:359] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.647439 12445 raft_consensus.cc:385] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:22.647755 12445 raft_consensus.cc:740] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Initialized, Role: FOLLOWER
I20260709 12:26:22.648433 12445 consensus_queue.cc:260] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.650533 12445 ts_tablet_manager.cc:1434] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.003s
I20260709 12:26:22.711611 12466 raft_consensus.cc:493] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.712177 12466 raft_consensus.cc:515] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.714064 12466 leader_election.cc:290] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.714944 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.715142 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:22.715816 12156 raft_consensus.cc:2468] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:22.715859 12305 raft_consensus.cc:2468] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 0.
I20260709 12:26:22.716902 12195 leader_election.cc:304] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.717681 12466 raft_consensus.cc:2804] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:22.718058 12466 raft_consensus.cc:493] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.718384 12466 raft_consensus.cc:3060] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.720739 12467 consensus_queue.cc:1048] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.722239 12467 raft_consensus.cc:493] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.722711 12467 raft_consensus.cc:515] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.725531 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.725819 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36" is_pre_election: true
I20260709 12:26:22.726344 12230 raft_consensus.cc:2393] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62022a2fa120494cbe56659b74b966f8 in current term 1: Already voted for candidate 3a09a0f9c16e402b8237f06400fc1b36 in this term.
I20260709 12:26:22.726311 12466 raft_consensus.cc:515] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.727780 12467 leader_election.cc:290] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.728106 12122 leader_election.cc:304] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:22.728883 12467 raft_consensus.cc:3060] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.730415 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.731083 12156 raft_consensus.cc:3060] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.731552 12466 leader_election.cc:290] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.731739 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:22.732508 12306 raft_consensus.cc:3060] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.739041 12156 raft_consensus.cc:2468] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:22.740525 12195 leader_election.cc:304] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.741238 12391 raft_consensus.cc:493] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.741852 12391 raft_consensus.cc:515] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.744046 12391 leader_election.cc:290] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:22.744225 12354 raft_consensus.cc:2804] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:22.744813 12354 raft_consensus.cc:697] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:22.745519 12472 consensus_queue.cc:1048] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:22.746054 12306 raft_consensus.cc:2468] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 1.
I20260709 12:26:22.745649 12354 consensus_queue.cc:237] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.747779 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588391676e574efa9b7da64a5ca954aa" is_pre_election: true
I20260709 12:26:22.748613 12230 raft_consensus.cc:2393] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a09a0f9c16e402b8237f06400fc1b36 in current term 1: Already voted for candidate 62022a2fa120494cbe56659b74b966f8 in this term.
I20260709 12:26:22.749619 12156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62022a2fa120494cbe56659b74b966f8" is_pre_election: true
I20260709 12:26:22.751103 12269 leader_election.cc:304] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36; no voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.751228 12363 consensus_queue.cc:1048] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.752023 12391 raft_consensus.cc:3060] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.770556 12055 catalog_manager.cc:5697] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 1 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:22.773605 12391 raft_consensus.cc:2749] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:22.774927 12466 consensus_queue.cc:1048] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:22.780279 12467 raft_consensus.cc:2749] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:22.800587 12156 raft_consensus.cc:3060] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.803229 12391 consensus_queue.cc:1048] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.809269 12466 consensus_queue.cc:1048] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.814179 12230 raft_consensus.cc:3060] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.831557 12391 consensus_queue.cc:1048] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.832333 12466 consensus_queue.cc:1048] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:22.836546 12230 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
mode: GRACEFUL
new_leader_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:57238
I20260709 12:26:22.837324 12230 raft_consensus.cc:606] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Received request to transfer leadership to TS 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.851837 12465 consensus_queue.cc:1048] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:22.861202 12306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
mode: GRACEFUL
new_leader_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:43432
I20260709 12:26:22.861927 12306 raft_consensus.cc:606] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Received request to transfer leadership to TS 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.875653 12391 consensus_queue.cc:1048] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.877574 12232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
mode: GRACEFUL
new_leader_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:57238
I20260709 12:26:22.878454 12232 raft_consensus.cc:606] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Received request to transfer leadership to TS 62022a2fa120494cbe56659b74b966f8
I20260709 12:26:22.889509 12306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
mode: GRACEFUL
new_leader_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:43432
I20260709 12:26:22.890081 12306 raft_consensus.cc:606] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Received request to transfer leadership to TS 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:22.896013 12306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "984de35a63d14587ae1333d131c23433"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
mode: GRACEFUL
new_leader_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:43432
I20260709 12:26:22.896577 12306 raft_consensus.cc:606] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Received request to transfer leadership to TS 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:22.905102 12155 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
mode: GRACEFUL
new_leader_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:37902
I20260709 12:26:22.905660 12155 raft_consensus.cc:606] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Received request to transfer leadership to TS 588391676e574efa9b7da64a5ca954aa
I20260709 12:26:22.911037 12231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
mode: GRACEFUL
new_leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
 from {username='slave'} at 127.0.0.1:57238
I20260709 12:26:22.911682 12231 raft_consensus.cc:606] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Received request to transfer leadership to TS 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:22.919804 12391 consensus_queue.cc:1048] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.929384 12231 raft_consensus.cc:3060] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:22.938674 12465 consensus_queue.cc:1048] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.944000 12466 consensus_queue.cc:1048] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:22.954103 12465 raft_consensus.cc:993] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: : Instructing follower 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:22.954442 12472 raft_consensus.cc:993] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: : Instructing follower 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:22.954602 12465 raft_consensus.cc:1081] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Signalling peer 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:22.954965 12472 raft_consensus.cc:1081] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Signalling peer 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:22.956643 12156 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:37884
I20260709 12:26:22.958969 12156 raft_consensus.cc:493] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:22.959425 12156 raft_consensus.cc:3060] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:22.963411 12354 consensus_queue.cc:1048] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:22.964819 12231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:26:22.965569 12231 raft_consensus.cc:493] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:22.966220 12231 raft_consensus.cc:3060] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:22.968986 12156 raft_consensus.cc:515] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.977118 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:22.977734 12232 raft_consensus.cc:3055] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:22.978107 12232 raft_consensus.cc:740] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:22.978960 12232 consensus_queue.cc:260] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.979929 12232 raft_consensus.cc:3060] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:22.980166 12231 raft_consensus.cc:515] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.982450 12231 leader_election.cc:290] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.983283 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:22.983959 12155 raft_consensus.cc:3060] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:22.984575 12304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7ca2294c8845eabca8500e7ba7a22a" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:22.985208 12304 raft_consensus.cc:3055] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:22.985531 12304 raft_consensus.cc:740] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:22.986340 12304 consensus_queue.cc:260] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.987705 12304 raft_consensus.cc:3060] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:22.990371 12232 raft_consensus.cc:2468] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:22.991483 12155 raft_consensus.cc:2468] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:22.991808 12156 leader_election.cc:290] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:22.993057 12306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9b5d597d28b46bebede3f34afc92b92" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:22.993407 12195 leader_election.cc:304] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.994513 12122 leader_election.cc:304] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:22.994889 12466 raft_consensus.cc:2804] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:22.995400 12363 raft_consensus.cc:2804] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:22.995450 12466 raft_consensus.cc:697] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:22.996083 12363 raft_consensus.cc:697] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:22.996589 12466 consensus_queue.cc:237] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.996994 12363 consensus_queue.cc:237] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:22.993750 12306 raft_consensus.cc:3060] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.002147 12304 raft_consensus.cc:2468] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:23.005827 12053 catalog_manager.cc:5697] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 1 to 2, leader changed from 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3) to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 2 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:23.007385 12052 catalog_manager.cc:5697] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 1 to 2, leader changed from 588391676e574efa9b7da64a5ca954aa (127.25.202.2) to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:23.010430 12306 raft_consensus.cc:2468] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:23.126981 12363 consensus_queue.cc:1048] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:23.127874 12354 consensus_queue.cc:1048] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:23.128141 12467 consensus_queue.cc:1048] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:23.136039 12467 raft_consensus.cc:993] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: : Instructing follower 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:23.136478 12467 raft_consensus.cc:1081] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Signalling peer 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:23.139596 12231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:57224
I20260709 12:26:23.140192 12231 raft_consensus.cc:493] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:23.140586 12231 raft_consensus.cc:3060] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.141798 12466 consensus_queue.cc:1048] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:23.149955 12231 raft_consensus.cc:515] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.153364 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:23.154014 12155 raft_consensus.cc:3055] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:23.154350 12155 raft_consensus.cc:740] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:23.155295 12155 consensus_queue.cc:260] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.156280 12155 raft_consensus.cc:3060] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.159193 12231 leader_election.cc:290] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:23.163237 12305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f59313bf8b2a4592b508d3fe3f261445" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:23.166028 12155 raft_consensus.cc:2468] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:23.167403 12195 leader_election.cc:304] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:23.168253 12354 raft_consensus.cc:2804] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:23.168526 12466 raft_consensus.cc:993] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: : Instructing follower 3a09a0f9c16e402b8237f06400fc1b36 to start an election
I20260709 12:26:23.168751 12354 raft_consensus.cc:697] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:23.169354 12472 raft_consensus.cc:1081] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Signalling peer 3a09a0f9c16e402b8237f06400fc1b36 to start an election
I20260709 12:26:23.169688 12354 consensus_queue.cc:237] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.173197 12304 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
 from {username='slave'} at 127.0.0.1:43416
I20260709 12:26:23.173890 12304 raft_consensus.cc:493] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:23.174528 12304 raft_consensus.cc:3060] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.178372 12054 catalog_manager.cc:5697] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 1 to 2, leader changed from 62022a2fa120494cbe56659b74b966f8 (127.25.202.1) to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 2 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:23.181190 12304 raft_consensus.cc:515] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.184314 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:23.184513 12231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2802579dd4404e269ebfc771503f56e9" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:23.185078 12304 leader_election.cc:290] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:23.185230 12231 raft_consensus.cc:3055] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:23.185688 12231 raft_consensus.cc:740] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:23.186084 12155 raft_consensus.cc:3060] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.186626 12231 consensus_queue.cc:260] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.187865 12231 raft_consensus.cc:3060] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.193567 12155 raft_consensus.cc:2468] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 2.
I20260709 12:26:23.194831 12269 leader_election.cc:304] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:23.196034 12476 raft_consensus.cc:2804] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:23.196532 12476 raft_consensus.cc:697] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:23.197351 12476 consensus_queue.cc:237] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.205752 12231 raft_consensus.cc:2468] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 2.
I20260709 12:26:23.206791 12054 catalog_manager.cc:5697] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 1 to 2, leader changed from 588391676e574efa9b7da64a5ca954aa (127.25.202.2) to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:23.317425 12354 raft_consensus.cc:993] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: : Instructing follower 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:23.317924 12466 raft_consensus.cc:1081] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Signalling peer 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:23.319465 12155 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:37884
I20260709 12:26:23.320130 12155 raft_consensus.cc:493] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:23.320546 12155 raft_consensus.cc:3060] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.326771 12155 raft_consensus.cc:515] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.328761 12155 leader_election.cc:290] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:23.329774 12231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:23.329982 12304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6953952a4d8e4a1b91693794a5c5c486" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:23.330569 12231 raft_consensus.cc:3055] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:23.330803 12304 raft_consensus.cc:3060] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.331008 12231 raft_consensus.cc:740] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:23.332113 12231 consensus_queue.cc:260] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.333220 12231 raft_consensus.cc:3060] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.337780 12476 raft_consensus.cc:993] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: : Instructing follower 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:23.338491 12391 raft_consensus.cc:1081] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Signalling peer 588391676e574efa9b7da64a5ca954aa to start an election
I20260709 12:26:23.341441 12232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "984de35a63d14587ae1333d131c23433"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:26:23.342089 12232 raft_consensus.cc:493] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:23.342473 12232 raft_consensus.cc:3060] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.346256 12231 raft_consensus.cc:2468] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:23.347975 12122 leader_election.cc:304] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:23.340770 12304 raft_consensus.cc:2468] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:23.348907 12454 raft_consensus.cc:2804] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:23.349568 12454 raft_consensus.cc:697] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:23.349406 12232 raft_consensus.cc:515] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.350598 12454 consensus_queue.cc:237] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.352269 12232 leader_election.cc:290] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:23.352851 12304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:23.353410 12155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "984de35a63d14587ae1333d131c23433" candidate_uuid: "588391676e574efa9b7da64a5ca954aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:23.353579 12304 raft_consensus.cc:3055] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:23.354051 12304 raft_consensus.cc:740] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:23.354104 12155 raft_consensus.cc:3060] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.354950 12304 consensus_queue.cc:260] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.356309 12304 raft_consensus.cc:3060] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.361002 12155 raft_consensus.cc:2468] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:23.362531 12195 leader_election.cc:304] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:23.362015 12054 catalog_manager.cc:5697] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 1 to 2, leader changed from 588391676e574efa9b7da64a5ca954aa (127.25.202.2) to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:23.363164 12304 raft_consensus.cc:2468] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588391676e574efa9b7da64a5ca954aa in term 2.
I20260709 12:26:23.363690 12466 raft_consensus.cc:2804] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:23.364519 12466 raft_consensus.cc:697] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Leader. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:23.365475 12466 consensus_queue.cc:237] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.373651 12052 catalog_manager.cc:5697] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa reported cstate change: term changed from 1 to 2, leader changed from 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3) to 588391676e574efa9b7da64a5ca954aa (127.25.202.2). New cstate: current_term: 2 leader_uuid: "588391676e574efa9b7da64a5ca954aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:23.404716 12391 raft_consensus.cc:993] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: : Instructing follower 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:23.405211 12476 raft_consensus.cc:1081] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Signalling peer 62022a2fa120494cbe56659b74b966f8 to start an election
I20260709 12:26:23.406857 12155 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
 from {username='slave'} at 127.0.0.1:37892
I20260709 12:26:23.407437 12155 raft_consensus.cc:493] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:23.407765 12155 raft_consensus.cc:3060] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.413075 12155 raft_consensus.cc:515] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.414677 12155 leader_election.cc:290] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Requested vote from peers 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351), 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3:46063)
I20260709 12:26:23.415470 12232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:23.415755 12304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56ab8e7a609e421c83735ff7134913a0" candidate_uuid: "62022a2fa120494cbe56659b74b966f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
I20260709 12:26:23.416167 12232 raft_consensus.cc:3060] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.416291 12304 raft_consensus.cc:3055] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:23.416703 12304 raft_consensus.cc:740] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:23.417546 12304 consensus_queue.cc:260] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.418804 12304 raft_consensus.cc:3060] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.423273 12232 raft_consensus.cc:2468] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:23.424666 12122 leader_election.cc:304] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588391676e574efa9b7da64a5ca954aa, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:23.425467 12454 raft_consensus.cc:2804] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:23.425848 12304 raft_consensus.cc:2468] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62022a2fa120494cbe56659b74b966f8 in term 2.
I20260709 12:26:23.425945 12454 raft_consensus.cc:697] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Becoming Leader. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:23.427024 12454 consensus_queue.cc:237] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:23.428809 12155 raft_consensus.cc:1275] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.430131 12466 consensus_queue.cc:1048] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.434942 12052 catalog_manager.cc:5697] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 reported cstate change: term changed from 1 to 2, leader changed from 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3) to 62022a2fa120494cbe56659b74b966f8 (127.25.202.1). New cstate: current_term: 2 leader_uuid: "62022a2fa120494cbe56659b74b966f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:23.437764 12304 raft_consensus.cc:1275] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.440093 12472 consensus_queue.cc:1048] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.441370 12304 raft_consensus.cc:1275] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.443740 12454 consensus_queue.cc:1048] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.460546 12232 raft_consensus.cc:1275] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.461848 12467 consensus_queue.cc:1048] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.562261 12155 raft_consensus.cc:1275] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.563560 12472 consensus_queue.cc:1048] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.571012 12304 raft_consensus.cc:3060] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:23.578801 12304 raft_consensus.cc:1275] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.580153 12472 consensus_queue.cc:1048] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:23.714797 12232 raft_consensus.cc:1275] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Refusing update from remote peer 3a09a0f9c16e402b8237f06400fc1b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.716145 12476 consensus_queue.cc:1048] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.728192 12155 raft_consensus.cc:1275] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 3a09a0f9c16e402b8237f06400fc1b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.729902 12476 consensus_queue.cc:1048] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.831270 12304 raft_consensus.cc:1275] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.832857 12472 consensus_queue.cc:1048] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.840968 12156 raft_consensus.cc:1275] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 588391676e574efa9b7da64a5ca954aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.843216 12472 consensus_queue.cc:1048] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.845096 12304 raft_consensus.cc:1275] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.846836 12467 consensus_queue.cc:1048] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.862442 12231 raft_consensus.cc:1275] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.863925 12467 consensus_queue.cc:1048] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.879148 12231 raft_consensus.cc:1275] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.881063 12467 consensus_queue.cc:1048] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:23.888620 12306 raft_consensus.cc:1275] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Refusing update from remote peer 62022a2fa120494cbe56659b74b966f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:23.890861 12363 consensus_queue.cc:1048] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:25.914806 12054 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37416:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:26:25.935840 12054 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37416:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:26:25.972204 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:26.148629 12572 tablet.cc:1724] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.153482 12568 tablet.cc:1724] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.155118 12555 tablet.cc:1724] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.115491 12556 tablet.cc:1724] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.194402 12578 tablet.cc:1724] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:26.192322 12554 log.cc:927] Time spent T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: Append to log took a long time: real 0.058s	user 0.004s	sys 0.000s
I20260709 12:26:26.194689 12581 tablet.cc:1724] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.220043 12576 tablet.cc:1724] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.235106 12582 tablet.cc:1724] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.297358 12568 tablet.cc:1724] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:26.350409 12584 log.cc:927] Time spent T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:26:26.353840 12228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744"
dest_uuid: "588391676e574efa9b7da64a5ca954aa"
mode: GRACEFUL
new_leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
 from {username='slave'} at 127.0.0.1:53140
I20260709 12:26:26.362138 12228 raft_consensus.cc:606] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Received request to transfer leadership to TS 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:26.390202 12588 tablet.cc:1724] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.396392 12578 tablet.cc:1724] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.360138 12592 tablet.cc:1724] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.345847 12556 tablet.cc:1724] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.333070 12581 tablet.cc:1724] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:26:26.414749 12590 log.cc:927] Time spent T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:26:26.327358 12576 tablet.cc:1724] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.420106 12555 tablet.cc:1724] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.423643 12582 tablet.cc:1724] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.427757 12572 tablet.cc:1724] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.435024 12587 tablet.cc:1724] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.439898 12556 tablet.cc:1724] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.466236 12595 tablet.cc:1724] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.469326 12592 tablet.cc:1724] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.478178 12578 tablet.cc:1724] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.490965 12587 tablet.cc:1724] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.498416 12595 tablet.cc:1724] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.507117 12587 tablet.cc:1724] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.525666 12588 tablet.cc:1724] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:26:26.528970 12154 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "61488d77945649d59f085e9f2035be1a"
dest_uuid: "62022a2fa120494cbe56659b74b966f8"
mode: GRACEFUL
new_leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:26:26.529727 12154 raft_consensus.cc:606] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Received request to transfer leadership to TS 3a09a0f9c16e402b8237f06400fc1b36
I20260709 12:26:26.531383 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:26:26.532044 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:26.544075 12053 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc: soft_deleted_table [id=49a347de1a09436b94c9a418479dfda6] alter complete (version 1)
I20260709 12:26:26.553215 12569 raft_consensus.cc:993] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: : Instructing follower 3a09a0f9c16e402b8237f06400fc1b36 to start an election
I20260709 12:26:26.553879 12585 raft_consensus.cc:1081] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Signalling peer 3a09a0f9c16e402b8237f06400fc1b36 to start an election
I20260709 12:26:26.555392 12304 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
 from {username='slave'} at 127.0.0.1:43416
I20260709 12:26:26.556077 12304 raft_consensus.cc:493] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:26.556435 12304 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:26.562188 12304 raft_consensus.cc:515] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:26.563939 12304 leader_election.cc:290] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 3 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:26.565083 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:26.565109 12154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9e43f88aef4273a7a00ced5ba8e744" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:26.565817 12230 raft_consensus.cc:3055] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:26.566004 12154 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:26.566138 12230 raft_consensus.cc:740] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 588391676e574efa9b7da64a5ca954aa, State: Running, Role: LEADER
I20260709 12:26:26.567000 12230 consensus_queue.cc:260] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:26.568054 12230 raft_consensus.cc:3060] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:26.571774 12154 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 3.
I20260709 12:26:26.573076 12230 raft_consensus.cc:2468] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 3.
I20260709 12:26:26.573053 12269 leader_election.cc:304] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 62022a2fa120494cbe56659b74b966f8; no voters: 
I20260709 12:26:26.574126 12476 raft_consensus.cc:2804] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:26.574618 12476 raft_consensus.cc:697] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 3 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:26.575433 12476 consensus_queue.cc:237] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:26.584123 12052 catalog_manager.cc:5697] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 2 to 3, leader changed from 588391676e574efa9b7da64a5ca954aa (127.25.202.2) to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 3 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:26.665465 12571 raft_consensus.cc:993] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: : Instructing follower 3a09a0f9c16e402b8237f06400fc1b36 to start an election
I20260709 12:26:26.666133 12525 raft_consensus.cc:1081] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Signalling peer 3a09a0f9c16e402b8237f06400fc1b36 to start an election
I20260709 12:26:26.668001 12304 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "61488d77945649d59f085e9f2035be1a"
dest_uuid: "3a09a0f9c16e402b8237f06400fc1b36"
 from {username='slave'} at 127.0.0.1:43406
I20260709 12:26:26.668627 12304 raft_consensus.cc:493] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:26.668973 12304 raft_consensus.cc:3060] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:26.674374 12304 raft_consensus.cc:515] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:26.675899 12304 leader_election.cc:290] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 2 election: Requested vote from peers 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853), 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351)
I20260709 12:26:26.676888 12230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61488d77945649d59f085e9f2035be1a" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "588391676e574efa9b7da64a5ca954aa"
I20260709 12:26:26.676800 12154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61488d77945649d59f085e9f2035be1a" candidate_uuid: "3a09a0f9c16e402b8237f06400fc1b36" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "62022a2fa120494cbe56659b74b966f8"
I20260709 12:26:26.677628 12230 raft_consensus.cc:3060] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:26.677801 12154 raft_consensus.cc:3055] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:26.678277 12154 raft_consensus.cc:740] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62022a2fa120494cbe56659b74b966f8, State: Running, Role: LEADER
I20260709 12:26:26.679024 12154 consensus_queue.cc:260] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:26.680084 12154 raft_consensus.cc:3060] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:26.682924 12230 raft_consensus.cc:2468] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 2.
I20260709 12:26:26.684051 12271 leader_election.cc:304] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09a0f9c16e402b8237f06400fc1b36, 588391676e574efa9b7da64a5ca954aa; no voters: 
I20260709 12:26:26.684912 12565 raft_consensus.cc:2804] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:26.685400 12565 raft_consensus.cc:697] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 LEADER]: Becoming Leader. State: Replica: 3a09a0f9c16e402b8237f06400fc1b36, State: Running, Role: LEADER
I20260709 12:26:26.685868 12154 raft_consensus.cc:2468] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09a0f9c16e402b8237f06400fc1b36 in term 2.
I20260709 12:26:26.686305 12565 consensus_queue.cc:237] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } }
I20260709 12:26:26.696287 12052 catalog_manager.cc:5697] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 reported cstate change: term changed from 1 to 2, leader changed from 62022a2fa120494cbe56659b74b966f8 (127.25.202.1) to 3a09a0f9c16e402b8237f06400fc1b36 (127.25.202.3). New cstate: current_term: 2 leader_uuid: "3a09a0f9c16e402b8237f06400fc1b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09a0f9c16e402b8237f06400fc1b36" member_type: VOTER last_known_addr { host: "127.25.202.3" port: 46063 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:27.016909 12153 raft_consensus.cc:1275] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 3 FOLLOWER]: Refusing update from remote peer 3a09a0f9c16e402b8237f06400fc1b36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:27.018375 12565 consensus_queue.cc:1048] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:27.040639 12230 raft_consensus.cc:1275] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 3 FOLLOWER]: Refusing update from remote peer 3a09a0f9c16e402b8237f06400fc1b36: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:27.042778 12565 consensus_queue.cc:1048] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:26:27.142472 12230 raft_consensus.cc:1275] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Refusing update from remote peer 3a09a0f9c16e402b8237f06400fc1b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:27.143805 12565 consensus_queue.cc:1048] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588391676e574efa9b7da64a5ca954aa" member_type: VOTER last_known_addr { host: "127.25.202.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:27.154186 12153 raft_consensus.cc:1275] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Refusing update from remote peer 3a09a0f9c16e402b8237f06400fc1b36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:27.157845 12608 consensus_queue.cc:1048] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62022a2fa120494cbe56659b74b966f8" member_type: VOTER last_known_addr { host: "127.25.202.1" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:27.533134 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:27.542573 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:27.543468 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:28.544906 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:28.552484 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:28.552963 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:29.554034 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:29.560705 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:29.561156 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:30.598367 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:30.606001 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:30.606532 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:26:30.894238 12086 debug-util.cc:398] Leaking SignalData structure 0x7b080013a6c0 after lost signal to thread 26411
W20260709 12:26:30.895680 12086 debug-util.cc:398] Leaking SignalData structure 0x7b0800090980 after lost signal to thread 12089
W20260709 12:26:30.896855 12086 debug-util.cc:398] Leaking SignalData structure 0x7b08002be080 after lost signal to thread 12181
W20260709 12:26:30.897886 12086 debug-util.cc:398] Leaking SignalData structure 0x7b08001f3180 after lost signal to thread 12256
W20260709 12:26:30.898939 12086 debug-util.cc:398] Leaking SignalData structure 0x7b080024d020 after lost signal to thread 12330
I20260709 12:26:31.607754 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:31.616292 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:31.616899 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:32.618199 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:32.626999 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:32.627743 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:33.628849 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:33.637812 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:33.638293 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:34.639427 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:34.645313 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:34.645828 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:35.646914 26408 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:35.652107 26408 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:35.652587 26408 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:36.714316 26408 tablet_server.cc:179] TabletServer@127.25.202.1:0 shutting down...
W20260709 12:26:36.720573 12269 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 16 similar messages]
W20260709 12:26:36.724826 12269 consensus_peers.cc:597] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Couldn't send request to peer 62022a2fa120494cbe56659b74b966f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.760624 12195 consensus_peers.cc:597] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa -> Peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Couldn't send request to peer 62022a2fa120494cbe56659b74b966f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.765126 12269 consensus_peers.cc:597] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Couldn't send request to peer 62022a2fa120494cbe56659b74b966f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.769268 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:36.770341 26408 tablet_replica.cc:333] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.771857 26408 raft_consensus.cc:2243] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.772825 26408 raft_consensus.cc:2272] T e2f95c3487484c0fb9e5b270a4623904 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.775919 26408 tablet_replica.cc:333] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.776726 26408 raft_consensus.cc:2243] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.777390 26408 raft_consensus.cc:2272] T 2802579dd4404e269ebfc771503f56e9 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.781291 26408 tablet_replica.cc:333] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.782305 26408 raft_consensus.cc:2243] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.783125 26408 raft_consensus.cc:2272] T f59313bf8b2a4592b508d3fe3f261445 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.787422 26408 tablet_replica.cc:333] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.788056 26408 raft_consensus.cc:2243] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.788975 26408 raft_consensus.cc:2272] T a9b5d597d28b46bebede3f34afc92b92 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.791523 26408 tablet_replica.cc:333] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.792084 26408 raft_consensus.cc:2243] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.792558 26408 raft_consensus.cc:2272] T 9f7ca2294c8845eabca8500e7ba7a22a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.795591 26408 tablet_replica.cc:333] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.796209 26408 raft_consensus.cc:2243] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.796896 26408 raft_consensus.cc:2272] T 984de35a63d14587ae1333d131c23433 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.799412 26408 tablet_replica.cc:333] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.799973 26408 raft_consensus.cc:2243] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.800874 26408 raft_consensus.cc:2272] T 25f7a207a27d4252b403a7aa91597d3c P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.803287 26408 tablet_replica.cc:333] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.804008 26408 raft_consensus.cc:2243] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.804497 26408 raft_consensus.cc:2272] T 781cab6f3dbf45a58fa529278460248f P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.807003 26408 tablet_replica.cc:333] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.807662 26408 raft_consensus.cc:2243] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.808588 26408 raft_consensus.cc:2272] T 6953952a4d8e4a1b91693794a5c5c486 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.811092 26408 tablet_replica.cc:333] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.811750 26408 raft_consensus.cc:2243] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.812286 26408 raft_consensus.cc:2272] T 503e4fb15a654c938a923e5c6808c209 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.814858 26408 tablet_replica.cc:333] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.815505 26408 raft_consensus.cc:2243] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.817024 26408 raft_consensus.cc:2272] T 56ab8e7a609e421c83735ff7134913a0 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:36.820544 12195 consensus_peers.cc:597] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa -> Peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Couldn't send request to peer 62022a2fa120494cbe56659b74b966f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.820552 26408 tablet_replica.cc:333] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.821626 26408 raft_consensus.cc:2243] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.822153 26408 raft_consensus.cc:2272] T 8d871c96be864188b0b1da87fd438d00 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.824841 26408 tablet_replica.cc:333] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.825378 26408 raft_consensus.cc:2243] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.826344 26408 raft_consensus.cc:2272] T e9689cd771ee44b2808c2d7c52d83766 P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.828905 26408 tablet_replica.cc:333] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.829485 26408 raft_consensus.cc:2243] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.829952 26408 raft_consensus.cc:2272] T 2a9e43f88aef4273a7a00ced5ba8e744 P 62022a2fa120494cbe56659b74b966f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.833168 26408 tablet_replica.cc:333] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.833813 26408 raft_consensus.cc:2243] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.834733 26408 raft_consensus.cc:2272] T d499c7e4809645de92dfc91281414bdd P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.837212 26408 tablet_replica.cc:333] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.837796 26408 raft_consensus.cc:2243] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.838685 26408 raft_consensus.cc:2272] T cf6db4244c274ed2b604f852a7267a95 P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.841128 26408 tablet_replica.cc:333] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.841583 26408 raft_consensus.cc:2243] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.842021 26408 raft_consensus.cc:2272] T 3ecef6444dfd41e1bbb5b1534a4626be P 62022a2fa120494cbe56659b74b966f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.844470 26408 tablet_replica.cc:333] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8: stopping tablet replica
I20260709 12:26:36.844933 26408 raft_consensus.cc:2243] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.845402 26408 raft_consensus.cc:2272] T 61488d77945649d59f085e9f2035be1a P 62022a2fa120494cbe56659b74b966f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.895232 26408 tablet_server.cc:196] TabletServer@127.25.202.1:0 shutdown complete.
W20260709 12:26:36.938252 12269 consensus_peers.cc:597] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Couldn't send request to peer 62022a2fa120494cbe56659b74b966f8. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:44853: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.944198 26408 tablet_server.cc:179] TabletServer@127.25.202.2:0 shutting down...
I20260709 12:26:36.980561 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:36.981554 26408 tablet_replica.cc:333] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:36.982507 26408 raft_consensus.cc:2243] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.983888 26408 raft_consensus.cc:2272] T 9f7ca2294c8845eabca8500e7ba7a22a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:36.985572 12271 consensus_peers.cc:597] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Couldn't send request to peer 588391676e574efa9b7da64a5ca954aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.987142 26408 tablet_replica.cc:333] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:36.987993 26408 raft_consensus.cc:2243] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.988740 26408 raft_consensus.cc:2272] T 2802579dd4404e269ebfc771503f56e9 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.991166 26408 tablet_replica.cc:333] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:36.991797 26408 raft_consensus.cc:2243] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:36.992705 26408 raft_consensus.cc:2272] T f59313bf8b2a4592b508d3fe3f261445 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.995450 26408 tablet_replica.cc:333] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:36.996224 26408 raft_consensus.cc:2243] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.997201 26408 raft_consensus.cc:2272] T a9b5d597d28b46bebede3f34afc92b92 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:36.998796 12271 consensus_peers.cc:597] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Couldn't send request to peer 588391676e574efa9b7da64a5ca954aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.000211 26408 tablet_replica.cc:333] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.000936 26408 raft_consensus.cc:2243] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.001827 26408 raft_consensus.cc:2272] T e2f95c3487484c0fb9e5b270a4623904 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.004298 26408 tablet_replica.cc:333] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.004849 26408 raft_consensus.cc:2243] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.005321 26408 raft_consensus.cc:2272] T 25f7a207a27d4252b403a7aa91597d3c P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.007753 26408 tablet_replica.cc:333] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.008318 26408 raft_consensus.cc:2243] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.009409 26408 raft_consensus.cc:2272] T 984de35a63d14587ae1333d131c23433 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.011721 26408 tablet_replica.cc:333] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.012271 26408 raft_consensus.cc:2243] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.013095 26408 raft_consensus.cc:2272] T 781cab6f3dbf45a58fa529278460248f P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.015517 26408 tablet_replica.cc:333] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.016067 26408 raft_consensus.cc:2243] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.016535 26408 raft_consensus.cc:2272] T e9689cd771ee44b2808c2d7c52d83766 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.018930 26408 tablet_replica.cc:333] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.019507 26408 raft_consensus.cc:2243] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.019923 26408 raft_consensus.cc:2272] T 2a9e43f88aef4273a7a00ced5ba8e744 P 588391676e574efa9b7da64a5ca954aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.022315 26408 tablet_replica.cc:333] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.022930 26408 raft_consensus.cc:2243] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.023494 26408 raft_consensus.cc:2272] T 56ab8e7a609e421c83735ff7134913a0 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.025808 26408 tablet_replica.cc:333] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.026371 26408 raft_consensus.cc:2243] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.027006 26408 raft_consensus.cc:2272] T 6953952a4d8e4a1b91693794a5c5c486 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.029361 26408 tablet_replica.cc:333] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.029925 26408 raft_consensus.cc:2243] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.030457 26408 raft_consensus.cc:2272] T 503e4fb15a654c938a923e5c6808c209 P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.032889 26408 tablet_replica.cc:333] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.033447 26408 raft_consensus.cc:2243] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.034399 26408 raft_consensus.cc:2272] T 8d871c96be864188b0b1da87fd438d00 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.036808 26408 tablet_replica.cc:333] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.037366 26408 raft_consensus.cc:2243] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.037858 26408 raft_consensus.cc:2272] T d499c7e4809645de92dfc91281414bdd P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.040619 26408 tablet_replica.cc:333] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.041170 26408 raft_consensus.cc:2243] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.041805 26408 raft_consensus.cc:2272] T cf6db4244c274ed2b604f852a7267a95 P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.044142 26408 tablet_replica.cc:333] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.044622 26408 raft_consensus.cc:2243] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.045502 26408 raft_consensus.cc:2272] T 3ecef6444dfd41e1bbb5b1534a4626be P 588391676e574efa9b7da64a5ca954aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.047767 26408 tablet_replica.cc:333] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa: stopping tablet replica
I20260709 12:26:37.048243 26408 raft_consensus.cc:2243] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.048748 26408 raft_consensus.cc:2272] T 61488d77945649d59f085e9f2035be1a P 588391676e574efa9b7da64a5ca954aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.093215 26408 tablet_server.cc:196] TabletServer@127.25.202.2:0 shutdown complete.
I20260709 12:26:37.143766 26408 tablet_server.cc:179] TabletServer@127.25.202.3:0 shutting down...
W20260709 12:26:37.157678 12269 consensus_peers.cc:597] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 62022a2fa120494cbe56659b74b966f8 (127.25.202.1:44853): Couldn't send request to peer 62022a2fa120494cbe56659b74b966f8. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.1:44853: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.182221 26408 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:37.182972 26408 tablet_replica.cc:333] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.184072 26408 raft_consensus.cc:2243] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.184752 26408 raft_consensus.cc:2272] T f59313bf8b2a4592b508d3fe3f261445 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.187100 26408 tablet_replica.cc:333] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.187716 26408 raft_consensus.cc:2243] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.188534 26408 raft_consensus.cc:2272] T 2802579dd4404e269ebfc771503f56e9 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.190948 26408 tablet_replica.cc:333] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.191766 26408 raft_consensus.cc:2243] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.192339 26408 raft_consensus.cc:2272] T a9b5d597d28b46bebede3f34afc92b92 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.194865 26408 tablet_replica.cc:333] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
W20260709 12:26:37.195248 12271 consensus_peers.cc:597] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 -> Peer 588391676e574efa9b7da64a5ca954aa (127.25.202.2:34351): Couldn't send request to peer 588391676e574efa9b7da64a5ca954aa. Status: Network error: Client connection negotiation failed: client connection to 127.25.202.2:34351: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.195554 26408 raft_consensus.cc:2243] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.196197 26408 raft_consensus.cc:2272] T 9f7ca2294c8845eabca8500e7ba7a22a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.198428 26408 tablet_replica.cc:333] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.199038 26408 raft_consensus.cc:2243] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.199632 26408 raft_consensus.cc:2272] T e2f95c3487484c0fb9e5b270a4623904 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.202060 26408 tablet_replica.cc:333] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.202618 26408 raft_consensus.cc:2243] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.203094 26408 raft_consensus.cc:2272] T 25f7a207a27d4252b403a7aa91597d3c P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.205273 26408 tablet_replica.cc:333] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.205832 26408 raft_consensus.cc:2243] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.206344 26408 raft_consensus.cc:2272] T 984de35a63d14587ae1333d131c23433 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.208593 26408 tablet_replica.cc:333] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.209200 26408 raft_consensus.cc:2243] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.210052 26408 raft_consensus.cc:2272] T 2a9e43f88aef4273a7a00ced5ba8e744 P 3a09a0f9c16e402b8237f06400fc1b36 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.212453 26408 tablet_replica.cc:333] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.213027 26408 raft_consensus.cc:2243] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.213471 26408 raft_consensus.cc:2272] T e9689cd771ee44b2808c2d7c52d83766 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.215618 26408 tablet_replica.cc:333] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.216198 26408 raft_consensus.cc:2243] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.216667 26408 raft_consensus.cc:2272] T 781cab6f3dbf45a58fa529278460248f P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.218894 26408 tablet_replica.cc:333] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.219507 26408 raft_consensus.cc:2243] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.220000 26408 raft_consensus.cc:2272] T 56ab8e7a609e421c83735ff7134913a0 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.222354 26408 tablet_replica.cc:333] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.222898 26408 raft_consensus.cc:2243] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.223416 26408 raft_consensus.cc:2272] T 6953952a4d8e4a1b91693794a5c5c486 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.225581 26408 tablet_replica.cc:333] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.226092 26408 raft_consensus.cc:2243] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.226676 26408 raft_consensus.cc:2272] T 8d871c96be864188b0b1da87fd438d00 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.228878 26408 tablet_replica.cc:333] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.229424 26408 raft_consensus.cc:2243] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.230321 26408 raft_consensus.cc:2272] T 503e4fb15a654c938a923e5c6808c209 P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.232441 26408 tablet_replica.cc:333] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.232971 26408 raft_consensus.cc:2243] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.233417 26408 raft_consensus.cc:2272] T d499c7e4809645de92dfc91281414bdd P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.235526 26408 tablet_replica.cc:333] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.235957 26408 raft_consensus.cc:2243] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.236372 26408 raft_consensus.cc:2272] T cf6db4244c274ed2b604f852a7267a95 P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.238312 26408 tablet_replica.cc:333] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.238713 26408 raft_consensus.cc:2243] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.239094 26408 raft_consensus.cc:2272] T 3ecef6444dfd41e1bbb5b1534a4626be P 3a09a0f9c16e402b8237f06400fc1b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.241216 26408 tablet_replica.cc:333] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36: stopping tablet replica
I20260709 12:26:37.241623 26408 raft_consensus.cc:2243] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.242540 26408 raft_consensus.cc:2272] T 61488d77945649d59f085e9f2035be1a P 3a09a0f9c16e402b8237f06400fc1b36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.285933 26408 tablet_server.cc:196] TabletServer@127.25.202.3:0 shutdown complete.
I20260709 12:26:37.326275 26408 master.cc:562] Master@127.25.202.62:41237 shutting down...
W20260709 12:26:37.352358 12103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Catalog manager is not initialized. State: Closing
I20260709 12:26:37.353497 26408 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.354101 26408 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.354419 26408 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4672144bf9f64bccaf81431afd23fabc: stopping tablet replica
I20260709 12:26:37.395048 26408 master.cc:584] Master@127.25.202.62:41237 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (20665 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (41680 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (609453 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 12:26:37.461597 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2225 messages since previous log ~31 seconds ago
I20260709 12:26:37.461846 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2225 messages since previous log ~31 seconds ago
I20260709 12:26:37.462033 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2223 messages since previous log ~31 seconds ago
I20260709 12:26:37.462212 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~50 seconds ago
I20260709 12:26:37.462375 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~50 seconds ago
I20260709 12:26:37.462581 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 7 messages since previous log ~206 seconds ago
I20260709 12:26:37.462742 26408 logging.cc:436] LogThrottler Service Queue Overflow: suppressed but not reported on 1 messages since previous log ~296 seconds ago
I20260709 12:26:37.462903 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/service_pool.cc:136: suppressed but not reported on 1 messages since previous log ~296 seconds ago
I20260709 12:26:37.463115 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 10 messages since previous log ~0 seconds ago
I20260709 12:26:37.463457 26408 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 52 messages since previous log ~19 seconds ago
